Image foreign matter detection method for reducing false detection probability

Abstract

The invention provides an image foreign matter detection method capable of reducing false detection probability. The method comprises the following steps: searching a difference region between a to-be-detected image and a standard image; binarizing the image according to the difference region to obtain a binarized abnormal image; inputting a to-be-detected image into the false detection model, and marking false detection points and non-false detection points; binarizing the image according to the false detection points to obtain a binarized false detection image; searching a connected domain Ai and a connected domain Bj in the two binarized images respectively, wherein the connected domains are recorded as the connected domain Ai and the connected domain Bj; if the consistency of a certain connected domain Ai and any connected domain Bj meets the requirement, deleting the connected domain Ai; marking the connected domain Ai which is not deleted in the to-be-detected image as a foreign matter region; according to the method, a deep learning model classification method and an image processing method are fused, the robustness of the foreign matter detection method is enhanced, suspected areas are screened through fusion of the two methods, and the false detection rate of foreign matter detection is effectively reduced; the anti-interference capability is stronger, and the foreign matter area positioning is more accurate.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to an image foreign object detection method that reduces the probability of false detection. Background Technology

[0002] Foreign object detection in the industrial field mainly includes: Abnormal conditions during the manufacturing process, such as missing parts, dents and protrusions on the surface of parts; Intrusion of abnormal objects during equipment operation, such as tree branches and leaves, animal carcasses, balloons, plastic bags, and construction debris.

[0003] These process defects and foreign object intrusions not only cause economic losses but also pose serious safety hazards, and even injuries or fatalities. Traditional manual inspection suffers from problems such as low efficiency, reliance on experience, strong subjectivity, and inability to conduct comprehensive inspections. For certain items to be inspected at heights, high-altitude operation qualifications are required, which is difficult and dangerous. Therefore, relying on computer vision technology to replace manual inspection is an inevitable trend.

[0004] Currently, foreign object detection methods based on visual detection mainly fall into the following two categories: Type 1: Foreign Object Detection Methods Based on Image Information Analysis. For example, patent document CN118097215A proposes an image processing method based on a template image set. This method determines the category of the image to be detected by comparing the similarity between the image to be detected and the template image set. This method has two problems: first, there are few defect image samples in industry, and it is difficult to create a template image set; second, the algorithm struggles to identify anomalies not included in the template image set. Another example is patent document CN115100191B, which proposes a defect recognition method based on image grayscale. This method obtains abrupt change regions in the image by analyzing the peaks and troughs of the image's grayscale histogram, and locates the defect position by analyzing these regions. This method does not require a large amount of abnormal data for training, but the algorithm cannot accurately identify parts with complex surface grayscale value changes. Furthermore, when ambient light changes or other factors affect the grayscale histogram, the algorithm suffers from severe false detections.

[0005] Category Two: Foreign Object Detection Methods Based on Deep Learning Models. For example, patent document CN117557841A proposes a classifier-based detection method. In this method, a first classifier determines the category of the input image, and a second classifier determines whether the input image belongs to an unknown category. Another example is patent document CN115424074A, which proposes a detection method based on deep learning and feature space transformation. This method inputs the image into a classification model, extracts multi-dimensional feature vectors, maps these vectors to a linear feature space, and further maps the linear feature space to an angular feature space. The classifier is then trained to classify the angular space features. Both of these methods share the same problems: first, the classifier needs to be trained using real abnormal images, but the amount of abnormal data samples in actual industrial scenarios is insufficient to support the training of deep learning classifiers; second, the classifier can only classify images and cannot locate the pixel set of the abnormal region within the image.

[0006] In summary, current methods for detecting foreign objects in industrial samples still have the following shortcomings: Traditional image processing and template comparison methods have low robustness and are too sensitive to ambient light or other external interference. When the grayscale of the surface of the item to be inspected changes, it is easy to cause false detection. Deep learning-based anomaly detection methods require a large number of anomaly samples, but in industrial scenarios, there are too few anomaly samples to drive the training of deep learning models. Furthermore, manual annotation is time-consuming and labor-intensive, and cannot achieve the desired detection results. For foreign object data that has never appeared before, false detections and false negatives are likely to occur. Usually, only the input image is classified, which cannot accurately locate the abnormal region. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides an image foreign object detection method that reduces the probability of false detection. This method integrates deep learning model classification and traditional digital image processing methods, enhancing the robustness of foreign object detection. By fusing these two methods to filter suspected areas, the false detection rate of foreign objects is effectively reduced; it also exhibits stronger anti-interference capabilities and more accurate foreign object region localization. Compared to existing methods, this solution demonstrates a lower false detection rate and more outstanding performance when applied to foreign object detection scenarios such as outdoor environments, areas with significant lighting variations, and poor foreground-background contrast.

[0008] The technical solution is as follows: A foreign object detection method for reducing the probability of false detection involves a visual sensor acquiring multiple images of the object to be detected in a batch visual inspection scenario or a visual monitoring scenario of the same object at different time periods, thereby detecting whether there are foreign objects on the surface of the object when the images are acquired. The foreign object detection process includes the following steps: Step 1: Compare the image to be detected with the standard image to find the difference regions between them; based on the difference regions and non-difference regions, binarize the image to be detected to obtain a binarized abnormal image; the standard image is a pre-stored image of the object under normal conditions. Meanwhile, the image to be detected is input into a pre-trained false detection model, which is a semantic segmentation model. It is used to classify each pixel in the image to be detected and label whether the pixel belongs to a false detection point or not. Based on the false detection points and not false detection points, the image to be detected is binarized to obtain a binarized false detection image. The false detection model is trained using a false detection dataset, which is obtained as follows: multiple images of the object under normal conditions are collected, each image is compared with a standard image, and the difference regions between each image and the standard image are identified; based on the difference regions and non-difference regions, each image is binarized to obtain multiple binarized label images, and the binarized label images are stored in correspondence with the collected images to form a false detection dataset. Step 2: Find connected components in the binarized outlier image and the binarized false detection image respectively, and denote the found connected components as connected component A. i Connected component B j ; Step 3: Analyze connected component A i Connected component B j The consistency among the connected components in the equation, if a certain connected component A i With any connected component B j If the consistency requirement is met, then the connected component A i This is a false positive region; delete the connected component A. i Otherwise, the connected component A i For foreign object regions, retain the connected component A. i ; Step 4: If connected component A i If all objects are deleted, then the image to be detected is free of foreign objects; otherwise, in the image to be detected, mark the connected components A that were not deleted. i The area is designated as a foreign object zone; foreign object detection is then performed.

[0009] Further, in step three, analyze the connected component A. i Connected component B j The consistency among connected components is achieved through the following methods: Based on the characteristics of connected components, analyze the consistency between them; The features of the connected component include: the location of the connected component, morphological features, intersection-union ratio, and one or more features in the feature map output by the false detection model.

[0010] Furthermore, we first analyze the consistency between connected components based on their positions, as follows: Obtain each connected component A separately i The centroid / mass; Connect the coordinates of each centroid / mass center to the connected component B. j Perform position matching if the coordinates of a certain centroid / mass are located in any connected region B. j / connected region B j Within the smallest bounding rectangle: Then denote the connected region A corresponding to the centroid / mass coordinates respectively. i Let A' be the connected region and B be the connected region where the centroid / mass coordinates are located. j Let A' be a connected component B'; the consistency between connected component A' and connected component B' satisfies the requirements. Otherwise, the connected region A corresponding to the centroid / mass coordinates i With connected component B j The consistency requirement is not met. i .

[0011] Preferably, if the coordinates of a certain centroid / mass are located in any connected region B j / connected region B j Within the minimum bounding rectangle, further based on the morphological characteristics of the connected components, it is determined whether the consistency between connected component A' and connected component B' meets the requirements: Obtain the morphological features of connected components A' and B' respectively; Calculate the similarity between morphological features; determine whether the consistency of connected components A' and B' meets the requirements based on the similarity. The morphological features include a combination of one or more of the following: circularity, rectangularity, convexity, ratio of minor axis to major axis, angle between major axis and minor axis, width-to-length ratio, and circumference ratio of the connected region. Convexity is the ratio of the area of ​​the connected region to the area of ​​the convex hull of the connected region. The minor axis and major axis are defined as follows: draw multiple straight lines at the centroid / mass of the connected domain, intersect the boundary of the connected domain, calculate the distance between the two intersection points, and denote the line segment with the longest distance as the major axis and the line segment with the shortest distance as the minor axis. The aspect ratio is the ratio of the major axis to the minor axis of the smallest bounding rectangle of the connected region; The perimeter ratio is the ratio of the perimeter of a connected region to the longer side of the smallest bounding rectangle of the connected region.

[0012] Preferably, if the coordinates of a certain centroid / mass center are located in any connected region B j / connected region B j Within the smallest bounding rectangle, further determination is made based on the intersection-union ratio to determine whether the consistency between connected component A' and connected component B' meets the requirements: Obtain the intersection-union ratio of connected components A' and B':

[0013] Where, represents the number of pixels in the intersection of two connected components, and represents the number of pixels in the union of two connected components; When the intersection-union ratio is greater than a preset threshold, the consistency of connected components A' and B' is satisfied.

[0014] Preferably, if the coordinates of a certain centroid / mass center are located in any connected region B j / connected region B j Within the minimum bounding rectangle, further analysis is conducted based on the feature map output by the false detection model to determine whether the consistency between connected component A' and connected component B' meets the requirements. In the image to be detected, the gray values ​​of the pixels in the connected region A' are kept unchanged, while the gray values ​​of the pixels in other regions are set to zero, thus obtaining image I; In the image to be detected, the gray values ​​of the pixels in the connected region B' are kept unchanged, while the gray values ​​of the pixels in other regions are set to zero, thus obtaining image II; Image I and image II are input into the false detection model, and the encoder in the false detection model is used to downsample image I and image II respectively, and output the corresponding feature map I and feature map II. Calculate the similarity between feature graph I and feature graph II, and determine whether the consistency of connected components A' and B' meets the requirements based on the similarity.

[0015] Preferably, the method for calculating the similarity between feature map I and feature map II is as follows: Flatten feature map I and feature map II into one-dimensional vectors respectively; Calculate the cosine similarity between two one-dimensional vectors, map the cosine similarity to the range [0, 1], and denote it as the degree of similarity.

[0016] Furthermore, in step one, the training steps for the false detection model are as follows: I. Obtain the false detection dataset; II. Input the acquired images from the false detection dataset into the false detection model. The false detection model includes an encoder and a decoder. The encoder downsamples the acquired images input into the model and outputs a feature map. The decoder upsamples the feature map and restores the feature map to the size of the acquired image. The encoder has two output channels, which represent the scores for classification as false detections and non-false detections, respectively. Based on the scores, each pixel in the restored feature map is classified: when the false detection score is greater than the non-false detection score, the pixel is classified as a false detection point; otherwise, the pixel is classified as a non-false detection point. Based on the false detection points and non-false detection points, a binarized false detection image is obtained. Using the corresponding binarized label image as the ground truth, calculate the cross-entropy loss for each pixel in the binarized false detection image, classifying it as a foreground or background pixel; and record the average of the losses of all pixels as the total loss. The network parameters are updated through backpropagation, the total loss is minimized, the false positive model parameters are stored, and the model training is complete.

[0017] Furthermore, the encoder can be a ResNet network model, an EfficientNet network model, a Vit network model, or a SwingTransformer network model. The decoder can be a U-Net network model, a PSPNet network model, a SegNet network model, or an FCN network model.

[0018] Preferably, in step one, the method for comparing the image to be detected with the standard image to find the difference region between the image to be detected and the standard image is: optical flow method, frame difference method or background difference method. In step two, the method for finding connected components is any one of the following: four-neighbor labeling, eight-neighbor labeling, or Two-Pass algorithm.

[0019] This method has the following characteristics: 1) The algorithm uses a deep learning module to predict false detection areas and then integrates it with traditional digital image processing methods, which enhances the robustness of the algorithm. By screening suspected areas, the false detection rate of foreign objects is greatly reduced.

[0020] 2) No need to use a large number of abnormal samples. Since the design idea of ​​this invention is to use a deep learning model to predict false detection regions, only positive samples (normal images) need to be collected when collecting samples. False detection regions are generated based on normal images for training, which is suitable for industrial detection scenarios where the number of abnormal samples is small and difficult to obtain. In addition, this solution does not require manual annotation when obtaining the training dataset, which shortens the model training time and cost.

[0021] 3) This solution can not only classify images as having or not having foreign objects, but also accurately locate the pixel set of the foreign object area, thus enabling the marking and positioning of the foreign object area. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the foreign object detection process in a specific implementation method; Figure 2 This is a schematic diagram illustrating the process of acquiring false detection datasets in a specific implementation method; Figure 3 This is a schematic diagram of the false detection model training process in a specific implementation method; Figure 4 This is a schematic diagram illustrating the process of analyzing connected component consistency using feature maps in a specific implementation. Figure 5 This is a schematic diagram of the pantograph detection process in a specific implementation method; Figure 6 In Figure a, it is a binary representation of the pantograph foreign object detection result; in Figure b, it is a representation of marking the pantograph foreign object detection area on a grayscale image; and in Figure c, it is a representation of selecting the pantograph foreign object area. Detailed Implementation

[0023] For ease of understanding, this implementation provides the following explanation of the semantic segmentation model: The semantic segmentation model (multi-class semantic segmentation model) is mainly used to classify each pixel in an image. This type of model has wide applications in fields such as image analysis, autonomous driving, and satellite image processing.

[0024] The network architecture is based on an encoder-decoder structure. The encoder uses pre-trained models such as ResNet and EfficientNet to extract features, while the decoder recovers spatial information through upsampling. It typically adopts the skip connection structure of U-Net, which effectively fuses features from deep and shallow layers, making it suitable for image segmentation.

[0025] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] A foreign object detection method for reducing the probability of false detection involves a visual sensor acquiring multiple images of the object to be detected in a batch visual inspection scenario or a visual monitoring scenario of the same object at different time periods, thereby detecting whether there are foreign objects on the surface of the object at the time of image acquisition. Among them, such as Figure 1 As shown, the foreign object detection process includes the following steps: Step 1: Compare the image to be detected with the standard image to find the difference regions between them; based on the difference regions (foreground) and the non-difference regions (background), binarize the image to be detected to obtain a binarized abnormal image; the standard image is a pre-stored image of the object under normal conditions. Meanwhile, the image to be detected is input into a pre-trained false detection model, which is a semantic segmentation model. It is used to classify each pixel in the image to be detected and label whether the pixel belongs to a false detection point or a non-false detection point. Based on the false detection points (foreground points) and non-false detection points (background points), the image to be detected is binarized to obtain a binarized false detection image. The false detection model is trained using a false detection dataset, such as... Figure 2As shown, the false detection dataset is obtained in the following way: multiple images of the test object under normal conditions are collected, each image is compared with a standard image, and the difference regions between each image and the standard image are found; according to the difference regions (foreground) and the non-difference regions (background), each image is binarized to obtain multiple binarized label images, and the binarized label images are stored in correspondence with the collected images to form the false detection dataset; Step 2: Find connected components in the binarized outlier image and the binarized false detection image respectively, and denote the found connected components as connected component A. i Connected component B j ; Step 3: Analyze connected component A i Connected component B j The consistency among the connected components in the equation, if a certain connected component A i With any connected component B j If the consistency requirement is met, then the connected component A i This is a false positive region; delete the connected component A. i Otherwise, the connected component A i For foreign object regions, retain the connected component A. i ; Step 4: If connected component A i If all objects are deleted, then the image to be detected is free of foreign objects; otherwise, in the image to be detected, mark the connected components A that were not deleted. i The area is designated as a foreign object zone; foreign object detection is then performed.

[0027] Specifically, step three involves analyzing the connected component A. i Connected component B j The consistency among connected components is achieved through the following methods: Based on the characteristics of connected components, analyze the consistency between them; The features of the connected component include: the location of the connected component, morphological features, intersection-union ratio, and one or more features in the feature map output by the false detection model.

[0028] In practice, the scheme for analyzing the consistency between connected components based on their positions will be denoted as Method 1. The scheme that analyzes the consistency between connected components based on their morphological characteristics is denoted as Method 2. The scheme that analyzes the consistency between connected components based on the intersection-union ratio of the connected components is denoted as Method 3; The scheme that analyzes the consistency between connected components based on the feature maps output by the connected components in the false detection model is denoted as Method 4. Method 1 is a coarse analysis, which can be performed first to roughly locate one or more pairs of connected components with consistent positions, shortening the analysis time. Methods 2 through 4 are deep analyses, used in conjunction with Method 1 to further analyze whether the consistency between the two meets the requirements, improving accuracy. More specifically, Methods 2 through 4 can be combined with Method 1 individually or in combination with all of them.

[0029] In this embodiment, Method 1 is performed first, followed by Methods 2 through 4.

[0030] Specifically, we first analyze the consistency between connected components based on their location, as follows: Obtain each connected component A separately i The centroid / mass; Connect the coordinates of each centroid / mass center to the connected component B. j Perform position matching if the coordinates of a certain centroid / mass are located in any connected region B. j / connected region B j Within the smallest bounding rectangle: Then denote the connected region A corresponding to the centroid / mass coordinates respectively. i Let A' be the connected region and B be the connected region where the centroid / mass coordinates are located. j Let A' be a connected component B'; the consistency between connected component A' and connected component B' satisfies the requirements. Otherwise, the connected region A corresponding to the centroid / mass coordinates i With connected component B j The consistency requirement is not met. i .

[0031] To improve accuracy, if a centroid / mass coordinate lies in any connected region B j / connected region B j Within the minimum bounding rectangle, further based on the morphological characteristics of the connected components, it is determined whether the consistency between connected component A' and connected component B' meets the requirements: Obtain the morphological features of connected components A' and B' respectively; Calculate the similarity between morphological features; determine whether the consistency of connected components A' and B' meets the requirements based on the similarity. The morphological features include a combination of one or more of the following: circularity, rectangularity, convexity, ratio of minor axis to major axis, angle between major axis and minor axis, width-to-length ratio, and circumference ratio of the connected region. Convexity is the ratio of the area of ​​the connected region to the area of ​​the convex hull of the connected region. The minor axis and major axis are defined as follows: draw multiple straight lines at the centroid / mass of the connected domain, intersect the boundary of the connected domain, calculate the distance between the two intersection points, and denote the line segment with the longest distance as the major axis and the line segment with the shortest distance as the minor axis. The aspect ratio is the ratio of the major axis to the minor axis of the smallest bounding rectangle of the connected region; The perimeter ratio is the ratio of the perimeter of a connected region to the longer side of the smallest bounding rectangle of the connected region.

[0032] To improve accuracy, if a centroid / mass coordinate lies in any connected region B j / connected region B j Within the smallest bounding rectangle, further determination is made based on the intersection-union ratio to determine whether the consistency between connected component A' and connected component B' meets the requirements: Obtain the intersection-union ratio of connected components A' and B':

[0033] Where, represents the number of pixels in the intersection of two connected components, and represents the number of pixels in the union of two connected components; When the intersection-union ratio is greater than a preset threshold, the consistency of connected components A' and B' is satisfied.

[0034] To improve accuracy, if a centroid / mass coordinate lies in any connected region B j / connected region B j Within the minimum bounding rectangle, further analysis is conducted based on the feature map output by the false detection model to determine whether the consistency between connected component A' and connected component B' meets the requirements. like Figure 4 As shown, in the image to be detected, the gray values ​​of the pixels in the connected region A' are kept unchanged, while the gray values ​​of the pixels in other regions are set to zero, thus obtaining image I; In the image to be detected, the gray values ​​of the pixels in the connected region B' are kept unchanged, while the gray values ​​of the pixels in other regions are set to zero, thus obtaining image II; Image I and image II are input into the false detection model, and the encoder in the false detection model is used to downsample image I and image II respectively, and output the corresponding feature map I and feature map II. Calculate the similarity between feature graph I and feature graph II, and determine whether the consistency of connected components A' and B' meets the requirements based on the similarity.

[0035] The preferred method for calculating the similarity between feature map I and feature map II is as follows: Flatten feature map I and feature map II into one-dimensional vectors respectively; Calculate the cosine similarity between two one-dimensional vectors, map the cosine similarity to the range [0, 1], and denote it as the degree of similarity.

[0036] In step one, such as Figure 3 As shown, the training steps for the false detection model are as follows: I. Obtain the false detection dataset; II. Input the acquired images from the false detection dataset into the false detection model. The false detection model includes an encoder and a decoder. The encoder downsamples the acquired images input into the model and outputs a feature map. The decoder upsamples the feature map and restores the feature map to the size of the acquired image. The encoder has two output channels, which represent the scores for classification as false detection (foreground) and non-false detection (background), respectively. Based on the scores, each pixel in the restored feature map is classified: when the false detection score is greater than the non-false detection score, the pixel is classified as a false detection point (foreground point); otherwise, the pixel is classified as a non-false detection point (background point). Based on the false detection points and non-false detection points, the image is binarized to obtain a binarized false detection image. Using the corresponding binarized label image as the ground truth, calculate the cross-entropy loss for each pixel in the binarized false detection image, classifying it as a foreground or background pixel; and record the average of the losses of all pixels as the total loss. The network parameters are updated through backpropagation, the total loss is minimized, the false positive model parameters are stored, and the model training is complete.

[0037] Depending on the complexity of the image to be detected, a deep learning model is selected, wherein the encoder is a ResNet network model, an EfficientNet network model, a Vit network model, or a Swin Transformer network model. The decoder can be a U-Net network model, a PSPNet network model, a SegNet network model, or an FCN network model.

[0038] In specific implementation, in step one, the method to compare the image to be detected with the standard image and find the difference area between the image to be detected and the standard image is: optical flow method, frame difference method or background difference method. In step two, the method for finding connected components is any one of the following: four-neighbor labeling, eight-neighbor labeling, or Two-Pass algorithm.

[0039] To better understand the design concept of this solution, the foreign object detection process is illustrated below: An image-based foreign object detection method to reduce the probability of false detection involves training a false detection model before the actual foreign object detection begins. The steps are as follows: I. In a visual inspection scenario involving batches of objects to be tested or a visual monitoring scenario involving the same object at different time periods, collect 1000 images of the object under normal conditions. Select one image as the standard image and compare each collected image with the standard image to find the difference regions between each collected image and the standard image. Based on the difference regions (foreground point grayscale value 255) and the non-difference regions (background point grayscale value 0), binarize each collected image to obtain multiple binarized label images. Store the binarized label images and the collected images accordingly to form a false detection dataset. II. Input the acquired images from the false detection dataset into the false detection model, which includes an encoder ResNet50 and a decoder U-net. The encoder ResNet50 downsamples the acquired images input into the model and outputs a feature map; the decoder upsamples the feature map and restores the feature map to the size of the acquired image. The encoder U-net has two output channels, which represent the scores for classification as false detection and non-false detection, respectively. Based on the scores, each pixel in the restored feature map is classified: when the false detection score is greater than the non-false detection score, the pixel is classified as a false detection point; otherwise, the pixel is classified as a non-false detection point. Based on the false detection points (foreground point grayscale value 255) and the non-false detection points (background point grayscale value 0), a binarized false detection image is obtained. Using the corresponding binarized label image as the ground truth, calculate the cross-entropy loss for each pixel in the binarized false detection image, classifying it as a foreground or background pixel; and record the average of the losses of all pixels as the total loss. The network parameters are updated through backpropagation, the total loss is minimized, the false positive model parameters are stored, and the model training is complete.

[0040] The following is the formal testing process, including the following steps: Step 1: Compare the image to be detected with the standard image to find the difference regions between them; based on the difference regions (foreground point grayscale value 255) and the non-difference regions (background point grayscale value 0), binarize the image to be detected to obtain a binarized abnormal image; the standard image is a pre-stored image of the object under normal conditions. Simultaneously, the image to be detected is input into a pre-trained false detection model, which is a semantic segmentation model used to classify each pixel in the image to be detected, labeling pixels as false detection points or non-false detection points. Based on the false detection points (foreground point grayscale value 255) and non-false detection points (background point grayscale value 0), the image to be detected is binarized to obtain a binarized false detection image. Step two: Connected components are searched in the binarized abnormal image and the binarized false detection image respectively, and the found connected components are denoted as connected component A. i Connected component B j ; Step 3: First, analyze the consistency between connected components based on their location (coarse analysis). The steps are as follows: Obtain each connected component A separately i The centroid / mass; Connect the coordinates of each centroid / mass center to the connected component B. j Perform position matching if the coordinates of a certain centroid / mass are located in any connected region B. j / connected region B j Within the smallest bounding rectangle: Then denote the connected region A corresponding to the centroid / mass coordinates respectively. i Let A' be the connected region and B be the connected region where the centroid / mass coordinates are located. j Let A' be a connected component B'. Initially, it is assumed that the consistency between connected components A' and B' meets the requirements. Further analysis is conducted based on morphological features and intersection-union ratio to determine whether the consistency between the two meets the requirements. If it does, further analysis is conducted based on the feature map output in the false detection model to determine whether the consistency between the two meets the requirements. Otherwise, the connected region A corresponding to the centroid / mass coordinates i With connected component B j The consistency requirement is not met. i .

[0041] Specifically, based on the morphological characteristics and intersection-union ratio of the connected components, it is further determined whether the consistency between connected component A' and connected component B' meets the requirements. The process is as follows: Obtain the morphological features of connected components A' and B' respectively; Specifically, the circularity of a connected component is obtained as the ratio of the area of ​​the connected component to the area of ​​its smallest circumcircle, with a range of [0,1]. Rectangularity of a connected component: the ratio of the area of ​​the connected component to the area of ​​its smallest bounding rectangle, with a range of [0,1]. Convexity of a connected component: the ratio of the area of ​​the connected component to the area of ​​its convex hull, with a value range of (0,1]. The ratio of the minor axis to the major axis of a connected component, with a value range of [0,1]. The angle between the major and minor axes of a connected domain is denoted as θ[0-180], in degrees. Let θ / 180 to map the angle to the interval [0-1]. Width-to-length ratio of a connected region: The ratio of the major axis to the minor axis of the smallest bounding rectangle of the connected region; Perimeter ratio of a connected region: The ratio of the perimeter of a connected region to the longest side of the smallest bounding rectangle of the connected region; The above 7 morphological features are stored sequentially to form a one-dimensional feature vector, and the similarity δ between the one-dimensional feature vectors corresponding to connected region A' and connected region B' is calculated. Obtain the intersection-union ratio of connected components A' and B':

[0042] Where, represents the number of pixels in the intersection of two connected components, and represents the number of pixels in the union of two connected components; Take the mean of similarity δ and intersection-union ratio. When the mean is greater than the preset threshold, the consistency of connected component A' and connected component B' meets the requirements.

[0043] To improve accuracy, the consistency between connected component A' and connected component B' is further determined based on the feature map output by the false detection model. In the image to be detected, the gray values ​​of the pixels in the connected region A' are kept unchanged, while the gray values ​​of the pixels in other regions are set to zero, thus obtaining image I; In the image to be detected, the gray values ​​of the pixels in the connected region B' are kept unchanged, while the gray values ​​of the pixels in other regions are set to zero, thus obtaining image II; Image I and image II are input into the false detection model, and the encoder in the false detection model is used to downsample image I and image II respectively, and output the corresponding feature map I and feature map II. Flatten feature map I and feature map II into one-dimensional vectors respectively; Calculate the cosine similarity ω between two one-dimensional vectors. Let Ω = (ω + 1) / 2, map the cosine similarity to the range [0, 1], and denote Ω as the degree of similarity.

[0044] When the similarity Ω is greater than the preset value, it is determined that the connection components A' and B' meet the consistency requirement.

[0045] like Figure 5 This is a schematic diagram of the pantograph image detection process; Figure 6 a~ Figure 6 c represents the detection results of different types of foreign object regions (the original image boundaries have been cropped for easier display). Figure 6 a is a binary representation of the foreign object detection results for the pantograph; Figure 6 b is a schematic diagram showing the foreign object detection area of ​​the pantograph marked on a grayscale image; Figure 6 Figure c shows a schematic diagram of the foreign object area selected by the pantograph. As can be seen from the figure, this method can not only classify the image as having or not having foreign objects, but also accurately locate the pixel set of the foreign object area, realizing the marking and location of the foreign object area, and the detection results are more intuitive.

[0046] This method and approach eliminates the need for manual annotation when acquiring the training dataset, thus shortening model training time and reducing costs. By using a deep learning module to predict false detection regions and then integrating it with traditional digital image processing methods, the robustness of the algorithm is enhanced. By screening suspected regions, the false detection rate of foreign objects is greatly reduced.

[0047] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and descriptive purposes. It is not intended to be exhaustive, nor to limit the invention to the precise forms disclosed; obviously, many changes and variations are possible in accordance with the foregoing teachings. The exemplary embodiments were chosen and described to explain the specific principles of the invention and its practical application, thereby enabling others skilled in the art to implement and utilize various exemplary embodiments of the invention, as well as their different alternatives and modifications. The scope of the invention is intended to be defined by the appended claims and their equivalents.

Claims

1. An image foreign object detection method to reduce the probability of false detection, wherein in a visual inspection scenario of batch objects to be tested or a visual monitoring scenario of the same object to be tested at different time periods, a visual sensor acquires images of the objects to be tested multiple times, and detects whether there are foreign objects on the surface of the objects to be tested when the images are acquired. Its features are, The foreign object detection process includes the following steps: Step 1: Compare the image to be detected with the standard image to find the difference regions between them; based on the difference regions and non-difference regions, binarize the image to be detected to obtain a binarized abnormal image; the standard image is a pre-stored image of the object under normal conditions. Meanwhile, the image to be detected is input into a pre-trained false detection model, which is a semantic segmentation model. It is used to classify each pixel in the image to be detected and label whether the pixel belongs to a false detection point or not. Based on the false detection points and not false detection points, the image to be detected is binarized to obtain a binarized false detection image. The false detection model is trained using a false detection dataset, which is obtained as follows: multiple images of the object under normal conditions are collected, each image is compared with a standard image, and the difference regions between each image and the standard image are identified; based on the difference regions and non-difference regions, each image is binarized to obtain multiple binarized label images, and the binarized label images are stored in correspondence with the collected images to form a false detection dataset. Step 2: Find connected components in the binarized outlier image and the binarized false detection image respectively, and denote the found connected components as connected component A. i Connected component B j ; Step 3: Analyze connected component A i Connected component B j The consistency among the connected components in the equation, if a certain connected component A i With any connected component B j If the consistency requirement is met, then the connected component A i This is a false positive region; delete the connected component A. i Otherwise, the connected component A i For foreign object regions, retain the connected component A. i ; Step 4: If the connected component A i If all objects are deleted, the image to be detected is free of foreign objects; otherwise, in the image to be detected, mark the connected components A that were not deleted. i The area is designated as a foreign object zone; foreign object detection is then performed.

2. The image foreign object detection method for reducing false detection probability as described in claim 1, characterized in that: Step 3: Analyze connected component A i Connected component B j The consistency among connected components is achieved through the following methods: Based on the characteristics of connected components, analyze the consistency between them; The features of the connected component include: the location of the connected component, morphological features, intersection-union ratio, and one or more features in the feature map output by the false detection model.

3. The image foreign object detection method for reducing false detection probability as described in claim 2, characterized in that: First, analyze the consistency between connected components based on their location. The steps are as follows: Obtain each connected component A separately i The centroid / mass; Connect the coordinates of each centroid / mass center to the connected component B. j Perform position matching if the coordinates of a certain centroid / mass are located in any connected region B. j / connected region B j Within the smallest bounding rectangle: Then denote the connected region A corresponding to the centroid / mass coordinates respectively. i Let A' be the connected region and B be the connected region where the centroid / mass coordinates are located. j Let A' be a connected component B'; the consistency between connected component A' and connected component B' satisfies the requirements. Otherwise, the connected region A corresponding to the centroid / mass coordinates i With connected component B j The consistency requirement is not met. i .

4. The image foreign object detection method for reducing false detection probability as described in claim 3, characterized in that: If the coordinates of a centroid / mass center lie in any connected region B j / connected region B j Within the minimum bounding rectangle, further based on the morphological characteristics of the connected components, it is determined whether the consistency between connected component A' and connected component B' meets the requirements: Obtain the morphological features of connected components A' and B' respectively; Calculate the similarity between morphological features; determine whether the consistency of connected components A' and B' meets the requirements based on the similarity. The morphological features include a combination of one or more of the following: circularity, rectangularity, convexity, ratio of minor axis to major axis, angle between major axis and minor axis, width-to-length ratio, and circumference ratio of the connected region. Convexity is the ratio of the area of ​​the connected region to the area of ​​the convex hull of the connected region. The minor axis and major axis are defined as follows: draw multiple straight lines at the centroid / mass of the connected domain, intersect the boundary of the connected domain, calculate the distance between the two intersection points, and denote the line segment with the longest distance as the major axis and the line segment with the shortest distance as the minor axis. The aspect ratio is the ratio of the major axis to the minor axis of the smallest bounding rectangle of the connected region; The perimeter ratio is the ratio of the perimeter of a connected region to the longer side of the smallest bounding rectangle of the connected region.

5. The image foreign object detection method for reducing false detection probability as described in claim 3, characterized in that: If the coordinates of a centroid / mass center lie in any connected region B j / connected region B j Within the smallest bounding rectangle, further determination is made based on the intersection-union ratio to determine whether the consistency between connected component A' and connected component B' meets the requirements: Obtain the intersection-union ratio of connected components A' and B': ； Where, represents the number of pixels in the intersection of two connected components, and represents the number of pixels in the union of two connected components; When the intersection-union ratio is greater than a preset threshold, the consistency of connected components A' and B' is satisfied.

6. The image foreign object detection method for reducing false detection probability as described in any one of claims 3 to 5, characterized in that: If the coordinates of a centroid / mass center lie in any connected region B j / connected region B j Within the minimum bounding rectangle, further analysis is conducted based on the feature map output from the false detection model to determine whether the consistency between connected component A' and connected component B' meets the requirements. In the image to be detected, the gray values ​​of the pixels in the connected region A' are kept unchanged, while the gray values ​​of the pixels in other regions are set to zero, thus obtaining image I; In the image to be detected, the gray values ​​of the pixels in the connected region B' are kept unchanged, while the gray values ​​of the pixels in other regions are set to zero, thus obtaining image II; Image I and image II are input into the false detection model, and the encoder in the false detection model is used to downsample image I and image II respectively, and output the corresponding feature map I and feature map II. Calculate the similarity between feature graph I and feature graph II, and determine whether the consistency of connected components A' and B' meets the requirements based on the similarity.

7. The image foreign object detection method for reducing false detection probability as described in claim 6, characterized in that: The method for calculating the similarity between feature map I and feature map II is as follows: Flatten feature map I and feature map II into one-dimensional vectors respectively; Calculate the cosine similarity between two one-dimensional vectors, map the cosine similarity to the range [0, 1], and denote it as the degree of similarity.

8. The image foreign object detection method for reducing false detection probability as described in claim 1, characterized in that: In step one, the training steps for the false detection model are as follows: I. Obtain the false detection dataset; II. Input the acquired images from the false detection dataset into the false detection model. The false detection model includes an encoder and a decoder. The encoder downsamples the acquired images input into the model and outputs a feature map. The decoder upsamples the feature map and restores the feature map to the size of the acquired image. The encoder has two output channels, which represent the scores for classification as false detection and non-false detection, respectively. Based on the scores, each pixel in the restored feature map is classified: when the false detection score is greater than the non-false detection score, the pixel is classified as a false detection point; otherwise, the pixel is classified as a non-false detection point. Based on the false detection points and non-false detection points, a binarized false detection image is obtained; Using the corresponding binarized label image as the ground truth, calculate the cross-entropy loss for each pixel in the binarized false detection image, classifying it as a foreground or background pixel; and record the average of the losses of all pixels as the total loss. The network parameters are updated through backpropagation, the total loss is minimized, the false positive model parameters are stored, and the model training is complete.

9. The image foreign object detection method for reducing false detection probability as described in claim 8, characterized in that: The encoder is a ResNet network model, an EfficientNet network model, a Vit network model, or a Swing Transformer network model. The decoder can be a U-Net network model, a PSPNet network model, a SegNet network model, or an FCN network model.

10. The image foreign object detection method for reducing false detection probability as described in claim 1, characterized in that: In step one, the method for comparing the image to be detected with the standard image and finding the difference region between the image to be detected and the standard image is: optical flow method, frame difference method or background difference method. In step two, the method for finding connected components is any one of the following: four-neighbor labeling, eight-neighbor labeling, or Two-Pass algorithm.

Images