An industrial scene adaptive continuous learning foreign matter detection method

Abstract

The application discloses an industrial scene adaptive continuous learning foreign matter detection method, which comprises the following steps: S1, motion detection based on long-short frame combination, candidate frame acquisition and confirmation; S2, binary processing is performed on the surrounding of the candidate frame, and edge detection is performed, so that the corresponding position article taking confirmation is completed; S3, candidate area AI detection, the candidate area is cropped based on an AI network, and is matched with a feature library, if the matching is successful, it is determined that it is a corresponding type of foreign matter, and the foreign matter feature library is updated. The application extracts the foreground through GMM, compares and identifies new targets through multiple frames, then performs semantic level judgment with the help of an AI recognition mechanism, and introduces a continuous learning mechanism to continuously optimize the recognition ability, adapt to foreign matter changes, and improve the accuracy, flexibility and intelligent level of the detection system in the actual environment.

Description

Technical Field

[0001] This application belongs to the field of computer vision and image processing technology, and in particular relates to an adaptive continuous learning method for foreign object detection in industrial scenarios. Background Technology

[0002] In rail transit, road transport, airport management, and various industrial plants, foreign objects (such as lost tools, packaging fragments, litter, and fallen parts) can easily cause traffic congestion, equipment damage, operational delays, and even serious safety accidents if they appear on roads or tracks. To ensure operational safety and efficiency, various transportation and industrial scenarios are increasingly relying on video surveillance systems or inspection robots for real-time foreign object detection and alarms.

[0003] Currently, mainstream foreign object detection methods are mainly divided into two categories: one is traditional rule-based methods based on image processing, such as background subtraction, inter-frame subtraction, and edge extraction; the other is object detection models based on deep learning, such as YOLO, Faster R-CNN, and SSD. The former relies on fixed thresholds or template matching, is extremely sensitive to scene changes (such as lighting, shadows, rain, and snow), has poor stability, and a high false positive and false negative rate. The latter, while possessing strong object recognition capabilities, has weak ability to identify "unseen foreign objects," and its model generalization is limited. In addition, AI models typically require a large number of labeled samples for training, making it difficult to cover the constantly changing new foreign objects in the real environment, resulting in high model update costs and difficulty in guaranteeing real-time performance.

[0004] Especially in open environments (such as railway lines, open roads, and airport runways), the type of foreign object is unpredictable and the environment changes drastically. Static recognition methods relying solely on pre-trained models are unlikely to meet the safety requirement of "instant detection and judgment." Traditional inter-frame difference methods also struggle to distinguish between actual foreign objects and environmental noise, failing to provide semantic-level judgment criteria.

[0005] Therefore, there is an urgent need for a detection mechanism that integrates dynamic foreground extraction and intelligent recognition. By utilizing background modeling methods (such as GMM) to compare multiple images, newly emerging targets can be quickly identified. Combined with AI recognition mechanisms, their attributes are determined, thus forming a joint decision-making method for "whether the foreign object exists" and "what the foreign object is." Simultaneously, it needs to possess continuous learning and adaptive update capabilities, enabling the system to gradually improve its recognition ability when faced with new types of foreign objects, thereby achieving truly robust foreign object detection for real-world environments. Summary of the Invention

[0006] The purpose of this application is to overcome the problems of existing technologies and disclose an adaptive continuous learning foreign object detection method for industrial scenarios. The method uses GMM for foreground extraction, multi-frame comparison to identify new targets, and then uses AI recognition mechanism for semantic-level judgment. In addition, a continuous learning mechanism is introduced to continuously optimize the recognition capability, adapt to changes in foreign objects, and improve the accuracy, flexibility and intelligence of the detection system in real environment.

[0007] The objective of this application is achieved through the following technical solution: An adaptive continuous learning foreign object detection method for industrial scenarios, the method comprising: S1: Based on motion detection combining long and short frames, candidate boxes are acquired and confirmed; S2: Binarize the area around the candidate box and perform edge detection to confirm the item being picked up at the corresponding location; S3: Candidate region AI detection. Based on the AI ​​network, candidate regions are cropped and matched with the feature library. If the match is successful, it is identified as a foreign object of the corresponding type, and the foreign object feature library is updated.

[0008] According to a preferred embodiment, step S1 includes generating short-frame candidate boxes and generating long-frame candidate boxes, and using short-frame candidate boxes superimposed with long-frame candidate boxes as candidate boxes of interest.

[0009] According to a preferred embodiment, the candidate box generation process includes: S11: Frame difference modeling. Receive continuous video frames. Set the interval for short frames to 5 to 10 frames and the interval for long frames to 50 to 60 frames. Perform frame difference analysis. Then, overlay the n nearest frame difference images. Use the OTSU algorithm to determine the optimal threshold and perform binarization. S12: Gaussian mixture modeling, using the Gaussian mixture model (GMM) encapsulated in OpenCV to obtain the background binarization map of the received continuous video frames; S13: After weighted fusion of the results obtained in S11 and the background binarized image obtained in S12, edge detection processing is performed on each of them, and the detected edges are thickened. S14: Connected component extraction. The images obtained in S13 are processed by a two-step scanning method to obtain their respective connected components. Based on the obtained connected components, a bounding box is calculated for each one. The calculated bounding box is the smallest bounding box containing the corresponding connected component, thereby obtaining short frame candidate boxes and long frame candidate boxes.

[0010] According to a preferred embodiment, the two-scan method includes: a first scan initially marking connected components for each pixel, and a second scan organizing and merging the initial markings to finally determine the unique identifier of each connected component.

[0011] According to a preferred embodiment, in step S14, the first scanning process includes: Starting from the top left corner of the image, scan each pixel row by row and column by column. When a pixel with the target value is encountered, check the markings of its adjacent pixels above and to the left. If neither of the pixels above nor to the left is labeled, a new unique label is assigned to the current pixel. If one or more of the pixels above and to the left have been labeled, the current pixel is labeled with the smallest label value among these adjacent pixels. By taking the smallest label value, it can be ensured that the same connected components have the same label.

[0012] According to a preferred embodiment, in step S14, the second scanning process includes: Scan each pixel of the image row by row and column by column. For each labeled pixel, check if its label value is the same as the label value of its neighboring pixel. If they are different, the two labels may belong to the same connected component and are merged. An equivalence table is used to record the equivalence relationships between labels. When two different labels are found to belong to the same connected component, their equivalence relationship is recorded in the equivalence table. After the scan is completed, all the labels are organized according to the equivalence table, and different labels belonging to the same connected component are unified into a single representative label.

[0013] According to a preferred embodiment, the edge detection process in step S2 includes: The detected edge information and candidate boxes are compared using IOU (Interval of Usage). If the IOU exceeds the threshold, the verification is passed, indicating that an object with a certain outline was placed at the corresponding location. Otherwise, it is considered that an object has been taken.

[0014] According to a preferred embodiment, in step S3, the AI ​​network adopts a Siamese network, and the matching process with the feature library is to perform feature comparison using metric learning.

[0015] According to a preferred embodiment, the Siamese network comprises two identical subnetworks, each with its own input and shared weights. Given a pair of images, the two subnetworks extract features from the two images respectively.

[0016] According to a preferred embodiment, the sub-network is a convolutional neural network architecture, which extracts feature representations of the image through several convolutional layers, pooling layers and fully connected layers. Finally, the features extracted by the two sub-networks are compared by calculating the cosine values ​​of the two vectors to calculate their similarity.

[0017] The aforementioned main solution and its various further alternative solutions can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed in this application. Those skilled in the art, after understanding the solution of this application, will realize that there are many combinations based on the prior art and common general knowledge, all of which are technical solutions to be protected in this application, and will not be exhaustively listed here.

[0018] The beneficial effects of this application are: This application's method is based on a joint modeling mechanism of long and short frame sequences. By capturing image changes at different time scales, it enhances the system's ability to perceive the presence or movement of foreign objects, thereby significantly improving the recall rate. Simultaneously, by combining edge detection and region matching techniques, this invention can effectively distinguish between "left behind" and "removed" object states, further reducing the false alarm rate and enhancing the system's robustness and reliability in complex scenarios. Furthermore, it fully leverages the fine-grained perception advantages of traditional algorithms and the semantic understanding capabilities of AI models, forming a complementary and synergistic approach, providing an efficient, stable, and practically valuable solution for industrial foreign object detection. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the adaptive continuous learning foreign object detection method for industrial scenarios in this application. Detailed Implementation

[0020] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0022] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this application is in use. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] Furthermore, terms such as "horizontal," "vertical," and "sag" do not imply that components must be absolutely horizontal or suspended, but rather that they can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0024] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0025] Furthermore, it should be noted that unless otherwise specified in this application, the specific structures, connections, positions, power sources, etc. involved are all things that a person skilled in the art can know without creative effort based on the prior art.

[0026] Example 1 refer to Figure 1 As shown, this application discloses an adaptive continuous learning foreign object detection method for industrial scenarios. This application uses GMM for foreground extraction, multi-frame comparison to identify new targets, and then uses an AI recognition mechanism for semantic-level judgment. Furthermore, it introduces a continuous learning mechanism to continuously optimize the recognition capability, adapt to changes in foreign objects, and improve the accuracy, flexibility, and intelligence level of the detection system in real-world environments.

[0027] The adaptive continuous learning foreign object detection method for industrial scenarios proposed in this application includes the following steps.

[0028] Step S1: Based on motion detection combining short and long frames, candidate boxes are acquired and confirmed. Step S1 includes generating short-frame candidate boxes and generating long-frame candidate boxes, and using the superposition of short-frame candidate boxes and long-frame candidate boxes as the candidate boxes of interest.

[0029] Specifically, it includes: (1) Frame difference modeling Short frames are set with an interval of 5 to 10 frames, and long frames are set with an interval of 50 to 60 frames. Frame differences are calculated, and then the n nearest frame difference images are superimposed (weighted according to weights). The OTSU algorithm is used to determine the optimal threshold and then binarize the images.

[0030] (2) Gaussian mixture modeling In this application, the Gaussian Mixture Model (GMM) functionality encapsulated in OpenCV is used. The training data for short frame modeling is 100 historical data sets, and the training data for long frame modeling is 500 historical data sets. The background binarized image obtained from Gaussian Mixture Modeling is weighted and fused with the result obtained from frame difference modeling, and then used together for edge detection and subsequent processing.

[0031] Gaussian Mixture Model (GMM) is an algorithm that models pixel color changes using multiple Gaussian distributions, often used for foreground segmentation in dynamic scenes. Its core idea is to establish multiple Gaussian distributions for each pixel, describing the multimodal changes in the background (such as lighting fluctuations and dynamic textures). Pixels that do not match the distributions are considered foreground. The algorithm adaptively adjusts the background model by continuously updating the distribution's mean, variance, and weights, thus distinguishing between stationary backgrounds and moving targets. GMM can effectively handle complex backgrounds (such as swaying leaves), but it is sensitive to parameters and has high computational cost.

[0032] (3) Edge detection Edge detection is performed on the binarized data, and the detected edges are coarsened to facilitate subsequent improvement of the recall rate of connected component detection.

[0033] (4) Connected component detection The images obtained by the aforementioned edge detection are processed by a two-step scanning method to obtain their respective connected regions. Based on the obtained connected regions, a bounding box is calculated for each region, and the calculated bounding box is the smallest bounding box containing the corresponding connected region, thereby obtaining short frame candidate boxes and long frame candidate boxes.

[0034] The two-scan method determines connected components based on pixel connectivity. In binary images, adjacent pixels with the same value (e.g., white pixels have a value of 255, and black pixels have a value of 0) are typically considered connected. By scanning the image twice, the first scan initially labels the connected components for each pixel, and the second scan organizes and merges these initial labels, ultimately determining a unique identifier for each connected component.

[0035] <1> First scan Starting from the top left corner of the image, scan each pixel of the image row by row and column by column.

[0036] When a pixel with a target value (such as white) is encountered, the labeling status of its upper and left adjacent pixels is checked. If neither of its upper nor left adjacent pixels is labeled (i.e., background pixels), a new unique label is assigned to the current pixel. If one or more of its upper and left adjacent pixels are already labeled, the current pixel is labeled with the smallest label value among these adjacent pixels. This is because, during the scanning process, earlier labels are considered to have higher priority, and taking the smallest label value ensures that connected components have the same label.

[0037] <2> Second scan Scan each pixel of the image again, row by row and column by column. For each labeled pixel, check if its label value is the same as the label value of its neighboring pixels. If they are different, it means that the two labels may belong to the same connected component and need to be merged. Record the equivalence relationships between labels using an equivalence table. When two different labels are found to belong to the same connected component, record their equivalence relationship in the equivalence table.

[0038] After the scan is complete, all labels are organized according to the equivalence table, and different labels belonging to the same connected component are unified into a single representative label. Typically, the smallest label in the equivalence class is chosen as the representative label.

[0039] <3> Generate candidate boxes Based on the found connected regions, a bounding box is calculated for each region. The bounding box to be calculated is the smallest bounding box that contains the connected region.

[0040] Step S2: Binarize the area around the candidate box and perform edge detection to confirm the item being picked up at the corresponding location.

[0041] The detected edge information and candidate boxes are compared using IOU (Interval in Context). If the IOU exceeds the threshold, the verification is passed, indicating that an item with a certain outline was placed at that location; otherwise, it is considered that an item has been taken.

[0042] Step S3: Candidate Area AI Detection. Having identified candidate areas and filtered out the possibility of object removal, we next crop the candidate areas and match them against the feature library. If a match is successful, it is considered a foreign object of that type, and the foreign object feature library is updated. The feature library contains feature data for various types of foreign objects and is continuously iterated, allowing for periodic manual review.

[0043] The matching process with the feature library employs metric learning for feature comparison. The AI ​​network uses a Siamese network, which typically consists of two identical sub-networks, each with its own input but shared weights. The input is usually a pair of images, and the two sub-networks extract features from each image separately. The sub-networks can be various common convolutional neural network architectures, extracting feature representations of the images through multiple convolutional layers, pooling layers, and fully connected layers. Finally, the features extracted by the two sub-networks are compared to calculate their similarity, which is done by calculating the cosine of the two vectors. This application's method is based on a joint modeling mechanism of long and short frame sequences. By capturing image changes at different time scales, it enhances the system's ability to perceive the presence or movement of foreign objects, thereby significantly improving the recall rate. Simultaneously, by combining edge detection and region matching techniques, this invention can effectively distinguish between "left behind" and "removed" object states, further reducing the false alarm rate and enhancing the system's robustness and reliability in complex scenarios. Furthermore, it fully leverages the fine-grained perception advantages of traditional algorithms and the semantic understanding capabilities of AI models, forming a complementary and synergistic approach, providing an efficient, stable, and practically valuable solution for industrial foreign object detection.

[0044] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An adaptive continuous learning foreign object detection method for industrial scenarios, characterized in that, The industrial scenario adaptive continuous learning foreign object detection method includes: S1: Based on motion detection combining long and short frames, candidate boxes are acquired and confirmed; Step S1 includes generating short-frame candidate boxes and long-frame candidate boxes, and using short-frame candidate boxes superimposed with long-frame candidate boxes as candidate boxes of interest; The candidate box generation process includes: S11: Frame difference modeling. Receive continuous video frames. Set the interval for short frames to 5 to 10 frames and the interval for long frames to 50 to 60 frames. Perform frame difference analysis. Then, overlay the n nearest frame difference images. Use the OTSU algorithm to determine the optimal threshold and perform binarization. S12: Gaussian mixture modeling, using the Gaussian mixture model (GMM) encapsulated in OpenCV to obtain the background binarization map of the received continuous video frames; S13: After weighted fusion of the results obtained in S11 and the background binarized image obtained in S12, edge detection processing is performed on each of them, and the detected edges are thickened. S14: Connected component extraction. The images obtained in S13 are processed by a two-scan method to obtain their respective connected components. Based on the obtained connected components, a bounding box is calculated for each one. The obtained bounding box is the smallest bounding box containing the corresponding connected component, thereby obtaining short frame candidate boxes and long frame candidate boxes. S2: Binarize the area around the candidate box and perform edge detection to confirm the item being picked up at the corresponding location; S3: Candidate region AI detection. Based on the AI ​​network, the candidate region is cropped and matched with the feature library. If the match is successful, it is identified as a foreign object of the corresponding type and the foreign object feature library is updated. In step S3, the AI ​​network uses a Siamese network, and the matching process with the feature library is performed by comparing features using metric learning.

2. The industrial scene adaptive continuous learning foreign object detection method as described in claim 1, characterized in that, The two-stage scanning method includes: the first scan initially marks the connected components for each pixel, and the second scan organizes and merges the initial marks to finally determine the unique identifier of each connected component.

3. The industrial scene adaptive continuous learning foreign object detection method as described in claim 2, characterized in that, In step S14, the first scanning process includes: Starting from the top left corner of the image, scan each pixel row by row and column by column. When a pixel with the target value is encountered, check the markings of its adjacent pixels above and to the left. If neither of the pixels above nor to the left is labeled, a new unique label is assigned to the current pixel. If one or more of the pixels above or to the left have been labeled, the current pixel is labeled with the smallest label value among these adjacent pixels. By taking the smallest label value, it is ensured that the same connected component has the same label.

4. The industrial scene adaptive continuous learning foreign object detection method as described in claim 3, characterized in that, In step S14, the second scanning process includes: Scan each pixel of the image row by row and column by column. For each labeled pixel, check if its label value is the same as the label value of its neighboring pixel. If they are different, the two labels may belong to the same connected component and are merged. An equivalence table is used to record the equivalence relationships between labels. When two different labels are found to belong to the same connected component, their equivalence relationship is recorded in the equivalence table. After the scan is completed, all the labels are organized according to the equivalence table, and different labels belonging to the same connected component are unified into a single representative label.

5. The industrial scene adaptive continuous learning foreign object detection method as described in claim 1, characterized in that, The edge detection process in step S2 includes: The detected edge information and candidate boxes are compared using IOU (Interval of Usage). If the IOU exceeds the threshold, the verification is passed, indicating that an object with a certain outline was placed at the corresponding location. Otherwise, it is considered that an object has been taken.

6. The industrial scene adaptive continuous learning foreign object detection method as described in claim 1, characterized in that, A Siamese network consists of two identical subnetworks, each with its own input and shared weights. Given a pair of images, the two subnetworks extract features from each image respectively.

7. The industrial scene adaptive continuous learning foreign object detection method as described in claim 6, characterized in that, The sub-network is a convolutional neural network architecture, which extracts feature representations of the image through several convolutional layers, pooling layers and fully connected layers. Finally, the features extracted by the two sub-networks are compared by calculating the cosine values ​​of the two vectors to calculate their similarity.

Images