Defect detection method based on multi-glass intelligent recognition and real-time processing
By identifying multiple glass regions through dynamic background modeling and differential processing, and assigning a unique identifier to each region, and combining spatiotemporal correlation matching for splicing and merging, the problem of confusion and misjudgment in the detection of multiple glass pieces is solved, and efficient defect detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN KELUODE TECH CO LTD
- Filing Date
- 2026-02-05
- Publication Date
- 2026-06-12
Smart Images
Figure CN122199392A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of defect detection technology, and in particular to a defect detection method based on multi-glass intelligent identification and real-time processing. Background Technology
[0002] During glass production, various defects such as scratches, bubbles, stones, and dirt may occur on the glass surface due to factors such as raw materials, processes, and equipment. These defects not only affect the product's appearance but may also reduce the glass's mechanical strength and optical properties. Therefore, efficient and accurate defect detection is a crucial step in ensuring product quality. Traditional manual visual inspection methods suffer from low efficiency, poor consistency, and high missed detection rates, making them unsuitable for the demands of modern large-scale production. Automated inspection technology based on machine vision is gradually becoming the mainstream configuration for glass production lines.
[0003] Currently, most mainstream glass defect detection systems adopt a single-piece detection mode. In this mode, the system acquires images of a single piece of glass through a camera, completes the detection through steps such as image preprocessing, defect identification, and result output, and then proceeds to detect the next piece of glass only after the current piece of glass has been inspected.
[0004] However, when multiple pieces of glass are present on the production line at the same time, the detection system must detect them one by one. The detection equipment is idle while waiting for the glass to enter and leave the detection area. When dealing with the situation where multiple pieces of glass appear in the camera's field of view at the same time, it lacks an effective multi-target recognition and separation mechanism, which can easily lead to missed detections or false detections. It cannot meet the high-efficiency and large-volume production needs of the glass production line. Summary of the Invention
[0005] The main objective of this invention is to propose a defect detection method based on intelligent identification and real-time processing of multiple glass panes, aiming to solve the problem of lacking an effective multi-target identification and separation mechanism when multiple glass panes appear simultaneously within the camera's field of view.
[0006] To achieve the above objectives, this invention proposes a defect detection method based on intelligent multi-glass identification and real-time processing. This method is applied to a glass production line, which includes multiple cameras arranged along the glass movement direction. The defect detection method based on intelligent multi-glass identification and real-time processing includes: Acquire target images from multiple cameras; Dynamically create a reference background for the target image; Differential processing is performed on the target image based on the reference background to identify at least one effective glass region within the field of view of a single camera; Filter out valid glass areas and assign a unique identifier to each valid glass area; Based on spatiotemporal correlation matching, glass segments from adjacent cameras are stitched together and merged; Defect detection is performed in parallel on each of the effective glass regions, and the glass region to which each defect belongs is determined.
[0007] In some embodiments, dynamically establishing a reference background for the target image includes: The target image is modeled to obtain an initial reference background; The reference background is updated in real time according to the changes in the target image; The reference background is adaptively adjusted according to environmental changes.
[0008] In some embodiments, the step of performing background modeling on the target image to obtain an initial reference background includes: A Gaussian mixture model is used to establish a real-time reference background; The parameters of the Gaussian mixture model are updated in real time.
[0009] In some embodiments, the differential processing of the target image based on a reference background to identify at least one effective glass region within the field of view of a single camera includes: The target image and the reference background are subjected to difference processing to obtain a difference image; Adaptive threshold segmentation is applied to the difference image; One or more valid glass regions can be identified simultaneously in a single frame image using a connected component labeling algorithm.
[0010] In some embodiments, the process of screening valid glass areas and assigning a unique identifier to each valid glass area includes: Calculate the area, aspect ratio, and convex hull area ratio of each effective glass region; Valid glass areas are selected based on preset geometric constraints.
[0011] In some embodiments, the process of screening valid glass areas and assigning a unique identifier to each valid glass area further includes: Each identified valid glass region is assigned a unique identifier combining a timestamp and its spatial location.
[0012] In some embodiments, the spatiotemporal correlation matching-based stitching of glass segments from adjacent cameras includes: Kalman filtering is used to predict the location and time of the glass's appearance in the next camera. The matching correctness was verified by analyzing the continuity of edge gradient direction and intensity.
[0013] In some embodiments, the spatiotemporal correlation matching-based stitching of glass segments from adjacent cameras includes: A scale-invariant feature transformation algorithm is used for feature matching, and a random sampling consistency algorithm is used to ensure splicing accuracy. Segment splicing is performed in real time during the glass movement.
[0014] In some embodiments, performing defect detection in parallel on each of the effective glass regions and determining the glass region to which each defect belongs includes: A rapid preliminary inspection of the effective glass area is performed based on edge detection and morphological manipulation algorithms to obtain candidate defects; Candidate defects are input into a convolutional neural network classifier for defect type classification, and then accurately identified through a semantic segmentation network. The glass region to which each defect belongs is determined based on the ray method and / or the number of turns algorithm; For defects located at the edge of the glass, their attribution is determined through a buffer mechanism and distance transformation.
[0015] In some embodiments, the defect detection method based on multi-glass intelligent identification and real-time processing further includes: The image is processed line by line based on the sliding window mechanism, and the glass region boundary and geometric features are updated in real time. Each effective glass region is assigned an independent processing thread to perform defect detection in parallel. Track the processing progress of each piece of glass and trigger the output after all detection calculations are completed; Batch synchronous output is achieved through a double buffering mechanism, preparing the next batch of data while processing the current batch.
[0016] The beneficial effects of the technical solution of this invention are as follows: By dynamically establishing a reference background and performing differential processing on the target image, multiple effective glass areas can be identified simultaneously within the field of view of a single camera, thereby breaking through the limitations of the traditional single-piece detection mode and realizing the simultaneous identification and processing of multiple pieces of glass on the production line; on this basis, a unique identifier is assigned to each effective glass area, enabling the system to effectively distinguish and independently manage multiple pieces of glass, avoiding confusion and misjudgment in multi-target scenes; at the same time, when the glass spans multiple camera fields of view, the glass segments of adjacent cameras are spliced and merged based on spatiotemporal correlation matching to ensure the integrity identification of large-size glass; furthermore, defect detection is performed in parallel on each effective glass area and the glass area to which each defect belongs is accurately determined, realizing the parallel processing of multiple pieces of glass and the accurate attribution of defects, greatly improving detection efficiency and equipment utilization, and meeting the high-efficiency, large-volume production needs of modern glass production lines. Attached Figure Description
[0017] Figure 1 This is a flowchart of an embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing according to the present invention; Figure 2 This is a flowchart of an embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing according to the present invention; Figure 3 This is a flowchart of another embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing of the present invention; Figure 4 This is a flowchart of an embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing according to the present invention; Figure 5 This is a flowchart of an embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing according to the present invention; Figure 6 This is a flowchart of an embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing according to the present invention; Figure 7 This is a flowchart of an embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing according to the present invention; Figure 8 This is a flowchart of another embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing of the present invention; Figure 9 This is a flowchart of another embodiment of the defect detection method based on multi-glass intelligent identification and real-time processing of the present invention.
[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0019] The solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.
[0021] It should also be noted that when a component is described as "fixed to" or "set on" another component, it can be directly on the other component or there may be an intervening component present. When a component is described as "connected to" another component, it can be directly connected to the other component or there may be an intervening component present.
[0022] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.
[0023] In related technologies, when multiple pieces of glass exist simultaneously on a production line, the detection system must inspect them one by one, leaving the detection equipment idle while waiting for glass to enter and leave the detection area. Furthermore, when dealing with multiple pieces of glass appearing simultaneously within the camera's field of view, existing systems lack effective multi-target recognition and separation mechanisms, easily leading to missed or false detections, and failing to meet the high-efficiency, high-volume production demands of glass production lines. Therefore, this embodiment proposes a defect detection method based on intelligent multi-glass recognition and real-time processing. By dynamically establishing a reference background for the target image and performing differential processing, multiple valid glass areas can be identified simultaneously within a single camera's field of view, thus breaking through the limitations of traditional single-piece detection modes and enabling simultaneous identification and processing of multiple pieces of glass on the production line. Furthermore, a unique identifier is assigned to each valid glass area, enabling the system to effectively distinguish and independently manage multiple pieces of glass, avoiding confusion and misjudgment in multi-target scenarios. Simultaneously, when glass spans multiple camera fields of view, spatiotemporal correlation matching is used to stitch together glass segments from adjacent cameras, ensuring the integrity of large-size glass for identification.
[0024] See Figure 1 This invention proposes a defect detection method based on intelligent multi-glass identification and real-time processing. This method is applied to a glass production line, which includes multiple cameras arranged along the glass movement direction. The defect detection method based on intelligent multi-glass identification and real-time processing includes: Step S10: Acquire target images captured by multiple cameras; Step S20: Dynamically establish a reference background for the target image; Step S30: Perform differential processing on the target image based on the reference background to identify at least one effective glass region within the field of view of a single camera; Step S40: Filter valid glass areas and assign a unique identifier to each valid glass area; Step S50: Based on spatiotemporal correlation matching, the glass segments of adjacent cameras are stitched together and merged; Step S60: Perform defect detection in parallel on each valid glass area and determine the glass area to which each defect belongs.
[0025] In this embodiment, a defect detection method based on multi-glass intelligent identification and real-time processing is applied to a glass production line, which includes multiple cameras arranged along the glass movement direction. Specifically, multiple linear array industrial cameras are arranged laterally along the glass movement direction, with a certain overlap in the field of view between adjacent cameras to ensure that large-size glass can be completely captured. The cameras achieve precise displacement equidistant sampling through a hard trigger controller, with triggering modes including encoder linkage triggering mode and fixed period triggering mode. The encoder linkage triggering mode is suitable for scenarios equipped with encoders, ensuring spatial equidistant sampling through displacement triggering; the fixed period triggering mode is suitable for scenarios with stable roller speeds, triggering acquisition through fixed time intervals. The light source system uses strip LED light sources arranged below the conveyor belt to provide uniform and stable transmitted illumination to the glass.
[0026] The purpose of this step is to acquire raw image data of the glass on the glass production line, providing a data foundation for subsequent background modeling and defect detection. In other embodiments, the camera can also be an area scan camera, the triggering method can be software triggering or free-running mode, and the light source can be reflective illumination or coaxial illumination. Those skilled in the art can choose according to the actual production environment and detection requirements.
[0027] In this embodiment, a Gaussian Mixture Model (GMM) can be used to model the background of the target image to obtain an initial reference background. The Gaussian Mixture Model models the grayscale distribution of each pixel in the image as a weighted sum of multiple Gaussian distributions, effectively adapting to the multimodal characteristics of the background. During system operation, the parameters of the Gaussian Mixture Model are updated in real time according to changes in the target image, enabling the reference background to adapt to environmental factors such as changes in illumination and conveyor belt status.
[0028] The purpose of this step is to establish a stable and reliable background reference, providing a benchmark for subsequent foreground segmentation. Compared to static background templates, dynamic background modeling has stronger environmental adaptability and can effectively cope with background changes caused by factors such as fluctuations in lighting and conveyor belt wear in the production environment. In other embodiments, background modeling can also employ other dynamic background modeling algorithms such as median filtering, Kalman filtering, and codebook models. Those skilled in the art can choose the appropriate algorithm based on the complexity of the actual scene and available computational resources.
[0029] In this embodiment, the currently acquired target image and the reference background are differentially processed to obtain a difference image. Adaptive threshold segmentation is applied to the difference image to separate the foreground region from the background. The segmented binary image is analyzed using a connected component labeling algorithm to simultaneously identify one or more glass regions in a single frame. The connected component labeling algorithm can divide pixels with the same pixel value and adjacent positions in an image into the same region and assign different labels to each independent region, thereby achieving simultaneous identification and separation of multiple glass regions.
[0030] This step aims to extract the region containing the glass from the target image and achieve simultaneous recognition of multiple targets. Unlike traditional single-block detection modes, this step can simultaneously recognize multiple pieces of glass within the field of view of a single camera, making it a key step in achieving parallel processing of multiple pieces of glass. In other embodiments, foreground segmentation can also employ methods such as inter-frame difference or optical flow, and threshold segmentation can also use methods such as Otsu thresholding or fixed thresholding. Those skilled in the art can choose the appropriate method based on image characteristics and detection accuracy requirements.
[0031] In this embodiment, the identified glass regions are filtered based on geometric feature constraints. Specifically, the geometric features of each glass region, such as area, aspect ratio, and convex hull area ratio, are calculated. Valid glass regions are filtered according to preset geometric constraints, while noise regions and interfering objects that do not meet the constraints are excluded. For each valid glass region after filtering, a unique identifier composed of a timestamp and spatial location is assigned, and lifecycle management of the glass instance is established.
[0032] This step aims to eliminate false detection areas and establish an independent management mechanism for multiple glass panels. Geometric feature constraints effectively filter out false areas caused by interference factors such as conveyor belt edges and dust particles; unique identifiers enable independent tracking and management of each glass panel in subsequent processing. In other embodiments, geometric features may also include other shape descriptors such as roundness, rectangularity, and Euler number, and unique identifiers may use other encoding methods such as UUIDs and hash values. Those skilled in the art can choose according to the glass product specifications and system architecture.
[0033] In this embodiment, when the glass is large enough to span multiple camera fields of view, a spatiotemporal correlation matching algorithm is used to stitch together glass segments captured by adjacent cameras. Specifically, a prediction model is established based on the glass movement speed, and Kalman filtering is used to predict the location and time of the glass's appearance in the next camera, establishing a correlation between glass segments across cameras. The matching correctness of glass segments between adjacent cameras is verified by analyzing the continuity of edge gradient direction and intensity. SIFT (Scale-Invariant Feature Transform) feature points in overlapping areas are extracted, and the RANSAC (Random Sample Consensus) algorithm is used for geometric consistency verification to ensure stitching accuracy. Image registration and stitching are performed in real time during the glass movement to generate a complete glass image.
[0034] This step addresses the integrity issue in cross-camera inspection of large-size glass. Through spatiotemporal correlation matching and real-time stitching, glass fragments acquired by multiple cameras can be accurately merged into a complete glass image, ensuring the integrity and accuracy of defect detection. In other embodiments, feature matching can also employ other feature descriptors such as SURF and ORB, and motion prediction can use other prediction algorithms such as extended Kalman filtering and particle filtering. Those skilled in the art can choose according to the stitching accuracy requirements and computational efficiency.
[0035] Defect detection is performed in parallel across all valid glass regions. Specifically, firstly, a rapid preliminary inspection of the valid glass regions is conducted based on edge detection and morphological manipulation algorithms to obtain candidate defects. Then, the candidate defects are input into a Convolutional Neural Network (CNN) classifier for defect type classification, and precise pixel-level defect segmentation is performed through a semantic segmentation network. Finally, the glass region to which each defect belongs is determined based on the ray method and / or the number of turns algorithm. For defects located at the glass edge, their affiliation is determined through a buffer mechanism and distance transformation.
[0036] This step aims to achieve accurate defect detection and attribution determination. By combining traditional visual algorithms with deep learning algorithms, both detection speed and recognition accuracy are ensured. Spatial attribution algorithms can accurately determine the glass region to which each defect belongs in multi-glass scenarios, avoiding defect attribution confusion. In other embodiments, defect detection may employ only traditional visual algorithms or only deep learning algorithms, and defect attribution may use other methods such as nearest neighbor algorithms or region inclusion judgments. Those skilled in the art can choose according to the defect type and detection accuracy requirements.
[0037] The working process of this embodiment will be described below in conjunction with a specific application scenario. After the system starts, multiple line scan cameras begin to synchronously acquire images under the control of the trigger controller. When glass enters the detection area, each camera acquires glass images line by line and transmits them to the processing unit. The processing unit performs dynamic background modeling on the acquired target images, establishes and updates the reference background in real time through a Gaussian mixture model to adapt to environmental changes in the production site.
[0038] During image acquisition, the system performs differential processing on the target image based on a reference background. Through adaptive threshold segmentation and connected component labeling algorithms, it simultaneously identifies all glass regions within the field of view of a single camera. When multiple pieces of glass exist on the production line, the system can identify and separate multiple independent glass regions in the same frame image, breaking through the limitations of traditional single-piece detection modes. Subsequently, the system can filter out effective glass regions based on geometric feature constraints, eliminate noise and interference regions, and assign a unique identifier composed of a combination of timestamp and spatial location to each effective glass region, establishing lifecycle management for glass instances.
[0039] When the glass is large and needs to span multiple camera fields of view, the system can use Kalman filtering to predict the location and time of the glass in the next camera, establish the correlation between glass segments between adjacent cameras through edge continuity verification and SIFT feature matching, and perform image registration and stitching in real time during the glass movement to generate a complete glass image.
[0040] Furthermore, while identifying and stitching glass regions, the system allocates an independent processing thread for each valid glass region to perform defect detection in parallel. Each processing thread first performs a rapid preliminary inspection through edge detection and morphological operations to obtain candidate defects; then, the candidate defects are input into a convolutional neural network classifier and a semantic segmentation network for accurate identification; finally, the glass region to which each defect belongs is determined by the ray-mapping method and the number of turns algorithm, and for edge defects, their affiliation is determined by a buffer mechanism and distance transformation.
[0041] The system employs a sliding window mechanism to process images acquired line by line, starting to process the acquired portion even before the glass image is fully acquired, and updating the glass region boundaries and geometric features in real time. A state machine tracks the processing progress of each glass piece, ensuring that output is triggered only after all detection calculations are completed. The system uses a double-buffering mechanism for batch synchronous output, preparing the next batch of data while processing the current batch, ensuring high throughput and output integrity.
[0042] The beneficial effects of the technical solution of this invention are as follows: By dynamically establishing a reference background and performing differential processing on the target image, multiple effective glass areas can be identified simultaneously within the field of view of a single camera, thereby breaking through the limitations of the traditional single-piece detection mode and realizing the simultaneous identification and processing of multiple pieces of glass on the production line; on this basis, a unique identifier is assigned to each effective glass area, enabling the system to effectively distinguish and independently manage multiple pieces of glass, avoiding confusion and misjudgment in multi-target scenes; at the same time, when the glass spans multiple camera fields of view, the glass segments of adjacent cameras are spliced and merged based on spatiotemporal correlation matching to ensure the integrity identification of large-size glass; furthermore, defect detection is performed in parallel on each effective glass area and the glass area to which each defect belongs is accurately determined, realizing the parallel processing of multiple pieces of glass and the accurate attribution of defects, greatly improving detection efficiency and equipment utilization, and meeting the high-efficiency, large-volume production needs of modern glass production lines.
[0043] In a preferred embodiment, see [reference] Figure 2 Dynamically create a reference background for the target image, including: Step S21: Perform background modeling on the target image to obtain an initial reference background; Step S22: Update the reference background in real time according to the changes in the target image; Step S23: Adaptively adjust the reference background according to environmental changes.
[0044] Specifically, at the initial stage of system startup, several frames of images of the conveyor belt without glass are acquired as training samples to model the background of the target image and establish an initial reference background. During the background modeling process, the grayscale value distribution of each pixel in the image is statistically analyzed to extract the feature information of the background region and form a stable background model. The initial reference background reflects the image features of the detection area in the glass-free state, including information such as the surface texture of the conveyor belt and the distribution of light source illumination.
[0045] During system operation, the reference background is updated in real time based on changes in the target image. When a new image frame is acquired, the system compares the current frame with the reference background to determine whether each pixel belongs to the background region. For pixels determined to be background, their grayscale values are included in the background model update calculation, allowing the reference background to gradually adjust along with the slow changes in the image. The update process uses a weighted average method, which preserves the stability of historical background information while absorbing new background changes.
[0046] Simultaneously, the reference background is adaptively adjusted according to environmental changes. Environmental factors in the production site may change, such as fluctuations in light intensity over time, changes in the conveyor belt surface due to wear, and stains appearing in the detection area. The system monitors the changing trends of background model parameters and adaptively adjusts the background update rate and threshold. When a significant change in the environment is detected, the background update rate is accelerated to quickly adapt to the new environment; when the environment remains stable, the background update rate is reduced to maintain the stability of the background model.
[0047] Through the dynamic background modeling method described above, the system can establish a stable and reliable reference background and update and adaptively adjust it in real time according to changes in the target image and environment, providing accurate background reference for subsequent foreground segmentation and glass area recognition, and effectively coping with the complex and ever-changing environmental conditions in the production site.
[0048] In a preferred embodiment, see [reference] Figure 3 Background modeling is performed on the target image to obtain an initial reference background, including: Step S211: Establish a real-time reference background using a Gaussian mixture model; Step S222: Update the parameters of the Gaussian mixture model in real time.
[0049] Specifically, a Gaussian Mixture Model (GMM) is used to establish a real-time reference background. The GMM models the grayscale distribution of each pixel in an image as a weighted sum of K Gaussian distributions, each described by three parameters: mean, variance, and weight. In the inspection scenario of a glass production line, the value of K is typically set to 3 to 5 to accommodate the multimodal characteristics of the background. For example, the conveyor belt surface may have alternating light and dark textures, and light source illumination may produce a gradual brightness distribution; these complex background features can be effectively modeled by superimposing multiple Gaussian distributions.
[0050] When the system starts, it initializes K Gaussian distribution parameters for each pixel. As image frames are continuously acquired, the system determines the degree of matching between the actual grayscale value of each pixel and each Gaussian distribution. If the difference between the current pixel value and the mean of a certain Gaussian distribution is within a preset threshold range, it is considered a successful match, and that Gaussian distribution is considered part of the background. The Gaussian distributions are sorted according to their weights, and those with larger weights and smaller variances are selected as the background model to generate the real-time reference background.
[0051] During system operation, the parameters of the Gaussian mixture model are updated in real time. For successfully matched Gaussian distributions, their mean and variance are updated using a recursive formula, and their weights are increased. For unmatched Gaussian distributions, their mean and variance remain unchanged, but their weights are decreased. When the weight of a Gaussian distribution falls below a preset threshold, it is replaced with a new Gaussian distribution constructed with the current pixel value as the mean, a larger initial variance, and a smaller initial weight. Through this parameter update mechanism, the Gaussian mixture model can continuously learn the changing characteristics of the background and adapt to environmental factors such as gradual changes in illumination and changes in conveyor belt status.
[0052] During the update process, the learning rate is a key parameter controlling the model's adaptation speed. A larger learning rate allows the model to adapt quickly to background changes, but reduces its sensitivity to fleeting foreground objects; a smaller learning rate results in higher model stability, but a slower response to environmental changes. In this embodiment, the learning rate is dynamically adjusted based on the actual operating status of the production line. A larger learning rate is used during the system startup phase to quickly establish the background model, while a smaller learning rate is used during the stable operation phase to maintain model stability.
[0053] By employing a Gaussian mixture model for background modeling and updating model parameters in real time, the system can accurately distinguish between background and foreground glass areas, effectively addressing background changes caused by factors such as fluctuations in lighting and conveyor belt wear in the production environment, and providing a reliable background reference for subsequent multi-glass area identification.
[0054] In a preferred embodiment, see [reference] Figure 4 Differential processing of the target image based on a reference background, identifying at least one valid glass region within the field of view of a single camera includes: Step S31: Perform difference processing on the target image and the reference background to obtain a difference image; Step S32: Apply adaptive threshold segmentation to the difference image; Step S33: Identify one or more valid glass regions simultaneously in a single frame image using a connected component labeling algorithm.
[0055] Specifically, a pixel-by-pixel difference operation is performed between the currently acquired target image and the reference background generated by the Gaussian mixture model to obtain a difference image. In the difference image, the pixel values of the background region are close to zero, while the foreground region where the glass is located exhibits a significant grayscale difference.
[0056] Adaptive thresholding segmentation is applied to the difference image to separate the foreground region from the background. Adaptive thresholding segmentation dynamically calculates the segmentation threshold based on the gray-level distribution of local image regions. Compared with fixed thresholding methods, it can better adapt to the brightness differences in different regions of the image and obtain accurate binarized segmentation results.
[0057] The segmented binary image is analyzed using a connected component labeling algorithm. This algorithm scans the binary image, dividing adjacent pixels with the same pixel value into connected regions and assigning a different label to each independent connected region. When multiple pieces of glass exist simultaneously on the production line, each piece of glass appears as an independent connected region in the binary image. The connected component labeling algorithm can simultaneously identify one or more glass regions in a single frame, achieving simultaneous separation and identification of multiple targets. Through the aforementioned differential processing, adaptive threshold segmentation, and connected component labeling process, the system can simultaneously identify all glass regions within the field of view of a single camera, laying the foundation for subsequent geometric feature selection and unique identifier assignment.
[0058] In a preferred embodiment, see [reference] Figure 5 Filter valid glass areas and assign a unique identifier to each valid glass area, including: Step S34: Calculate the area, aspect ratio, and convex hull area ratio of each effective glass region; Step S35: Select valid glass areas based on preset geometric constraints.
[0059] Specifically, for each glass region identified by the connected component labeling algorithm, its geometric feature parameters are calculated. The area feature is obtained by counting the number of pixels contained in each connected region, reflecting the overall size of the glass region; the aspect ratio is obtained by calculating the aspect ratio of the minimum bounding rectangle of the region, reflecting the shape feature of the glass region; the convex hull area ratio is obtained by calculating the ratio of the actual area of the region to the area of its convex hull, reflecting the regularity of the glass region boundary.
[0060] Each glass region is filtered based on preset geometric constraints. Area constraints are used to exclude excessively small noise regions and excessively large abnormal regions. For example, a lower area limit is set to filter out small connected regions caused by dust particles, and an upper area limit is set to exclude abnormal regions formed by multiple pieces of glass sticking together. Aspect ratio constraints are used to exclude regions whose shapes obviously do not conform to the characteristics of glass, such as regions that are too long or too flat. Convex hull area ratio constraints are used to exclude regions with severely irregular boundaries. The convex hull area ratio of normal glass regions is usually close to 1, while the convex hull area ratio of regions with broken boundaries or large defects is significantly lower.
[0061] Regions that satisfy all geometric constraints are considered valid glass regions, while regions that do not satisfy any constraint are excluded. Through geometric feature constraint screening, the system can effectively filter out false regions caused by interference factors such as conveyor belt edge reflections, dust particles, and water stains, ensuring the accuracy of subsequent defect detection.
[0062] In a preferred embodiment, see [reference] Figure 5 The process of filtering valid glass areas and assigning a unique identifier to each valid glass area also includes: Step S36: Assign a unique identifier combining a timestamp and spatial location to each identified valid glass region.
[0063] Specifically, for each valid glass region selected through geometric feature constraints, the system assigns it a unique identifier composed of a timestamp, spatial location, and sequence number. The timestamp records the moment the glass region is first identified, accurate to the millisecond level, ensuring that glass entering the detection area at different times has different timestamps. The spatial location records the glass region's position information in the image coordinate system, including the horizontal and vertical coordinates of the region's centroid, used to distinguish multiple pieces of glass appearing at different spatial locations at the same time. The sequence number provides an additional distinguishing marker for glass regions appearing at similar spatial locations at the same time, ensuring the uniqueness of the identifier. An example of the unique identifier format is "20250121_143052_X1280_Y960_001", where "20250121_143052" is the timestamp, "X1280_Y960" is the spatial location, and "001" is the sequence number.
[0064] Based on unique identifier allocation, the system establishes a lifecycle management mechanism for glass instances. Each valid glass area is considered an independent glass instance, and its lifecycle includes four stages: creation, tracking, updating, and destruction. In the creation stage, when a new valid glass area is identified, the system assigns it a unique identifier and initializes instance information, including area boundaries, geometric features, and the camera to which it belongs. In the tracking stage, as the glass moves on the conveyor belt, the system continuously tracks the glass instance based on motion prediction and feature matching, maintaining its unique identifier. In the updating stage, the system updates the area boundaries and geometric features of the glass instance in real time based on newly acquired image data and records detected defect information. In the destruction stage, when the glass completely leaves the detection area and all detection calculations are completed, the system outputs the complete detection results for the glass instance and releases the system resources it occupied.
[0065] The lifecycle management mechanism also includes anomaly handling functionality. When a glass instance experiences brief occlusion or image acquisition anomalies during tracking, the system maintains the instance's tracking status based on the motion prediction model and continues updating after the anomaly is resolved. When a glass instance fails to match new image data for an extended period, the system marks it as lost and automatically destroys it after a preset timeout.
[0066] Through unique identifier allocation and lifecycle management, the system can effectively distinguish and manage multiple pieces of glass on the production line, ensuring that each piece of glass is accurately tracked from entering the inspection area to leaving the inspection area. This avoids confusion and misjudgment in multi-target scenarios and provides a reliable instance management foundation for subsequent cross-camera stitching and defect attribution determination.
[0067] In a preferred embodiment, see [reference] Figure 6 Based on spatiotemporal correlation matching, glass fragments from adjacent cameras are stitched together and include: Step S51: Use Kalman filtering to predict the location and time of the glass's appearance in the next camera. Step S52 verifies the matching correctness through continuity analysis of edge gradient direction and intensity.
[0068] Specifically, when the glass is large enough to span multiple camera fields of view, the system establishes a prediction model based on the glass's motion velocity and uses a Kalman filter to predict the glass's position and time of appearance in the next camera's field of view. The Kalman filter is a recursive optimal estimation algorithm that optimally estimates the target's state by fusing the system's motion model and actual observation data. In this embodiment, the system models the glass's motion as uniform linear motion, with the state vector including the glass's position and velocity along the conveyor belt. Based on the conveyor belt's speed and the distance between adjacent cameras, the Kalman filter predicts the position coordinates and time point of the glass's appearance in the next camera's field of view after it leaves the current camera's field of view.
[0069] Based on the prediction results of Kalman filtering, the system establishes a correlation between glass segments across cameras. When the next camera detects a new glass region within the prediction time window, and the deviation between the region's position and the predicted position is within an allowable range, the system establishes a preliminary correlation between that glass region and the glass instance in the previous camera. The deviation threshold between the predicted and actual positions is set according to the conveyor belt speed fluctuation range and the camera acquisition cycle to ensure accurate correlation establishment under normal production conditions.
[0070] After establishing an initial association, the system verifies the matching correctness through continuity analysis of edge gradient direction and intensity. It extracts the departure edge contour of the glass region from the previous camera and the arrival edge contour of the glass region from the next camera, calculating the gradient direction and intensity at each point on both edge contours. If the two edge contours belong to the same piece of glass, the gradient direction should remain consistent or exhibit a smooth transition at the stitching point, and the gradient intensity should also remain continuous or have only minor differences. The system calculates the consistency score of the edge gradient direction and the continuity score of the gradient intensity. When both scores exceed a preset threshold, the match is considered correct; otherwise, the system re-searches for possible matching candidates or identifies the current glass region as a new glass instance.
[0071] Through a dual mechanism of Kalman filter prediction and edge continuity verification, the system can accurately establish the correspondence between glass segments of adjacent cameras, providing a reliable matching basis for subsequent image stitching and effectively avoiding matching errors caused by conveyor belt speed fluctuations or multiple glass segments crossing the camera's field of view simultaneously.
[0072] In a preferred embodiment, see [reference] Figure 7 Based on spatiotemporal correlation matching, glass fragments from adjacent cameras are stitched together and include: Step S53: The scale-invariant feature transformation algorithm is used for feature matching, and the random sampling consistency algorithm is used to ensure the splicing accuracy. Step S54: Real-time segment splicing is performed during the glass movement process.
[0073] Specifically, after establishing preliminary associations through Kalman filtering prediction and edge continuity verification, the system employs the SIFT (Scale-Invariant Feature Transform) algorithm for feature matching to further ensure stitching accuracy. The SIFT algorithm extracts feature points within the overlapping region of glass images acquired by adjacent cameras. Each feature point contains information such as location, scale, orientation, and a 128-dimensional feature descriptor. SIFT features are scale-invariant and rotation-invariant, maintaining stability under different camera imaging conditions, making them suitable for feature matching across camera images. The system performs descriptor matching on the feature points extracted from the two images, using the nearest neighbor distance ratio method to filter matching point pairs, thus initially establishing the feature correspondence in the overlapping region.
[0074] Because mismatched point pairs may exist during feature matching, the system employs the RANSAC (Random Sample Consensus) algorithm for geometric consistency verification, eliminating mismatches and ensuring stitching accuracy. The RANSAC algorithm randomly selects a minimum subset from the initially matched feature point pairs, calculates the geometric transformation matrix, and then counts the number of interior points in all matching point pairs that conform to this transformation matrix. After multiple iterations, the transformation matrix with the highest number of interior points is selected as the optimal solution, and matching point pairs that do not conform to this transformation matrix are judged as mismatches and eliminated. Through the RANSAC algorithm, the system can robustly estimate the geometric transformation relationship between adjacent camera images even in the presence of noise and mismatches, ensuring the geometric accuracy of the stitching.
[0075] After completing feature matching and geometric transformation estimation, the system performs real-time segment stitching during glass movement. As the glass continues to move on the conveyor belt, the system registers and fuses glass segments captured by adjacent cameras based on the estimated geometric transformation matrix to generate a complete glass image. The stitching process uses a weighted fusion method to handle overlapping areas, avoiding obvious brightness jumps or seam marks at the stitching boundaries. Simultaneously, the system updates the geometric parameters and boundary information of the glass instances based on the stitching results, ensuring that subsequent defect detection is based on the complete glass image.
[0076] The real-time stitching process employs a pipelined architecture, executing feature extraction, feature matching, transform estimation, and image fusion in parallel. While feature extraction is being performed on a glass segment from the previous camera, the system simultaneously matches and stitches segments whose features have already been extracted, ensuring that the real-time performance of the stitching process meets the production line's cycle time requirements.
[0077] With the dual guarantee of SIFT feature matching and RANSAC geometric verification, the system can achieve high-precision cross-camera image stitching with geometric accuracy error controlled within 0.2mm. Through the real-time stitching processing mechanism, the system can complete segment merging during glass movement, ensuring the integrity identification and defect detection of large-size glass.
[0078] In a preferred embodiment, see [reference] Figure 8 Defect detection is performed in parallel on each valid glass area, and the glass area to which each defect belongs is determined, including: Step S61: Perform a rapid preliminary inspection of the effective glass area based on edge detection and morphological manipulation algorithms to obtain candidate defects; Step S62: Input the candidate defects into the convolutional neural network classifier to classify the defect types, and then use the semantic segmentation network for accurate identification. Step S63: Determine the glass region to which each defect belongs based on the ray method and / or the number of turns algorithm; Step S64: For defects located at the edge of the glass, their attribution is determined through a buffer mechanism and distance transformation.
[0079] Specifically, the system first performs a rapid preliminary inspection of the effective glass area based on edge detection and morphological operation algorithms to obtain candidate defects. Within each effective glass area, the Canny edge detection algorithm or the Sobel operator is used to extract edge information from the image, marking areas with significant gray-scale abrupt changes as potential defect locations. Subsequently, morphological operations are applied to process the edge detection results, including erosion to remove small noise points, dilation to connect broken defect edges, opening to smooth defect contours, and closing to fill internal holes in defects. After morphological operations, the extracted areas are initially screened based on features such as area and shape, filtering out noise areas with excessively small areas and areas whose shapes clearly do not conform to defect characteristics, resulting in a set of candidate defects.
[0080] The system inputs candidate defects into a Convolutional Neural Network (CNN) classifier for defect type classification and then performs accurate identification using a semantic segmentation network. For each candidate defect, the system extracts an image patch of its location, normalizes its size, and inputs it into a pre-trained CNN classifier. The CNN classifier extracts deep features of the defect through multiple layers of convolution, pooling, and fully connected operations and outputs classification results, categorizing defects into scratches, bubbles, stones, dirt, and chipped edges. For regions classified as defects, a semantic segmentation network is further applied for pixel-level defect segmentation, accurately delineating the defect's boundary contours and obtaining precise location, area, and shape information. The semantic segmentation network employs an encoder-decoder structure, enabling accurate pixel-level classification while maintaining spatial resolution.
[0081] After defect detection, the system determines the glass region to which each defect belongs based on the ray casting method and / or the number of turns algorithm. The ray casting method emits a ray from the defect point in any direction and counts the number of intersections between the ray and the glass region boundary. If the number of intersections is odd, the defect point is determined to be inside the glass region; if the number of intersections is even, the defect point is determined to be outside the glass region. The number of turns algorithm calculates the number of times the glass region boundary surrounds the defect point. If the number of turns is not zero, the defect point is determined to be inside the glass region. Both the ray casting method and the number of turns algorithm support the classification of complex-shaped glass, including concave polygons and glass regions with internal holes. Based on the classification results, the system establishes a mapping relationship between defects and unique glass identifiers, accurately associating each defect with its corresponding glass instance.
[0082] For defects located at the glass edge, the system determines their classification using a buffer mechanism and distance transformation. A buffer zone is set within a certain range inward from the glass region boundary. When a defect is within this buffer, a distance transformation is used to calculate the shortest distance from the defect point to the boundaries of adjacent glass regions, assigning the defect to the nearest glass region. For cases where boundary ambiguity makes classification difficult, the system applies morphological dilation to appropriately expand the glass region boundaries, ensuring that boundary defects are correctly classified. When a defect is located within the buffer zones of multiple glass regions simultaneously, the system determines its classification based on the proportion of overlap between the defect and each glass region, or associates the defect with multiple glass instances and marks it as a boundary defect.
[0083] In a preferred embodiment, see [reference] Figure 9 Defect detection methods based on multi-glass intelligent recognition and real-time processing also include: Step S70: Process the images acquired line by line based on the sliding window mechanism to update the glass region boundary and geometric features in real time; Step S71: Assign an independent processing thread to each valid glass area and perform defect detection in parallel; Step S72: Track the processing progress of each piece of glass and trigger the output after all detection calculations are completed; Step S73: Batch synchronous output is performed through a double buffering mechanism, preparing the next batch of data while processing the current batch.
[0084] Specifically, the system processes images acquired line by line using a sliding window mechanism, updating the glass region boundaries and geometric features in real time. Since the linear scan camera acquires images line by line, a complete image of the glass needs to be formed gradually as the conveyor belt moves. The system uses a fixed-size sliding window that covers several recently acquired lines of image data. Whenever a new line of image data is acquired, the sliding window moves forward, and the system performs incremental analysis on the image data within the window, updating the boundary positions and geometric features of each glass region in real time. When the front end of the glass enters the field of view, the system identifies and creates a new glass instance; as the glass moves within the field of view, the system continuously updates the region boundaries of that instance; when the rear end of the glass leaves the field of view, the system determines the final boundary of the instance and marks it as acquired. Through the sliding window mechanism, the system can begin processing the acquired portion even before the glass image is fully acquired, achieving real-time processing of a portion of the image.
[0085] The system allocates an independent processing thread to each valid glass area, performing defect detection in parallel. Once a new valid glass area is identified and assigned a unique identifier, the system allocates an independent processing thread from the thread pool to handle the defect detection task for that glass instance. Each processing thread operates independently, allowing for simultaneous defect detection on multiple glass pieces, achieving true parallel processing. The system supports GPU-accelerated multi-threaded parallel computing, scheduling computationally intensive tasks such as convolutional neural network classification and semantic segmentation to the GPU for execution, fully utilizing the GPU's parallel computing capabilities to improve processing efficiency. When multiple glass pieces exist on the production line simultaneously, the defect detection tasks for each glass instance are executed in parallel on different threads without interference, significantly reducing the overall detection time.
[0086] The system establishes a state machine to track the processing progress of each glass piece, triggering output upon completion of all detection calculations. The state machine maintains a set of state flags for each glass instance, including region identification status, defect detection status, geometric measurement status, and cross-camera stitching status. Each processing module updates its corresponding state flags after completing its task, and the state machine monitors the changes in these flags in real time. When all state flags for a glass instance change to the completed state, the state machine determines that all detection calculations for that instance are complete and triggers the result output process. Through this state machine management mechanism, the system ensures that the detection results for each glass piece are complete upon output, preventing situations where partial calculations are not completed before output.
[0087] The system employs a double-buffering mechanism for batch synchronous output, preparing the next batch of data while processing the current batch. This mechanism uses two output buffers: a front-end buffer and a back-end buffer. After the detection calculation for a glass instance is completed, the detection results are first written to the back-end buffer. When the back-end buffer accumulates a certain number of detection results or reaches a preset output cycle, the system swaps the front-end and back-end buffers. The output module then reads data from the front-end buffer for batch output, while new detection results continue to be written to the back-end buffer. This double-buffering mechanism decouples detection calculations from result output, preventing output operations from blocking the detection process and ensuring high system throughput and output integrity.
[0088] Thus, through the comprehensive application of sliding window mechanism, multi-threaded parallel computing, state machine management and double buffered output, the system achieves true real-time pipeline processing, maximizing detection efficiency while ensuring detection integrity and accuracy, and meeting the real-time detection needs of high-speed glass production lines.
[0089] The above description is only a part or preferred embodiment of the present invention. Neither the text nor the drawings should limit the scope of protection of the present invention. All equivalent structural transformations made using the content of the present invention specification and drawings under the overall concept of the present invention, or direct / indirect applications in other related technical fields, are included within the scope of protection of the present invention.
Claims
1. A defect detection method based on intelligent multi-glass identification and real-time processing, wherein the defect detection method based on intelligent multi-glass identification and real-time processing is applied to a glass production line, the glass production line comprising multiple cameras arranged along the glass movement direction, characterized in that, The defect detection method based on multi-glass intelligent identification and real-time processing includes: Acquire target images from multiple cameras; Dynamically create a reference background for the target image; Differential processing is performed on the target image based on the reference background to identify at least one effective glass region within the field of view of a single camera; Filter out valid glass areas and assign a unique identifier to each valid glass area; Based on spatiotemporal correlation matching, glass segments from adjacent cameras are stitched together and merged; Defect detection is performed in parallel on each of the effective glass regions, and the glass region to which each defect belongs is determined.
2. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 1, characterized in that, The step of dynamically establishing a reference background for the target image includes: The target image is modeled to obtain an initial reference background; The reference background is updated in real time according to the changes in the target image; The reference background is adaptively adjusted according to environmental changes.
3. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 2, characterized in that, The step of performing background modeling on the target image to obtain an initial reference background includes: A Gaussian mixture model is used to establish a real-time reference background; The parameters of the Gaussian mixture model are updated in real time.
4. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 1, characterized in that, The step of performing differential processing on the target image based on the reference background to identify at least one effective glass region within the field of view of a single camera includes: The target image and the reference background are subjected to difference processing to obtain a difference image; Adaptive threshold segmentation is applied to the difference image; One or more valid glass regions can be identified simultaneously in a single frame image using a connected component labeling algorithm.
5. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 1, characterized in that, The process of selecting valid glass areas and assigning a unique identifier to each valid glass area includes: Calculate the area, aspect ratio, and convex hull area ratio of each effective glass region; Valid glass areas are selected based on preset geometric constraints.
6. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 5, characterized in that, The process of screening valid glass areas and assigning a unique identifier to each valid glass area also includes: Each identified valid glass region is assigned a unique identifier combining a timestamp and its spatial location.
7. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 1, characterized in that, The spatiotemporal correlation matching method for stitching together glass segments from adjacent cameras includes: Kalman filtering is used to predict the location and time of the glass's appearance in the next camera. The matching correctness was verified by analyzing the continuity of edge gradient direction and intensity.
8. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 7, characterized in that, The spatiotemporal correlation matching method for stitching together glass segments from adjacent cameras includes: A scale-invariant feature transformation algorithm is used for feature matching, and a random sampling consistency algorithm is used to ensure splicing accuracy. Segment splicing is performed in real time during the glass movement.
9. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 1, characterized in that, The process of performing defect detection in parallel on each of the effective glass regions and determining the glass region to which each defect belongs includes: A rapid preliminary inspection of the effective glass area is performed based on edge detection and morphological manipulation algorithms to obtain candidate defects; Candidate defects are input into a convolutional neural network classifier for defect type classification, and then accurately identified through a semantic segmentation network. The glass region to which each defect belongs is determined based on the ray method and / or the number of turns algorithm; For defects located at the edge of the glass, their attribution is determined through a buffer mechanism and distance transformation.
10. The defect detection method based on multi-glass intelligent identification and real-time processing according to claim 1, characterized in that, The defect detection method based on multi-glass intelligent identification and real-time processing also includes: The image is processed line by line based on the sliding window mechanism, and the glass region boundary and geometric features are updated in real time. Each effective glass region is assigned an independent processing thread to perform defect detection in parallel. Track the processing progress of each piece of glass and trigger the output after all detection calculations are completed; Batch synchronous output is achieved through a double buffering mechanism, preparing the next batch of data while processing the current batch.