Glass surface defect detection method and system based on visual detection

Abstract

The application relates to the technical field of glass detection, in particular to a glass surface defect detection method and system based on visual detection, which comprises the following steps: comparing an initial gray image set with a preset static defect feature template library to detect whether the initial gray image set meets primary matching; if the initial gray image set does not meet the primary matching, extracting preliminary defect feature information of a glass surface according to the initial gray image set; and dynamically adjusting the focal length of a liquid lens and the exit angle of a polarized light source according to geometric shape parameters and initial position coordinates in the preliminary defect feature information to optimize the optical parameters of a glass surface defect area. The application dynamically adjusts the optical parameters by using the preliminary defect feature information, so that the best imaging effect can be obtained under different conditions, and the defects can be more clearly identified.

Description

Technical Field

[0001] This invention relates to the field of glass inspection technology, specifically to a method and system for detecting glass surface defects based on visual inspection. Background Technology

[0002] Currently, many traditional detection methods rely on a single viewpoint for image capture, resulting in a limited detection range for glass surface defects and the potential to miss some defects that are difficult to identify from a single viewpoint. Furthermore, traditional methods are unstable under different lighting conditions and cannot dynamically adjust optical parameters to adapt to environmental changes, thus affecting image quality and the accuracy of defect identification. In addition, traditional methods usually employ relatively simple feature extraction techniques, which cannot deeply analyze the complexity of defects and make it difficult to achieve efficient matching and classification.

[0003] Furthermore, due to the lack of systematic data analysis and utilization of historical images, traditional methods are prone to missed detections, which may lead to defective products entering the market. In addition, traditional detection systems often lack real-time feedback, making it difficult to adjust detection strategies in a timely manner according to new situations that arise during the production process. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: a glass surface defect detection method based on visual inspection, comprising:

[0005] A multi-view industrial camera array is used to acquire full-frame images of the glass substrate during transmission, obtaining an initial set of grayscale images of the glass surface. The multi-view industrial camera array includes a line scan camera and an area scan camera. The line scan camera is used to capture the edge contour information of the glass, and the area scan camera is used to capture the texture information of the glass body.

[0006] The initial set of grayscale images is compared with a preset static defect feature template library to detect whether the initial set of grayscale images meets the primary matching criteria. The primary matching criteria include edge contour matching, texture grayscale contrast, and specific shape geometry matching.

[0007] If the initial matching is not met, preliminary defect feature information of the glass surface is extracted based on the initial grayscale image set; wherein, the preliminary defect feature information includes the grayscale contrast, geometric shape parameters and initial position coordinates of the defect;

[0008] Based on the geometric parameters and initial position coordinates in the preliminary defect feature information, the focal length of the liquid lens and the emission angle of the polarization light source are dynamically adjusted to optimize the optical parameters of the defect area on the glass surface. Based on the multi-view industrial camera array with optimized optical parameters, high frame rate image tracking and acquisition are performed on the defect area to obtain fine texture and depth images of the defect area.

[0009] The refined texture image and depth image are input into the depth feature extraction model to extract the current depth feature vector. The current depth feature vector includes color space distribution, texture complexity, and local gradient direction. The model then checks whether the current depth feature vector matches the latest dynamic defect feature library in a high-level matching manner.

[0010] If the high-level matching is met, it is determined that there is a defect on the glass surface, and the current depth feature vector and the corresponding acquisition time parameters are added to the dynamic defect feature library.

[0011] Preferably, the method further includes:

[0012] Based on the refined texture image and depth image, and combined with the grayscale contrast in the preliminary defect feature information, a three-dimensional topological reconstruction model of glass surface defects is constructed.

[0013] The volume parameters and edge sharpness of the defect are calculated based on the three-dimensional topological reconstruction model;

[0014] The volume parameters, edge sharpness, and current depth feature vector are input into a pre-trained defect classification neural network, which outputs the probability distribution of defects.

[0015] Based on the probability distribution and the usage scenario parameters of the glass substrate, a preset level mapping table is consulted to determine the final defect level of the glass surface.

[0016] Preferably, a three-dimensional topological reconstruction model of the glass surface defect is constructed based on the refined texture image and depth image, as well as the grayscale contrast in the preliminary defect feature information, including:

[0017] The refined texture image is mapped onto the surface mesh of the 3D topology reconstruction model to obtain texture mapping data;

[0018] The depth image is converted into height field data of a 3D topological reconstruction model, and noise interference in the height field is corrected based on the gray-scale contrast in the preliminary defect feature information.

[0019] Based on the texture mapping data and the corrected height field data, the surface normal vector and curvature change of the defect region are calculated iteratively.

[0020] Based on the surface normal vector and curvature changes, the three-dimensional topological connectivity of the defect is determined, and a three-dimensional topological reconstruction model is generated.

[0021] Preferably, before acquiring detailed texture and depth images of the defect area by high-frame-rate image tracking and acquisition of the defect area using a multi-view industrial camera array optimized with optical parameters, the method further includes:

[0022] Based on the geometric shape parameters in the preliminary defect feature information, determine whether the defect is located in the blind zone of the overlapping field of view of adjacent cameras;

[0023] If so, the blind zone compensation angle is calculated based on the geometric parameters, and a specific camera in the multi-view industrial camera array is controlled to perform micro-displacement or rotation until the defect completely enters the field of view overlap area.

[0024] Based on the camera's pose after displacement or rotation, the image acquisition coordinate system is recalibrated, and a calibrated acquisition area mapping table is generated.

[0025] Among them, a multi-view industrial camera array optimized with optical parameters is used to perform high-frame-rate image tracking and acquisition of defect areas, including:

[0026] Based on the calibrated acquisition area mapping table, the multi-view industrial camera array with optimized optical parameters is used to perform high frame rate image tracking and acquisition of the defect area.

[0027] Preferably, the validity period of the latest dynamic defect feature library is within a preset production time sliding window corresponding to the acquisition time parameter;

[0028] Detect whether the current deep feature vector meets the advanced matching criteria with the latest dynamic defect feature library, including:

[0029] Identify whether the latest dynamic defect feature library contains neighboring feature vectors belonging to the same cluster as the current deep feature vector;

[0030] Compare whether the density values ​​of neighboring feature vectors appearing in the dynamic defect feature library exceed the corresponding density threshold;

[0031] When the density threshold is exceeded and the confidence level of the current depth feature vector is lower than the average confidence level in the database, the current depth feature vector is determined to meet the high-level matching criteria.

[0032] Preferably, the method further includes:

[0033] Re-extract the historical depth feature vectors corresponding to historical image frames that were not identified as defects within the production time sliding window;

[0034] Calculate the feature space distance between the historical deep feature vector and each feature vector in the latest dynamic defect feature library;

[0035] Historical image frames corresponding to historical depth feature vectors whose feature space distance is less than a preset distance threshold are identified as images with missed defects.

[0036] The dynamic defect feature library is updated based on the historical depth feature vectors of images identified as missed defects.

[0037] Preferably, the method further includes:

[0038] Based on the volume parameters and edge sharpness corresponding to the final defect level, the exposure time and gain parameters of the multi-view industrial camera array are adjusted in reverse.

[0039] If the final defect level is severe, the sorting mechanism of the glass transmission line will be controlled to accurately reject the defect based on the defect location coordinates in the three-dimensional topology reconstruction model.

[0040] If the final defect level is minor, the laser energy required for repair is calculated based on the defect's volume parameters, and the laser repair equipment is controlled to perform in-situ repair.

[0041] Preferably, if the final defect level is severe, the sorting mechanism of the glass transmission line is controlled to precisely reject defects based on the defect location coordinates in the three-dimensional topology reconstruction model, including:

[0042] Obtain real-time operating speed and acceleration information of the glass transmission line;

[0043] Based on the location coordinates of the defect and the real-time running speed of the line, predict the time delay for the defect to reach the sorting mechanism;

[0044] Calculate the lead offset based on the time delay and the action response time of the sorting mechanism;

[0045] The timing of the pneumatic valve opening of the sorting mechanism is controlled by the lead offset, and glass substrates with serious defects are rejected and sent to the waste area.

[0046] Preferably, the method further includes:

[0047] Obtain a sample set of glass surface images; the sample set of glass surface images includes real defect labels and defect-free labels;

[0048] Multi-scale defect feature extraction from image sample set;

[0049] A static defect feature template library is constructed based on the extracted multi-scale defect features;

[0050] The depth feature extraction model is fine-tuned online based on the image frames identified as defective, and the fine-tuned depth feature extraction model is used for feature extraction of the next current image frame.

[0051] A vision-based glass surface defect detection system, applicable to the aforementioned vision-based glass surface defect detection method, includes:

[0052] The image acquisition unit is used to acquire full-frame images of the glass substrate in transit through a multi-view industrial camera array, and obtain an initial grayscale image set of the glass surface; wherein, the multi-view industrial camera array includes a line scan camera and an area scan camera, the line scan camera is used to capture the glass edge contour information, and the area scan camera is used to capture the glass body texture information.

[0053] The image comparison unit is used to compare the initial set of grayscale images with a preset static defect feature template library to detect whether the initial set of grayscale images meets the primary matching criteria; wherein, the primary matching criteria include edge contour matching, texture grayscale contrast and specific shape geometric matching;

[0054] The defect extraction unit is used to extract preliminary defect feature information of the glass surface based on the initial grayscale image set if the initial matching condition is not met. The preliminary defect feature information includes the grayscale contrast, geometric shape parameters and initial position coordinates of the defect.

[0055] The parameter optimization unit is used to dynamically adjust the focal length of the liquid lens and the emission angle of the polarization light source based on the geometric shape parameters and initial position coordinates in the preliminary defect feature information to optimize the optical parameters of the defect area on the glass surface; and to perform high frame rate image tracking and acquisition of the defect area based on the multi-view industrial camera array after optical parameter optimization to obtain the fine texture image and depth image of the defect area.

[0056] The vector extraction unit is used to input the refined texture image and depth image into the depth feature extraction model to extract the current depth feature vector. The current depth feature vector includes color space distribution, texture complexity, and local gradient direction. It also detects whether the current depth feature vector meets the high-level matching requirement with the latest dynamic defect feature library.

[0057] The defect matching unit is used to determine that there is a defect on the glass surface if the high-level matching is met, and adds the current depth feature vector and the corresponding acquisition time parameters to the dynamic defect feature library.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] This invention utilizes a combination of multi-view industrial camera arrays to comprehensively capture the contour and texture information of glass surfaces, thereby improving the accuracy and coverage of defect detection. Furthermore, by dynamically adjusting optical parameters using preliminary defect feature information, optimal imaging results can be obtained under different conditions, which helps to identify defects more clearly.

[0060] When a serious defect is detected, the system can automatically control the sorting mechanism to reject the defect, thereby improving the automation level of the production line and reducing manual intervention. By constructing a three-dimensional topology reconstruction model and combining it with a deep feature extraction model, advanced matching and classification of defects can be achieved, improving the accuracy and reliability of defect identification. Furthermore, by re-extracting and analyzing historical image frames, the risk of missed detection can be effectively reduced, ensuring that defective products do not enter the market. Attached Figure Description

[0061] Figure 1 This is a schematic flowchart of the overall method in one embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the overall system architecture in one embodiment of the present invention.

[0063] In the diagram: 1. Image acquisition unit; 2. Image comparison unit; 3. Defect extraction unit; 4. Parameter optimization unit; 5. Vector extraction unit; 6. Defect matching unit. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Example 1, please refer to Figure 1 This invention provides a technical solution: a method for detecting glass surface defects based on visual inspection, comprising:

[0066] S1. Full-frame image acquisition of the glass substrate during transmission is performed using a multi-view industrial camera array to obtain an initial grayscale image set of the glass surface; wherein, the multi-view industrial camera array includes a line scan camera and an area scan camera, the line scan camera is used to capture the edge contour information of the glass, and the area scan camera is used to capture the texture information of the glass body.

[0067] S2. Compare the initial grayscale image set with the preset static defect feature template library to detect whether the initial grayscale image set meets the primary matching criteria; wherein, the primary matching criteria include edge contour matching, texture grayscale contrast and specific shape geometry matching;

[0068] S3. If the initial matching is not met, the preliminary defect feature information of the glass surface is extracted based on the initial grayscale image set; wherein, the preliminary defect feature information includes the grayscale contrast, geometric shape parameters and initial position coordinates of the defect;

[0069] S4. Based on the geometric shape parameters and initial position coordinates in the preliminary defect feature information, dynamically adjust the focal length of the liquid lens and the emission angle of the polarization light source to optimize the optical parameters of the defect area on the glass surface; based on the multi-view industrial camera array with optimized optical parameters, perform high frame rate image tracking and acquisition on the defect area to obtain a refined texture image and depth image of the defect area.

[0070] S5. Input the refined texture image and depth image into the depth feature extraction model to extract the current depth feature vector; wherein, the current depth feature vector includes color space distribution, texture complexity and local gradient direction; detect whether the current depth feature vector meets the high-level matching with the latest dynamic defect feature library;

[0071] S6. If the high-level matching is met, it is determined that there is a defect on the glass surface, and the current depth feature vector and the corresponding acquisition time parameters are added to the dynamic defect feature library.

[0072] It should be noted that, firstly, a multi-view industrial camera array, including line scan cameras and area scan cameras, is set up to comprehensively capture images of the glass surface; the line scan cameras focus on capturing the edge contour information of the glass, which is very important for identifying the boundaries and shapes of defects; the area scan cameras are responsible for capturing the texture information of the glass body, so as to better analyze the details and texture features of the surface.

[0073] The acquired initial grayscale images are compared with a preset static defect feature template library to confirm whether these images meet the primary matching criteria. The primary matching criteria mainly include three aspects: edge contour matching, texture grayscale comparison, and geometric matching of specific shapes. This step is to initially filter out images that do not meet the standards.

[0074] If some images fail to meet the initial matching criteria, the system will extract preliminary defect feature information from these images. This includes the grayscale contrast of the defect (i.e., the brightness difference between the defect and the normal area), geometric parameters (such as the size and shape of the defect), and initial position coordinates (the specific location of the defect on the glass surface).

[0075] Based on the extracted preliminary defect feature information, the system adjusts the focal length of the liquid lens and the emission angle of the polarized light source to optimize optical parameters so that the defect area can be observed more clearly; the purpose of this step is to improve the quality of subsequent imaging, thereby making it easier to obtain detailed information about the defect.

[0076] After optimizing the optical parameters, the system will again use a multi-view industrial camera array to perform high-frame-rate image tracking and acquisition of the defect area to obtain more refined texture and depth images; these images can provide more detailed information about the defect.

[0077] Subsequently, these refined texture and depth images are input into a depth feature extraction model to extract the current depth feature vector. This feature vector contains information such as color space distribution, texture complexity, and local gradient direction, which are important indicators for further evaluation of defects.

[0078] Finally, the system checks whether the current depth feature vector matches the latest dynamic defect feature library with advanced matching. If it does, the system will determine that there is a defect on the glass surface and add the feature vector and its corresponding acquisition time parameters to the dynamic defect feature library for future detection.

[0079] For example, suppose that during the production process, a worker discovers a noticeable scratch on the surface of a piece of glass. Through the process described above, the system first captures a full-frame image of the glass, the line scan camera identifies the edge of the scratch, and the area scan camera acquires the texture information around the scratch. After preliminary matching detection, the system may find that these features are similar to scratch features in the static defect template library. Therefore, the system extracts the scratch feature information and adjusts the optical parameters to analyze the nature of the scratch in more detail. Finally, the system successfully confirms the existence of the scratch and records the relevant information in the dynamic defect feature library to support subsequent quality control.

[0080] In an optional embodiment, the method further includes:

[0081] Based on the refined texture image and depth image, and combined with the grayscale contrast in the preliminary defect feature information, a three-dimensional topological reconstruction model of glass surface defects is constructed.

[0082] The volume parameters and edge sharpness of the defect are calculated based on the three-dimensional topological reconstruction model;

[0083] The volume parameters, edge sharpness, and current depth feature vector are input into a pre-trained defect classification neural network, which outputs the probability distribution of defects.

[0084] Based on the probability distribution and the usage scenario parameters of the glass substrate, a preset level mapping table is consulted to determine the final defect level of the glass surface.

[0085] It should be noted that by using refined texture images and depth images, combined with grayscale contrast from preliminary defect feature information, a three-dimensional topological reconstruction model of the glass surface can be generated. This model can accurately reflect the shape and location of defects on the glass surface. The three-dimensional reconstruction model can not only show the height variation of defects, but also provide information about the spatial distribution of defects. For example, if there is a small pit on a piece of glass, the three-dimensional model will show the depth of the pit, the surrounding edges, and its relationship with the normal surface.

[0086] Based on the three-dimensional topological reconstruction model, the volume parameters and edge sharpness of the defect are calculated. The volume parameters reflect the degree to which the defect occupies space, while the edge sharpness describes the clarity and sharpness of the defect edge. A sharp edge may mean that the defect is more obvious and may affect the overall quality of the glass. For example, a small crack with a very sharp edge may mean that it has a greater impact on the strength of the product, while a less obvious flaw may have a smaller impact.

[0087] Then, the calculated volume parameters, edge sharpness, and current depth feature vector are input into a pre-trained defect classification neural network. This neural network, by learning from a large amount of labeled defect data, can identify different types of defects and provide corresponding probability distributions. For example, the network might output a probability distribution indicating the likelihood that the defect is a scratch, dent, or other type of defect, with probabilities of 70%, 20%, and 10%, respectively.

[0088] Finally, based on the output probability distribution and the usage scenario parameters of the glass substrate (such as usage environment, load-bearing capacity, etc.), a preset level mapping table is consulted to determine the final defect level of the glass surface. The level mapping table combines the probability distribution with quality standards to help determine whether the defect is within an acceptable range. For example, if the detected defect probability distribution shows that it is a scratch, and according to the mapping table, the danger level of the scratch is "high", then in practical applications, this glass may not be suitable for load-bearing occasions.

[0089] For example, suppose a defect is found in a piece of transparent glass during the production process. Through the above process, the system first constructs a 3D model of the defect, finding that it is a small, sharp dent. After calculation, its volume is determined to be 0.05 cubic millimeters, with relatively sharp edges. After being input into a classification neural network, the system outputs that there is an 80% probability that the dent is caused by mechanical impact. Combining the usage scenario parameters, assuming that this glass is to be used for car windows, the final judgment is that the defect level is "serious," which may affect safety, so it is recommended to replace the glass. In this way, through this series of analyses and judgments, the quality and safety of the product are ensured.

[0090] In an optional embodiment, a three-dimensional topological reconstruction model of the glass surface defect is constructed based on the refined texture image and depth image, and by combining the grayscale contrast in the preliminary defect feature information, including:

[0091] The refined texture image is mapped onto the surface mesh of the 3D topology reconstruction model to obtain texture mapping data;

[0092] The depth image is converted into height field data of a 3D topological reconstruction model, and noise interference in the height field is corrected based on the gray-scale contrast in the preliminary defect feature information.

[0093] Based on the texture mapping data and the corrected height field data, the surface normal vector and curvature change of the defect region are calculated iteratively.

[0094] Based on the surface normal vector and curvature changes, the three-dimensional topological connectivity of the defect is determined, and a three-dimensional topological reconstruction model is generated.

[0095] It should be noted that the first step is to match the acquired refined texture image with the surface mesh of the 3D topology reconstruction model. This means mapping 2D image data (such as color and texture information) onto the surface of the 3D model to form a complete texture map. This map allows for a better visual understanding of the model's appearance. For example, if a glass surface has tiny spots or scratches, these features will be clearly displayed on the 3D model through the texture map, making subsequent analysis more intuitive.

[0096] Depth images are converted into height field data, which reflects the height variation of the surface in space. By using grayscale contrast information, noise in the height field can be corrected. For example, if there are some errors or noise in the depth image (such as misjudgment caused by uneven lighting), the influence of these noises can be identified and reduced by comparing grayscale contrast, so that the height field can more accurately reflect the true defect morphology.

[0097] Once the correct height field data and texture maps are available, the surface normals, i.e., the vertical direction at each point, can be calculated. This is crucial for analyzing the geometry of defects, as the normals provide information about the surface tilt and orientation. Furthermore, through iterative calculations, curvature variations can be obtained, indicating the degree of curvature of the surface in the defect area. For example, the curvature may be higher in concave areas and lower in flat areas. Such calculations help in further analyzing the nature and severity of the defects.

[0098] Finally, based on the calculated normal vector and curvature change, the three-dimensional topological connectivity of the defect can be determined; this means that it is possible to determine whether the defect is isolated or connected to the surrounding surface; this information is very important for assessing the overall impact of the defect; by comprehensively considering these factors, the final generated three-dimensional topological reconstruction model can accurately reflect the defect characteristics of the glass surface.

[0099] For example, suppose that during the production process, multiple small pits are found on the surface of a piece of glass. Through the above process, firstly, a refined texture image is mapped to clearly show the pits and the texture around them. Next, a height field is generated using a depth image, and the noise caused by illumination is corrected, thus obtaining more accurate height data. Then, the calculated normal vectors and curvature changes show that the edges of these pits are relatively sharp, and there is a certain degree of connectivity between them. Finally, the generated three-dimensional topological reconstruction model clearly shows the specific morphology of these defects, providing an important basis for subsequent quality assessment.

[0100] In an optional embodiment, before acquiring a refined texture image and depth image of the defect region by high-frame-rate image tracking and acquisition of the defect region using a multi-view industrial camera array with optimized optical parameters, the method further includes:

[0101] Based on the geometric shape parameters in the preliminary defect feature information, determine whether the defect is located in the blind zone of the overlapping field of view of adjacent cameras;

[0102] If so, the blind zone compensation angle is calculated based on the geometric parameters, and a specific camera in the multi-view industrial camera array is controlled to perform micro-displacement or rotation until the defect completely enters the field of view overlap area.

[0103] Based on the camera's pose after displacement or rotation, the image acquisition coordinate system is recalibrated, and a calibrated acquisition area mapping table is generated.

[0104] Among them, a multi-view industrial camera array optimized with optical parameters is used to perform high-frame-rate image tracking and acquisition of defect areas, including:

[0105] Based on the calibrated acquisition area mapping table, the multi-view industrial camera array with optimized optical parameters is used to perform high frame rate image tracking and acquisition of the defect area.

[0106] It should be noted that in this step, the geometry of the identified defect needs to be evaluated, such as its location, size, and shape. Then, based on these parameters, it is determined whether the defect is located outside the overlapping area of ​​the fields of view of the adjacent cameras, which is the so-called "blind zone". For example, if the overlapping area of ​​the fields of view of the two cameras is not enough to cover a large defect, then measures need to be taken to ensure that the defect can be effectively captured.

[0107] If the defect is indeed found to be within the blind zone, a compensation angle needs to be calculated to adjust the camera position. This process involves geometric calculations, which determine the specific angle of micro-displacement or rotation required by analyzing the relationship between the current camera's viewpoint and the defect's position. For example, if the defect is in the blind zone of the left camera while the right camera can capture part of the image, it may be necessary to move the left camera slightly to the left or upward by a certain angle so that the defect is fully within the overlapping field of view of the two cameras.

[0108] Once the compensation angle is calculated, the system will automatically control the camera to make fine adjustments. This may involve precise mechanical adjustments, such as using servo motors to move or rotate the camera so that the defect is fully within the overlapping field of view. In this process, precise camera control is crucial to ensure that the adjusted viewing angle can minimize errors.

[0109] After adjusting the camera position, the image acquisition coordinate system needs to be recalibrated. This is because the camera's field of view and geometric relationship have changed. By recalibrating, a new acquisition area mapping table can be generated, which records the new field of view information as well as the relative position and attitude of each camera. This ensures that subsequent image processing can accurately reflect the actual scene captured by each camera.

[0110] Finally, based on the newly generated mapping table, the system will control a multi-view camera array with optimized optical parameters to acquire high-frame-rate images of the defect area; high-frame-rate acquisition can capture dynamic or subtle changes in the defect area, thereby providing more detailed and comprehensive data for subsequent analysis.

[0111] For example, suppose a small scratch appears on the surface of a piece of glass in a production line. After initial inspection, it is found that the scratch is located in the blind zone of the overlapping fields of view of two adjacent cameras. The system first analyzes the geometry of the scratch to confirm that it is indeed not within the overlapping field of view. Next, the system calculates the required compensation angle, such as rotating the left camera 5 degrees to the right and moving it slightly by 1 centimeter. After performing these operations, the viewing angle of the left camera is adjusted so that the scratch is completely inside the overlapping area of ​​the two cameras.

[0112] At this point, the system recalibrates the camera's image acquisition coordinate system, generates a new region mapping table, and records the new position of each camera. Finally, using this new mapping table, the system begins to acquire images of the scratched area at a high frame rate, thus obtaining clearer and more detailed data for subsequent quality analysis and detection.

[0113] In an optional embodiment, the validity period of the latest dynamic defect feature library is within a preset production time sliding window corresponding to the acquisition time parameter;

[0114] Detect whether the current deep feature vector meets the advanced matching criteria with the latest dynamic defect feature library, including:

[0115] Identify whether the latest dynamic defect feature library contains neighboring feature vectors belonging to the same cluster as the current deep feature vector;

[0116] Compare whether the density values ​​of neighboring feature vectors appearing in the dynamic defect feature library exceed the corresponding density threshold;

[0117] When the density threshold is exceeded and the confidence level of the current depth feature vector is lower than the average confidence level in the database, the current depth feature vector is determined to meet the high-level matching criteria.

[0118] It should be noted that during defect detection, the first step is to convert the detected defect into a deep feature vector. These feature vectors are obtained by analyzing information such as the defect's geometry, texture, and color. Then, the system groups these feature vectors according to an algorithm (such as K-means clustering or other clustering methods). Next, the system checks whether the current deep feature vector is in the same cluster as some feature vectors in the feature library. For example, if the detected scratch is a new type, the system will search the feature library for similar scratch feature vectors.

[0119] If a neighboring feature vector belonging to the same cluster as the current feature vector is found, the system will calculate the density value of these neighboring feature vectors in the feature library. The density value can be understood as the degree of concentration of the feature vector distribution in the feature space, which is usually measured by the number of neighboring points. If the density value of these neighboring feature vectors exceeds the preset density threshold, it means that this type of defect is relatively common in historical data.

[0120] Finally, the system will also evaluate the confidence level of the current feature vector; confidence level refers to the degree of confidence the model has in the accuracy of the current detection result, which is usually derived from the output of the machine learning model; if the confidence level of the current deep feature vector is low, and the density value of the neighboring feature vectors has exceeded the specified threshold, then it can be determined that the current feature vector has high-level matching; this means that although the current detection result is not very certain, the existence of a large number of similar features in the feature library indicates that this type of defect is reasonable and can be further confirmed.

[0121] For example: Suppose that during a production process, the inspection system discovers a new defect on a piece of glass. After processing, the defect is converted into a depth feature vector, and the system begins to analyze this vector. The system searches a dynamic defect feature library and finds that defects similar to the current feature vector (e.g., minor scratches) already exist, and they are clustered into a cluster. The system calculates the density of this cluster and finds that there are many similar feature vectors in the cluster, and the density values ​​of these vectors significantly exceed a set threshold (e.g., at least 30 similar feature vectors). Then, the system measures the confidence of the current depth feature vector and finds that its confidence is lower than the average confidence of similar feature vectors in the feature library (e.g., the current feature vector has a confidence of 0.6, while the average confidence of the corresponding feature in the library is 0.8).

[0122] In this situation, since the current deep feature vector meets advanced matching criteria, the system may suggest further manual inspection or take measures to confirm the nature of the defect. In this way, by combining information from the dynamic feature library, the detection system can more effectively identify and respond to various defects, ensuring production quality and efficiency.

[0123] In an optional embodiment, the method further includes:

[0124] Re-extract the historical depth feature vectors corresponding to historical image frames that were not identified as defects within the production time sliding window;

[0125] Calculate the feature space distance between the historical deep feature vector and each feature vector in the latest dynamic defect feature library;

[0126] Historical image frames corresponding to historical depth feature vectors whose feature space distance is less than a preset distance threshold are identified as images with missed defects.

[0127] The dynamic defect feature library is updated based on the historical depth feature vectors of images identified as missed defects.

[0128] It should be noted that during the production process, the system continuously captures images and extracts the corresponding depth feature vectors. Within a certain time range, i.e., the "time sliding window", the system will retain all images and their feature vectors within this time range. If some images are not judged as defects during the initial inspection, the system will still retain them. These unjudged image frames and their corresponding depth feature vectors will be extracted for subsequent analysis.

[0129] Once these historical deep feature vectors are acquired, the system will begin to calculate their distances to all known defect feature vectors in the current dynamic defect feature library. Feature space distance is a measure that reflects the similarity between a feature vector and other feature vectors. It is conceivable that if two feature vectors are very close in the feature space, they may represent similar types of defects.

[0130] The system sets a distance threshold. Any historical depth feature vector whose distance to any feature vector in the dynamic defect feature library is less than this threshold will be considered as potentially missing. This means that there may be defects in these images that have been missed, even though they were not identified in the initial inspection.

[0131] Once a missed defect image is identified, the system uses these historical deep feature vectors to update the dynamic defect feature library. This can improve the overall performance of the detection system by adding these newly discovered defect features to the feature library, making future detections more accurate.

[0132] For example: Suppose that during a certain production cycle, the inspection system records multiple image frames, some of which are not marked as defects during the initial inspection; however, in the subsequent analysis, the system re-extracts the feature vectors of these historical image frames; the system identifies these historical image frames and extracts their depth feature vectors, for example, a piece of glass that appears to have a smooth surface, but after further analysis, its feature vectors show potential anomalies.

[0133] The system then compares these historical deep feature vectors with feature vectors in the dynamic defect feature library and finds that some of these feature vectors are very close to a known defect feature vector in the library (e.g., a distance of 0.1, which is below the set threshold of 0.2). Therefore, these historical image frames (e.g., some seemingly normal images) are identified as missed defect images.

[0134] Finally, the system adds these new deep feature vectors to the dynamic defect feature library to better identify similar defects in future detections; for example, this may lead the system to include previously undetected scratches or bubbles in new defect categories, thereby improving the accuracy and reliability of the entire monitoring system.

[0135] In an optional embodiment, the method further includes:

[0136] Based on the volume parameters and edge sharpness corresponding to the final defect level, the exposure time and gain parameters of the multi-view industrial camera array are adjusted in reverse.

[0137] If the final defect level is severe, the sorting mechanism of the glass transmission line will be controlled to accurately reject the defect based on the defect location coordinates in the three-dimensional topology reconstruction model.

[0138] If the final defect level is minor, the laser energy required for repair is calculated based on the defect's volume parameters, and the laser repair equipment is controlled to perform in-situ repair.

[0139] It should be noted that during the inspection process, the final defect level (e.g., minor, severe, etc.) is determined by analyzing the defect's volumetric parameters (size, volume, etc.) and edge sharpness (clarity of the defect edge); once the defect level is determined, the system will adjust the exposure time and gain of the multi-view industrial camera in reverse according to these parameters.

[0140] If the detected defect is significant, the exposure time may need to be shortened to avoid overexposure and loss of detail; conversely, if the defect is minor, the exposure time may need to be increased to capture more detail. Gain determines the degree to which the camera amplifies the light signal; in low light conditions, appropriately increasing the gain can help capture a clearer image, but excessive gain can introduce noise; therefore, adjusting the gain according to the nature of the defect is also necessary.

[0141] For example, suppose a small scratch (minor defect) is detected on a piece of glass with low edge sharpness; the system will increase the exposure time and moderately increase the gain to ensure that the details of the scratch can be captured more clearly in subsequent shots, facilitating further analysis;

[0142] When the system identifies a defect as "severe," it means that the defect may seriously affect the quality or safety of the product. At this time, the system will use the defect location coordinates determined in the three-dimensional topology reconstruction model to instruct the sorting mechanism on the glass conveyor line to accurately reject the glass. The sorting mechanism may use a robotic arm or other automated equipment that can quickly and accurately identify and reject glass with severe defects.

[0143] For example, if a large bubble (serious defect) is detected on a piece of glass, the system will immediately calculate the exact location of the bubble on the glass and remove the glass from the production line through a sorting mechanism to prevent it from flowing into the downstream processing or sales process.

[0144] For minor defects, such as small scratches or tiny bubbles, the system assesses their volume parameters and calculates the required laser energy for repair; the laser repair device can apply specific energy to the defective area to eliminate the defect without affecting the integrity of the surrounding area.

[0145] For example, if a minor scratch (minor defect) is detected on a piece of glass, the system will measure the length and width of the scratch and then calculate the laser energy required for repair. For example, the system may determine that 150 millijoules of laser energy is needed to repair the scratch and control the laser repair device to process the scratch at the location to achieve the repair effect.

[0146] In an optional embodiment, if the final defect level is severe, the sorting mechanism of the glass transmission line is controlled to precisely reject defects based on the defect location coordinates in the three-dimensional topology reconstruction model, including:

[0147] Obtain real-time operating speed and acceleration information of the glass transmission line;

[0148] Based on the location coordinates of the defect and the real-time running speed of the line, predict the time delay for the defect to reach the sorting mechanism;

[0149] Calculate the lead offset based on the time delay and the action response time of the sorting mechanism;

[0150] The timing of the pneumatic valve opening of the sorting mechanism is controlled by the lead offset, and glass substrates with serious defects are rejected and sent to the waste area.

[0151] It should be noted that the operating speed and acceleration of the glass transport line are dynamically changing during the production process. By installing sensors (such as photoelectric sensors or encoders), the system can acquire the speed (e.g., in meters per second) and acceleration (rate of change of speed) of the transport line in real time. This information is crucial for subsequent defect prediction and sorting control. For example, assuming the real-time operating speed of the transport line is 2 meters per second and the acceleration is 0.5 meters per second squared, this means that if the current speed remains constant, the glass substrate will continue to move at this speed for a period of time.

[0152] Once a defect is detected, the system records the specific location coordinates of the defect (e.g., a specific point on the conveyor line). Combined with the real-time operating speed, the system can calculate the distance of the defect from the sorting mechanism and predict the time required for the defect to reach the sorting mechanism. For example, if a defect is located 10 meters away from the sorting mechanism and the current speed of the conveyor line is 2 meters per second, then the time for the defect to reach the sorting mechanism is 10 meters divided by 2 meters per second, which equals 5 seconds.

[0153] The response time of a sorting mechanism refers to the time required for it to receive a signal and perform a rejection action. To ensure that the sorting mechanism can react promptly when a defect arrives, this response time needs to be taken into account, and a lead time offset needs to be calculated. For example, suppose the response time of the sorting mechanism is 1 second; therefore, the total time delay is 5 seconds, and the sorting mechanism needs to be ready in 4 seconds (i.e., 1 second in advance). This means that a lead time offset needs to be calculated to ensure that the sorting mechanism is ready before the defect arrives.

[0154] Based on the aforementioned lead time offset, the system will determine the opening time of the pneumatic valve of the sorting mechanism so that the defect can be promptly removed to the waste area when it arrives. This control is usually performed by an automated system to ensure the accuracy and timeliness of the response. For example, if the calculated lead time offset is 4 seconds, the system will send a signal 4 seconds before the defect arrives at the sorting mechanism to open the pneumatic valve and effectively push the glass substrate with the serious defect to the waste area.

[0155] In an optional embodiment, the method further includes:

[0156] Obtain a sample set of glass surface images; the sample set of glass surface images includes real defect labels and defect-free labels;

[0157] Multi-scale defect feature extraction from image sample set;

[0158] A static defect feature template library is constructed based on the extracted multi-scale defect features;

[0159] The depth feature extraction model is fine-tuned online based on the image frames identified as defective, and the fine-tuned depth feature extraction model is used for feature extraction of the next current image frame.

[0160] It should be noted that, firstly, a sample set of glass surface images containing different types of defects needs to be collected. This sample set should include images labeled with real defects (such as scratches, bubbles, impurities, etc.) and images without defects. These samples can provide the necessary data foundation for subsequent training of machine learning models. For example, suppose the research team collects 1000 images by taking pictures of glass from different production batches, of which 600 images show defects (such as small scratches) and 400 images are intact; these images will be used to train and validate the defect detection model.

[0161] In deep learning, multi-scale feature extraction refers to analyzing images at different resolutions and scales to capture defects of various sizes and shapes. This process may involve using convolutional neural networks (CNNs) to extract features and generate multi-level feature representations. For example, when processing an image of glass containing scratches, the model can identify features at different scales: at a small scale, it can detect subtle scratches, while at a large scale, it can identify overall pattern changes. In this way, the system can gain a more comprehensive understanding of the characteristics of the defects.

[0162] Features extracted from the image sample set will be integrated to construct a static defect feature template library. This template library will contain feature descriptions of various known defects for subsequent defect detection and classification. For example, assuming that after multi-scale feature extraction, the system identifies three main defects: scratches, bubbles, and impurities, each defect will have a unique set of feature vectors. These features are stored in the template library so that future detection models can identify defects by comparing the features of the input image with the features in the template library.

[0163] In practical applications, as more new images are input, the system can fine-tune the deep feature extraction model online. That is, when a new type of defect or sample is detected, the model will adjust based on this new data to improve the accuracy and robustness of detection. For example, if a new input image shows a new type of scratch that has not appeared in the template library before, the system can collect this information and use the sample to update the model parameters to make it more adaptable to the new type of defect. After that, the fine-tuned model will be used for the next feature extraction, thereby improving the overall detection effect.

[0164] Example 2, please refer to Figure 2 This invention provides a technical solution: a glass surface defect detection system based on visual inspection, applicable to the aforementioned glass surface defect detection method based on visual inspection, comprising:

[0165] Image acquisition unit 1 is used to acquire full-frame images of the glass substrate in transit through a multi-view industrial camera array to obtain an initial set of grayscale images of the glass surface; wherein, the multi-view industrial camera array includes a line scan camera and an area scan camera, the line scan camera is used to capture the glass edge contour information, and the area scan camera is used to capture the glass body texture information.

[0166] Image comparison unit 2 is used to compare the initial grayscale image set with a preset static defect feature template library to detect whether the initial grayscale image set meets the primary matching criteria; wherein, the primary matching criteria include edge contour matching, texture grayscale contrast and specific shape geometric matching;

[0167] The defect extraction unit 3 is used to extract preliminary defect feature information of the glass surface based on the initial grayscale image set if the initial matching is not met; wherein, the preliminary defect feature information includes the grayscale contrast, geometric shape parameters and initial position coordinates of the defect;

[0168] The parameter optimization unit 4 is used to dynamically adjust the focal length of the liquid lens and the emission angle of the polarization light source based on the geometric shape parameters and initial position coordinates in the preliminary defect feature information to optimize the optical parameters of the defect area on the glass surface; and to perform high frame rate image tracking and acquisition of the defect area based on the multi-view industrial camera array after optical parameter optimization to obtain the fine texture image and depth image of the defect area.

[0169] Vector extraction unit 5 is used to input the refined texture image and depth image into the depth feature extraction model to extract the current depth feature vector; wherein, the current depth feature vector includes color space distribution, texture complexity and local gradient direction; and to detect whether the current depth feature vector meets the high-level matching with the latest dynamic defect feature library;

[0170] The defect matching unit 6 is used to determine that there is a defect on the glass surface if the high-level matching is met, and to add the current depth feature vector and the corresponding acquisition time parameters to the dynamic defect feature library.

[0171] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A method for detecting glass surface defects based on visual inspection, characterized in that, include: A multi-view industrial camera array is used to acquire full-frame images of the glass substrate during transmission, obtaining an initial set of grayscale images of the glass surface. The multi-view industrial camera array includes a line scan camera and an area scan camera. The line scan camera is used to capture the edge contour information of the glass, and the area scan camera is used to capture the texture information of the glass body. The initial set of grayscale images is compared with a preset static defect feature template library to detect whether the initial set of grayscale images meets the primary matching criteria. The primary matching criteria include edge contour matching, texture grayscale contrast, and specific shape geometry matching. If the initial matching is not met, preliminary defect feature information of the glass surface is extracted based on the initial grayscale image set; wherein, the preliminary defect feature information includes the grayscale contrast, geometric shape parameters and initial position coordinates of the defect; Based on the geometric parameters and initial position coordinates in the preliminary defect feature information, the focal length of the liquid lens and the emission angle of the polarization light source are dynamically adjusted to optimize the optical parameters of the defect area on the glass surface. Based on the multi-view industrial camera array with optimized optical parameters, high frame rate image tracking and acquisition are performed on the defect area to obtain fine texture and depth images of the defect area. The refined texture image and depth image are input into the depth feature extraction model to extract the current depth feature vector. The current depth feature vector includes color space distribution, texture complexity, and local gradient direction. The model then checks whether the current depth feature vector matches the latest dynamic defect feature library in a high-level matching manner. If the high-level matching is met, it is determined that there is a defect on the glass surface, and the current depth feature vector and the corresponding acquisition time parameters are added to the dynamic defect feature library.

2. The method for detecting glass surface defects based on visual inspection according to claim 1, characterized in that, The method further includes: Based on the refined texture image and depth image, and combined with the grayscale contrast in the preliminary defect feature information, a three-dimensional topological reconstruction model of glass surface defects is constructed. The volume parameters and edge sharpness of the defect are calculated based on the three-dimensional topological reconstruction model; The volume parameters, edge sharpness, and current depth feature vector are input into a pre-trained defect classification neural network, which outputs the probability distribution of defects. Based on the probability distribution and the usage scenario parameters of the glass substrate, a preset level mapping table is consulted to determine the final defect level of the glass surface.

3. The method for detecting glass surface defects based on visual inspection according to claim 2, characterized in that, Based on the refined texture image and depth image, and combined with the grayscale contrast from the preliminary defect feature information, a three-dimensional topological reconstruction model of the glass surface defect is constructed, including: The refined texture image is mapped onto the surface mesh of the 3D topology reconstruction model to obtain texture mapping data; The depth image is converted into height field data of a 3D topological reconstruction model, and noise interference in the height field is corrected based on the gray-scale contrast in the preliminary defect feature information. Based on the texture mapping data and the corrected height field data, the surface normal vector and curvature change of the defect region are calculated iteratively. Based on the surface normal vector and curvature changes, the three-dimensional topological connectivity of the defect is determined, and a three-dimensional topological reconstruction model is generated.

4. The method for detecting glass surface defects based on visual inspection according to claim 3, characterized in that, Before acquiring detailed texture and depth images of the defect area by high-frame-rate image tracking and acquisition of the defect area using a multi-view industrial camera array optimized for optical parameters, the following steps are also included: Based on the geometric shape parameters in the preliminary defect feature information, determine whether the defect is located in the blind zone of the overlapping field of view of adjacent cameras; If so, the blind zone compensation angle is calculated based on the geometric parameters, and a specific camera in the multi-view industrial camera array is controlled to perform micro-displacement or rotation until the defect completely enters the field of view overlap area. Based on the camera's pose after displacement or rotation, the image acquisition coordinate system is recalibrated, and a calibrated acquisition area mapping table is generated. Among them, a multi-view industrial camera array optimized with optical parameters is used to perform high-frame-rate image tracking and acquisition of defect areas, including: Based on the calibrated acquisition area mapping table, the multi-view industrial camera array with optimized optical parameters is used to perform high frame rate image tracking and acquisition of the defect area.

5. The method for detecting glass surface defects based on visual inspection according to claim 4, characterized in that, The validity period of the latest dynamic defect feature library is within a preset production time sliding window corresponding to the data acquisition time parameter; Detect whether the current deep feature vector meets the advanced matching criteria with the latest dynamic defect feature library, including: Identify whether the latest dynamic defect feature library contains neighboring feature vectors belonging to the same cluster as the current deep feature vector; Compare whether the density values ​​of neighboring feature vectors appearing in the dynamic defect feature library exceed the corresponding density threshold; When the density threshold is exceeded and the confidence level of the current depth feature vector is lower than the average confidence level in the database, the current depth feature vector is determined to meet the high-level matching criteria.

6. The method for detecting glass surface defects based on visual inspection according to claim 5, characterized in that, The method further includes: Re-extract the historical depth feature vectors corresponding to historical image frames that were not identified as defects within the production time sliding window; Calculate the feature space distance between the historical deep feature vector and each feature vector in the latest dynamic defect feature library; Historical image frames corresponding to historical depth feature vectors whose feature space distance is less than a preset distance threshold are identified as images with missed defects. The dynamic defect feature library is updated based on the historical depth feature vectors of images identified as missed defects.

7. The method for detecting glass surface defects based on visual inspection according to claim 6, characterized in that, The method further includes: Based on the volume parameters and edge sharpness corresponding to the final defect level, the exposure time and gain parameters of the multi-view industrial camera array are adjusted in reverse. If the final defect level is severe, the sorting mechanism of the glass transmission line will be controlled to accurately reject the defect based on the defect location coordinates in the three-dimensional topology reconstruction model. If the final defect level is minor, the laser energy required for repair is calculated based on the defect's volume parameters, and the laser repair equipment is controlled to perform in-situ repair.

8. The method for detecting glass surface defects based on visual inspection according to claim 7, characterized in that, If the final defect level is severe, the sorting mechanism of the glass transmission line will be controlled to precisely reject the defect based on the defect location coordinates in the 3D topology reconstruction model, including: Obtain real-time operating speed and acceleration information of the glass transmission line; Based on the location coordinates of the defect and the real-time running speed of the line, predict the time delay for the defect to reach the sorting mechanism; Calculate the lead offset based on the time delay and the action response time of the sorting mechanism; The timing of the pneumatic valve opening of the sorting mechanism is controlled by the lead offset, and glass substrates with serious defects are rejected and sent to the waste area.

9. The method for detecting glass surface defects based on visual inspection according to claim 8, characterized in that, The method further includes: Obtain a sample set of glass surface images; the sample set of glass surface images includes real defect labels and defect-free labels; Multi-scale defect feature extraction from image sample set; A static defect feature template library is constructed based on the extracted multi-scale defect features; The depth feature extraction model is fine-tuned online based on the image frames identified as defective, and the fine-tuned depth feature extraction model is used for feature extraction of the next current image frame.

10. A glass surface defect detection system based on vision inspection, applicable to the glass surface defect detection method based on vision inspection as described in any one of claims 1-9, characterized in that, include: The image acquisition unit is used to acquire full-frame images of the glass substrate in transit through a multi-view industrial camera array, and obtain an initial grayscale image set of the glass surface; wherein, the multi-view industrial camera array includes a line scan camera and an area scan camera, the line scan camera is used to capture the glass edge contour information, and the area scan camera is used to capture the glass body texture information. The image comparison unit is used to compare the initial set of grayscale images with a preset static defect feature template library to detect whether the initial set of grayscale images meets the primary matching criteria; wherein, the primary matching criteria include edge contour matching, texture grayscale contrast and specific shape geometric matching; The defect extraction unit is used to extract preliminary defect feature information of the glass surface based on the initial grayscale image set if the initial matching condition is not met. The preliminary defect feature information includes the grayscale contrast, geometric shape parameters and initial position coordinates of the defect. The parameter optimization unit is used to dynamically adjust the focal length of the liquid lens and the emission angle of the polarization light source based on the geometric shape parameters and initial position coordinates in the preliminary defect feature information to optimize the optical parameters of the defect area on the glass surface; and to perform high frame rate image tracking and acquisition of the defect area based on the multi-view industrial camera array after optical parameter optimization to obtain the fine texture image and depth image of the defect area. The vector extraction unit is used to input the refined texture image and depth image into the depth feature extraction model to extract the current depth feature vector. The current depth feature vector includes color space distribution, texture complexity, and local gradient direction. It also detects whether the current depth feature vector meets the high-level matching requirement with the latest dynamic defect feature library. The defect matching unit is used to determine that there is a defect on the glass surface if the high-level matching is met, and adds the current depth feature vector and the corresponding acquisition time parameters to the dynamic defect feature library.

Images