Machine vision-based online detection and sorting system for surface defects of baked porcelain plate

Abstract

The invention relates to the technical field of industrial automation and machine vision, and particularly discloses a baked porcelain plate surface defect online detecting and sorting system based on machine vision. The system comprises an automatic feeding module, a multi-modal image acquisition module, a central data processing module and an intelligent execution sorting module which are arranged in sequence, synchronously acquiring surface information by adopting three imaging units of a bright field, a dark field and a 3D outline; constructing a multi-modal feature fusion defect classification model based on deep learning, and realizing feature fusion through a cross attention mechanism; the integrated model online learning unit is used for realizing incremental learning by adopting an elastic weight consolidation algorithm; the intelligent execution sorting module plans a track according to the defect position and performs sorting through a self-adaptive vacuum chuck array; the device realizes comprehensive detection of various defects such as scratches, pits and chromatic aberration, has the beneficial effects of high detection precision, strong self-adaption, avoidance of secondary damage and the like, and improves the production efficiency and the product quality.

Description

Technical Field

[0001] This invention relates to the fields of industrial automation and machine vision technology, and more specifically, to an online detection and sorting system for surface defects of ceramic plates based on machine vision. Background Technology

[0002] As a high-end building decoration and home renovation material, the surface quality of ceramic tile is a key factor in determining product grade, market value and brand reputation. During its production and manufacturing process, due to the influence of various complex factors such as raw material ratio, sintering process, glaze spraying, mechanical conveying and post-packaging, ceramic tile surface is prone to various types of defects, including but not limited to scratches, micro-cracks, pits, bumps, color difference, stains, chipped edges, orange peel texture and wavy deformation. Currently, most domestic ceramic tile manufacturers still rely mainly on manual visual inspection or automated inspection systems for surface quality inspection of finished products. However, manual inspection is inefficient, costly, and difficult to quantify and trace. Existing automated inspection systems mostly use a single light source and camera imaging mode, which can only effectively detect defects sensitive to specific imaging conditions. For defects such as scratches, tiny pits and bumps, and orange peel texture that require specific lighting angles or three-dimensional information to be highlighted, the detection rate is extremely low, which cannot meet the detection needs of all types of defects in ceramic tiles. This has the problem of one-sided detection capabilities. At the same time, the detection technology based on two-dimensional images cannot obtain the depth and height information of the surface, so it is powerless to detect crucial quality indicators such as flatness and tiny three-dimensional deformation. Therefore, based on the above problems, there is a need for an automated system that can comprehensively, accurately, efficiently, and intelligently detect and sort various defects on the surface of ceramic plates online, in order to overcome the inherent shortcomings of manual inspection and traditional machine vision technology. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this invention is to provide an online detection and sorting system for surface defects of ceramic plates based on machine vision, so as to solve the problems mentioned in the background technology.

[0004] The above-mentioned technical objective of the present invention is achieved through the following technical solution: an online detection and sorting system for surface defects of ceramic plates based on machine vision, comprising: an automatic feeding module for conveying ceramic plates, a multimodal image acquisition module, a central data processing module, and an intelligent execution sorting module arranged sequentially along the production line conveying direction; The multimodal image acquisition module includes three imaging units arranged in parallel within a closed detection chamber, namely: The first imaging unit consists of a first high-speed linear array camera and a high-angle uniform bright field light source that works in conjunction with the first high-speed linear array camera. It is configured to acquire macroscopic color and texture images of the surface of the ceramic plate and to detect color difference, stains, and edge chipping defects on the ceramic plate. The second imaging unit consists of a second high-speed linear array camera and a low-angle dark field linear light source that works in conjunction with the second high-speed linear array camera. It is configured to acquire microscopic scattering images of the surface of the ceramic plate for detecting scratches, microcracks, and orange peel defects on the ceramic plate. The third imaging unit consists of a line laser scanning 3D profilometer and a laser that works in conjunction with the line laser scanning 3D profilometer. It is configured to acquire three-dimensional point cloud data of the surface of the ceramic plate in real time by line scanning, in order to reconstruct the high-precision three-dimensional shape of the ceramic plate and detect pits, protrusions, and wave deformation defects. The central data processing module is communicatively connected to the multimodal image acquisition module. It has a built-in deep learning-based multimodal feature fusion defect classification model, which is used to analyze the data acquired by the multimodal image acquisition module and output the porcelain plate recognition result. It fits the information of the porcelain plate with defects to generate defect information and transmits it. The intelligent execution sorting module is communicatively connected to the central data processing module and includes a multi-axis industrial robot and an end effector connected thereto. The multi-axis industrial robot plans a motion trajectory based on the received defect information, and then drives the end effector to locate, grasp and sort the defective ceramic plates.

[0005] Optionally, the multimodal feature fusion defect classification model is configured as follows: Pixel-level registration is performed on the two-dimensional images from the first imaging unit and the second imaging unit, and spatiotemporal synchronization is performed with the three-dimensional point cloud data from the third imaging unit. Depth visual features of 2D images and geometric topological features of 3D point clouds are extracted respectively; By using a pre-defined feature fusion network based on a cross-attention mechanism, the deep visual features and geometric topological features are adaptively weighted and fused to generate a unified defect feature representation rich in multimodal information. Based on the defect feature representation, the existence, category, confidence level of the defect, as well as the precise pixel coordinates and physical size on the detected ceramic plate are output, and then the defect identification result of the ceramic plate is generated by fitting.

[0006] Optionally, the multimodal feature fusion defect classification model adopts a cascaded network architecture, which includes, in sequence: The feature encoder includes a 2D branch and a 3D branch; the 2D branch uses a cropped ConvNeXt network, which simultaneously receives the registered images of the first and second imaging units to perform dual-stream feature extraction and early fusion; the 3D branch uses a point cloud processing network based on PointNet++ to extract multi-scale three-dimensional geometric features. The cross-modal fusion module is based on a cross-attention module. It uses the image feature map output by the 2D branch as the query and the point cloud features output by the 3D branch as the key and value to calculate the correlation between the image region and the 3D point cloud, thereby realizing the guidance and enhancement of 2D image features by 3D geometric information. The defect decoder receives the fused features and employs a fully convolutional network structure to output two prediction heads in parallel: one for pixel-level defect semantic segmentation, generating an accurate mask for defects, and the other for multi-label defect classification based on global features.

[0007] Optionally, it also includes an online learning unit for the model, including: The confidence assessment module, integrated into the defect decoder, is used to calculate the confidence score of each ceramic tile defect identification result. The active learning module automatically marks the sample as a difficult case when the confidence score of any of the defects in the ceramic tile identification results is lower than a preset threshold, and temporarily stores the corresponding image, point cloud data and prediction results in the database to be reviewed. The manual review and annotation module provides a graphical tool for quality inspectors to quickly review and correctly annotate difficult cases; Incremental learning engine: The elastic weight consolidation algorithm, which is manually reviewed, is used periodically to fine-tune the multimodal feature fusion defect classification model to combat overfitting, and the fine-tuned model version is saved.

[0008] Optionally, the elastic weight consolidation algorithm introduces a regularization term when calculating the loss function; the regularization term is based on the Fisher information matrix of the model's old parameters on historical training data, and is used to penalize large modifications to important parameters of the old task during the learning process of the new task, thereby effectively mitigating catastrophic forgetting.

[0009] Optionally, the high-angle uniform bright field light source of the first imaging unit is an LED planar light source with a color temperature of 6000K±500K and a color rendering index greater than 95, and the illumination angle is 75°-90° with the normal direction of the ceramic plate surface. The low-angle dark-field linear light source of the second imaging unit is an infrared LED linear light source with a wavelength of 850nm. The illumination angle is 5°-15° with the normal direction of the ceramic plate surface, and the second high-speed linear array camera is equipped with a corresponding infrared bandpass filter.

[0010] Optionally, it also includes a high-precision synchronization triggering unit, the high-precision synchronization triggering unit comprising: A rotary encoder is mounted on the drive shaft of the production line conveyor belt to measure the precise displacement of the conveyor belt in real time. The FPGA synchronous control unit is used to receive the pulse signal from the rotary encoder, accurately calculate the trigger delay according to the pre-calibrated relative position of the camera and the ceramic plate, and send a main trigger signal to the first imaging unit, the second imaging unit and the third imaging unit, thereby ensuring that the three-modal images are acquired from the same physical area of ​​the same ceramic plate.

[0011] Optionally, the end effector of the intelligent sorting module is an adaptive vacuum suction cup array, which consists of several independently controlled vacuum suction cup units; each vacuum suction cup unit consists of an independent vacuum generator and a pressure sensor; the defect location information output by the central data processing module is used to dynamically generate an adsorption strategy and control the suction cup units far from the defect location to work, so as to avoid secondary damage to the defect area.

[0012] Optionally, a distributed optical calibration module is also integrated for initial calibration before system deployment and periodic self-calibration during operation, including: A 2D calibration board is used for the joint calibration of intrinsic parameters, extrinsic parameters, and distortion correction of the first and second imaging units. The 3D calibration block is a stepped standard block with a known precise height on its surface, used to calibrate the height measurement accuracy of the third imaging unit; Hand-eye calibration tool: used to establish precise transformation relationships between the robot coordinate system and all camera coordinate systems.

[0013] A method for online detection and sorting of surface defects in ceramic enamel plates based on the above-mentioned machine vision-based online detection and sorting system includes the following steps: Step S1: System initialization, complete the hand-eye calibration and measurement accuracy calibration of the first imaging unit, the second imaging unit and the third imaging unit through the distributed optical calibration module; Step S2: The automatic feeding module feeds the ceramic plates to be inspected to the high-speed conveyor belt at fixed intervals; Step S3: According to the encoder signal, when the ceramic plate enters the acquisition station, the high-precision synchronous triggering unit synchronously triggers the first imaging unit, the second imaging unit and the third imaging unit to acquire the bright field 2D image, dark field 2D image and 3D point cloud data of the ceramic plate respectively. Step S4: The central data processing module receives multimodal data and performs pixel-level registration and spatiotemporal synchronization. Step S5: The multimodal feature fusion defect classification model extracts features from the registered data, performs cross-modal fusion, and performs semantic segmentation and classification of defects. It outputs the porcelain plate recognition results containing defect category, location and confidence level, and fits the porcelain plate information with defects to generate defect information. Step S6: The intelligent sorting module receives defect information, the motion controller plans a collision-free motion trajectory, and drives the adaptive vacuum suction cup array to accurately grasp the defective board. Step S7: According to the preset classification rules, defective boards are sorted to the corresponding scrap bins or downgraded product areas, while qualified products continue to be conveyed along the main water line.

[0014] In summary, the present invention has the following beneficial effects: 1. By using a multimodal image acquisition module and a deep learning-based intelligent analysis model, the system replaces subjective human judgment. The identification, classification, and localization of defects are all performed based on a pre-trained, unified algorithm model, eliminating the interference of subjective factors such as fatigue and emotions. Regardless of the time or the operator, the system's judgment criteria for the same defect remain consistent, thereby improving the consistency and reliability of product quality and achieving objectivity, datafication, and standardization of detection results.

[0015] 2. The multimodal image acquisition module employs three complementary imaging modalities based on physical principles: bright field, dark field, and 3D contour, constructing a comprehensive perception system. The first imaging unit specializes in color and macroscopic texture defects, such as color difference and blemishes; the second imaging unit excels at capturing microscopic scattering defects, such as scratches and microcracks; and the third imaging unit directly measures three-dimensional morphological defects, such as pits, protrusions, and wavy deformation. This multimodal collaborative mechanism ensures that various defects, from two-dimensional to three-dimensional, from macroscopic to microscopic, and from color to geometry, can be effectively perceived, overcoming the limitations of single detection technologies, achieving full coverage of the detection range, and resulting in a low false negative rate.

[0016] 3. In the process of system identification and judgment, the traditional threshold and edge-based fragile algorithms in existing technologies were not used. Instead, a multimodal feature fusion defect classification model based on deep learning was adopted. This model, trained on massive amounts of data, can learn the essence and abstract features of defects and has strong fault tolerance and adaptability to lighting fluctuations, slight product position shifts, and complex background changes. In particular, the introduction of the cross-attention mechanism enables the model to intelligently fuse 2D appearance information and 3D geometric information for mutual verification. For example, 3D information is used to confirm whether the area suspected to be a scratch in the 2D image is a real physical groove, thereby reducing the probability of misjudging scratches, reflections, etc. as defects, improving the recognition accuracy and reducing the false detection rate. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system flow of the present invention; Figure 2 This is a flowchart illustrating the system execution method of the present invention. Detailed Implementation

[0018] To make the objectives, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0019] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0020] In this invention, unless otherwise expressly specified and limited, "above" or "below" a second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of a second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" of a second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature. The terms "vertical," "horizontal," "left," "right," "above," "below," and similar expressions are for illustrative purposes only and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0021] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] This invention provides an online detection and sorting system for surface defects in ceramic enamel plates based on machine vision, such as... Figure 1As shown, it includes: an automatic feeding module for conveying ceramic plates, a multimodal image acquisition module, a central data processing module, and an intelligent execution sorting module, which are arranged sequentially along the production line conveying direction; The multimodal image acquisition module includes three imaging units arranged in parallel within a closed detection chamber, namely: The first imaging unit consists of a first high-speed linear array camera and a high-angle uniform bright field light source that works in conjunction with the first high-speed linear array camera. It is configured to acquire macroscopic color and texture images of the surface of the ceramic plate and to detect color difference, stains, and edge chipping defects on the ceramic plate. The second imaging unit consists of a second high-speed linear array camera and a low-angle dark field linear light source that works in conjunction with the second high-speed linear array camera. It is configured to acquire microscopic scattering images of the surface of the ceramic plate for detecting scratches, microcracks, and orange peel defects on the ceramic plate. The third imaging unit consists of a line laser scanning 3D profilometer and a laser that works in conjunction with the line laser scanning 3D profilometer. It is configured to acquire three-dimensional point cloud data of the surface of the ceramic plate in real time by line scanning, in order to reconstruct the high-precision three-dimensional shape of the ceramic plate and detect pits, protrusions, and wave deformation defects. The central data processing module is communicatively connected to the multimodal image acquisition module. It has a built-in deep learning-based multimodal feature fusion defect classification model, which is used to analyze the data acquired by the multimodal image acquisition module and output the porcelain plate recognition result. It fits the information of the porcelain plate with defects to generate defect information and transmits it. The intelligent execution sorting module is communicatively connected to the central data processing module and includes a multi-axis industrial robot and an end effector connected thereto. The multi-axis industrial robot plans a motion trajectory based on the received defect information, and then drives the end effector to locate, grasp and sort the defective ceramic plates.

[0023] Specifically, the automatic feeding module in this system is a roller conveyor line driven by a servo motor, equipped with photoelectric sensors to detect the position of the ceramic plates. The PLC control system realizes the equal-spacing conveying of the ceramic plates, with the spacing set at 500mm. The conveying speed can be steplessly adjusted within the range of 1-3m / s to adapt to different production cycle requirements, so as to convey the ceramic plates according to the actual detection speed. The multimodal image acquisition module is installed in a closed detection chamber. The preferred size of the closed detection chamber is 2000mm×1500mm×1200mm. The interior is treated with matte blackening to eliminate stray light interference. The central data processing module has a built-in multimodal feature fusion defect classification model, which is used to analyze the collected data and then output defect information. The intelligent sorting module is a six-axis industrial robot with an end effector at the end, which is a customized 4×4 vacuum suction cup array with 16 independently controlled SMC ZH series vacuum suction cups. The vacuum system is an electric vacuum generator equipped with an SMCZSE30 pressure sensor, thereby enabling the sorting of defective ceramic plates.

[0024] Furthermore, the multimodal feature fusion defect classification model is configured as follows: Pixel-level registration is performed on the two-dimensional images from the first imaging unit and the second imaging unit, and spatiotemporal synchronization is performed with the three-dimensional point cloud data from the third imaging unit. Depth visual features of 2D images and geometric topological features of 3D point clouds are extracted respectively; By using a pre-defined feature fusion network based on a cross-attention mechanism, the deep visual features and geometric topological features are adaptively weighted and fused to generate a unified defect feature representation rich in multimodal information. Based on the defect feature representation, the existence, category, confidence level of the defect, as well as the precise pixel coordinates and physical size on the detected ceramic plate are output, and then the defect identification result of the ceramic plate is generated by fitting.

[0025] Furthermore, the multimodal feature fusion defect classification model adopts a cascaded network architecture, which includes, in sequence: The feature encoder includes a 2D branch and a 3D branch; the 2D branch uses a cropped ConvNeXt network, which simultaneously receives the registered images of the first and second imaging units to perform dual-stream feature extraction and early fusion; the 3D branch uses a point cloud processing network based on PointNet++ to extract multi-scale three-dimensional geometric features. The cross-modal fusion module is based on a cross-attention module. It uses the image feature map output by the 2D branch as the query and the point cloud features output by the 3D branch as the key and value to calculate the correlation between the image region and the 3D point cloud, thereby realizing the guidance and enhancement of 2D image features by 3D geometric information. The defect decoder receives the fused features and employs a fully convolutional network structure to output two prediction heads in parallel: one for pixel-level defect semantic segmentation, generating an accurate mask for defects, and the other for multi-label defect classification based on global features.

[0026] In a specific embodiment, the multimodal feature fusion defect classification model adopts a cascaded network architecture, comprising three main components: a feature encoder, a cross-modal fusion unit, and a defect decoder; wherein the feature encoder includes two parallel branches, specifically comprising: The 2D feature encoding branch takes as input the bright-field image acquired by the first imaging unit. Dark-field images acquired by the second imaging unit Where H represents the height of the image and W represents the width of the image; The 2D feature encoding branch adopts an improved ConvNeXt network structure, and its forward propagation process is as follows: Let the input image be... Where C is the number of channels, with C=3 for bright field and C=1 for dark field; the network consists of multiple convolutional blocks, and the operation of each convolutional block can be represented as:

[0027]

[0028] in, This is represented as the feature map of the l-th layer; This is represented as a layer normalization operation; Represented as depthwise separable convolution; It is represented as a multilayer perceptron, consisting of two fully connected layers and a GELU activation function; After the bright-field and dark-field images are passed through independent shallow convolutional layers, feature fusion is performed in the second stage of the network, as follows:

[0029] in, Represented as the feature extraction function for bright-field images, This is represented as a feature extraction function for dark-field images. This is represented as a channel-dimensional concatenation operation; the result is a fused 2D feature map. , ; The input to the 3D feature coding branch is the 3D point cloud data acquired by the third imaging unit. Where N represents the number of point cloud data, Let be the three-dimensional coordinates of the i-th point; The 3D feature encoding branch employs a multi-scale feature extraction network based on PointNet++, specifically implemented as follows: Suppose that point cloud P is divided into multiple local regions. Each region contains K points. For each local region, the feature extraction process is as follows:

[0030] in, Represented as a point coordinate transformation function, This is represented as a multilayer perceptron with shared weights, and MAX represents the max pooling operation. Furthermore, through a multi-scale grouping MSG strategy, features are extracted at different scales, as follows:

[0031] in, This is represented as the extracted local features within the search radius r. This is represented as the final extracted geometric features. ; After 2D and 3D processing, feature fusion is performed through a cross-modal fusion network. The cross-modal fusion network employs a cross-attention mechanism, and the specific implementation process is as follows: Let the feature map of the 2D branch output be... Flatten it into a sequence form ,in ; Let the feature vector of the 3D branch output be... The key and value are generated through a linear transformation, represented as follows:

[0032]

[0033] in, All of these are learnable weight matrices; In the process of cross-attention calculation, it is represented as:

[0034] in , which is represented as the square root of the attention dimension, used for scaling; This is represented as softmax normalization in the row direction; Simultaneously, a multi-head attention mechanism is adopted, with the number of heads being h, expressed as:

[0035] The calculation process for each attention head is represented as follows:

[0036] Finally, the output fused features are represented as:

[0037] Based on defect feature representation The process of outputting the defect identification results of porcelain enamel panels includes three main stages: feature decoding, multi-task prediction, and result fusion. In the feature decoding stage, the specific implementation process is as follows, assuming the fused defect features are represented as follows: First, it is restored to a spatial feature map through feature reshaping, represented as follows:

[0038] in, , which represents the spatial size after downsampling; In the multi-task prediction stage, defect semantic segmentation is performed first. The segmentation head adopts an encoder-decoder structure, and spatial resolution is restored through progressive upsampling, as shown below: Let the encoder feature pyramid be... ,in The decoding process is represented as follows: Level 1 Decoding: ; Second-level decoding: ; Third-level decoding:

[0039] Final upsampling: ; Segmentation map generation: ; in, This is represented as 2x bilinear upsampling; It is represented as a 3×3 convolutional layer + batch normalization + ReLU activation; K represents the number of defect categories, including the background class, and S represents the probability distribution of each pixel belonging to each category; Defect multi-label classification is performed synchronously and in parallel. The classification head generates image-level classification results by globally aggregating spatial features, as shown below: Global feature aggregation:

[0040] After global feature aggregation, multi-label classification of defect labels is performed, represented as follows:

[0041]

[0042] in, Indicates global average pooling. It is represented as a bias vector. Represented as a classification weight matrix, This represents the probability of the existence of the k-th type defect; In the result fusion stage, the input data includes: the semantic segmentation probability map output by the segmentation head, the multi-label classification probability output by the classification head, 3D point cloud data, and defect representation; In the data processing stage, the segmentation map output by the segmentation head is post-processed to generate accurate defect regions. First, an initial segmentation mask is generated through maximum probability classification, represented as... Subsequently, a conditional random field is applied for refinement, which is expressed as: ,in The function is implemented based on minimizing the following energy function. ,in, For univariate potential energy, based on segmentation probability, , It is a binary potential energy; The refined segmentation mask is used for instance segmentation. The specific implementation process is as follows: For each defect category, it is particularly important to skip the background category 0, extract the binary mask of the current category, perform connected component analysis, filter small area noise, and then identify the connected regions in the image to complete the post-processing of the segmentation results. The classification results are calibrated using a confidence level based on segmentation area. The spatial information provided by the segmentation results is used to calibrate the classification confidence level. For each defect category k, the segmentation area ratio is calculated, and the post-calibration confidence level is calculated, expressed as:

[0043] in, =0.7, representing the confidence weight of the original classification number. This can be expressed as an area percentage function. =8, which represents the scaling factor, controlling the sensitivity to the influence of area; After calculating the classification results, the physical rationality of the classification results is verified using 3D point cloud data. For each suspected defect area, the consistency of 3D features is checked, the 3D point cloud of the corresponding area is extracted, and the 3D feature anomaly score is calculated. Based on the calculated 3D feature anomaly score, the classification confidence is adjusted. When the score is higher than the preset value, the confidence is increased, with a typical value of 1.2 times the original value; when the score is lower than the preset value, the confidence is decreased, with a typical value of 0.8 times the original value. The detected defects are instantiated. For each detected defect instance, detailed geometric and physical properties are calculated, including but not limited to: the defect's bounding box, the defect's center coordinates, physical size calculation, and defect contour extraction. The physical size calculation can be based on 3D point cloud data or estimated based on pixel size and camera calibration parameters. Then, a comprehensive confidence score is calculated for each defect instance, including: a base confidence score and a calibrated classification confidence score for the corresponding category; a segmentation quality score, used to evaluate the reliability and accuracy of the segmentation results, calculated based on the following multi-factors, including but not limited to: boundary sharpness assessment, region consistency assessment, and contour smoothness assessment; a geometric consistency score, used to evaluate the consistency between 2D segmentation results and 3D geometric features, using 2D and 3D data for analysis, including but not limited to: height variation consistency, curvature consistency, normal vector consistency, and spatial location consistency; and a feature consistency score, used to evaluate the consistency of multimodal features within the defect region, including but not limited to: intermodal feature correlation, feature space clustering degree, anomaly feature detection, and feature stability; based on the above scores, a weighted calculation is used to obtain the comprehensive confidence score. Perform a final segmentation and classification consistency check, classify the defects, check categories with high classification confidence and large segmentation area, check categories with high classification confidence but small segmentation area, and check categories with large segmentation area but low classification confidence. Judgments with confidence below the preset threshold are considered false judgments. The corresponding conflict resolution strategies are used in parallel to distinguish the categories. For categories with high classification confidence but small segmentation area, they may be small target defects. They are retained but the confidence is reduced. For categories with large segmentation area but low classification confidence, they may be missegments. 3D features are checked for confirmation. If the 3D features confirm that the defect exists, the confidence is increased. If it is confirmed to be a missegment, the defect category is removed. This process enables the identification and judgment of defect categories. In Example 1, under normal circumstances, the input data is: segmentation map S, a scratched area is detected, with an area ratio of 0.8%; classification probability. The scratch class score was 0.85, while all other classes were <0.1; the 3D point cloud showed significant height variations in the corresponding regions; during the fusion process: accurate scratch masks were obtained through post-segmentation processing; classification calibration was performed as follows. Instantiation: Calculate bounding box, center coordinates, and physical dimensions; Confidence fusion, considering segmentation quality, geometric consistency, etc., with a final confidence level of 0.72; The fused output is a single scratch defect with high confidence. In Example 2, under the condition of multiple defects, the input data is: Segmentation map S: two independent regions, corresponding to scratches and dents respectively, and classification probabilities. Scratches: 0.65, pits: 0.55, color difference: 0.20; 3D point cloud: Region 1 has slight height changes, and Region 2 has obvious depressions; During the fusion process, two independent instances are identified through connected component analysis, and the two instances are instantiated and analyzed separately. The results are then fused and output as two defect instances: scratches (medium confidence) and pits (high confidence). In Example 3, under the condition of inconsistency, the input data is as follows: Segmentation map S: a large area is marked as scratch, accounting for 8% of the area; Classification probability y_cls: scratch 0.25, color difference 0.70; 3D point cloud: the corresponding area has no height change, but the color is abnormal; During the fusion process: Inconsistency is detected: the segmentation area is large but the classification confidence is low; 3D verification: there are no scratch features, but the color features are obvious; Reclassification: based on multimodal features, the instance is reclassified as color difference; Confidence adjustment: considering the initial inconsistency, the final confidence is 0.45; Output: color difference defect, medium confidence, marked as requiring manual review or misjudgment.

[0044] Furthermore, it also includes online learning units for models, including: The confidence assessment module, integrated into the defect decoder, is used to calculate the confidence score of each ceramic tile defect identification result. The active learning module automatically marks the sample as a difficult case when the confidence score of any of the defects in the ceramic tile identification results is lower than a preset threshold, and temporarily stores the corresponding image, point cloud data and prediction results in the database to be reviewed. The manual review and annotation module provides a graphical tool for quality inspectors to quickly review and correctly annotate difficult cases; Incremental learning engine: The elastic weight consolidation algorithm, which is manually reviewed, is used periodically to fine-tune the multimodal feature fusion defect classification model to combat overfitting, and the fine-tuned model version is saved.

[0045] Specifically, the confidence assessment module includes: an input interface that receives the segmentation and classification results output by the defect decoder; a confidence calculation unit that calculates the confidence of the results based on a multi-factor fusion confidence calculation algorithm; and a threshold management unit that has an adaptive mechanism for dynamically adjusting the confidence threshold. The active learning module includes: a difficult example identification unit, which identifies uncertain samples based on confidence assessment results; a data cache management unit, which uses a sample storage system with a circular buffer structure; and a metadata recording unit, which is used to save the context information of the samples and model prediction records. The manual review and annotation module includes: a web-based visual interface that provides multimodal data collaborative display; an interactive annotation tool that supports pixel-level annotation and category correction; and a review workflow management unit for quality inspection task allocation and progress tracking. The incremental learning engine includes a training data management unit, a balanced sampling strategy for new and old samples, a built-in model fine-tuning algorithm based on elastic weight consolidation, and a model versioning and rollback mechanism.

[0046] Furthermore, the elastic weight consolidation algorithm introduces a regularization term when calculating the loss function; the regularization term is based on the Fisher information matrix of the model's old parameters on historical training data, and is used to penalize significant modifications to important parameters of the old task during the learning process of the new task, thereby effectively mitigating catastrophic forgetting.

[0047] Furthermore, the high-angle uniform bright field light source of the first imaging unit is an LED planar light source with a color temperature of 6000K±500K and a color rendering index greater than 95, and the illumination angle is 75°-90° with the normal direction of the ceramic plate surface. The low-angle dark-field linear light source of the second imaging unit is an infrared LED linear light source with a wavelength of 850nm. The illumination angle is 5°-15° with the normal direction of the ceramic plate surface, and the second high-speed linear array camera is equipped with a corresponding infrared bandpass filter.

[0048] Furthermore, it also includes a high-precision synchronization triggering unit, which comprises: A rotary encoder is mounted on the drive shaft of the production line conveyor belt to measure the precise displacement of the conveyor belt in real time. The FPGA synchronous control unit is used to receive the pulse signal from the rotary encoder, accurately calculate the trigger delay according to the pre-calibrated relative position of the camera and the ceramic plate, and send a main trigger signal to the first imaging unit, the second imaging unit and the third imaging unit, thereby ensuring that the three-modal images are acquired from the same physical area of ​​the same ceramic plate.

[0049] Furthermore, the end effector of the intelligent sorting module is an adaptive vacuum suction cup array, which consists of several independently controlled vacuum suction cup units; each vacuum suction cup unit consists of an independent vacuum generator and a pressure sensor; the defect location information output by the central data processing module is used to dynamically generate an adsorption strategy and control the suction cup units far from the defect location to work, so as to avoid secondary damage to the defect area.

[0050] Specifically, based on the output defect identification results, the intelligent sorting module operates according to the following process: The input data includes: defect identification results: including defect category, location, confidence level, and physical dimensions; ceramic plate position information: real-time position tracking based on encoder; and system status information: robot current pose, conveyor belt speed, and sorting target position. The output data includes: robot motion trajectory, end effector control signal, and sorting destination selection; The specific implementation process is as follows: Sorting decisions are generated based on defect identification results. The intelligent sorting module's built-in analysis unit analyzes the severity of defects and determines the sorting destination based on the analyzed defect severity, including but not limited to: waste area, downgraded product area, or reprocessing line, and confirms the sorting priority based on the sorting destination. During the adsorption process of defective ceramic plates, an adaptive adsorption strategy based on the defect location on the ceramic plate is intelligently generated. The optimization conditions are: the number of adsorption points meets the minimum requirement, and the available point farthest from the defect is selected, thereby ensuring that secondary damage is avoided during the sorting process. The robot's gripping points are calculated based on the defect location and adsorption strategy, and then converted into the robot's coordinate system. Considering gripping safety, the gripping height is adjusted, and a dynamic trajectory optimization method is used to generate a path point sequence, thereby generating a time-optimal trajectory plan. Furthermore, real-time trajectory compensation based on conveyor belt motion is introduced into the optimization conditions to achieve further optimization of gripping. Based on the above calculation results, the suction cup array is controlled by the adsorption strategy, and the vacuum pressure is dynamically adjusted according to the weight of the board and the defect characteristics. Based on the gripping path under the time-optimal planning, the sorting action is completed, which includes the following stages: approach, precise positioning, adsorption execution, lifting and transfer and release and return. Furthermore, during the sorting process, real-time monitoring of vacuum pressure, robot load, and positional accuracy further improves the precision and efficiency of gripping, fulfilling the requirement for intelligent sorting of defective ceramic plates.

[0051] Furthermore, it also integrates a distributed optical calibration module for initial calibration before system deployment and periodic self-calibration during operation, including: A 2D calibration board is used for the joint calibration of intrinsic parameters, extrinsic parameters, and distortion correction of the first and second imaging units. The 3D calibration block is a stepped standard block with a known precise height on its surface, used to calibrate the height measurement accuracy of the third imaging unit; Hand-eye calibration tool: used to establish precise transformation relationships between the robot coordinate system and all camera coordinate systems.

[0052] A method for online detection and sorting of surface defects in ceramic enamel plates based on the above-mentioned machine vision-based online detection and sorting system includes the following steps: Step S1: System initialization, complete the hand-eye calibration and measurement accuracy calibration of the first imaging unit, the second imaging unit and the third imaging unit through the distributed optical calibration module; Step S2: The automatic feeding module feeds the ceramic plates to be inspected to the high-speed conveyor belt at fixed intervals; Step S3: According to the encoder signal, when the ceramic plate enters the acquisition station, the high-precision synchronous triggering unit synchronously triggers the first imaging unit, the second imaging unit and the third imaging unit to acquire the bright field 2D image, dark field 2D image and 3D point cloud data of the ceramic plate respectively. Step S4: The central data processing module receives multimodal data and performs pixel-level registration and spatiotemporal synchronization. Step S5: The multimodal feature fusion defect classification model extracts features from the registered data, performs cross-modal fusion, and performs semantic segmentation and classification of defects. It outputs the porcelain plate recognition results containing defect category, location and confidence level, and fits the porcelain plate information with defects to generate defect information. Step S6: The intelligent sorting module receives defect information, the motion controller plans a collision-free motion trajectory, and drives the adaptive vacuum suction cup array to accurately grasp the defective board. Step S7: According to the preset classification rules, defective boards are sorted to the corresponding scrap bins or downgraded product areas, while qualified products continue to be conveyed along the main water line.

[0053] This invention discloses an online detection and sorting system for surface defects in ceramic enamel plates based on machine vision. It replaces subjective human judgment with a multimodal image acquisition module and a deep learning-based intelligent analysis model. The system identifies, classifies, and locates defects based on a pre-trained, unified algorithm model, eliminating interference from subjective factors such as fatigue and emotions. Regardless of the time or operator, the system maintains consistent judgment criteria for the same defect, thereby improving product quality consistency and reliability, and achieving objectivity, datafication, and standardization of detection results. The multimodal image acquisition module employs three complementary imaging modalities based on physical principles: bright field, dark field, and 3D contour, constructing a comprehensive perception system. The first imaging unit specializes in color and macroscopic texture defects, such as color difference and stains; the second imaging unit excels at capturing microscopic scattering defects, such as scratches and microcracks; and the third imaging unit directly measures three-dimensional morphological defects, such as pits, protrusions, and wavy deformation. The modal collaborative working mechanism ensures that various defects, from two-dimensional to three-dimensional, from macroscopic to microscopic, and from color to geometry, can be effectively perceived, overcoming the limitations of single detection technologies, achieving full coverage of the detection range, and resulting in a low false negative rate. In the system's identification and judgment process, instead of using traditional, fragile algorithms such as thresholding and edge detection, a deep learning-based multimodal feature fusion defect classification model is employed. This model, trained on massive amounts of data, learns the essence and abstract features of defects, exhibiting strong fault tolerance and adaptability to lighting fluctuations, slight product position shifts, and complex background changes. In particular, the introduction of the cross-attention mechanism enables the model to intelligently fuse 2D appearance information and 3D geometric information for mutual verification. For example, 3D information is used to confirm whether a suspected scratch area in a 2D image is a real physical groove, thereby reducing the probability of misjudging scratches and reflections as defects, improving recognition accuracy, and reducing the false positive rate.

[0054] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A machine vision-based online detection and sorting system for surface defects in ceramic enamel plates, characterized in that, include: The automatic feeding module, multimodal image acquisition module, central data processing module, and intelligent sorting module for conveying ceramic plates are arranged sequentially along the production line conveying direction. The multimodal image acquisition module includes three imaging units arranged in parallel within a closed detection chamber, namely: The first imaging unit consists of a first high-speed linear array camera and a high-angle uniform bright field light source that works in conjunction with the first high-speed linear array camera. It is configured to acquire macroscopic color and texture images of the surface of the ceramic plate and to detect color difference, stains, and edge chipping defects on the ceramic plate. The second imaging unit consists of a second high-speed linear array camera and a low-angle dark field linear light source that works in conjunction with the second high-speed linear array camera. It is configured to acquire microscopic scattering images of the surface of the ceramic plate for detecting scratches, microcracks, and orange peel defects on the ceramic plate. The third imaging unit consists of a line laser scanning 3D profilometer and a laser that works in conjunction with the line laser scanning 3D profilometer. It is configured to acquire three-dimensional point cloud data of the surface of the ceramic plate in real time by line scanning, in order to reconstruct the high-precision three-dimensional shape of the ceramic plate and detect pits, protrusions, and wave deformation defects. The central data processing module is communicatively connected to the multimodal image acquisition module. It has a built-in deep learning-based multimodal feature fusion defect classification model, which is used to analyze the data acquired by the multimodal image acquisition module and output the porcelain plate recognition result. It fits the information of the porcelain plate with defects to generate defect information and transmits it. The intelligent execution sorting module is communicatively connected to the central data processing module and includes a multi-axis industrial robot and an end effector connected thereto. The multi-axis industrial robot plans a motion trajectory based on the received defect information, and then drives the end effector to locate, grasp and sort the defective ceramic plates.

2. The online detection and sorting system for surface defects of ceramic plates based on machine vision according to claim 1, characterized in that, The multimodal feature fusion defect classification model is configured as follows: Pixel-level registration is performed on the two-dimensional images from the first imaging unit and the second imaging unit, and spatiotemporal synchronization is performed with the three-dimensional point cloud data from the third imaging unit. Depth visual features of 2D images and geometric topological features of 3D point clouds are extracted respectively; By using a pre-defined feature fusion network based on a cross-attention mechanism, the deep visual features and geometric topological features are adaptively weighted and fused to generate a unified defect feature representation rich in multimodal information. Based on the defect feature representation, the existence, category, confidence level of the defect, as well as the precise pixel coordinates and physical size on the detected ceramic plate are output, and then the defect identification result of the ceramic plate is generated by fitting.

3. The online detection and sorting system for surface defects of ceramic plates based on machine vision according to claim 2, characterized in that, The multimodal feature fusion defect classification model adopts a cascaded network architecture, which includes, in sequence: The feature encoder includes a 2D branch and a 3D branch; the 2D branch uses a cropped ConvNeXt network, which simultaneously receives the registered images of the first and second imaging units to perform dual-stream feature extraction and early fusion; the 3D branch uses a point cloud processing network based on PointNet++ to extract multi-scale three-dimensional geometric features. The cross-modal fusion module is based on a cross-attention module. It uses the image feature map output by the 2D branch as the query and the point cloud features output by the 3D branch as the key and value to calculate the correlation between the image region and the 3D point cloud, thereby realizing the guidance and enhancement of 2D image features by 3D geometric information. The defect decoder receives the fused features and employs a fully convolutional network structure to output two prediction heads in parallel: one for pixel-level defect semantic segmentation, generating an accurate mask for defects, and the other for multi-label defect classification based on global features.

4. The online detection and sorting system for surface defects of ceramic plates based on machine vision according to claim 3, characterized in that, It also includes online learning units for models, including: The confidence assessment module, integrated into the defect decoder, is used to calculate the confidence score of each ceramic tile defect identification result. The active learning module automatically marks the sample as a difficult case when the confidence score of any of the defects in the ceramic tile identification results is lower than a preset threshold, and temporarily stores the corresponding image, point cloud data and prediction results in the database to be reviewed. The manual review and annotation module provides a graphical tool for quality inspectors to quickly review and correctly annotate difficult cases; Incremental learning engine: The elastic weight consolidation algorithm, which is manually reviewed, is used periodically to fine-tune the multimodal feature fusion defect classification model to combat overfitting, and the fine-tuned model version is saved.

5. The online detection and sorting system for surface defects of ceramic plates based on machine vision according to claim 4, characterized in that, The elastic weight consolidation algorithm introduces a regularization term when calculating the loss function. This regularization term is based on the Fisher information matrix of the model's old parameters on historical training data. It is used to penalize significant modifications to important parameters of the old task during the learning process of the new task, thereby effectively mitigating catastrophic forgetting.

6. The online detection and sorting system for surface defects of ceramic enamel plates based on machine vision according to claim 1, characterized in that, The high-angle uniform bright field light source of the first imaging unit is an LED planar light source with a color temperature of 6000K±500K and a color rendering index greater than 95, and the illumination angle is 75°-90° with the normal direction of the ceramic plate. The low-angle dark-field linear light source of the second imaging unit is an infrared LED linear light source with a wavelength of 850nm. The illumination angle is 5°-15° with the normal direction of the ceramic plate surface, and the second high-speed linear array camera is equipped with a corresponding infrared bandpass filter.

7. The online detection and sorting system for surface defects of ceramic enamel plates based on machine vision according to claim 1, characterized in that, It also includes a high-precision synchronization triggering unit, which comprises: A rotary encoder is mounted on the drive shaft of the production line conveyor belt to measure the precise displacement of the conveyor belt in real time. The FPGA synchronous control unit is used to receive the pulse signal from the rotary encoder, accurately calculate the trigger delay according to the pre-calibrated relative position of the camera and the ceramic plate, and send a main trigger signal to the first imaging unit, the second imaging unit and the third imaging unit, thereby ensuring that the three-modal images are acquired from the same physical area of ​​the same ceramic plate.

8. The online detection and sorting system for surface defects of ceramic enamel plates based on machine vision according to claim 1, characterized in that, The end effector of the intelligent sorting module is an adaptive vacuum suction cup array, which consists of several independently controlled vacuum suction cup units. Each vacuum suction cup unit consists of an independent vacuum generator and a pressure sensor. The defect location information output by the central data processing module is used to dynamically generate an adsorption strategy and control the suction cup units far from the defect location to work, so as to avoid secondary damage to the defect area.

9. The online detection and sorting system for surface defects of ceramic enamel plates based on machine vision according to claim 1, characterized in that, It also integrates a distributed optical calibration module for initial calibration before system deployment and periodic self-calibration during operation, including: A 2D calibration board is used for the joint calibration of intrinsic parameters, extrinsic parameters, and distortion correction of the first and second imaging units. The 3D calibration block is a stepped standard block with a known precise height on its surface, used to calibrate the height measurement accuracy of the third imaging unit; Hand-eye calibration tool: used to establish precise transformation relationships between the robot coordinate system and all camera coordinate systems.

10. A method for online detection and sorting of surface defects in porcelain enamel plates based on the machine vision-based online detection and sorting system for surface defects in porcelain enamel plates according to any one of claims 1-9, characterized in that, Includes the following steps: Step S1: System initialization, complete the hand-eye calibration and measurement accuracy calibration of the first imaging unit, the second imaging unit and the third imaging unit through the distributed optical calibration module; Step S2: The automatic feeding module feeds the ceramic plates to be inspected to the high-speed conveyor belt at fixed intervals; Step S3: According to the encoder signal, when the ceramic plate enters the acquisition station, the high-precision synchronous triggering unit synchronously triggers the first imaging unit, the second imaging unit and the third imaging unit to acquire the bright field 2D image, dark field 2D image and 3D point cloud data of the ceramic plate respectively. Step S4: The central data processing module receives multimodal data and performs pixel-level registration and spatiotemporal synchronization. Step S5: The multimodal feature fusion defect classification model extracts features from the registered data, performs cross-modal fusion, and performs semantic segmentation and classification of defects. It outputs the porcelain plate recognition results containing defect category, location and confidence level, and fits the porcelain plate information with defects to generate defect information. Step S6: The intelligent sorting module receives defect information, the motion controller plans a collision-free motion trajectory, and drives the adaptive vacuum suction cup array to accurately grasp the defective board. Step S7: According to the preset classification rules, defective boards are sorted to the corresponding scrap bins or downgraded product areas, while qualified products continue to be conveyed along the main water line.

Images