Automatic substrate glass surface defect detection method and system based on machine vision

By using multimodal data and machine vision technology in substrate glass surface detection, combined with graph neural network and multi-task learning network, the problem of insufficient defect positioning and feature fusion in the existing technology is solved, and the defects on substrate glass surface are realized Efficient and accurate detection.

CN119006469AActive Publication Date: 2024-11-22OPTICAL MEASUREMENT IND INTELLIGENT EQUIP (NANJING) CO LTD

Patent Information

Application Number
CN202411490084.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-24
Publication Date
2024-11-22
Estimated Expiration
2044-10-24

AI Technical Summary

Technical Problem

The prior art has shortcomings in defect positioning, multi-scale feature fusion and model optimization in substrate glass defect detection, which is difficult to meet the requirements of high precision.

Method used

Using a machine vision-based method, pre-processing and feature extraction is performed by acquiring multimodal data (image, acoustics, thermal imaging), combining graph neural networks and multi-task learning networks, the topological relationship between pixels is established and cross-modal Feature fusion and defect area segmentation are used to accurately locate the target detection model, and accurately identify it in combination with cloud detection model.

Benefits of technology

It improves the accuracy and robustness of defect detection, and can more accurately detect and locate complex defect forms, achieving efficient and accurate detection of defects on the surface of substrate glass.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006469A_ABST
    Figure CN119006469A_ABST
Patent Text Reader

Abstract

The invention provides an automatic substrate glass surface defect detection method and system based on machine vision, and relates to the technical field of defect detection.The method comprises the steps that multi-modal data of the surface of substrate glass are obtained and preprocessed, cross-modal feature fusion is conducted on extracted image features, acoustic features and thermal imaging features, and a multi-modal feature fusion model is obtained; inputting the fused feature graph into a pre-constructed graph neural network to obtain a topological feature graph representing a defect structure; cascading the topological feature map and the fused feature map, and inputting the cascaded topological feature map and the fused feature map into a multi-task learning network to obtain a segmented suspected defect region and a corresponding defect type; performing defect positioning by using a pre-trained target detection model, and determining an accurate position of a defect through bounding box regression to obtain a target defect region image; and uploading the target defect area image to a cloud end, and accurately identifying the defect type of the target defect area by using a defect detection model deployed in the cloud end to obtain a final defect detection result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to defect detection technology, and in particular to a method and system for automatically detecting defects on the surface of a substrate glass based on machine vision. Background Art

[0002] Substrate glass is an important component of panel display devices, and its surface quality directly affects the display effect. During the production process, various defects such as scratches, particles, stains, etc. are prone to appear on the surface of substrate glass, which will seriously affect the display quality and even cause the entire display device to be scrapped. Traditional manual visual inspection methods are time-consuming and labor-intensive, and it is difficult to ensure the consistency and accuracy of the inspection results. Therefore, an efficient and accurate automated inspection method is urgently needed to meet the requirements of modern production.

[0003] The development of machine vision technology has provided strong support for automatic defect detection. Existing defect detection methods mainly include traditional methods based on image processing and intelligent methods based on deep learning. Traditional methods usually use manually designed image processing algorithms such as image filtering and edge detection to locate and classify defect areas. However, these methods are sensitive to noise and complex backgrounds, and are difficult to adapt to the changing defect morphology.

[0004] In recent years, deep learning technology has made breakthrough progress in the field of image processing and pattern recognition, bringing new opportunities for defect detection tasks. Intelligent methods based on deep learning can automatically learn feature representations from large amounts of data and have strong adaptability to complex defect morphologies. However, existing deep learning methods still have certain deficiencies in defect localization, multi-scale feature fusion, and model optimization, and it is difficult to meet the high-precision requirements of substrate glass defect detection. Therefore, an innovative defect automatic detection method and system is needed that can effectively solve the above problems and achieve efficient and accurate detection of substrate glass surface defects. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for automatically detecting defects on the surface of a substrate glass based on machine vision, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention, A method for automatically detecting defects on a substrate glass surface based on machine vision is provided, comprising: Acquiring and preprocessing multimodal data of the surface of the substrate glass, the multimodal data including image data, acoustic data and thermal imaging data, respectively extracting features from the preprocessed image data, the preprocessed acoustic data and the preprocessed thermal imaging data, and performing cross-modal feature fusion on the extracted image features, acoustic features and thermal imaging features to obtain a fused feature map; The fused feature map is input into a pre-built graph neural network, and the topological relationship between pixels in the fused feature map is established through node embedding and graph convolution operations, and the structure and context information of the defect are extracted to obtain a topological feature map representing the defect structure; the topological feature map is cascaded with the fused feature map, and input into a multi-task learning network, and the multi-task learning network is used to simultaneously complete the segmentation of the defect area and the preliminary classification of the defect type in the cascaded feature map, and obtain the segmented suspected defect area and the corresponding defect type; For the suspected defect area after segmentation, the pre-trained target detection model is used to locate the defect, and the precise position of the defect is determined through bounding box regression to obtain the target defect area image; the target defect area image is uploaded to the cloud, and the defect detection model deployed on the cloud is used to accurately identify the defect type of the target defect area to obtain the final defect detection result.

[0007] In an optional embodiment, The fused feature map is input into the pre-built graph neural network model. Through node embedding and graph convolution operations, the topological relationship between pixels in the fused feature map is established, the structure and context information of the defect are extracted, and the topological feature map representing the defect structure is obtained, including: The fused feature map is divided into multiple local regions, each of which corresponds to a node in the map. The node features are obtained by averaging the pixel features in each local region. The adjacency matrix of the graph is constructed based on the similarity between node features. The elements of the adjacency matrix represent the topological connection relationship between nodes. Construct positive and negative sample pairs, where the positive sample pairs are composed of different pixel features in the same local area, and the negative sample pairs are composed of graph representations of different local areas. By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs, design a graph contrast loss function, and construct a joint loss function based on the obtained graph contrast loss function and graph classification loss function. Based on the joint loss function, update the parameters of the graph neural network model through a gradient optimization algorithm, and finally obtain a trained graph neural network model; Use the trained graph neural network model to update and propagate node features, and learn high-level feature representations of nodes by aggregating node neighborhood information; Graph pooling operation is used to aggregate high-level features of nodes into graph-level feature representation, extract the structure and context information of the defect, and obtain a topological feature map that represents the defect structure.

[0008] In an optional embodiment, The calculation formula for constructing the joint loss function based on the obtained image contrast loss function and image classification loss function is as follows: ; Among them, L represents the joint loss function, B represents the number of graph samples, and C represents the number of categories. y i,k Indicates g i The true label of category k, p(·) represents the predicted probability distribution, g i represents node features, λ represents the balance factor, s(·) represents the similarity, z i represents the i-th node feature in the positive sample pair, Indicates that z i Another node feature of the same local area, τ, represents the temperature parameter, Represents the jth node feature in the negative sample pair.

[0009] In an optional embodiment, For the suspected defect area after segmentation, the pre-trained target detection model is used to locate the defect, and the precise position of the defect is determined by bounding box regression. The target defect area image is obtained, including: Acquire a suspected defect area image, adjust the suspected defect area image to a fixed size, and obtain an adjusted suspected defect area image; The adjusted suspected defect area image is input into the pre-trained target detection model, the multi-scale feature map of the defect area image is extracted through the backbone network, and the multi-scale feature map is fused using the feature pyramid structure to obtain the fused feature map; The spatial attention weight map is introduced and element-wise multiplied with the fused feature map to obtain the feature map enhanced with spatial attention. At the same time, the channel attention weight vector is introduced and channel-wise multiplied with the feature map enhanced with spatial attention to obtain the enhanced target feature map. The enhanced target feature map is input into the detection head for defect target detection. The detection head generates preliminary detection results by predicting the target center point coordinates, bounding box width and height, and target confidence. The preliminary detection results are processed by non-maximum suppression to filter redundant detection frames to obtain filtered detection results, and bounding box regression optimization is performed on the filtered detection results to determine the precise location of the defect and obtain the final target defect area image.

[0010] In an optional embodiment, For the segmented suspected defect area, the steps before using the pre-trained target detection model to locate the defect include: Obtain labeled training samples containing category information and location information of defective areas, and perform data enhancement processing to obtain enhanced training samples; Inputting the enhanced training samples into the target detection model, the target detection model includes a feature extraction network, a region proposal network and a detection head, the feature extraction network uses a deep convolutional neural network to extract multi-scale feature maps of defect areas, the region proposal network generates candidate areas through a region proposal algorithm, and the detection head uses a fully connected layer to classify and regress candidate areas; According to the prediction results and true labels of the target detection model, the difference between the minimum closure area and the union of the predicted box and the true box is calculated. At the same time, the weight factor and modulation factor of the sample difficulty are introduced to dynamically adjust the weights of positive and negative samples and construct the target detection loss function. The stochastic gradient descent algorithm is used to iteratively optimize the parameters of the target detection model by minimizing the target detection loss function, so that the model can learn the feature representation of bounding box regression and difficult-to-classify samples. The iteration is repeated until the preset termination condition is met to obtain a trained target detection model.

[0011] In an optional embodiment, The target defect area image is uploaded to the cloud, and the defect detection model deployed on the cloud is used to accurately identify the defect type of the target defect area. The final defect detection results include: Upload the target defect area image to the cloud server; The cloud server uses the deployed feature extraction model to extract features from the received defect area image to obtain a multi-scale high-level semantic feature vector of the defect image; The extracted multi-scale high-level semantic feature vector is input into the first model based on the attention mechanism, and the multi-scale features are adaptively fused through the attention mechanism to obtain the fused defect feature vector, and the defect type soft label is output at the same time; The second model set is trained by minimizing the difference between the prediction result of the second model set and the soft label of the defect type according to the output soft label of the defect type as a supervisory signal; The fused defect feature vector is input into the trained second model set, and the defect type prediction results output by each second model are combined to obtain the final defect type recognition result.

[0012] In an optional embodiment, The extracted multi-scale high-level semantic feature vector is input into the first model based on the attention mechanism. The multi-scale features are adaptively fused through the attention mechanism to obtain the fused defect feature vector. At the same time, the defect type soft labels are output including: The normalization factor is introduced, and the calculation formula for the defect type soft label is as follows: ; Where T represents the soft label of the defect type, N represents the total number of defect types, and t a represents the probability of the a-th defect type, t d represents the probability of the d-th defect type; According to the output defect type soft label as the supervisory signal, the second model set is trained by minimizing the difference between the prediction result of the second model set and the defect type soft label, including: For each feature channel, the difference between the features extracted by the teacher model and the student model on the feature channel is calculated, and the KL divergence between the predicted probability of the first model and the predicted probability of the second model is introduced to construct the objective function. The calculation formula is as follows: ; Where J represents the objective function, α1 represents the weight of the KL divergence term, Q represents the dimension of the predicted probability distribution, KL(·) represents the KL divergence, and w s represents the predicted probability of the first model for the s-th defect category, q s represents the predicted probability of the second model for the s-th defect category, α2 represents the weight coefficient of feature map alignment, R represents the number of feature channels, and H m Represents the height of the feature map, W m Indicates the feature map width, x l Indicates the sample set l samples, Represents the first model for sample x l The features extracted on the mth channel, Represents the second model for sample x l The features extracted on the mth channel, ||·|| represents the norm.

[0013] According to a second aspect of the embodiments of the present invention, Provided is a substrate glass surface defect automatic detection system based on machine vision, comprising: The first unit is used to obtain and preprocess multimodal data of the surface of the substrate glass, wherein the multimodal data includes image data, acoustic data and thermal imaging data, and to extract features from the preprocessed image data, the preprocessed acoustic data and the preprocessed thermal imaging data, respectively, and to perform cross-modal feature fusion on the extracted image features, acoustic features and thermal imaging features to obtain a fused feature map; The second unit is used to input the fused feature map into a pre-built graph neural network, establish the topological relationship between pixels in the fused feature map through node embedding and graph convolution operations, extract the structure and context information of the defect, and obtain a topological feature map that characterizes the defect structure; cascade the topological feature map with the fused feature map, input them into a multi-task learning network, and use the multi-task learning network to simultaneously complete the segmentation of the defect area in the cascaded feature map and the preliminary classification of the defect type, and obtain the segmented suspected defect area and the corresponding defect type; The third unit is used to locate the defects in the segmented suspected defect area using a pre-trained target detection model, determine the precise position of the defect through bounding box regression, and obtain the target defect area image; upload the target defect area image to the cloud, and use the defect detection model deployed on the cloud to accurately identify the defect type of the target defect area to obtain the final defect detection result.

[0014] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0015] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0016] In this embodiment, by fusing the data of three modalities, image, acoustic and thermal imaging, the defect information on the surface of the substrate glass can be more comprehensively characterized, and the accuracy and robustness of defect detection can be improved. Different modal data can complement the different side features of the defect, which is conducive to improving the performance of defect detection. Using the graph neural network to establish the topological relationship between pixels, the structure and context information of the defect can be effectively extracted, and the overall characteristics of the defect can be captured, rather than just limited to local pixel information. This helps to more accurately detect and locate complex defect morphology. The defect detection task is divided into two stages. This end-cloud collaborative processing mode can balance computing resources and detection accuracy, complete preliminary processing on edge devices, reduce computing pressure on the cloud, and use the powerful computing power of the cloud for more refined defect identification. Through the multi-task learning network, the defect area segmentation and preliminary classification are completed at the same time, and the parameters of the feature extractor can be shared, which promotes each other and improves the generalization ability and detection efficiency of the model. Using the target detection model to accurately locate the defect area, and then combining the defect detection model in the cloud for accurate identification, more accurate defect type and location information can be obtained, providing a reliable basis for subsequent defect processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic flow chart of a method for automatically detecting defects on the surface of a glass substrate based on machine vision according to an embodiment of the present invention; Figure 2 It is a schematic structural diagram of a system for automatically detecting defects on a substrate glass surface based on machine vision according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0020] Figure 1 FIG. 1 is a flow chart of a method for automatically detecting defects on a substrate glass surface based on machine vision according to an embodiment of the present invention. Figure 1 As shown, the method includes: S101. Acquire multimodal data of the surface of the substrate glass and perform preprocessing, wherein the multimodal data includes image data, acoustic data, and thermal imaging data, perform feature extraction on the preprocessed image data, the preprocessed acoustic data, and the preprocessed thermal imaging data, respectively, perform cross-modal feature fusion on the extracted image features, acoustic features, and thermal imaging features, and obtain a fused feature map.

[0021] Exemplarily, multimodal data of the glass surface of the substrate is first obtained, including image data, acoustic data and thermal imaging data. Among them, the image data is collected by a high-resolution industrial camera, the acoustic data is collected by an ultrasonic sensor, and the thermal imaging data is collected by an infrared thermal imager. For image data, the preprocessing steps include: image denoising, image enhancement, image correction, etc. Among them, image denoising adopts a method combining wavelet transform and adaptive median filtering, which can effectively suppress Gaussian noise and salt and pepper noise; image enhancement adopts an adaptive histogram equalization algorithm to improve the contrast and clarity of the image; image correction adopts a perspective transformation method based on corner detection to eliminate the geometric distortion of the image. For acoustic data, the preprocessing steps include: signal filtering, signal enhancement, feature extraction, etc. Among them, signal filtering adopts a wavelet packet decomposition and reconstruction method, which can effectively remove high-frequency noise in the acoustic signal; signal enhancement adopts an envelope extraction method based on Hilbert transform to highlight the amplitude characteristics of the acoustic signal; feature extraction adopts a short-time Fourier transform (STFT) method to obtain the time-frequency characteristics of the acoustic signal. For thermal imaging data, the preprocessing steps include: thermal image denoising, thermal image enhancement, thermal image segmentation, etc. Among them, thermal image denoising uses an adaptive guided filtering algorithm to effectively suppress high-frequency noise in thermal imaging data; thermal image enhancement uses an adaptive plateaus histogram equalization method to improve the contrast and clarity of the thermal image; thermal image segmentation uses a binarization method based on Otsu threshold to extract high-temperature areas in the thermal image.

[0022] Feature extraction is performed on the preprocessed image data, acoustic data, and thermal imaging data. Image feature extraction uses a multi-scale residual convolutional neural network to extract multi-scale texture and structural features of the image by designing receptive fields and residual connections of different scales; acoustic feature extraction uses a long short-term memory network based on an attention mechanism to adaptively extract the temporal features of the acoustic signal by introducing an attention mechanism between the encoder and decoder of the LSTM; thermal imaging feature extraction uses a heat map pyramid pooling convolutional neural network to extract multi-scale regional features of the heat map through spatial pyramid pooling operations.

[0023] The extracted image features, acoustic features and thermal imaging features are cross-modal fused. First, the image features, acoustic features and thermal imaging features are aligned to the same scale and dimension through feature mapping; then, the self-attention fusion module is used to adaptively weight the aligned features, and the fusion weight matrix is ​​generated by calculating the correlation between different modal features to achieve cross-modal interaction and fusion of features; finally, the fused features are compressed and refined through convolutional layers and pooling layers to obtain a compact fusion feature map.

[0024] In this embodiment, suitable preprocessing methods such as denoising, enhancement, and correction are used for data of three different modalities, namely, image, acoustic, and thermal imaging, respectively, to improve the quality and feature separability of each modality data. A self-attention fusion module is used to adaptively generate a fusion weight matrix based on the correlation between different modal features, thereby realizing the interaction and fusion of features and making full use of the complementary advantages of multimodal data. Through technical means such as multimodal data preprocessing, deep learning feature extraction, cross-modal feature fusion, and end-cloud collaborative processing, the rich information of image, acoustic, and thermal imaging data can be fully mined and utilized, the accuracy, robustness, and efficiency of defect detection can be improved, and high-precision detection and classification of defects on the surface of substrate glass can be achieved.

[0025] S102. Input the fused feature map into a pre-built graph neural network, establish the topological relationship between pixels in the fused feature map through node embedding and graph convolution operations, extract the structure and context information of the defect, and obtain a topological feature map that characterizes the defect structure; cascade the topological feature map with the fused feature map, input them into a multi-task learning network, and use the multi-task learning network to simultaneously complete the segmentation of the defect area in the cascaded feature map and the preliminary classification of the defect type, and obtain the segmented suspected defect area and the corresponding defect type.

[0026] Among them, the multi-task learning network adopts an encoder-decoder architecture. The encoder part consists of several convolutional layers and pooling layers, which are used to extract high-level semantic features of the cascade feature map. The convolution layer uses a 3×3 convolution kernel to extract local features and expand the receptive field by performing convolution operations on the feature map. The pooling layer uses a 2×2 maximum pooling to reduce the spatial resolution of the feature map by downsampling the feature map, while increasing the receptive field. After multiple convolutions and pooling, the spatial resolution of the feature map continues to decrease, but the feature dimension continues to increase, and high-level semantic features with strong discriminative power are obtained. The decoder part consists of several upsampling layers and deconvolution layers, which are used to restore the spatial resolution of the feature map. The upsampling layer uses the nearest neighbor interpolation method to double the spatial resolution of the feature map. The deconvolution layer uses a 3×3 convolution kernel to extract local features and refine the feature representation by performing convolution operations on the upsampled feature map. After multiple upsampling and deconvolution, the spatial resolution of the feature map is gradually restored to the same as the input image, and a segmentation result with the same size as the input image is obtained.

[0027] In order to achieve multi-task learning, two parallel fully connected layers are introduced in the last layer of the encoder, corresponding to the defect area segmentation task and the defect type classification task respectively. The fully connected layer of the segmentation task maps high-level semantic features to pixel-level segmentation results and predicts the probability of each pixel belonging to the defect area. The fully connected layer of the classification task maps high-level semantic features to category-level classification results and predicts the probability of the input image belonging to each defect type.

[0028] The gradient average strategy is used for multi-task learning. By jointly optimizing the segmentation task and the classification task, end-to-end defect segmentation and classification is achieved. During the training process, the cascade feature map and the corresponding segmentation label and category label are input into the multi-task learning network, the loss function of the segmentation task and the classification task is calculated, and the network parameters are updated by gradient average. After multiple rounds of iterative training, a multi-task learning network with the ability to segment defect areas and classify defect types is obtained.

[0029] During the inference process, the cascade feature map of the substrate glass surface image to be tested is input into the trained multi-task learning network, and the segmentation result and classification result are obtained through forward propagation. The segmentation result is a probability map with the same size as the input image, indicating the probability that each pixel belongs to the defect area. The probability map is binarized to obtain the suspected defect area after segmentation. The classification result is a multi-dimensional probability vector, indicating the probability that the input image belongs to each defect type. The category with the highest probability is taken as the final defect type. Through the inference of the multi-task learning network, the suspected defect area and the corresponding defect type on the substrate glass surface can be obtained at the same time, realizing the automatic segmentation and identification of the defect area. Compared with the separate training of the segmentation network and the classification network, the multi-task learning network can make full use of the correlation between different tasks, realize the mutual promotion between tasks through joint optimization, and improve the accuracy and efficiency of defect segmentation and classification.

[0030] In an optional embodiment, The fused feature map is input into the pre-built graph neural network model. Through node embedding and graph convolution operations, the topological relationship between pixels in the fused feature map is established, the structure and context information of the defect are extracted, and the topological feature map representing the defect structure is obtained, including: The fused feature map is divided into multiple local regions, each of which corresponds to a node in the map. The node features are obtained by averaging the pixel features in each local region. The adjacency matrix of the graph is constructed based on the similarity between node features. The elements of the adjacency matrix represent the topological connection relationship between nodes. Construct positive and negative sample pairs, where the positive sample pairs are composed of different pixel features in the same local area, and the negative sample pairs are composed of graph representations of different local areas. By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs, design a graph contrast loss function, and construct a joint loss function based on the obtained graph contrast loss function and graph classification loss function. Based on the joint loss function, update the parameters of the graph neural network model through a gradient optimization algorithm, and finally obtain a trained graph neural network model; Use the trained graph neural network model to update and propagate node features, and learn high-level feature representations of nodes by aggregating node neighborhood information; Graph pooling operation is used to aggregate high-level features of nodes into graph-level feature representation, extract the structure and context information of the defect, and obtain a topological feature map that represents the defect structure.

[0031] Exemplarily, the fused feature map is divided into multiple local regions, each of which corresponds to a node in the graph. For each local region, the node feature vector is obtained by performing an average pooling operation on the pixel features therein. Based on the similarity between the node features, the adjacency matrix of the graph is constructed. The similarity between the node features can be calculated using a similarity metric (such as Euclidean distance, cosine similarity, etc.), and the similarity is converted into an element of the adjacency matrix to represent the topological connection relationship between the nodes.

[0032] Construct positive and negative sample pairs. The positive sample pair consists of different image representations of the same defect area, and the negative sample pair consists of image representations of different defect areas. Specifically, for a given defect area, multiple different image representations are generated through data enhancement techniques (such as rotation, flipping, cropping, etc.) as positive sample pairs. For different defect areas, their image representations are randomly selected as negative sample pairs.

[0033] Based on the obtained graph contrast loss function and graph classification loss function, a joint loss function is constructed. Based on the joint loss function, a gradient optimization algorithm (such as stochastic gradient descent) is used to update the parameters of the graph neural network model to minimize the joint loss function. Through the iterative training process, the model parameters are continuously adjusted to improve the performance of the model on graph classification tasks and graph contrast tasks. The trained graph neural network model is used to update and propagate node features. By aggregating the neighborhood information of the node, the high-level feature representation of the node is learned. Graph neural network structures such as graph convolutional networks and graph attention networks can be used to achieve the update and propagation of node features. The high-level features of the node are aggregated into a graph-level feature representation using graph pooling operations. The graph pooling operation can use methods such as graph convolution and graph attention to aggregate node features into a graph-level feature vector. In this way, the structure and context information of the defect can be extracted to obtain a topological feature map representing the defect structure.

[0034] In an optional embodiment, The calculation formula for constructing the joint loss function based on the obtained image contrast loss function and image classification loss function is as follows: ; Among them, L represents the joint loss function, B represents the number of graph samples, and C represents the number of categories. y i,k Indicates g i The true label of category k, p(·) represents the predicted probability distribution, g i represents node features, λ represents the balance factor, s(·) represents the similarity, z i represents the i-th node feature in the positive sample pair, Indicates that z i Another node feature of the same local area, τ, represents the temperature parameter, Represents the jth node feature in the negative sample pair.

[0035] In this embodiment, the representation method based on the graph structure can effectively capture the spatial proximity and structural association between pixels in the image, surpassing the limitation of traditional convolutional neural networks that only consider local neighborhoods. The feature learning method based on the graph structure helps to extract the structure and context information of the defects and capture the topological features of the defects in the overall image, rather than being limited to the features at the local pixel level. The topological feature map contains not only the local texture and edge information of the defects, but also the structural features and context information of the defects in the overall image, which helps to more accurately detect and classify complex defect morphologies. The optimization strategy based on contrastive learning enables the model to learn more discriminative feature representations, improve the accuracy and performance of defect detection, and thus achieve efficient detection and classification of defects on the surface of substrate glass.

[0036] S103. For the suspected defect area after segmentation, the defect is located using a pre-trained target detection model, and the precise position of the defect is determined by bounding box regression to obtain an image of the target defect area; the image of the target defect area is uploaded to the cloud, and the defect detection model deployed on the cloud is used to accurately identify the defect type of the target defect area to obtain the final defect detection result.

[0037] Among them, the target detection model adopts a two-stage detection framework, such as Faster R-CNN or Mask R-CNN, generates candidate regions through the region proposal network (RPN), and classifies and regresses the candidate regions. It mainly consists of three parts: the backbone network, the neck network, and the detection head. The backbone network extracts multi-scale features through residual connections and CSP (Cross Stage Partial) modules; the neck network adopts the PANet structure, and realizes the fusion of features of different scales through feature pyramids and adaptive feature fusion; the detection head adopts an anchor-free design to directly predict the center point and bounding box of the target.

[0038] In an optional embodiment, For the suspected defect area after segmentation, the pre-trained target detection model is used to locate the defect, and the precise position of the defect is determined by bounding box regression. The target defect area image is obtained, including: Acquire a suspected defect area image, adjust the suspected defect area image to a fixed size, and obtain an adjusted suspected defect area image; The adjusted suspected defect area image is input into the pre-trained target detection model, the multi-scale feature map of the defect area image is extracted through the backbone network, and the multi-scale feature map is fused using the feature pyramid structure to obtain the fused feature map; The spatial attention weight map is introduced and element-wise multiplied with the fused feature map to obtain the feature map enhanced with spatial attention. At the same time, the channel attention weight vector is introduced and channel-wise multiplied with the feature map enhanced with spatial attention to obtain the enhanced target feature map. The enhanced target feature map is input into the detection head for defect target detection. The detection head generates preliminary detection results by predicting the target center point coordinates, bounding box width and height, and target confidence. The preliminary detection results are processed by non-maximum suppression to filter redundant detection frames to obtain filtered detection results, and bounding box regression optimization is performed on the filtered detection results to determine the precise location of the defect and obtain the final target defect area image.

[0039] For example, first, an image of a suspected defect region is obtained and adjusted to a fixed size to meet the input requirements of the target detection model. Operations such as scaling and cropping are usually used to obtain the adjusted image of the suspected defect region.

[0040] The adjusted suspected defect area image is input into the pre-trained target detection model. The target detection model usually consists of a backbone network and a detection head. The backbone network uses a deep convolutional neural network, such as ResNet, VGG, etc., to extract multi-scale feature maps of defect area images. In order to fuse feature information of different scales, a feature pyramid structure (FPN) is introduced. Through a top-down path and lateral connections, high-level semantic features are fused with low-level detail features to obtain a fused feature map.

[0041] In order to further enhance the representation ability of defect areas, the attention mechanism is introduced. First, by generating a spatial attention weight map, the spatial attention weight map is element-wise multiplied with the fused feature map to obtain the feature map enhanced by spatial attention. The spatial attention weight map can be obtained through the convolution layer and the Sigmoid activation function to highlight the spatial position information of the defect area. At the same time, the channel attention weight vector is introduced and channel-wise multiplied with the feature map enhanced by spatial attention to obtain the enhanced target feature map. The channel attention weight vector can be obtained through global average pooling, fully connected layer and Sigmoid activation function to adaptively adjust the importance of different channels. The enhanced target feature map is input into the detection head for defect target detection. The detection head generates preliminary detection results by predicting the target center point coordinates, bounding box width and height, and target confidence. Commonly used detection heads include SSD, YOLO, Faster R-CNN, etc. The corresponding detection head is selected according to the specific model. The preliminary detection results are processed by non-maximum suppression. The non-maximum suppression algorithm (such as IOU threshold, confidence threshold) can be used to filter redundant detection boxes to obtain filtered detection results. The bounding box regression optimization is performed on the filtered detection results. Linear regression, smooth L1 loss and other methods can be used to adjust the bounding box position of the preliminary detection results according to the bounding box offset and scaling factor predicted by the model to obtain a more accurate defect location.

[0042] According to the optimized bounding box position, the defect area image is extracted from the original image, and an image processing library (such as OpenCV) can be used to perform image cropping operations to obtain the final target defect area image.

[0043] In this embodiment, by fusing multi-scale features through the feature pyramid structure, the different scale features of the defect area can be effectively captured. By predicting the target center point coordinates, the width and height of the bounding box and the target confidence, the preliminary detection results are generated, and the non-maximum suppression processing and bounding box regression optimization are performed, so that the position and range of the defect can be accurately located to obtain the final target defect area image. The attention mechanism can adaptively enhance the attention to the defect area and feature channel, improve the sensitivity of the model to the defect features, and thus improve the accuracy of defect location. The defect location task is deployed on the end side, and the defect location is performed using the pre-trained target detection model, which can complete the preliminary processing on the edge device and reduce the computing pressure on the cloud. Only after the target defect area is determined, the target defect area image is uploaded to the cloud for further accurate identification. This end-cloud collaborative processing mode can improve the efficiency and performance of the overall system. By extracting multi-scale feature maps through the backbone network and fusing multi-scale feature maps using the feature pyramid structure, the defect features at different scales can be effectively captured and fused, which improves the model's ability to detect defects.

[0044] In an optional embodiment, For the segmented suspected defect area, the steps before using the pre-trained target detection model to locate the defect include: Obtain labeled training samples containing category information and location information of defective areas, and perform data enhancement processing to obtain enhanced training samples; Inputting the enhanced training samples into the target detection model, the target detection model includes a feature extraction network, a region proposal network and a detection head, the feature extraction network uses a deep convolutional neural network to extract multi-scale feature maps of defect areas, the region proposal network generates candidate areas through a region proposal algorithm, and the detection head uses a fully connected layer to classify and regress candidate areas; According to the prediction results and true labels of the target detection model, the difference between the minimum closure area and the union of the predicted box and the true box is calculated. At the same time, the weight factor and modulation factor of the sample difficulty are introduced to dynamically adjust the weights of positive and negative samples and construct the target detection loss function. The stochastic gradient descent algorithm is used to iteratively optimize the parameters of the target detection model by minimizing the target detection loss function, so that the model can learn the feature representation of bounding box regression and difficult-to-classify samples. The iteration is repeated until the preset termination condition is met to obtain a trained target detection model.

[0045] Exemplarily, first obtain labeled training samples containing category information and location information of defective areas, perform data enhancement processing on the training samples, including random flipping, random cropping, random scaling and other operations, to obtain enhanced training samples. The enhanced training samples are input into the target detection model for training. The target detection model includes a feature extraction network, a region proposal network and a detection head. The feature extraction network uses a deep convolutional neural network to extract multi-scale feature maps of defective areas. The region proposal network generates candidate areas through a region proposal algorithm. The detection head uses a fully connected layer to classify and regress candidate areas. According to the prediction results of the target detection model and the true label, the difference between the minimum closure area and the union of the predicted box and the true box is calculated to construct the target detection loss function. At the same time, the weight factor and modulation factor of the sample difficulty are introduced to dynamically adjust the weights of positive and negative samples to strengthen the learning of difficult-to-classify samples.

[0046] Using the stochastic gradient descent algorithm, the parameters of the target detection model are iteratively optimized by minimizing the target detection loss function, so that the model can learn the feature representation of bounding box regression and difficult-to-classify samples. Repeat the iteration until the preset termination condition is met to obtain a trained target detection model.

[0047] In this embodiment, by performing data enhancement processing on the labeled training samples, such as flipping, rotating, scaling and other operations, the training data set can be effectively expanded to improve the diversity and richness of the data. The multi-scale feature extraction strategy helps the model detect and locate defects of different sizes and shapes, and improves the generalization ability of the model. The introduction of the region proposal network and the generation of candidate regions through the region proposal algorithm can effectively reduce the number of regions that need to be processed and improve computational efficiency. The dynamic sample weight adjustment strategy can make the model pay more attention to difficult-to-classify samples, improve the learning ability of bounding box regression and difficult-to-classify samples, and thus improve the accuracy of defect detection. Through technical means such as data enhancement, multi-scale feature extraction, region proposal algorithm, dynamic sample weight adjustment and end-cloud collaborative processing, a high-performance target detection model can be trained to achieve accurate positioning of defects on the surface of substrate glass. In an optional embodiment, The target defect area image is uploaded to the cloud, and the defect detection model deployed on the cloud is used to accurately identify the defect type of the target defect area. The final defect detection results include: Upload the target defect area image to the cloud server; The cloud server uses the deployed feature extraction model to extract features from the received defect area image to obtain a multi-scale high-level semantic feature vector of the defect image; The extracted multi-scale high-level semantic feature vector is input into the first model based on the attention mechanism, and the multi-scale features are adaptively fused through the attention mechanism to obtain the fused defect feature vector, and the defect type soft label is output at the same time; The second model set is trained by minimizing the difference between the prediction result of the second model set and the soft label of the defect type according to the output soft label of the defect type as a supervisory signal; The fused defect feature vector is input into the trained second model set, and the defect type prediction results output by each second model are combined to obtain the final defect type recognition result.

[0048] Exemplarily, an image of an object to be detected is collected by a terminal device, and the collected image of the object to be detected is transmitted to an edge server; the edge server uses a deployed multi-task learning network to segment the defect area of ​​the received image of the object to be detected to obtain a segmented defect area image, and uses a deployed target detection model to accurately locate the segmented defect area image to obtain a located defect area image; the located defect area image obtained in the edge server is uploaded to a cloud server.

[0049] The cloud server uses the deployed feature extraction model to extract features from the received defect area image. The feature extraction model usually uses a pre-trained deep convolutional neural network, such as EfficientNet, ResNet, etc., to extract multi-scale high-level semantic feature vectors of the defect image. By outputting feature maps at different convolutional layers, feature representations of different scales are obtained to capture the multi-scale semantic information of the defect area.

[0050] The extracted multi-scale high-level semantic feature vector is input into the first model based on the attention mechanism. The first model adaptively fuses multi-scale features through the attention mechanism to obtain a fused defect feature vector. Specifically, by generating an attention weight vector and performing a weighted summation with the multi-scale feature vector, the importance of features at different scales is highlighted to obtain a fused defect feature vector. At the same time, the first model also outputs a soft label of the defect type as a supervisory signal for subsequent model training.

[0051] The second model set is trained according to the defect type soft label output by the first model. The second model set consists of multiple independent defect type recognition models, each corresponding to a defect type. By minimizing the difference between the prediction results of the second model set and the defect type soft label, the second model set is trained using the cross entropy loss function or the KL divergence loss function to learn the discriminative features of the defect type.

[0052] The fused defect feature vector is input into the trained second model set to identify the defect type. Each second model independently classifies the defect feature vector and outputs the predicted probability of the corresponding defect type. By combining the prediction results of each second model, a voting mechanism or weighted average method is used to obtain the final defect type identification result.

[0053] In an optional embodiment, The extracted multi-scale high-level semantic feature vector is input into the first model based on the attention mechanism. The multi-scale features are adaptively fused through the attention mechanism to obtain the fused defect feature vector. At the same time, the defect type soft labels are output including: The normalization factor is introduced, and the calculation formula for the defect type soft label is as follows: ; Where T represents the soft label of the defect type, N represents the total number of defect types, and t a represents the probability of the a-th defect type, t d represents the probability of the d-th defect type; According to the output defect type soft label as the supervisory signal, the second model set is trained by minimizing the difference between the prediction result of the second model set and the defect type soft label, including: For each feature channel, the difference between the features extracted by the teacher model and the student model on the feature channel is calculated, and the KL divergence between the predicted probability of the first model and the predicted probability of the second model is introduced to construct the objective function. The calculation formula is as follows: ; Where J represents the objective function, α1 represents the weight of the KL divergence term, Q represents the dimension of the predicted probability distribution, KL(·) represents the KL divergence, and w s represents the predicted probability of the first model for the s-th defect category, q s represents the predicted probability of the second model for the s-th defect category, α2 represents the weight coefficient of feature map alignment, R represents the number of feature channels, and H m Represents the height of the feature map, W m Indicates the feature map width, x l Indicates the sample set l samples, Represents the first model for sample x l The features extracted on the mth channel, Represents the second model for sample x l The features extracted on the mth channel, ||·|| represents the norm.

[0054] In this embodiment, by uploading the target defect area image to the cloud server, using the powerful computing power and storage resources of the cloud, a more complex and high-performance defect detection model can be deployed to achieve accurate identification of defect types. The cloud server uses the deployed feature extraction model to extract multi-scale high-level semantic feature vectors from the defect area image, capturing the texture, edge and structural information of the defect at different scales. The first model based on the attention mechanism can adaptively focus on the important parts of the features of different scales, improve the sensitivity to the defect features, and thus improve the accuracy of defect type identification. At the same time, the first model can also output the defect type soft label to provide an effective supervision signal for subsequent model training. The fused defect feature vector is input into the trained second model set, and combined with the output results of each model, model integration can be achieved to improve the robustness and generalization ability of defect type identification. The end-cloud collaborative processing mode is adopted to complete the preliminary defect location on the edge device, and the target defect area image is uploaded to the cloud for fine defect type identification. By leveraging the powerful computing power of the cloud, multi-scale feature extraction and fusion, attention mechanism enhanced feature representation, model integration and knowledge distillation, and end-cloud collaborative processing, we can achieve high-precision type identification of surface defects on substrate glass, providing key support for subsequent defect repair and quality control.

[0055] Figure 2FIG. 1 is a schematic diagram of the structure of a system for automatically detecting defects on a substrate glass surface based on machine vision according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to obtain and preprocess multimodal data of the surface of the substrate glass, wherein the multimodal data includes image data, acoustic data and thermal imaging data, and to extract features from the preprocessed image data, the preprocessed acoustic data and the preprocessed thermal imaging data, respectively, and to perform cross-modal feature fusion on the extracted image features, acoustic features and thermal imaging features to obtain a fused feature map; The second unit is used to input the fused feature map into a pre-built graph neural network, establish the topological relationship between pixels in the fused feature map through node embedding and graph convolution operations, extract the structure and context information of the defect, and obtain a topological feature map that characterizes the defect structure; cascade the topological feature map with the fused feature map, input them into a multi-task learning network, and use the multi-task learning network to simultaneously complete the segmentation of the defect area in the cascaded feature map and the preliminary classification of the defect type, and obtain the segmented suspected defect area and the corresponding defect type; The third unit is used to locate the defects in the segmented suspected defect area using a pre-trained target detection model, determine the precise position of the defect through bounding box regression, and obtain the target defect area image; upload the target defect area image to the cloud, and use the defect detection model deployed on the cloud to accurately identify the defect type of the target defect area to obtain the final defect detection result.

[0056] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0057] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0058] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically detecting defects on the surface of a substrate glass based on machine vision, characterized in that: include: Acquiring and preprocessing multimodal data of the surface of the substrate glass, the multimodal data including image data, acoustic data and thermal imaging data, respectively extracting features from the preprocessed image data, the preprocessed acoustic data and the preprocessed thermal imaging data, and performing cross-modal feature fusion on the extracted image features, acoustic features and thermal imaging features to obtain a fused feature map; The fused feature map is input into a pre-built graph neural network, and the topological relationship between pixels in the fused feature map is established through node embedding and graph convolution operations, and the structure and context information of the defect are extracted to obtain a topological feature map representing the defect structure; the topological feature map is cascaded with the fused feature map, and input into a multi-task learning network, and the multi-task learning network is used to simultaneously complete the segmentation of the defect area and the preliminary classification of the defect type in the cascaded feature map, and obtain the segmented suspected defect area and the corresponding defect type; For the suspected defect area after segmentation, the pre-trained target detection model is used to locate the defect, and the precise position of the defect is determined by bounding box regression to obtain the target defect area image; The target defect area image is uploaded to the cloud, and the defect detection model deployed on the cloud is used to accurately identify the defect type of the target defect area to obtain the final defect detection result.

2. The method according to claim 1, characterized in that The fused feature map is input into the pre-built graph neural network model. Through node embedding and graph convolution operations, the topological relationship between pixels in the fused feature map is established, the structure and context information of the defect are extracted, and the topological feature map representing the defect structure is obtained, including: The fused feature map is divided into multiple local regions, each of which corresponds to a node in the map. The node features are obtained by averaging the pixel features in each local region. The adjacency matrix of the graph is constructed based on the similarity between node features. The elements of the adjacency matrix represent the topological connection relationship between nodes. Construct positive and negative sample pairs, where the positive sample pairs are composed of different pixel features in the same local area, and the negative sample pairs are composed of graph representations of different local areas. By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs, design a graph contrast loss function, and construct a joint loss function based on the obtained graph contrast loss function and graph classification loss function. Based on the joint loss function, update the parameters of the graph neural network model through a gradient optimization algorithm, and finally obtain a trained graph neural network model; Use the trained graph neural network model to update and propagate node features, and learn high-level feature representations of nodes by aggregating node neighborhood information; Graph pooling operation is used to aggregate high-level features of nodes into graph-level feature representation, extract the structure and context information of the defect, and obtain a topological feature map that represents the defect structure.

3. The method according to claim 2, characterized in that The calculation formula for constructing the joint loss function based on the obtained image contrast loss function and image classification loss function is as follows: ; Among them, L represents the joint loss function, B represents the number of graph samples, and C represents the number of categories. y i,k Indicates g i The true label of category k, p(·) represents the predicted probability distribution, g i represents node features, λ represents the balance factor, s(·) represents the similarity, z i represents the i-th node feature in the positive sample pair, Indicates that z i Another node feature of the same local area, τ, represents the temperature parameter, Represents the jth node feature in the negative sample pair.

4. The method according to claim 1, characterized in that: For the suspected defect area after segmentation, the pre-trained target detection model is used to locate the defect, and the precise position of the defect is determined by bounding box regression. The target defect area image is obtained, including: Acquire a suspected defect area image, adjust the suspected defect area image to a fixed size, and obtain an adjusted suspected defect area image; The adjusted suspected defect area image is input into the pre-trained target detection model, the multi-scale feature map of the defect area image is extracted through the backbone network, and the multi-scale feature map is fused using the feature pyramid structure to obtain the fused feature map; The spatial attention weight map is introduced and element-wise multiplied with the fused feature map to obtain the feature map enhanced with spatial attention. At the same time, the channel attention weight vector is introduced and channel-wise multiplied with the feature map enhanced with spatial attention to obtain the enhanced target feature map. The enhanced target feature map is input into the detection head for defect target detection. The detection head generates preliminary detection results by predicting the target center point coordinates, bounding box width and height, and target confidence. The preliminary detection results are processed by non-maximum suppression to filter redundant detection frames to obtain filtered detection results, and bounding box regression optimization is performed on the filtered detection results to determine the precise location of the defect and obtain the final target defect area image.

5. The method according to claim 1, characterized in that For the segmented suspected defect area, the steps before using the pre-trained target detection model to locate the defect include: Obtain labeled training samples containing category information and location information of defective areas, and perform data enhancement processing to obtain enhanced training samples; Inputting the enhanced training samples into the target detection model, the target detection model includes a feature extraction network, a region proposal network and a detection head, the feature extraction network uses a deep convolutional neural network to extract multi-scale feature maps of defect areas, the region proposal network generates candidate areas through a region proposal algorithm, and the detection head uses a fully connected layer to classify and regress candidate areas; According to the prediction results and true labels of the target detection model, the difference between the minimum closure area and the union of the predicted box and the true box is calculated. At the same time, the weight factor and modulation factor of the sample difficulty are introduced to dynamically adjust the weights of positive and negative samples and construct the target detection loss function. The stochastic gradient descent algorithm is used to iteratively optimize the parameters of the target detection model by minimizing the target detection loss function, so that the model can learn the feature representation of bounding box regression and difficult-to-classify samples. The iteration is repeated until the preset termination condition is met to obtain a trained target detection model.

6. The method according to claim 1, characterized in that The target defect area image is uploaded to the cloud, and the defect detection model deployed on the cloud is used to accurately identify the defect type of the target defect area. The final defect detection results include: Upload the target defect area image to the cloud server; The cloud server uses the deployed feature extraction model to extract features from the received defect area image to obtain a multi-scale high-level semantic feature vector of the defect image; The extracted multi-scale high-level semantic feature vector is input into the first model based on the attention mechanism, and the multi-scale features are adaptively fused through the attention mechanism to obtain the fused defect feature vector, and the defect type soft label is output at the same time; The second model set is trained by minimizing the difference between the prediction result of the second model set and the soft label of the defect type according to the output soft label of the defect type as a supervisory signal; The fused defect feature vector is input into the trained second model set, and the defect type prediction results output by each second model are combined to obtain the final defect type recognition result.

7. The method according to claim 6, characterized in that The extracted multi-scale high-level semantic feature vector is input into the first model based on the attention mechanism. The multi-scale features are adaptively fused through the attention mechanism to obtain the fused defect feature vector. At the same time, the defect type soft labels are output including: The normalization factor is introduced, and the calculation formula for the defect type soft label is as follows: ; Where T represents the soft label of the defect type, N represents the total number of defect types, and t a represents the probability of the a-th defect type, t d represents the probability of the d-th defect type; According to the output defect type soft label as the supervisory signal, the second model set is trained by minimizing the difference between the prediction result of the second model set and the defect type soft label, including: For each feature channel, the difference between the features extracted by the teacher model and the student model on the feature channel is calculated, and the KL divergence between the predicted probability of the first model and the predicted probability of the second model is introduced to construct the objective function. The calculation formula is as follows: ; Where J represents the objective function, α1 represents the weight of the KL divergence term, Q represents the dimension of the predicted probability distribution, KL(·) represents the KL divergence, and w s represents the predicted probability of the first model for the s-th defect category, q s represents the predicted probability of the second model for the s-th defect category, α2 represents the weight coefficient of feature map alignment, R represents the number of feature channels, and H m Represents the height of the feature map, W m Indicates the feature map width, x l Indicates the sample set l samples, Represents the first model for sample x l The features extracted on the mth channel, Represents the second model for sample x l The features extracted on the mth channel, ||·|| represents the norm.

8. A system for automatically detecting defects on the surface of a substrate glass based on machine vision, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain and preprocess multimodal data of the surface of the substrate glass, wherein the multimodal data includes image data, acoustic data and thermal imaging data, and to extract features from the preprocessed image data, the preprocessed acoustic data and the preprocessed thermal imaging data, respectively, and to perform cross-modal feature fusion on the extracted image features, acoustic features and thermal imaging features to obtain a fused feature map; The second unit is used to input the fused feature map into a pre-built graph neural network, establish the topological relationship between pixels in the fused feature map through node embedding and graph convolution operations, extract the structure and context information of the defect, and obtain a topological feature map that characterizes the defect structure; cascade the topological feature map with the fused feature map, input them into a multi-task learning network, and use the multi-task learning network to simultaneously complete the segmentation of the defect area in the cascaded feature map and the preliminary classification of the defect type, and obtain the segmented suspected defect area and the corresponding defect type; The third unit is used to locate the defect in the segmented suspected defect area using a pre-trained target detection model, determine the exact position of the defect through bounding box regression, and obtain the target defect area image; The target defect area image is uploaded to the cloud, and the defect detection model deployed on the cloud is used to accurately identify the defect type of the target defect area to obtain the final defect detection result.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Surface defect detection method based on multi-scale information fusion

    CN113610822A

  • Glass panel surface defect detection method based on small sample learning

    CN114092389A

  • Glass defect detection method and device based on deformable convolution and feature fusion

    CN114170230A

  • Glass defect detection method and system based on multi-modal feature fusion

    CN116664586A

  • Method for feature detection of complex defects based on multimodal data

    US20230316736A1

Cited By

  • Method and system for automatically detecting surface defects of flexible touch screen

    CN119205777A

  • A method and system for automatically detecting surface defects of a flexible touch screen

    CN119205777B

  • Seamless steel tube defect detection method and device based on machine vision and storage medium

    CN119246811A

  • Product detection method and system based on machine vision

    CN119323659A

  • A product detection method and system based on machine vision

    CN119323659B