A glass defect monitoring method based on multi-sensor data fusion and classification

Abstract

The application discloses a glass defect monitoring method based on multi-sensor data fusion and classification, comprising the following steps: S1, collecting the detection signals of glass through two or more sensors, and performing deep learning fusion on the detection signals collected by the two or more sensors; S2, judging whether there is a defect according to the fused detection signals through a trained neural network, and determining the type and position of the defect type when judging that there is a defect. Deep learning fusion is combined with the detection information of multiple sensors to detect and position the glass defects, which can provide multi-dimensional information about the defect position, type and the like, has strong adaptability, good stability and reliability in different environments and application scenarios, and can provide more accurate and complete glass defect detection and positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of glass monitoring, and particularly relates to a glass defect monitoring method based on multi-sensor data fusion and classification. BACKGROUND

[0002] Glass is a widely used material, and its quality is crucial for many application scenarios. Glass defects can cause structural weakening, reduced optical performance, and aesthetic problems, so it is important to detect defects in glass. Existing glass defect detection methods mainly rely on traditional optical imaging technology and machine vision algorithms. These methods usually need to use lenses for imaging and rely on external light sources for illumination. However, these methods have the following problems.

[0003] Dependence on light sources and imaging equipment: Traditional detection methods require dependence on external light sources and imaging equipment, which increases the complexity and cost of the system. At the same time, this dependence leads to sensitivity to changes in environmental lighting, thereby affecting the detection effect (sensitive to lighting conditions, expensive equipment).

[0004] Limited detection dimensions: Existing methods can only detect whether there are defects in the glass, but cannot provide multi-dimensional information about the location and type of defects. This limits the effectiveness of the detection method in practical applications.

[0005] Poor adaptability to environments and scenarios: Due to the dependence on light sources and imaging equipment, existing methods often require parameter adjustment and equipment replacement when facing different environments and application scenarios, resulting in poor adaptability of the system. SUMMARY

[0006] The purpose of the present application is to solve at least one of the above problems in the prior art, and to provide a glass defect monitoring method based on multi-sensor data fusion and classification.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0008] A glass defect monitoring method based on multi-sensor data fusion and classification, comprising the following steps: S1, collecting detection signals of glass through two or more sensors, and performing deep learning fusion on the detection signals collected by the two or more sensors; S2, determining whether there is a defect according to the fused detection signals through a trained neural network, and determining the type and location of the defect type when a defect is determined.

[0009] The sensors are arranged on the side edges of the glass, one of which is an optical sensor, and the others include one or more of an infrared sensor, an electromagnetic wave sensor, an acoustic sensor, and a capacitive sensor, and the number of each sensor is one or more; the defects can be detected in real time during the use of the glass.

[0010] In some embodiments of the present application, the deep learning fusion comprises:

[0011] Feature extraction is performed on different types of detection signals using a deep neural network, and the signals are converted into a common feature space;

[0012] In the common feature space, a feature fusion strategy is applied to fuse the features of different types of detection signals;

[0013] The fused feature map is mapped to restore the spatial resolution;

[0014] The fused feature map is converted to obtain a fusion output.

[0015] In some embodiments of the present application, the neural network is a multi-task graph neural network MT-GNN; wherein the glass surface is divided into a set of grids, and each grid unit is regarded as a node in the graph, the connection between the nodes represents the relationship between adjacent grid units, and each node is assigned a feature vector containing feature data from the sensor, thereby constructing a graph structure of the glass surface; wherein the graph neural network has an input layer, a plurality of layers of graph convolution layers, and an output layer, the input layer receives a feature vector obtained from the physical signal, the graph convolution layer is used to learn the relationship between nodes and propagate information between nodes, and the output layer is used to process the output of the defect prediction task.

[0016] In some embodiments of the present application, the graph neural network includes the output layers of the following four tasks:

[0017] The defect presence output layer, which performs binary classification on each node to determine whether the node has a defect;

[0018] The defect approximate location output layer, which performs regression on each node to predict the relative location of the defect within the node;

[0019] The defect type classification output layer, which performs multi-classification on each node to identify the type of defect within the node;

[0020] The defect detailed location output layer, which performs dense regression on each node to predict the detailed location of the defect within the node;

[0021] Preferably, each task has a corresponding loss function, and the total loss function is:

[0022] L_total=λ1*L1+λ2*L2+λ3*L3+λ4*L4

[0023] Wherein, L_total is a total loss function, L1, L2, L3, L4 are loss functions of four tasks respectively, and λ1, λ2, λ3, λ4 are weights of loss functions of each task respectively.

[0024] Losses are calculated for each task respectively, and combined to obtain a total loss.

[0025] Preferably, one or more of the following loss function settings is adopted:

[0026] Defect presence prediction adopts a binary cross-entropy loss function BCELoss:

[0027] L1=BCELoss)=-Σ[y*log(p)+(1-y)*log(1-p)]

[0028] Wherein, y represents an actual label (0 or 1), and p represents a probability predicted by a model.

[0029] Defect approximate position prediction adopts a mean squared error loss function MSELoss:

[0030] L2=MSELoss=Σ(y-p)^2 / N

[0031] Wherein, y represents an actual position, p represents a position predicted by a model, and N represents a sample quantity.

[0032] Defect type classification prediction adopts a categorical cross-entropy loss function CCELoss:

[0033] L3=CCELoss=-ΣΣy_i*log(p_i)

[0034] Wherein, y_i represents a one-hot encoding of an actual category, and p_i represents a probability predicted by a model.

[0035] Defect detailed position prediction adopts a mean squared error loss MSELoss:

[0036] L4=MSELoss=Σ(y1-p1)^2 / N

[0037] Wherein, y1 represents an actual detailed position, p1 represents a detailed position predicted by a model, and N represents a sample quantity.

[0038] In some embodiments of the present application, each graph convolutional layer is according to the following formula:

[0039] X^(l+1)=σ(AX^(l)W^(l))

[0040] Wherein, X^(l) represents a node feature matrix of the lth layer, A represents an adjacency matrix, W^(l) represents a weight matrix of the lth layer, and σ represents an activation function.

[0041] In some embodiments of the present application, the following steps are further included:

[0042] S3, performing a warning operation according to the judgment result of step S2, specifically including:

[0043] Receiving defect prediction results from the multi-task graph neural network, and integrating to form a defect information list;

[0044] Traversing the defect information list, and determining whether to trigger a warning according to a preset warning threshold and a confidence of the defect;

[0045] When the warning is triggered, one or more of the following operations are performed: display, sending, and recording of warning information.

[0046] In some embodiments, an adaptive graph topology construction method is used to optimize the performance of the multi-task graph neural network MT-GNN by dynamically modifying the edge weights of the graph.

[0047] In some embodiments, a feature fusion technology based on multi-modal data is used to absorb and integrate information from multiple sensors, improving the classification accuracy of the multi-task graph neural network MT-GNN.

[0048] In some embodiments, the multi-task graph neural network MT-GNN uses a loss function design based on confusion matrix optimization to adaptively adjust the weight of each task and achieve balance between multiple tasks. During training, the confusion matrix of each task is first calculated, and then the loss of each task is calculated according to the confusion matrix; if the performance of a task is poor, its loss is increased so that the model pays more attention to that task in subsequent training, and vice versa.

[0049] The present application has the following advantages:

[0050] The glass defect monitoring method based on multi-sensor data fusion and classification provided by the present application collects detection signals of glass through two or more sensors, and performs deep learning fusion on the detection signals collected by the two or more sensors; then, through a trained neural network, it determines whether there is a defect and determines the type and position of the defect when a defect is found; by combining the detection information of multiple sensors and fusing multi-sensor data, the defect can be detected and positioned, providing multi-dimensional information about the defect position, type, etc. The method is highly adaptable and has good stability and reliability in different environments and application scenarios, and can provide more accurate and complete glass defect detection and positioning information, thereby enabling accurate identification and positioning of glass defects.

[0051] The advantages of the present application are as follows:

[0052] 1. Improve detection accuracy: Different types of sensors can provide multiple information sources, such as optical sensors can provide optical characteristics, infrared sensors can provide thermal characteristics, electromagnetic wave sensors can provide electromagnetic characteristics, etc. By fusing the data of multiple sensors, different characteristics and information can be comprehensively utilized, thereby improving the accuracy of defect detection.

[0053] 2. Multi-dimensional defect information: Traditional glass defect detection methods can usually only determine whether the glass has defects, while the method based on multi-sensor data fusion can provide more dimensional defect information, such as defect location, type, size, etc. This enables the operator to comprehensively understand the nature of the defect, so as to take corresponding measures.

[0054] 3. Enhance system robustness: Multi-sensor fusion can reduce the dependence on a single sensor and reduce the influence of environmental light and other factors. By combining the information of different sensors, the adaptability of the system to different environments and scenes can be improved, and the stability and robustness of defect detection can be enhanced.

[0055] Further, the sensors are installed on the side of the glass on various devices to monitor the defects of the glass in situ and in real time, without the need for additional lenses and light sources and other equipment. During the monitoring process, the dependence on light sources and imaging equipment can be eliminated, thereby realizing lens-free and external light source-free detection, which can significantly reduce the monitoring cost and improve the real-time performance.

[0056] By monitoring the glass in real time, the glass defects can be discovered and identified in a timely manner, thereby effectively avoiding the impact and safety hazards caused by the defects, and the method has the advantages of low cost, high real-time performance, and easy implementation.

[0057] Other beneficial effects of the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is a step flowchart of the embodiments of the present application;

[0059] Figure 2 is a step flowchart of the deep learning fusion in the embodiments of the present application;

[0060] Figure 3 is a step flowchart in embodiment 2;

[0061] Figure 4 is a schematic diagram of the installation of the sensor of the glass defect monitoring device of the embodiments of the present application on the side of the glass;

[0062] The reference signs are as follows:

[0063] 1 is glass, and 2 is a sensor. DETAILED DESCRIPTION

[0064] The application will be further described below with reference to the drawings and in conjunction with preferred embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0065] It should be noted that the terms of left, right, up, down, top, bottom and the like in the embodiments are only relative concepts or are referenced to the normal use state of the product, and should not be considered as limiting.

[0066] The following embodiments of the application propose a glass defect monitoring method based on multi-sensor data fusion and classification, referring to Figure 1 , comprising the following steps: S1, collecting detection signals of the glass 1 through two or more sensors 2, and performing deep learning fusion on the detection signals collected by the two or more sensors 2; S2, determining whether there is a defect according to the fused detection signals through a trained neural network, and determining the type and position of the defect type when a defect is determined.

[0067] Referring to Figure 2 , in the preferred embodiments, the deep learning fusion comprises:

[0068] using a deep neural network to extract features of different types of detection signals and convert them into a common feature space;

[0069] In the common feature space, a feature fusion strategy is applied to fuse the features of different types of detection signals;

[0070] mapping the fused features to restore the spatial resolution;

[0071] converting the fused features to obtain a fusion output.

[0072] Embodiment 1

[0073] Sensor:

[0074] The sensor 2 is the core part of the entire glass defect monitoring device, responsible for collecting detection signals on the surface of the glass 1. The sensor 2 needs to maintain a certain distance from the surface of the glass 1 to obtain the best signal quality. The output signal of the sensor module is usually an analog signal.

[0075] The embodiments of the application use multiple sensors 2 to collect data to realize multi-dimensional glass defect detection. Referring to Figure 4According to the above scheme, at least two sensors 2 are required, one of which is an optical sensor for capturing optical information of the glass surface, and the other is an infrared sensor, an acoustic sensor, an electromagnetic wave sensor, or a capacitance sensor, etc. according to different schemes, for detecting the heat, acoustic wave, electromagnetic wave, or capacitance distribution information of the glass surface, so as to obtain multi-dimensional feature information related to defects. The type of sensor can be selected according to actual needs.

[0076] The functions of different sensors are as follows:

[0077] Optical sensor: collects light signals in the glass and converts them into electrical signals. Here, the light signal is not only the intensity, but also the spectral distribution, phase distribution, and polarization. Optical sensor: captures optical information of the glass surface, such as surface reflection, transmission, scattering, etc., which can be used to detect surface defects, bubbles, foreign matter, etc.

[0078] Infrared sensor: detects the temperature distribution of the glass surface, which can obtain thermal characteristic information related to defects, such as thermal cracks, hot spots, etc.

[0079] Acoustic sensor: can be used to detect the sound reflection of the glass surface, and by analyzing the acoustic reflection characteristics of the glass surface, the existence and approximate position of defects such as glass cracks, internal bubbles, etc. can be determined.

[0080] Electromagnetic wave sensor: analyzes the electromagnetic wave reflection characteristics of the glass surface, which can determine the existence and approximate position of defects such as scratches and micro-cracks on the glass surface.

[0081] Capacitance sensor: detects the capacitance distribution of the glass surface, thereby obtaining electrical characteristic information related to defects, such as defects and foreign matter on the glass surface.

[0082] Magnetic sensor: can detect the magnetic field distribution in the glass to determine whether there is a defect.

[0083] Thermal sensor: can detect the temperature change of the glass surface to determine whether there is a defect.

[0084] Pressure sensor: can be used to detect the pressure change of the glass surface to determine whether there is a defect.

[0085] Electromagnetic sensor: can detect the electromagnetic field distribution in the glass to determine whether there is a defect.

[0086] In specific embodiments, the above-mentioned sensors are not limited, and even the optical sensor has different types. These sensors can realize defect detection by converting different properties of light into electrical signals, such as

[0087] Photodiode: converts light intensity into an electrical current or voltage signal.

[0088] Photoelectric Cell: converts the energy of light into electrical energy, producing a voltage.

[0089] Photomultiplier Tube: converts photons into electrons, amplifying the light signal through electron multiplication.

[0090] Optical Fiber Sensor: uses the transmission properties of light in optical fibers to convert external physical quantities (such as temperature, pressure, etc.) into light intensity, frequency, or phase parameters, and then converts them into electrical signals.

[0091] Grating Sensor: converts the wavelength or frequency of light into an electrical signal.

[0092] CCD (Charge-Coupled Device) Sensor: converts photons into electric charges, and then converts them into electrical signals through charge transfer and amplification.

[0093] In a multi-dimensional glass defect monitoring system, the data types and characteristics collected by different sensors may have significant differences. In order to obtain more accurate results in subsequent analysis and processing, it is necessary to preprocess and extract features from these data. The embodiments of the present invention explain how to preprocess and extract features from the data collected by different sensors for glass defect detection scenarios to obtain detection signals:

[0094] 1. Optical sensor

[0095] Preprocessing: For the data collected by the optical sensor, first perform denoising processing, such as using a median filter to remove noise points in the image. Next, perform contrast enhancement to make it easier to detect defects in subsequent steps.

[0096] Feature extraction: extract edge features (such as Canny edge detection), texture features (such as gray level co-occurrence matrix), and local binary pattern (LBP) in the optical image. These features help locate and identify the type of defect.

[0097] 2. Infrared sensor

[0098] Preprocessing: The data collected by the infrared sensor may be affected by factors such as temperature fluctuations. First, perform normalization to map the image data to a fixed range. Then, perform denoising processing, such as using a Gaussian filter.

[0099] Feature Extraction: Extract features from the infrared signal, such as temperature distribution characteristics, hot spot characteristics, etc. By calculating the temperature gradient in the infrared image, find the area where the defect may exist, use hot spot detection algorithm (such as threshold segmentation method) to identify abnormal hot spots related to defects.

[0100] 3. Electromagnetic wave sensor

[0101] Preprocessing: For the signal collected by the electromagnetic wave sensor, first denoising processing, such as using filter to remove noise components in the signal. Then signal normalization, the signal amplitude is mapped to a fixed range.

[0102] Feature Extraction: Extract frequency features (such as Fourier transform) and time domain features (such as signal amplitude, signal strength, etc.) from electromagnetic wave signals. These features can help identify glass defects.

[0103] 4. Acoustic sensor

[0104] Preprocessing: For the signal collected by the acoustic sensor, first denoising processing, such as using band-pass filter to remove noise components in the signal. Then signal normalization, the signal amplitude is mapped to a fixed range.

[0105] Feature Extraction: Extract frequency features (such as Fourier transform) and time domain features (such as signal amplitude, signal strength, etc.) from acoustic signals. By analyzing the spectral characteristics of the sound signal, you can find sound abnormalities related to glass defects. In addition, extract the time-frequency features of the signal, such as short-time Fourier transform (STFT) or wavelet transform (WT), which can provide more information about the time and frequency of the defect.

[0106] 5. Capacitive sensor

[0107] Preprocessing: For the signal collected by the capacitive sensor, first denoising processing, such as using low-pass filter to remove high-frequency noise components in the signal. Then signal normalization, the signal amplitude is mapped to a fixed range.

[0108] Feature Extraction: Extract time domain features (such as signal amplitude, signal strength, etc.) and frequency features (such as Fourier transform) from capacitive signals. By analyzing the changes in capacitive signals, you can find abnormal capacitive fluctuations related to glass defects.

[0109] 6. Vibration sensor:

[0110] Preprocessing: For the data collected by the vibration sensor, due to the weak vibration signal generated by the glass defect, wavelet denoising algorithm can be used for preprocessing, effectively eliminating environmental noise and retaining vibration information related to glass defects.

[0111] Feature extraction: Extract features of the vibration signal, such as signal energy, time-domain features, etc. For the glass defect detection scenario, the time-domain sliding window method can be used to calculate the energy of the vibration signal, combined with the energy threshold to determine whether there is a defect, and the approximate position of the defect is determined by the distribution of the energy.

[0112] Sensor installation and calibration

[0113] Calibration is a process to determine the relationship between the sensor output signal and the actual physical quantity. In this embodiment, the role of calibration is to ensure the relationship between the sensor installation position and the subsequent algorithm performance.

[0114] The specific method is as follows: First, use a laser tracker or similar equipment to measure the precise position of the sensor on the side of the glass. Then, place sample glasses with known defects at different positions and collect signal data through the sensor. Then, according to the known defect position and sensor output signal, establish a relationship model between the sensor output signal and the actual physical quantity (such as defect position, size). These model parameters will be used as input for subsequent algorithms to ensure the accuracy and stability of the algorithm.

[0115] Establishing a relationship model between the sensor output signal and the actual physical quantity (such as defect position, size) requires the use of certain data processing and machine learning techniques, and the specific steps are as follows:

[0116] i. Data collection: Place sample glasses with known defects and use the sensor to collect signal data at different positions. Multiple sets of data can be collected to improve the accuracy and robustness of the model.

[0117] ii. Feature extraction: Process and analyze the collected signal data and extract feature quantities. The selection of feature quantities should fully consider the sensitivity to defect position and size, while minimizing the sensitivity to noise and interference.

[0118] iii. Establish relationship model: Use machine learning algorithms (such as regression analysis, support vector machines, etc.) to establish a relationship model between the sensor output signal and the actual physical quantity. When training the model, certain data preprocessing and feature selection techniques need to be used, and cross-validation methods need to be used to evaluate the accuracy and generalization ability of the model.

[0119] iv. Model verification: Use a new sample glass data set to verify the established relationship model. The verification results can be used to evaluate the accuracy and stability of the model, and can be used to adjust and optimize the model parameters.

[0120] 3. Practical standards and formulas for the number and distribution of sensors

[0121] The number and distribution of sensors need to consider factors such as signal strength and algorithm complexity. The following formula can be used to determine the number of sensors:

[0122] N = (L * H) / (S_min * S_max)

[0123] Where:

[0124] N represents the number of sensors, L represents the length of the glass, H represents the height of the glass, S_min represents the minimum distance between sensors, S_max represents the maximum distance between sensors,

[0125] Determination of S_min: The minimum distance S_min between sensors is limited by the range of received signal strength. When the sensors are too close, signal overlap may cause different defects to be indistinguishable. Therefore, S_min should satisfy the following condition:

[0126] S_min >= k * d

[0127] Where k is a coefficient greater than 1, and d is the signal attenuation distance in the glass. The value of k can be determined by experiment to ensure that the minimum distance between sensors can effectively avoid signal overlap.

[0128] The larger the value of k, the larger the minimum distance between sensors, which can effectively avoid signal overlap. When the signal attenuation distance d is constant, if the value of k is smaller, the value of S_min is smaller, and the minimum distance between sensors is smaller, which is easy to cause signal overlap and is not conducive to accurate signal collection. Therefore, in order to avoid signal overlap, a coefficient k greater than 1 needs to be selected to make the minimum distance S_min between sensors large enough.

[0129] Determination of S_max:

[0130] The maximum distance S_max between sensors is limited by the computing power. When the sensors are too far apart, the computational complexity of the algorithm is reduced, but some defects may not be detected. Therefore, a suitable S_max value needs to be determined to ensure real-time performance while maximizing defect detection rate. S_max can be determined by the following formula:

[0131] S_max = f * R_max

[0132] Where f is a coefficient less than 1, and R_max is the maximum detection range of the sensor. The value of f can be adjusted according to the actual application scenario and computing resources.

[0133] The size of the f value determines the maximum effective detection distance S_max of the sensor. A smaller f value will result in a smaller maximum detection distance of the sensor, but can improve the accuracy and response speed of the sensor. A larger f value can expand the maximum detection distance of the sensor, but may reduce the accuracy and response speed of the sensor. Therefore, it is necessary to make reasonable adjustments according to the needs in the actual application scene.

[0134] In summary, the embodiment can determine the appropriate number and distribution of sensors using the formula N = (L * H) / (S_min * S_max) according to the limitations of S_min and S_max and the size of the glass. In this way, while ensuring real-time and detection accuracy, effective detection of glass defects can be achieved.

[0135] The glass defect monitoring method based on multi-sensor data fusion and classification provided by the embodiment of the application fuses the detection signals collected by different sensors based on a graph neural network (GNN), and performs deep learning fusion on the detection signals collected by the two or more sensors, and through the trained neural network, the tasks of judging whether there is a defect, determining the approximate position of the defect, classifying the defect type, and determining the detailed position of the defect can be simultaneously realized, and specifically includes the following contents:

[0136] 1. Data preprocessing and feature extraction: For the detection signals collected by the above-mentioned multiple sensors, the embodiment of the application can perform data preprocessing and feature extraction, combine these features into a feature vector, and input the feature vector into a graph neural network.

[0137] 2. Constructing a graph structure: the surface of the glass is divided into a set of regular grids, and each grid cell is regarded as a node in the graph, and the connection between the nodes represents the relationship between adjacent grid cells. Each node is assigned a feature vector containing feature data from all sensors, thereby constructing a graph structure of the glass surface.

[0138] 3. Designing a graph neural network: a multi-task graph neural network (MT-GNN) is designed, which includes an input layer, a plurality of graph convolution layers, and an output layer. The input layer receives the feature vector obtained from the physical signal, the graph convolution layer is used to learn the relationship between nodes and propagate information, and the output layer is used to process the output of the defect prediction task.

[0139] Preferably, the embodiment of the application adds four task-specific output layers at the top of the graph neural network to realize different tasks.

[0140] 3.1 Defect existence output layer: binary classification is performed on each node to determine whether the node has a defect.

[0141] 3.2 Defect rough location output layer: Perform regression on each node to predict the relative location of defects within that node (relative to the node's coordinates).

[0142] 3.3 Defect type classification output layer: Perform multi-classification on each node to identify the type of defects within that node.

[0143] 3.4 Defect detailed location output layer: Perform dense regression on each node to predict the detailed location of defects within that node.

[0144] The network structure, computation flow, and calculation formulas of the specific multi-task graph neural network (MT-GNN), including the objective function and loss function design, are as follows:

[0145] 1. Network structure:

[0146] Input layer: Receives feature vectors for each node, which contain pre-processed and feature-extracted data from five sensors.

[0147] Graph convolution layer: Contains several layers of graph convolution operations to learn and propagate information between nodes. Each layer of graph convolution follows the following formula:

[0148] X^(l+1) = σ(AX^(l)W^(l))

[0149] Where X^(l) represents the node feature matrix of the lth layer, A represents the adjacency matrix, W^(l) represents the weight matrix of the lth layer, and σ represents the activation function (such as ReLU).

[0150] Task-specific output layer: Add four task-specific output layers at the top of the graph neural network, for defect presence, defect rough location, defect type, and defect detailed location. These output layers can be fully connected layers, respectively handling the outputs of different tasks.

[0151] 2. Computation flow as follows:

[0152] 2.1 The input layer receives node feature vectors as input;

[0153] 2.2 Learn and propagate information between nodes through the graph convolution layer;

[0154] 2.3 Task-specific output layers handle the outputs of different tasks respectively.

[0155] Where "information between nodes" refers to the relationship and transmission of information between the feature vectors of each node and its surrounding nodes. These feature vectors include pre-processed and feature-extracted data from five sensors, while the transmitted information is learned and propagated by the graph convolution layer, including the adjacency relationship and feature similarity between nodes.

[0156] 3. Objective function and loss function:

[0157] In MT-GNN, each task has a corresponding loss function. The embodiments of the present application can calculate the loss for each task separately and combine them to get the total loss.

[0158] The following is the detailed formula of the loss function of each task:

[0159] 3.1 Defect presence prediction loss function:

[0160] This is a binary classification problem, and the binary cross-entropy loss is used as the loss function. It can effectively measure the difference between the model's predicted probability and the actual label, and make the model pay attention to misclassified samples during optimization.

[0161] L1 = Binary Cross-Entropy Loss (BCELoss) = -Σ[y*log(p)+(1-y)*log(1-p)]

[0162] Where y represents the actual label (0 or 1), and p represents the model's predicted probability.

[0163] 3.2 Defect approximate location prediction loss function:

[0164] This is a regression problem, and the embodiments of the present application need to predict the approximate location of the defect. Therefore, the mean squared error loss is used to measure the difference between the predicted value and the actual value. This loss function can effectively optimize the model's performance in location prediction.

[0165] L2 = Mean Squared Error Loss (MSELoss) = Σ(y-p)^2 / N

[0166] Where y represents the actual location, p represents the model's predicted location, and N represents the number of samples.

[0167] 3.3 Defect type classification prediction loss function:

[0168] This is a multi-classification problem, and the embodiments of the present application need to classify the defect type. Therefore, the categorical cross-entropy loss is used as the loss function. It can measure the difference between the model's predicted probability and the actual label in multiple categories, and help improve the model's performance in classification tasks.

[0169] L3 = Categorical Cross-Entropy Loss = -∑∑y_i*log(p_i)

[0170] where y_i represents the one-hot encoding of the actual class, and p_i represents the probability predicted by the model.

[0171] 3.4 Defect Detailed Location Prediction Loss Function:

[0172] This is a regression problem, which requires predicting the detailed location of defects. Therefore, Mean Squared Error Loss is used to measure the difference between the predicted value and the actual value. This loss function can also effectively optimize the model's performance in location prediction.

[0173] L4 = Mean Squared Error Loss (MSE Loss) = ∑(y-p)^2 / N

[0174] where y represents the actual detailed location, p represents the detailed location predicted by the model, and N represents the number of samples.

[0175] 3.5 Combine the loss functions of the above four tasks to get the total loss function:

[0176] L_total = λ1*L1 + λ2*L2 + λ3*L3 + λ4*L4

[0177] where λ1, λ2, λ3, and λ4 are the weights of each task loss function, which can be adjusted according to the actual problem and data.

[0178] Setting the loss function as above has the following benefits:

[0179] (1) By calculating the loss for each task separately, we can ensure that the model contributes to learning each task. This helps improve the model's performance on each task.

[0180] (2) Using the total loss function, we can balance the importance of different tasks. By adjusting the weights of each task loss function, we can optimize the model's performance on different tasks.

[0181] (3) The total loss function helps to realize end-to-end training, so that the model learns multiple tasks in a unified framework, improving the efficiency of training and prediction.

[0182] 4、Training and Prediction:

[0183] Use the labeled dataset to train MT-GNN, minimizing the total loss function.

[0184] In the prediction phase, the sensor feature vector of the glass to be detected is input, and the prediction results of each task are obtained by MT-GNN:

[0185] a) Defect presence prediction: output the binary classification probability of each node, and set a threshold (such as 0.5) to determine whether there is a defect.

[0186] b) Defect approximate location prediction: output the relative position of the defect within each node (relative to the coordinates of the node).

[0187] c) Defect type classification prediction: output the multi-classification probability of each node, and determine the defect type according to the class with the highest probability.

[0188] d) Defect detailed location prediction: output the detailed location of the defect within each node.

[0189] Analyze the prediction results:

[0190] According to the defect presence prediction result, determine the nodes on the glass that have defects.

[0191] For nodes with defects, use the defect approximate location and defect detailed location prediction results to determine the actual location of the defect on the glass.

[0192] The multi-task graph neural network uses MT-GNN, and the embodiments of the present application realize a unified framework that can simultaneously process data from different sensors and implement multiple tasks of defect detection. This method takes advantage of the spatial information processing capabilities of graph neural networks, effectively fusing and analyzing sensor data. At the same time, the design of multi-task learning enables the network to learn multiple tasks in a unified framework, improving the efficiency of training and prediction.

[0193] In the preferred embodiment, we use an adaptive graph topology construction method to optimize the performance of the multi-task graph neural network MT-GNN by dynamically modifying the edge weights of the graph.

[0194] We propose an adaptive graph topology construction method to optimize the performance of the multi-task graph neural network MT-GNN. In actual operation, we dynamically adjust the edge weights of the graph according to the actual situation of the data and the distribution of defects. For example, if the data from two sensor locations is more similar or related, we can increase the edge weight between them, and vice versa. This method can better capture the internal structure of the data, enabling the MT-GNN to better learn and judge the type and severity of defects.

[0195] In the preferred embodiment, a feature fusion technology based on multi-modal data is used to absorb and integrate information from multiple sensors, improving the classification accuracy of the multi-task graph neural network MT-GNN.

[0196] We adopt a feature fusion technique based on multi-modal data. Specifically, we first acquire data from various types of sensors such as optical sensors, thermal sensors, etc., and then preprocess these data to extract useful features. Next, we use feature fusion techniques to combine these features together to form a unified feature representation. Finally, we input this feature representation into the MT-GNN for defect classification. This method can integrate multiple types of data, provide more comprehensive and accurate information, and thus improve the accuracy of defect classification.

[0197] In the preferred embodiment, the multi-task graph neural network MT-GNN adopts a loss function design based on confusion matrix optimization to adaptively adjust the weight of each task and achieve balance between multiple tasks. During training, the confusion matrix of each task is first calculated, and then the loss of each task is calculated according to the confusion matrix; if the performance of a task is poor, its loss is increased so that the model pays more attention to this task in subsequent training, and vice versa.

[0198] In our method, MT-GNN adopts a loss function design based on confusion matrix optimization. During training, we first calculate the confusion matrix of each task, and then calculate the loss of each task according to the confusion matrix. If the performance of a task is poor, we increase its loss so that the model pays more attention to this task in subsequent training. Conversely, if the performance of a task is good, we reduce its loss to prevent it from occupying too much training resources. This method can adaptively adjust the weight of each task and achieve balance between multiple tasks, improving the overall performance.

[0199] Signal processing and defect detection algorithm

[0200] First, acquire signal data from sensors, then extract information about glass defects through signal processing techniques. Next, through defect detection algorithms, including using machine learning and / or other methods for recognition:

[0201] 1. Signal acquisition and preprocessing

[0202] (1) Adaptive filter: Least Mean Square (LMS) algorithm can be used, which can update filter coefficients at each iteration to minimize the mean square error of the output signal. The LMS algorithm update formula is:

[0203] W(n+1) = W(n) + μ * e(n) * X(n)

[0204] where W(n) is the filter coefficient, μ is the step parameter, e(n) is the error signal, and X(n) is the input signal.

[0205] (2) Empirical Mode Decomposition (EMD): First, find all the extreme points (including maxima and minima) of the signal. Then, calculate the envelope curves between the maxima and minima by cubic spline interpolation. Next, calculate the mean of the upper and lower envelope curves, and subtract the mean curve from the original signal to obtain the first intrinsic mode function (IMF). Repeat this process until the signal is decomposed into multiple IMFs.

[0206] Adaptive filtering: Adaptive filtering is a nonlinear filtering method that can automatically adjust the parameters of the filter according to the characteristics of the signal, so as to filter out noise while preserving important information of the signal. In glass defect detection, adaptive filtering can remove signal noise and interference caused by uneven light, camera angle, etc., thereby improving the clarity and accuracy of the signal.

[0207] Empirical Mode Decomposition (EMD): Empirical Mode Decomposition is a signal decomposition method based on local signal characteristics, which can decompose the original signal into several intrinsic mode functions and a residual term. In glass defect detection, empirical mode decomposition can decompose the original signal into different local signal characteristics, thereby better extracting defect information from the signal and reducing the influence of interference on defect detection. At the same time, empirical mode decomposition can effectively remove noise and interference in the signal, improving the accuracy and stability of the signal.

[0208] 2. Defect detection algorithm

[0209] 2.1. Physics-based method

[0210] (1) Mutual Information (MI): Calculate the MI of two signals X and Y, the formula is as follows:

[0211] MI(X,Y) = H(X) + H(Y) - H(X,Y)

[0212] where H(X) and H(Y) represent the entropy of signals X and Y, respectively, and H(X,Y) represents the joint entropy.

[0213] Method for determining whether there is a defect: If there is a defect, the defect will cause a change in the correlation between signals, which in turn will cause a change in the MI value. Therefore, by calculating the MI value between signals, we can determine whether the correlation between signals has changed, thereby indirectly determining whether there is a defect.

[0214] (2) Support Vector Machine (SVM): First, the fused signal features are used as input data. "Signal features" refer to the features extracted in the previous preprocessing steps, such as signal amplitude, frequency, spectral lines, etc. These features are used as input data for training and testing the SVM classifier. When training the SVM classifier, each data point needs to be assigned a class label (defect or no defect). During training, the SVM classifier finds a hyperplane that maximizes the separation between the two classes. In the testing phase, the SVM classifier classifies new data points based on the hyperplane.

[0215] Method for determining the presence of defects: By training an SVM classifier with fused signal features as input data, defects and non-defects can be classified into two categories. In the testing phase, new data points are classified, and if they are classified as defects, it indicates the presence of defects.

[0216] 2.2 Learning-based methods

[0217] (1) Attention mechanism: A weight vector α is introduced to calculate the weight between different signal channels. Channel signals can include temperature, humidity, and light sensor information collected by different signal channels. Attention weights are calculated by the Softmax function:

[0218] α_i = exp(e_i) / Σ(exp(e_j))

[0219] where e_i is the original weight value of the i-th channel, and j represents the channel index.

[0220] (2) Long Short Term Memory (LSTM): LSTM consists of input gates, output gates, and forget gates, which can effectively handle long sequence data. At each time step, LSTM updates the state of each gate based on the current input and the previous hidden state. The output gate determines which information will be passed to the hidden state at the next time step, the forget gate determines which information will be discarded from the cell state, and the input gate determines which information will be added to the cell state. The key formulas of LSTM are as follows:

[0221] f_t = σ(W_f * [h_(t-1), x_t] + b_f) Forget gate

[0222] i_t = σ(W_i * [h_(t-1), x_t] + b_i) Input gate

[0223] o_t = σ(W_o * [h_(t-1), x_t] + b_o) Output gate

[0224] c_t = f_t * c_(t-1) + i_t * tanh(W_c * [h_(t-1), x_t] + b_c) cell state update

[0225] h_t = o_t * tanh(c_t) hidden state update

[0226] These formulas describe a neural network model called LSTM, which is used to process sequential data such as time series data or text data. Here, t represents the time step, h_t represents the hidden state at time step t, x_t represents the input data. f_t, i_t, o_t are mechanisms called gates that control the flow of information in the model. The forget gate (f_t) decides how much information from the previous time step's cell state to keep, the input gate (i_t) decides how much new information to add to the cell state at the current time step, and the output gate (o_t) decides how much information from the current time step's hidden state to pass to the next time step. c_t represents the cell state at the current time step, which is a combination of the cell state from the previous time step and the input information at the current time step. Finally, the LSTM model calculates the cell state and hidden state at the current time step using the above formulas, which are used for the calculation of the next time step.

[0227] where σ denotes the sigmoid activation function, f_t, i_t, o_t are the states of the forget gate, input gate, and output gate, respectively, c_t is the cell state, h_t is the hidden state, W and b represent the weight matrix and bias term, respectively.

[0228] For the glass defect detection problem, first, the signals from each sensor are concatenated to form a multi-channel signal. Then, the attention mechanism is introduced, so that the LSTM network can pay attention to the importance of different channel signals at each time step. During training, the network will automatically learn the signal features and timing information related to defects.

[0229] The attention-based LSTM model is based on the long short-term memory network (LSTM) and attention mechanism. In this model, the signals collected by multiple sensors are first concatenated to form a multi-channel signal and passed as input to the LSTM model. The LSTM model consists of an input gate, a forget gate, and an output gate, and can process time series data. At each time step, the LSTM model updates the state of each gate based on the current input and the hidden state at the previous time step, and outputs the hidden state at the current time step.

[0230] To better focus on the importance of different channel signals, the model introduces an attention mechanism. The attention mechanism introduces a weighting vector a to calculate the weight between different signal channels. The attention weight is calculated by the Softmax function, and then weighted with the input signal to get the weighted input signal. In this way, the LSTM model can pay more attention to the importance of different channel signals at each time step.

[0231] During training, the model updates the weight parameters through the backpropagation algorithm, thereby automatically learning the signal features and timing information related to defects. In the testing phase, for each new data point, the model inputs it into the LSTM network and calculates the activation value of the output layer. The activation value of the output layer is normalized by the Softmax function.

[0232] The trained Attention-based LSTM model can be used to predict whether new data has defects. Specifically, for each new data point, it can be input into the LSTM network and the activation value of the output layer is calculated. The activation value of the output layer can be normalized by the Softmax function to obtain the probability distribution of the presence or absence of defects.

[0233] In summary, these specific preprocessing and feature extraction schemes can process different types of sensor data, thereby supporting multi-dimensional glass defect detection. By integrating these features together, the approximate location of the defect, the defect type, and the precise location can be more accurately determined.

[0234] Embodiment 2

[0235] The other contents of this embodiment are the same as Embodiment 1, and the same contents will not be repeated.

[0236] Reference Figure 3 Preferably, this embodiment includes a warning operation according to the above judgment result, specifically including: receiving the defect prediction result from the multi-task graph neural network, and integrating to form a defect information list; traversing the defect information list, and judging whether to trigger a warning according to a preset warning threshold and a confidence of the defect; when the warning is triggered, performing one or more operations of displaying, sending, and recording the warning information.

[0237] For the specific scheme of the warning operation, the following steps can be implemented:

[0238] (1) Set the warning threshold: set a warning threshold for each defect type according to historical data and experience, such as a general warning when the confidence is greater than 0.8, and a serious warning when the confidence is greater than 0.9.

[0239] (2) Defect Information Integration: The early warning module receives defect prediction results from the multi-task graph neural network (MT-GNN). It integrates information such as defect type, location, and confidence level to form a defect information list.

[0240] (3) Design an early warning strategy: Traverse the defect information list and determine whether to trigger an early warning based on a preset threshold. For example, if the confidence level of a defect is greater than 0.8, a general early warning is triggered; if the confidence level is greater than 0.9, a severe early warning is triggered.

[0241] (4) Early warning information presentation: Design a real-time monitoring interface to display defect information. The defect type, location, confidence level, etc. can be listed in a table format, and different levels of early warning can be represented by colors or icons.

[0242] (5) Early Warning Notification: When an early warning is triggered, relevant personnel should be notified promptly. Early warning information can be sent via SMS gateways, email servers, or mobile app push notifications. Simultaneously, the early warning information should be recorded in a database or log file for subsequent analysis.

[0243] For specific scenarios, such as automotive glass, existing methods cannot meet the needs of real-time detection. During vehicle operation, glass may develop defects due to impacts, abrasion, or other reasons. These defects can obstruct vision, reduce optical performance, and even pose safety hazards. Therefore, for applications such as automotive glass, a method is needed that can detect defects in real time during use, enabling timely warnings and intervention when problems are detected.

[0244] This section will also describe how to transmit detected glass defect information to users in real time so that warnings can be issued at any time during use, and introduce the specific implementation method of the real-time warning system, as well as how to adapt it to different scenarios (such as automotive glass).

[0245] 1. Warning Level Classification

[0246] To better convey early warning information, the warning level is divided into three levels, each representing a different degree of urgency of the glass defect.

[0247] (1) Low risk (green): The defect is small and does not affect the normal use of the glass, but it needs to be monitored to prevent the problem from worsening.

[0248] (2) Medium risk (yellow): The defects are obvious and may affect the normal use of the glass. They need to be resolved in the short term.

[0249] (3) High risk (red): The defect is serious and may cause damage to the glass, requiring immediate action.

[0250] 2. Issue an early warning

[0251] According to different scenarios and warning levels, the way of issuing warnings needs to be adjusted. The following are examples:

[0252] (1) Vehicle-mounted glass:

[0253] Low risk: Display a green icon on the car's dashboard to prompt the driver to pay attention;

[0254] Medium risk: Display a yellow icon on the car's dashboard with a slight alarm sound to prompt the driver to solve the problem as soon as possible;

[0255] High risk: Display a red icon on the car's dashboard with a continuous emergency alarm sound to prompt the driver to stop and check immediately.

[0256] (2) Building glass:

[0257] Low risk: Send a short message or email notification to the relevant responsible person through the building property management system;

[0258] Medium risk: Send a short message or email notification to the relevant responsible person through the building property management system and require the problem to be solved within a certain time;

[0259] High risk: Send a short message or email notification to the relevant responsible person through the building property management system and require immediate organization of professional personnel for inspection and repair.

[0260] 3. Real-time warning notification

[0261] Vehicle-mounted glass: Send warning information to the car's dashboard through the car's built-in communication module (such as vehicle-mounted Bluetooth, Wi-Fi (Wireless Fidelity, Wi-Fi) wireless fidelity, etc.), display the warning level and prompt information in real time.

[0262] Building glass: Send warning information to the building property management system through wireless network (such as Wi-Fi, (Long Range Radio, LoRa) long-distance wireless power supply, etc.), and the system automatically sends the corresponding notification according to the warning level.

[0263] Through the design and implementation of the above real-time warning system, appropriate warning methods can be taken according to different scenarios and warning levels to ensure that warning information is delivered to users in a timely manner when glass defects occur, improving users' safety awareness and reducing potential risks.

[0264] 4. Automatic update and tracking of warning information

[0265] In order to ensure that users always know the latest status of glass defects, the real-time warning system also needs to have the function of automatic update and tracking.

[0266] Automatic updates: The system will periodically collect and process signal data from sensors to update warning information in real-time. When the warning level changes, the system will automatically adjust the warning notification to ensure that users are always aware of the current risk situation.

[0267] Tracking processing progress: For glass defects that have been issued warnings, the system can track the processing progress to help users monitor the resolution of the problem. For example, for building glass, the property management system can update the warning information and notify the responsible person after processing is completed. For vehicle glass, users can check the processing progress through the car dashboard and receive a notification when the problem is resolved.

[0268] 5. Adaptability to different environments and scenarios

[0269] The real-time warning system should have good adaptability to work normally in different environments and scenarios. Specifically, the system needs to consider the following factors:

[0270] (1) Transmission distance: Depending on different scenarios (such as car glass, building glass, etc.), the transmission distance of warning information may vary. The system should be able to support data transmission at different distances, such as through Bluetooth, Wi-Fi, and other communication methods.

[0271] (2) Environmental factors: The system needs to cope with various environmental factors such as temperature, humidity, light, etc. For example, for outdoor building glass, the system should have waterproof, dustproof, etc. functions to ensure normal operation in harsh environments.

[0272] (3) Device compatibility: The real-time warning system needs to be compatible with various devices such as car dashboards, property management systems, etc. In addition, it needs to consider different brands and models of devices to ensure that warning information can be correctly displayed and transmitted.

[0273] By considering the above factors, a real-time warning system with strong adaptability can be designed to provide timely and accurate warning information to users in various scenarios and environments.

[0274] Example 3

[0275] The other contents of this example are the same as those of Example 1 and 2, and the same contents are not repeated.

[0276] In order to improve the performance of the glass defect detection system, including reducing false positives and false negatives, improving detection speed and accuracy, etc., the following optimization contents are proposed:

[0277] (1) Optimizing algorithms: By continuously optimizing signal processing and defect detection algorithms, detection accuracy and speed can be improved. This may include using more efficient signal processing techniques, developing more powerful machine learning models, etc.

[0278] Use more efficient signal processing techniques: Methods such as multi-channel signal processing, wavelet transform, and Fourier transform can be used to improve signal processing efficiency and accuracy.

[0279] Develop more powerful machine learning models: Methods such as deep learning and reinforcement learning can be used to improve the accuracy and speed of defect detection. In addition, methods such as transfer learning and incremental learning can be used to cope with real-time changes in the environment and data.

[0280] (2) Sensor calibration: Regularly calibrate and maintain sensors to ensure their stable performance. This helps reduce false positives and false negatives, improving detection accuracy.

[0281] Regularly calibrate and maintain sensors: Standard calibration boards and regular maintenance can be used to ensure the stable performance and accuracy of sensors.

[0282] Use multi-sensor fusion: Multi-sensor fusion techniques such as Kalman filtering can be used to eliminate errors and uncertainties between sensors and improve detection accuracy.

[0283] (3) Dynamic threshold adjustment: Dynamically adjust the defect detection threshold according to the actual application environment and conditions. This helps reduce false positives and false negatives, improving detection accuracy.

[0284] Adjust the defect detection threshold dynamically according to the actual application environment and conditions: Adaptive algorithms such as sliding window and dynamic threshold can be used to improve detection accuracy and robustness.

[0285] Use multi-threshold decision: Multi-threshold decision techniques such as interval-based multi-threshold decision and confidence-based multi-threshold decision can be used to further improve detection accuracy and robustness.

[0286] (4) Data fusion: By fusing data from multiple sensors, detection accuracy and robustness can be improved. This can be achieved through weighted averaging, Kalman filtering, and other data fusion techniques

[0287] Weighted averaging: Weighted averaging of multiple sensor data can reduce noise and errors and improve detection accuracy.

[0288] Kalman filtering: Use Kalman filtering to fuse multi-sensor data, eliminate sensor noise and errors, and improve detection accuracy and robustness.

[0289] Adaptive Warning Strategy: Design adaptive warning strategies for different scenarios and application requirements. For example, during vehicle driving, adjust the warning level and notification method in real time according to vehicle speed, road conditions, etc.; in the building property management scenario, set appropriate emergency level and notification strategy for defects according to floor, area, etc.

[0290] (6) Model updating and online learning: Through continuous collection and analysis of new data, online learning and updating of defect detection model. This helps to improve the performance and adaptability of the system, and reduce the false positive and false negative rates.

[0291] Establish a real-time data acquisition and analysis system to collect real-time data from sensors and process and analyze them.

[0292] Use machine learning algorithms and model updating techniques to learn and update the model in real time with new data, improving the accuracy and robustness of the model.

[0293] Customize specific learning strategies and updating methods for different application scenarios and devices, such as incremental learning or online learning-based model updating techniques.

[0294] (7) Performance monitoring and feedback: Real-time monitoring of system performance and collection of user feedback. Through the analysis of performance indicators and user needs, continuously optimize the system and improve user experience.

[0295] Collect user feedback and suggestions, use user surveys, user behavior analysis, etc. to analyze user needs and preferences, and continuously optimize the system based on user feedback.

[0296] Based on the monitoring results and user feedback, design appropriate performance indicators and evaluation methods to evaluate system performance and user experience to continuously improve system performance and user satisfaction.

[0297] Through optimization algorithms, sensor calibration, dynamic threshold adjustment, data fusion, adaptive warning strategies, model updating and online learning, performance monitoring and feedback, etc., improve the performance and user experience of the glass defect detection system.

[0298] The glass defect detection method in the embodiment of the present application can be used for various types of glass, for example:

[0299] 1. Building glass: used for building exterior walls, curtain walls, etc., needs to detect whether the surface is flat, whether there are bubbles, cracks, etc.

[0300] 2. Optical glass: used to manufacture various optical devices such as lenses, prisms, filters, etc., needs to detect whether the surface is smooth, whether there are scratches, bumps, etc.

[0301] 3. Intelligent glass: for scenarios such as electronic displays, smart homes, etc., it is necessary to detect whether the surface is flat, whether there are scratches, stains and other defects.

[0302] The embodiment of the present application proposes a glass defect monitoring method based on multi-sensor data fusion and classification. The data collected by multiple sensors are combined with the position information of multiple sensors to organically fuse the multi-sensor data. The defect detection algorithm is used to detect and locate the defects, and multi-dimensional information about the defect position, type, etc. is provided. This method not only realizes lens-free and external light source-free detection, but also innovates in system optimization and adaptability, and has good stability and reliability in different environments and application scenarios, providing more detailed information about glass defects.

[0303] The multi-sensor data fusion and classification glass defect monitoring method of the embodiment of the present application uses advanced fusion and classification algorithms to provide more accurate and complete glass defect detection and positioning information. In addition, this method has innovations in system optimization and adaptability, and can adapt to different environments and application scenarios, having high practical value and advancement. With the continuous development of the automotive industry and the increasing demand for glass quality, the application prospect of this technology is broad, and it is expected to become an important innovative technology in the field of glass defect monitoring. At the same time, this technology also has strong popularization value and can be widely used in defect monitoring and positioning in other fields, having great commercial value and social value.

[0304] The present application also proposes a computer readable storage medium storing a computer program, which is executed by a processor to implement the method as described above.

[0305] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent alternatives or obvious variations can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A glass defect monitoring method based on multi-sensor data fusion and classification, characterized in that, Includes the following steps: S1. Collect detection signals of the glass using two or more sensors, and fuse the detection signals collected by the two or more sensors using deep learning. S2. Using a trained neural network, determine the presence or absence of defects based on the fused detection signals, and when a defect is detected, determine the type and location of the defect. The neural network is a multi-task graph neural network (MT-GNN). The glass surface is divided into a grid, with each grid cell considered a node in the graph. Connections between nodes represent relationships between adjacent grid cells. A feature vector is assigned to each node, containing feature data from the sensor, thereby constructing the graph structure of the glass surface. The graph neural network has an input layer, several graph convolutional layers, and an output layer. The input layer receives feature vectors obtained from physical signals. The graph convolutional layers learn relationships between nodes and propagate information between nodes. The output layer processes the output of the defect prediction task. The graph neural network includes output layers for the following four tasks: a defect presence / absence output layer, which performs binary classification on each node to determine whether a defect exists in that node; an approximate defect location output layer, which performs regression on each node to predict the relative location of the defect within that node; a defect type classification output layer, which performs multi-classification on each node to identify the type of defect within that node; and a detailed defect location output layer, which performs dense regression on each node to predict the detailed location of the defect within that node.

2. The glass defect monitoring method as described in claim 1, characterized in that, The sensors are located on the side of the glass. One type is an optical sensor, and the others include one or more of infrared sensors, electromagnetic wave sensors, acoustic sensors, and capacitive sensors, with one or more of each type of sensor. The sensors can detect defects in real time during the use of the glass.

3. The glass defect monitoring method as described in claim 1, characterized in that, The deep learning fusion includes: Deep neural networks are used to extract features from different types of detection signals and transform them into a common feature space. In the common feature space, a feature fusion strategy is applied to fuse the features of different types of detection signals; The fused feature maps are restored to their spatial resolution. The fused feature maps are transformed to obtain the fused output.

4. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, Each task has a corresponding loss function, and the total loss function is: L_total = λ1 L1 + λ2 L2 + λ3 L3 + λ4 L4 Where L_total is the total loss function, L1, L2, L3, and L4 are the loss functions of the four tasks, and λ1, λ2, λ3, and λ4 are the weights of the loss functions of each task. Calculate the loss for each task separately, and combine them to obtain the total loss; Specifically, one or more of the following loss functions can be used: Whether defects are predicted is determined using the binary cross-entropy loss function BCELoss: L1 = BCELoss = -Σ [y log(p) + (1 - y) log(1 - p)] Where y represents the actual label 0 or 1, and p represents the probability predicted by the model; Defect location is roughly predicted using the mean squared error loss function MSELoss. L2 = MSELoss= Σ (y - p)^2 / N Where y represents the actual location, p represents the location predicted by the model, and N represents the number of samples; Defect type classification prediction uses the category cross-entropy loss function CCELoss: L3 =CCELoss= -Σ Σ y_i log(p_i) Where y_i represents the one-hot encoding of the actual class, and p_i represents the probability predicted by the model; Detailed defect location prediction using mean squared error loss (MSELoss): L4 = MSELoss= Σ (y1 - p1)^2 / N Where y1 represents the actual detailed location, p1 represents the detailed location predicted by the model, and N represents the number of samples.

5. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, Each graph convolutional layer performs feature processing according to the following formula: X^(l+1) = σ(AX^(l)W^(l)) Where X^(l) represents the node feature matrix of the l-th layer, X^(l+1) represents the node feature matrix of the (l+1)-th layer, A represents the adjacency matrix, W^(l) represents the weight matrix of the l-th layer, and σ represents the activation function.

6. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, It also includes the following steps: S3. Based on the judgment result of step S2, perform an early warning operation, specifically including: Receive defect prediction results from a multi-task graph neural network and integrate them to form a defect information list; Traverse the list of defect information and determine whether to trigger an alert based on the preset warning threshold and the confidence level of the defect; When an alert is triggered, perform one or more of the following operations: displaying, sending, or recording the alert information.

7. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, It also includes: using an adaptive graph topology construction method to optimize the performance of the multi-task graph neural network MT-GNN by dynamically modifying the edge weights of the graph.

8. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, Also includes: By using feature fusion technology based on multimodal data, information from multiple sensors is absorbed and integrated to improve the classification accuracy of the multi-task graph neural network MT-GNN.

9. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, Also includes: The multi-task graph neural network MT-GNN adopts a loss function design based on confusion matrix optimization to adaptively adjust the weight of each task and achieve a balance among multiple tasks; During training, the confusion matrix for each task is calculated first, and then the loss for each task is calculated based on the confusion matrix. If the performance of a task is poor, its loss is increased so that the model pays more attention to the task in subsequent training, and vice versa.

Images