A deep learning-based glass defect monitoring method

Abstract

The application discloses a kind of glass defect monitoring methods based on deep learning, comprising the following steps: S1, the detection signal of glass is collected by more than two sensors;S2, using trained long short-term memory network (LSTM) model based on attention mechanism, the signal collected by the more than two sensors is fused in series, forms a multi-channel signal, and is passed to the LSTM model as input to predict whether glass has defects.The application combines the detection signals of multiple sensors, fuses multiple sensor data in series to detect defects, has strong adaptability, good stability and reliability in different environments and application scenarios, can provide more accurate and complete glass defect detection information, so as to realize accurate identification of glass defects.

Description

Technical Field

[0001] The present invention relates to the technical field of glass monitoring, and in particular to a deep learning-based method for monitoring glass defects. Background Technology

[0002] Glass, as a widely used material, is crucial to the quality of many applications. Glass defects can lead to structural weakening, reduced optical performance, and aesthetic problems, making defect detection in glass of great significance. Existing glass defect detection methods mainly rely on traditional optical imaging techniques and machine vision algorithms. These methods typically require lenses for imaging and depend on external light sources for illumination. However, these methods suffer from the following problems.

[0003] Dependence on light sources and imaging equipment: Traditional detection methods rely on external light sources and imaging equipment, which increases the complexity and cost of the system. Furthermore, this dependence leads to sensitivity to changes in ambient lighting, thus affecting detection performance (sensitive to lighting conditions, expensive equipment).

[0004] Limited detection dimensions: Existing methods often only detect the presence of defects in glass, but cannot provide multi-dimensional information such as the location and type of defects. This limits the effectiveness of detection methods in practical applications.

[0005] Poor adaptability to environment and scenario: Due to the dependence on light source and imaging equipment, existing methods often require parameter adjustment and equipment replacement when facing different environments and application scenarios, resulting in poor system adaptability. Summary of the Invention

[0006] The purpose of this invention is to solve at least one of the problems mentioned above in the prior art and to provide a glass defect monitoring method based on deep learning.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A deep learning-based glass defect monitoring method includes the following steps: S1, acquiring detection signals of the glass through two or more sensors; S2, using a trained attention-based Long Short-Term Memory (LSTM) network model to fuse and concatenate the signals acquired by the two or more sensors to form a multi-channel signal, which is then used as input to the LSTM model to predict whether the glass has defects.

[0009] In some embodiments of the present invention, the LSTM model includes an input gate, a forget gate, and an output gate. At each time step of processing time series data, the LSTM model updates the state of each gate based on the current input and the hidden state of the previous time step, and outputs the hidden state of the current time step. The attention mechanism introduces a weighted vector to determine the weights between different signal channels, thereby enabling the LSTM model to focus on the importance of different channel signals at each time step.

[0010] In some embodiments of the present invention, the attention weights are calculated using the Softmax function and then weighted and summed with the input signal to obtain the weighted input signal.

[0011] In some embodiments of the present invention, during the training process, the LSTM model updates the weight parameters through the backpropagation algorithm, thereby automatically learning the signal features and temporal information related to the defects; during the testing phase, for each new input data point, the LSTM model calculates the activation value of the output layer and normalizes it through the Softmax function to obtain the probability distribution of the presence or absence of defects.

[0012] In some embodiments of the present invention, step S2 involves preprocessing the detection signal, specifically including one or more of the following:

[0013] Adaptive filtering removes noise while preserving important signal information.

[0014] Empirical mode decomposition (EMD) decomposes the original signal into different local signal features in order to extract defect information from the signal and reduce interference.

[0015] In some embodiments of the present invention, the two or more sensors include two types of sensors disposed on the side of the glass, one of which is an optical sensor, and the others include one or more of infrared sensors, electromagnetic wave sensors, acoustic sensors, and capacitive sensors, with one or more of each type of sensor; it is possible to detect defects in real time during the use of the glass.

[0016] In some embodiments of the present invention, step S2 further includes extracting data from different sensors using a multi-task graph neural network (MT-GNN); wherein the glass surface is divided into a set of grids, and each grid cell is regarded as a node in a graph, the connection between nodes represents the relationship between adjacent grid cells, and a feature vector is assigned to each node, which contains feature data from the sensors, thereby constructing a graph structure of the glass surface; wherein the graph neural network has an input layer, several graph convolutional layers and an output layer, the input layer receives feature vectors obtained according to the physical signal, the graph convolutional layers are used to learn the relationship between nodes and propagate information between nodes, and the output layer outputs data about the defect prediction task.

[0017] In some embodiments of the present invention, step S1 includes acquiring a spectral image of the glass using a spectral image sensor; in step S2, the physical properties of the glass surface and interior are identified based on the spectral distribution of light reflected, absorbed, or transmitted by the glass using the characteristics of the spectral image, thereby detecting defects in the glass.

[0018] Preferably, in step S1, a set of multispectral cameras is set up so that they can capture spectral images of the glass from different angles and distances; the multispectral cameras scan along multiple different directions, thereby enabling comprehensive detection of the surface and interior of the glass.

[0019] In some embodiments of the present invention, step S2 further includes: using an Adaptive Convolutional Neural Network (ACNN) to extract and classify spectral image features to achieve defect identification; the Adaptive Convolutional Neural Network includes a series of convolutional layers and fully connected layers, wherein the convolutional layers are used to extract image features, and the fully connected layers are used to integrate the features for classification; the Adaptive Convolutional Neural Network has adaptive convolutional layers, which can automatically adjust the size and stride of their convolutional kernels according to the characteristics of the input image to extract important features in the image.

[0020] In some embodiments of the present invention, step S2 further includes: processing the feature sequence of the spectral image using a Temporal Convolutional Neural Network (TCNN) to extract temporal information reflecting the defect development process; preferably, the continuous spectral image is first converted into a feature sequence, and then these sequences are input into the temporal convolutional neural network; the temporal convolutional neural network uses multiple convolutional layers and recurrent layers to process the input feature sequence, thereby capturing the temporal information in the sequence; preferably, the training of TCNN uses the backpropagation algorithm and stochastic gradient descent method, and the network parameters are updated by optimizing the loss function so that the network can better identify and predict the development of defects.

[0021] The present invention has the following beneficial effects:

[0022] The deep learning-based glass defect monitoring method proposed in this invention collects detection signals from glass using two or more sensors. Then, a trained Long Short-Term Memory (LSTM) network model based on an attention mechanism is used to fuse and concatenate the signals from the two or more sensors, forming a multi-channel signal, which is then input to the LSTM model to predict the presence of defects in the glass. By combining the detection signals from multiple sensors and using the LSTM model to fuse and concatenate the data to predict defects, this method exhibits strong adaptability, good stability and reliability in different environments and application scenarios, and provides more accurate and complete glass defect detection information, thus enabling accurate identification of glass defects. Because the LSTM model also incorporates an attention mechanism, it allows the network to focus on the importance of different channel signals at each time step. During training, the network automatically learns the signal features and temporal information related to defects, achieving focused attention on key information and improving the accuracy of defect detection.

[0023] In some embodiments, the method of the present invention achieves accurate and efficient glass defect monitoring by integrating lensless and external light source-free detection, adaptive deep convolutional neural networks, and temporal convolutional neural networks.

[0024] In addition, some embodiments also have the following beneficial effects:

[0025] Furthermore, by mounting sensors on the side of glass on various devices, in-situ, real-time defect monitoring of the glass can be performed without the need for additional lenses and light sources. The monitoring process can be freed from dependence on light sources and imaging equipment, thereby achieving lensless and external light source-free detection, which can significantly reduce monitoring costs and improve real-time performance.

[0026] By monitoring glass in real time, defects can be detected and identified in a timely manner, thereby effectively avoiding the impact of defects on use and safety hazards. It has the advantages of low cost, high real-time performance and ease of implementation.

[0027] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0028] Figure 1 This is a flowchart of the steps in an embodiment of the present invention;

[0029] Figure 2 Flowchart of steps in the embodiment;

[0030] Figure 3 This is a schematic diagram of the sensor installation on the side of the glass in the glass defect monitoring device according to an embodiment of the present invention;

[0031] The attached figures are labeled as follows:

[0032] 1 is glass, 2 is a sensor. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0034] It should be noted that the directional terms such as left, right, up, down, top, and bottom used in this embodiment are only relative concepts or are based on the normal use of the product, and should not be considered as restrictive.

[0035] Example 1

[0036] A deep learning-based method for glass defect monitoring, see [reference]. Figure 1 The process includes the following steps: S1, acquiring detection signals of glass 1 through two or more sensors 2; S2, using a trained attention-based long short-term memory network (LSTM) model to fuse and connect the signals acquired by the two or more sensors 2 to form a multi-channel signal, which is then used as input to the LSTM model to predict whether there are defects in glass 1.

[0037] Preferably, the glass 1 mentioned above includes glass for vehicles, ships, aircraft, buildings, industrial equipment, photovoltaic equipment, or household appliances.

[0038] sensor

[0039] Sensor 2 is the core component of the entire glass defect monitoring device, responsible for collecting detection signals from the surface of glass 1. Sensor 2 needs to maintain a certain distance from the surface of glass 1 to obtain optimal signal quality. The output signal of the sensor module is typically an analog signal.

[0040] See Figure 3 This invention employs multiple sensors 2 for data acquisition to achieve multi-dimensional glass defect detection. According to the above scheme, at least two types of sensors 2 are required. One is an optical sensor to capture optical information from the glass surface; the other, depending on the scheme, can be an infrared sensor, acoustic sensor, electromagnetic wave sensor, or capacitive sensor, etc., to detect the thermal, acoustic, electromagnetic wave, or capacitance distribution information of the glass surface, thereby obtaining multi-dimensional feature information related to defects. The sensor type can be selected according to actual needs.

[0041] The functions of the different sensors are as follows:

[0042] Optical sensors collect light signals from the glass and convert them into electrical signals. These light signals include not only intensity but also spectral distribution, phase distribution, and polarization. Optical sensors also capture optical information from the glass surface, such as surface reflection, transmission, and scattering, and can be used to detect surface defects, bubbles, and foreign objects.

[0043] Infrared sensor: Detects the temperature distribution on the glass surface and can obtain thermal characteristic information related to defects, such as hot cracks and hot spots.

[0044] Acoustic sensors: These can be used to detect sound reflections from glass surfaces. By analyzing the characteristics of sound wave reflections from the glass surface, the presence and approximate location of defects can be determined, such as glass cracks or internal bubbles.

[0045] Electromagnetic wave sensor: By analyzing the electromagnetic wave reflection characteristics of the glass surface, the presence and approximate location of defects can be determined, such as scratches and microcracks on the glass surface.

[0046] Capacitive sensor: detects the capacitance distribution on the glass surface to obtain electrical characteristic information related to defects, such as defects and foreign objects on the glass surface.

[0047] Magnetic sensors: These can detect the distribution of magnetic fields in glass in order to determine whether defects exist.

[0048] Thermal sensors: These can detect temperature changes on the glass surface to determine if defects are present.

[0049] Pressure sensor: can be used to detect pressure changes on the glass surface in order to determine whether there are defects.

[0050] Electromagnetic sensors: These can detect the distribution of electromagnetic fields in glass in order to determine whether defects exist.

[0051] In specific embodiments, the sensors are not limited to those described above; even optical sensors come in different types. These sensors can achieve defect detection by converting different properties of light into electrical signals: for example...

[0052] Photodiode: Converts light intensity into current or voltage signals;

[0053] Photoelectric cell: Converts the energy of light into electrical energy, generating voltage;

[0054] Photomultiplier tube: converts photons into electrons, and enhances the light signal through electron multiplication;

[0055] Optical fiber sensor: Utilizing the light transmission characteristics in optical fibers, it converts external physical quantities (such as temperature, pressure, etc.) into parameters such as light intensity, frequency, or phase, and then converts them into electrical signals;

[0056] Grating sensor: converts the wavelength or frequency of light into an electrical signal;

[0057] CCD (Charge-Coupled Device) sensor: Converts photons into electrical charges, and then converts them into electrical signals through charge transfer and amplification.

[0058] In a multi-dimensional glass defect monitoring system, the data types and characteristics collected by different sensors 2 may vary significantly. To obtain more accurate results in subsequent analysis and processing, these data need to be preprocessed and feature extracted. This invention illustrates how to perform data preprocessing and feature extraction for different sensors in a glass defect detection scenario to obtain the detection signal:

[0059] 1. Optical Sensor

[0060] Preprocessing: For data acquired by the optical sensor, denoising is first performed, for example, using a median filter to remove noise from the image. Next, contrast enhancement is performed to make defects easier to detect in subsequent steps.

[0061] Feature extraction: Extracting edge features (such as Canny edge detection), texture features (such as gray-level co-occurrence matrix), and local binary patterns (LBP) from optical images. These features help to locate and identify defect types.

[0062] 2. Infrared sensor

[0063] Preprocessing: Data acquired by infrared sensors may be affected by factors such as temperature fluctuations. First, normalization is performed to map the image data to a fixed range. Then, noise reduction is performed, for example, using a Gaussian filter.

[0064] Feature extraction: Extracting features from infrared signals, such as temperature distribution and hotspot features. By calculating the temperature gradient in the infrared image, regions with potential defects are identified, and hotspot detection algorithms (such as threshold segmentation) are used to identify abnormal hotspots associated with defects.

[0065] 3. Electromagnetic wave sensor

[0066] Preprocessing: For signals acquired by electromagnetic wave sensors, noise reduction is first performed, for example, by using filters to remove noise components from the signal. Then, signal normalization is performed to map the signal amplitude to a fixed range.

[0067] Feature extraction: Extracting frequency features (such as Fourier transform) and time-domain features (such as signal amplitude and signal strength) from electromagnetic wave signals can help identify glass defects.

[0068] 4. Acoustic Sensor

[0069] Preprocessing: For signals acquired by acoustic sensors, noise reduction is first performed, for example, by using a bandpass filter to remove noise components from the signal. Then, signal normalization is performed to map the signal amplitude to a fixed range.

[0070] Feature extraction: Extracting frequency features (such as Fourier transform) and time-domain features (such as signal amplitude and signal intensity) from acoustic signals. By analyzing the spectral characteristics of sound signals, acoustic anomalies related to glass defects can be detected. Furthermore, extracting time-frequency features of the signal, such as Short-Time Fourier Transform (STFT) or Wavelet Transform (WT), can provide more information about the timing and frequency of defect occurrence.

[0071] 5. Capacitive sensor

[0072] Preprocessing: For signals acquired by capacitive sensors, noise reduction is first performed, for example, by using a low-pass filter to remove high-frequency noise components from the signal. Then, signal normalization is performed to map the signal amplitude to a fixed range.

[0073] Feature extraction: Extracting time-domain features (such as signal amplitude and signal strength) and frequency features (such as Fourier transform) from the capacitance signal. By analyzing changes in the capacitance signal, abnormal capacitance fluctuations related to glass defects can be detected.

[0074] 6. Vibration sensor:

[0075] Preprocessing: For the data collected by the vibration sensor, since the vibration signal generated by the glass defect is weak, wavelet denoising algorithm can be used for preprocessing to effectively eliminate environmental noise and retain the vibration information related to the glass defect.

[0076] Feature extraction: Extracting features of the vibration signal, such as signal energy and time-domain features. For glass defect detection scenarios, the time-domain sliding window method can be used to calculate the energy of the vibration signal, and the presence of a defect can be determined by combining the energy threshold. At the same time, the approximate location of the defect can be determined by the energy distribution.

[0077] Sensor Installation and Calibration

[0078] Calibration is a process of determining the relationship between a sensor's output signal and the actual physical quantity. In this embodiment, the purpose of calibration is to ensure the relationship between the sensor's installation location and the performance of subsequent algorithms.

[0079] The specific method is as follows: First, use a laser tracker or similar device to measure the precise position of the sensor on the side of the glass. Then, place sample glass with known defects at different locations and collect signal data through the sensor. Next, based on the known defect locations and the sensor output signals, establish a relationship model between the sensor output signals and actual physical quantities (such as defect location and size). These model parameters will serve as inputs for subsequent algorithms to ensure their accuracy and stability.

[0080] Establishing a model relating sensor output signals to actual physical quantities (such as defect location and size) requires the use of data processing and machine learning techniques. The specific steps are as follows:

[0081] i. Data Acquisition: By placing sample glass with known defects, sensors are used to collect signal data at different locations. Multiple sets of data can be collected to improve the accuracy and robustness of the model.

[0082] ii. Feature Extraction: The acquired signal data is processed and analyzed to extract features. The selection of features should fully consider the sensitivity to defect location and size, while minimizing sensitivity to noise and interference as much as possible.

[0083] iii. Establishing a Relationship Model: Utilize machine learning algorithms (such as regression analysis, support vector machines, etc.) to establish a relationship model between the sensor output signal and the actual physical quantity. During model training, certain data preprocessing and feature selection techniques are required, along with methods such as cross-validation to evaluate the model's accuracy and generalization ability.

[0084] iv. Model Validation: The established relational model is validated using a new sample glass dataset. The validation results can be used to evaluate the accuracy and stability of the model and to adjust and optimize the model parameters.

[0085] Practical standards and formulas for the number and distribution of sensors

[0086] The number and distribution of sensors need to take into account factors such as signal strength and algorithm computational complexity. The following formula can be used to determine the number of sensors:

[0087] N = (L * H) / (S_min * S_max)

[0088] in:

[0089] N represents the number of sensors, L represents the length of the glass, H represents the height of the glass, S_min represents the minimum spacing between sensors, and S_max represents the maximum spacing between sensors.

[0090] Determining S_min: The minimum spacing S_min between sensors is limited by the range of received signal strength. When sensors are too close together, signal overlap may prevent the differentiation of different defects. Therefore, S_min should satisfy the following condition:

[0091] S_min>=k*d

[0092] Where k is a coefficient greater than 1, and d is the signal attenuation distance in the glass. The value of k can be determined experimentally to ensure that the minimum spacing between sensors can effectively avoid signal overlap.

[0093] A larger value of k results in a larger minimum spacing between sensors, effectively preventing signal overlap. When the signal attenuation distance d is constant, a smaller value of k leads to a smaller value of S_min, resulting in a smaller minimum spacing between sensors and increasing the likelihood of signal overlap, which is detrimental to accurate signal acquisition. Therefore, to avoid signal overlap, a coefficient k greater than 1 needs to be chosen to ensure that the minimum spacing S_min between sensors is sufficiently large.

[0094] Determining S_max:

[0095] The maximum spacing S_max between sensors is limited by computing power. When the sensors are too far apart, the computational complexity of the algorithm decreases, but some defects may go undetected. Therefore, it is necessary to determine a suitable value for S_max to maximize the defect detection rate while ensuring real-time performance. S_max can be determined using the following formula:

[0096] S_max=f*R_max

[0097] Where f is a coefficient less than 1, and R_max is the maximum detection range of the sensor. The value of f can be adjusted according to the actual application scenario and computing resources.

[0098] The value of f determines the sensor's maximum effective detection distance S_max. A smaller f value will reduce the sensor's maximum detection distance but can improve the sensor's accuracy and response speed; a larger f value can increase the sensor's maximum detection distance but may reduce the sensor's accuracy and response speed. Therefore, it is necessary to make reasonable adjustments according to the needs of the actual application scenario.

[0099] In summary, this embodiment can determine the appropriate number and distribution of sensors using the formula N = (L*H) / (S_min*S_max) based on the limitations of S_min and S_max and the size of the glass. This allows for effective detection of glass defects while ensuring real-time performance and detection accuracy.

[0100] This invention provides a deep learning-based glass defect monitoring method. A Long Short-Term Memory (LSTM) network model based on an attention mechanism fuses detection signals from different sensors in series to form a multi-channel signal, which is then input to the LSTM model. This method can predict whether the glass has defects. Specifically, it includes the following:

[0101] 1. Data preprocessing and feature extraction: For the detection signals collected by the various sensors mentioned above, the embodiments of the present invention can perform data preprocessing and feature extraction, and combine these features into a feature vector for input into the graph neural network.

[0102] 2. Constructing the Graph Structure: The glass surface is divided into a set of regular grids, and each grid cell is considered a node in the graph. The connections between nodes represent the relationships between adjacent grid cells. A feature vector is assigned to each node, containing feature data from all sensors, thereby constructing the graph structure of the glass surface.

[0103] 3. Design a graph neural network: Design a multi-task graph neural network (MT-GNN). The network includes an input layer, several graph convolutional layers, and an output layer. The input layer receives the feature vector obtained from the physical signal. The graph convolutional layers are used to learn the relationships between nodes and propagate information. The output layer is used to process the output of the defect prediction task.

[0104] Preferably, in this embodiment of the invention, four task-specific output layers are added on top of the graph neural network to achieve different tasks:

[0105] 3.1 Defect Existence Output Layer: Perform binary classification on each node to determine whether the node has a defect.

[0106] 3.2 Defect approximate location output layer: Perform regression on each node to predict the relative location of the defect within that node (relative to the node's coordinates).

[0107] 3.3 Defect Type Classification Output Layer: Performs multiple classifications on each node to identify the type of defect within that node.

[0108] 3.4 Detailed Defect Location Output Layer: Perform dense regression on each node to predict the detailed location of defects within that node.

[0109] The specific network structure, computation process, and calculation formulas of the Multi-Task Graph Neural Network (MT-GNN), including the design of the objective function and loss function, are as follows:

[0110] 1. Network structure:

[0111] Input layer: Receives the feature vector of each node, which contains preprocessed and feature extraction data from five sensors.

[0112] Graph convolutional layers: These layers contain several graph convolution operations used to learn the relationships between nodes and propagate information. Each graph convolutional layer follows the formula:

[0113] X^(l+1)=σ(AX^(l)W^(l))

[0114] Where X^(l) represents the node feature matrix of the l-th layer, A represents the adjacency matrix, W^(l) represents the weight matrix of the l-th layer, and σ represents the activation function (such as ReLU).

[0115] Task-specific output layers: Four task-specific output layers are added on top of the graph neural network, representing the presence / absence of defects, approximate location of defects, defect type, and detailed location of defects. These output layers can be fully connected layers, each handling the output for a different task.

[0116] 2. The calculation process is as follows:

[0117] 2.1 The input layer receives node feature vectors as input;

[0118] 2.2 Learning and propagating information between nodes through graph convolutional layers;

[0119] 2.3 Task-specific output layers handle the output of different tasks respectively.

[0120] Here, "inter-node information" refers to the relationship and information transmitted between the feature vector of each node and the feature vectors of its surrounding nodes. These feature vectors include preprocessed and feature-extracted data from five sensors, while the transmitted information is learned and propagated by the graph convolutional layers, including adjacency relationships and feature similarities between nodes.

[0121] 3. Objective function and loss function:

[0122] In MT-GNN, each task has a corresponding loss function. This invention can calculate the loss for each task separately and combine them to obtain the total loss.

[0123] The following are the detailed formulas for the loss function for each task:

[0124] 3.1 Does the defect have a predictive loss function?

[0125] This is a binary classification problem, using binary cross-entropy loss as the loss function. It can effectively measure the difference between the model's predicted probabilities and the actual labels, and make the model pay attention to misclassified samples during the optimization process.

[0126] L1=Binary Cross-Entropy Loss(BCELoss)=-Σ[y*log(p)+(1-y)*log(1-p)]

[0127] Where y represents the actual label (0 or 1), and p represents the probability predicted by the model.

[0128] 3.2 Loss function for predicting approximate defect location:

[0129] This is a regression problem, and this embodiment of the invention needs to predict the approximate location of the defect. Therefore, this embodiment of the invention uses Mean Squared Error Loss to measure the difference between the predicted and actual values. This loss function can effectively optimize the model's performance in location prediction.

[0130] L2=Mean Squared Error Loss(MSELoss)=Σ(yp)^2 / N

[0131] Where y represents the actual location, p represents the location predicted by the model, and N represents the number of samples.

[0132] 3.3 Defect Type Classification Prediction Loss Function:

[0133] This is a multi-class classification problem, and this embodiment of the invention requires classification of defect types. Therefore, Categorical Cross-Entropy Loss is used as the loss function. It measures the difference between the model's predicted probabilities and the actual labels across multiple classes, and helps improve the model's performance on classification tasks.

[0134] L3=Categorical Cross-Entropy Loss=-ΣΣy_i*log(p_i)

[0135] Where y_i represents the one-hot encoding of the actual class, and p_i represents the probability predicted by the model.

[0136] 3.4 Loss function for predicting detailed defect location:

[0137] This is a regression problem that requires predicting the precise location of defects. Therefore, Mean Squared Error Loss is used to measure the difference between the predicted and actual values. This loss function can also effectively optimize the model's performance in location prediction.

[0138] L4=Mean Squared Error Loss(MSELoss)=Σ(yp)^2 / N

[0139] Where y represents the actual detailed location, p represents the detailed location predicted by the model, and N represents the number of samples.

[0140] 3.5 Combining the loss functions of the four tasks above, we obtain the total loss function:

[0141] L_total=λ1*L1+λ2*L2+λ3*L3+λ4*L4

[0142] Among them, λ1, λ2, λ3 and λ4 are the weights of the loss function for each task, which can be adjusted according to the actual problem and data.

[0143] Setting the loss function as described above has the following advantages:

[0144] (1) By calculating the loss for each task separately, it can be ensured that the model contributes to the learning of each task. This helps to improve the model's performance on each task.

[0145] (2) Using the total loss function can balance the importance of different tasks. By adjusting the weights of the loss functions for each task, the performance of the model on different tasks can be optimized.

[0146] (3) The total loss function helps to achieve end-to-end training, enabling the model to learn multiple tasks in a unified framework, thus improving the efficiency of training and prediction.

[0147] 4. Training and Prediction:

[0148] Train the MT-GNN using a labeled dataset and minimize the total loss function.

[0149] In the prediction phase, the sensor feature vector of the glass to be detected is input, and the prediction results for each task are obtained through MT-GNN:

[0150] a) Defect Prediction: Output the binary classification probability of each node. A threshold (e.g., 0.5) can be set to determine whether a defect exists.

[0151] b) Defect approximate location prediction: Output the relative position of the defect within each node (coordinates relative to the node).

[0152] c) Defect type classification prediction: Output the multi-class probability of each node, and determine the defect type based on the category with the highest probability.

[0153] d) Detailed defect location prediction: Outputs the detailed location of defects within each node.

[0154] Analysis and prediction results:

[0155] Based on whether or not a defect is predicted, the nodes on the glass where defects exist are identified.

[0156] For defective nodes, the approximate and detailed defect location prediction results are used to determine the actual location of the defect on the glass.

[0157] The multi-task graph neural network, implemented in this embodiment of the invention via MT-GNN, provides a unified framework capable of simultaneously processing data from different sensors and performing multiple defect detection tasks. This method leverages the advantages of graph neural networks in spatial information processing to effectively fuse and analyze sensor data. Furthermore, the multi-task learning design allows the network to learn multiple tasks within a unified framework, improving training and prediction efficiency.

[0158] Example 2

[0159] This embodiment utilizes the characteristics of spectral images to perform optical inspection of the glass without lenses or external light sources using natural or ambient light, thereby enabling the identification of surface and internal defects in the glass.

[0160] In this embodiment, in step S1, a spectral image sensor is used to acquire a spectral image of the glass; in step S2, the physical properties of the glass surface and interior are identified based on the spectral distribution of light reflected, absorbed, or transmitted by the glass, using the characteristics of the spectral image, thereby detecting defects in the glass.

[0161] Specifically, in step S1, a set of multispectral cameras is set up to capture spectral images of the glass from different angles and distances; the multispectral cameras scan along multiple different directions. To ensure comprehensive detection of both the surface and interior of the glass, the cameras are designed to scan in multiple directions, such as up and down, left and right, and front and back.

[0162] In this embodiment, step S2 employs an Adaptive Convolutional Neural Network (ACNN) to extract and classify spectral image features, thereby achieving accurate defect identification. The ACNN comprises a series of convolutional layers and fully connected layers. Each convolutional layer extracts specific features of the image, such as lines, colors, and textures, while the fully connected layers integrate these features for classification. The ACNN features adaptive convolutional layers that automatically adjust the kernel size and stride based on the characteristics of the input image, thus better extracting important features and improving recognition accuracy.

[0163] In some embodiments of the present invention, step S2 further includes: processing the feature sequences of the spectral images using a Temporal Convolutional Neural Network (TCNN) to extract temporal information reflecting the defect development process. First, continuous spectral images are converted into feature sequences, which are then input into the TCNN. In the TCNN structure, multiple convolutional and recurrent layers are used to process the input feature sequences, thus capturing temporal information in the sequences, such as the development trend of defects. For training the TCNN, backpropagation and stochastic gradient descent are used to update the network parameters by optimizing the loss function, thereby enabling the network to better identify and predict defect development.

[0164] In summary, the method in this embodiment achieves accurate and efficient glass defect monitoring by integrating lensless and external light source-free detection, adaptive deep convolutional neural networks, and temporal convolutional neural networks.

[0165] Example 3

[0166] The other contents of this embodiment are the same as those in embodiments 1 and 2, and the same contents will not be repeated.

[0167] See Figure 2 Preferably, the embodiments of the present invention include performing an early warning operation based on the above judgment result, specifically including: receiving defect prediction results from a multi-task graph neural network and integrating them to form a defect information list; traversing the defect information list and determining whether to trigger an early warning based on a preset early warning threshold and the confidence level of the defect; when an early warning is triggered, performing one or more operations among displaying, sending, and recording the early warning information.

[0168] The specific plan for early warning operations can be implemented using the following steps:

[0169] (1) Set warning thresholds: Set a warning threshold for each type of defect based on historical data and experience. For example, a confidence level greater than 0.8 is a general warning and a confidence level greater than 0.9 is a severe warning.

[0170] (2) Defect Information Integration: The early warning module receives defect prediction results from the multi-task graph neural network (MT-GNN). It integrates information such as defect type, location, and confidence level to form a defect information list.

[0171] (3) Design an early warning strategy: Traverse the defect information list and determine whether to trigger an early warning based on a preset threshold. For example, if the confidence level of a defect is greater than 0.8, a general early warning is triggered; if the confidence level is greater than 0.9, a severe early warning is triggered.

[0172] (4) Early warning information presentation: Design a real-time monitoring interface to display defect information. The defect type, location, confidence level, etc. can be listed in a table format, and different levels of early warning can be represented by colors or icons.

[0173] (5) Early Warning Notification: When an early warning is triggered, relevant personnel should be notified promptly. Early warning information can be sent via SMS gateways, email servers, or mobile app push notifications. Simultaneously, the early warning information should be recorded in a database or log file for subsequent analysis.

[0174] The embodiments of the present invention will be further described below with reference to the following examples. It should be emphasized that the embodiments described below are merely exemplary embodiments of the present invention, and the present invention is not limited to this embodiment.

[0175] Example 4

[0176] The other contents of this embodiment are the same as those of embodiments 1, 2 and 3, and the same contents will not be repeated.

[0177] The glass defect detection method in this embodiment can be used for various types of glass, as exemplified below:

[0178] 1. Architectural glass: Used in building exterior walls, curtain walls, etc., it is necessary to inspect whether its surface is flat and whether there are defects such as bubbles and cracks.

[0179] 2. Optical glass: Used to manufacture various optical devices, such as lenses, prisms, and filters. It is necessary to inspect whether its surface is smooth and whether there are defects such as scratches or bumps.

[0180] 3. Smart glass: Used in electronic displays, smart homes and other scenarios, it is necessary to detect whether its surface is flat and whether there are defects such as scratches and stains.

[0181] I. Sensors

[0182] See Figure 3 First, a suitable sensor 2 needs to be selected for different types of glass 1. The sensor 2 is then installed on the side of the glass 1. The installation of the sensor includes the following aspects:

[0183] 1. Sensor Selection Principles and Examples

[0184] When selecting appropriate sensors for different types of glass, it is generally believed that for automotive glass defect detection, the sensor sensitivity should reach 0.1mm and the accuracy should reach approximately 0.01mm. Of course, specific analysis and evaluation are required based on different application scenarios and needs. In addition to considering the sensor's sensitivity and accuracy for defect detection, the following aspects also need to be considered:

[0185] (1) Sensor size: The size of the sensor will affect the installation process and the actual application scenario. In a limited space, choosing a smaller sensor may be easier to install. At the same time, a smaller sensor also helps to reduce the impact on the overall aesthetics of the glass.

[0186] (2) Ease of installation: The installation method of the sensor may vary depending on the application scenario. Choosing an easy-to-install sensor can reduce installation difficulty and cost, and improve the practicality of the entire system. For example, for glass defect detection, an optical sensor integrated on a small board can be selected. This sensor is of moderate size and easy to install on the glass surface. At the same time, the cable of this sensor is relatively short, which is convenient for wiring, and it has an automatic calibration function, which can avoid the time and labor costs of manual calibration.

[0187] (3) Environmental adaptability: Sensors need to adapt to various environmental conditions, such as temperature, humidity and light. When selecting a sensor, its stability and reliability in the actual application environment should be considered.

[0188] For sensors with good temperature adaptability, they should have a certain temperature range, and the output value within this range should be stable and not excessively affected by temperature. For example, some temperature sensors are typically used in the range of -40℃ to +125℃ and have temperature compensation functions to ensure output accuracy and stability.

[0189] For sensors with good humidity adaptability, they should be able to provide stable output under different humidity levels. For example, some humidity sensors can measure humidity in the range of 0-100% and can compensate for changes in environmental factors such as temperature and air pressure.

[0190] For sensors with good light adaptability, they should be able to operate under different light intensities, while maintaining good stability and accuracy under different colors of light. For example, some light sensors can measure light intensity under different light levels and have the function of automatically adjusting sensitivity to adapt to different lighting environments.

[0191] Example:

[0192] Suppose we need to design a defect detection system for a car's windshield. In this case, a compact optical sensor can be considered. This type of sensor is small in size, making it easy to install on the side of the glass with minimal impact on the overall aesthetics. Furthermore, compact optical sensors typically have good environmental adaptability, operating normally under various temperature, humidity, and lighting conditions. Finally, considering the vibrations and shocks that a car's windshield may encounter, the stability and reliability of the compact optical sensor are also important factors.

[0193] 2. Sensor Installation and Calibration

[0194] Calibration is a process of determining the relationship between a sensor's output signal and the actual physical quantity. In this embodiment, the purpose of calibration is to ensure the relationship between the sensor's installation location and the performance of subsequent algorithms.

[0195] The specific method is as follows: First, use a laser tracker or similar device to measure the precise position of the sensor on the side of the glass. Then, place sample glass with known defects at different locations and collect signal data through the sensor. Next, based on the known defect locations and the sensor output signals, establish a relationship model between the sensor output signals and actual physical quantities (such as defect location and size). These model parameters will serve as inputs for subsequent algorithms to ensure their accuracy and stability.

[0196] Establishing a model relating sensor output signals to actual physical quantities (such as defect location and size) requires the use of data processing and machine learning techniques. The specific steps are as follows:

[0197] i. Data Acquisition: By placing sample glass with known defects, sensors are used to collect signal data at different locations. Multiple sets of data can be collected to improve the accuracy and robustness of the model.

[0198] ii. Feature Extraction: The acquired signal data is processed and analyzed to extract features. The selection of features should fully consider the sensitivity to defect location and size, while minimizing sensitivity to noise and interference as much as possible.

[0199] iii. Establishing a Relationship Model: Utilize machine learning algorithms (such as regression analysis, support vector machines, etc.) to establish a relationship model between the sensor output signal and the actual physical quantity. During model training, certain data preprocessing and feature selection techniques are required, along with methods such as cross-validation to evaluate the model's accuracy and generalization ability.

[0200] iv. Model Validation: The established relational model is validated using a new sample glass dataset. The validation results can be used to evaluate the accuracy and stability of the model and to adjust and optimize the model parameters.

[0201] 3. Practical standards and formulas for the number and distribution of sensors

[0202] The number and distribution of sensors need to take into account factors such as signal strength and algorithm computational complexity. The following formula can be used to determine the number of sensors:

[0203] N = (L * H) / (S_min * S_max)

[0204] in:

[0205] N represents the number of sensors, L represents the length of the glass, H represents the height of the glass, S_min represents the minimum spacing between sensors, and S_max represents the maximum spacing between sensors.

[0206] Determining S_min: The minimum spacing S_min between sensors is limited by the range of received signal strength. When sensors are too close together, signal overlap may prevent the differentiation of different defects. Therefore, S_min should satisfy the following condition:

[0207] S_min>=k*d

[0208] Where k is a coefficient greater than 1, and d is the signal attenuation distance in the glass. The value of k can be determined experimentally to ensure that the minimum spacing between sensors can effectively avoid signal overlap.

[0209] A larger value of k results in a larger minimum spacing between sensors, effectively preventing signal overlap. When the signal attenuation distance d is constant, a smaller value of k leads to a smaller value of S_min, resulting in a smaller minimum spacing between sensors and increasing the likelihood of signal overlap, which is detrimental to accurate signal acquisition. Therefore, to avoid signal overlap, a coefficient k greater than 1 needs to be chosen to ensure that the minimum spacing S_min between sensors is sufficiently large.

[0210] Determining S_max:

[0211] The maximum spacing S_max between sensors is limited by computing power. When the sensors are too far apart, the computational complexity of the algorithm decreases, but some defects may go undetected. Therefore, it is necessary to determine a suitable value for S_max to maximize the defect detection rate while ensuring real-time performance. S_max can be determined using the following formula:

[0212] S_max=f*R_max

[0213] Where f is a coefficient less than 1, and R_max is the maximum detection range of the sensor. The value of f can be adjusted according to the actual application scenario and computing resources.

[0214] The value of f determines the sensor's maximum effective detection distance S_max. A smaller f value will reduce the sensor's maximum detection distance but can improve the sensor's accuracy and response speed; a larger f value can increase the sensor's maximum detection distance but may reduce the sensor's accuracy and response speed. Therefore, it is necessary to make reasonable adjustments according to the needs of the actual application scenario.

[0215] In summary, this embodiment can determine the appropriate number and distribution of sensors using the formula N = (L*H) / (S_min*S_max) based on the limitations of S_min and S_max and the size of the glass. This allows for effective detection of glass defects while ensuring real-time performance and detection accuracy.

[0216] II. Signal Processing and Defect Detection Algorithms

[0217] First, signal data is acquired from sensors, and then information about glass defects is extracted using signal processing techniques. Next, defect detection algorithms are used for identification, including the use of machine learning and / or other methods.

[0218] 1. Signal Acquisition and Preprocessing

[0219] (1) Adaptive Filter: The Least Mean Square (LMS) algorithm can be used, which updates the filter coefficients in each iteration to minimize the mean square error of the output signal. The LMS algorithm update formula is:

[0220] W(n+1) = W(n) + μ*e(n)*X(n)

[0221] Where W(n) are the filter coefficients, μ is the step size parameter, e(n) is the error signal, and X(n) is the input signal.

[0222] (2) Empirical Mode Decomposition (EMD): First, find all extreme points of the signal (including maxima and minima). Then, calculate the envelope curves between the maxima and minima using cubic spline interpolation. Next, calculate the mean of the upper and lower envelope curves, and subtract the mean curve from the original signal to obtain the first intrinsic mode function (IMF). Repeat this process until the signal is decomposed into multiple IMFs.

[0223] Adaptive filtering: Adaptive filtering is a nonlinear filtering method that automatically adjusts the filter parameters according to the characteristics of the signal, thereby removing noise while retaining important signal information. In glass defect detection, adaptive filtering can remove signal noise and interference caused by uneven lighting, camera angle, etc., thereby improving signal clarity and accuracy.

[0224] Empirical Mode Decomposition (EMD): Empirical Mode Decomposition is a signal decomposition method based on local signal features. It can decompose the original signal into several intrinsic mode functions (EMFs) and a residual term. In glass defect detection, EMD can decompose the original signal into different local signal features, thereby better extracting defect information from the signal and reducing the impact of interference on defect detection. Simultaneously, EMD can effectively remove noise and interference from the signal, improving its accuracy and stability.

[0225] 2. Defect Detection Algorithm

[0226] Using learning-based methods

[0227] (1) Attention Mechanism: A weighted vector α is introduced to calculate the weights between different signal channels. The channel signals can include information collected by sensors such as temperature, humidity, and light intensity. Different channels represent different signal channels. The attention weights are calculated using the Softmax function.

[0228] α_i = exp(e_i) / Σ(exp(e_j))

[0229] Where e_i is the original weight value of the i-th channel, and j represents the channel index.

[0230] (2) Long Short-Term Memory (LSTM) Network: LSTM consists of input gates, output gates, and forget gates, and can effectively process long sequences of data. At each time step, LSTM updates the state of each gate based on the current input and the hidden state of the previous time step. The output gate determines which information will be passed to the hidden state of the next time step, the forget gate determines which information will be discarded from the cell state, and the input gate determines which information will be added to the cell state. The key formula of LSTM is as follows:

[0231] f_t=σ(W_f *[h_(t-1),x_t]+b_f) Forget Gate

[0232] i_t=σ(W_i*[h_(t-1),x_t]+b_i) Input gate

[0233] o_t=σ(W_o*[h_(t-1),x_t]+b_o) Output gate

[0234] c_t=f_t*c_(t-1)+i_t*tanh(W_c*[h_(t-1),x_t]+b_c) Cell state update

[0235] h_t = o_t * tanh(c_t) Hidden state update

[0236] These formulas describe the LSTM neural network model, used to process sequential data, such as time series data or text data. Here, t represents a time step, h_t represents the hidden state at time step t, and x_t represents the input data. f_t, i_t, and o_t are mechanisms called gates, used to control the flow of information in the model. The forget gate (f_t) determines how much information is retained in the cell state of the previous time step, the input gate (i_t) determines how much new information is added to the cell state at the current time step, and the output gate (o_t) determines how much information from the hidden state of the current time step is passed to the next time step. c_t represents the cell state of the current time step, which is a combination of the cell state of the previous time step and the input information of the current time step. Finally, the LSTM model calculates the cell state and hidden state of the current time step using the above formulas for use in the calculation of the next time step.

[0237] Where σ represents the sigmoid activation function, f_t, i_t, and o_t are the states of the forget gate, input gate, and output gate, respectively, c_t is the cell state, h_t is the hidden state, and W and b represent the weight matrix and bias term, respectively.

[0238] For glass defect detection, the signals from various sensors are first concatenated to form a multi-channel signal. Then, an attention mechanism is introduced, enabling the LSTM network to focus on the importance of different channel signals at each time step. During training, the network automatically learns the signal characteristics and temporal information related to defects.

[0239] The attention-based LSTM model in this invention is based on Long Short-Term Memory (LSTM) networks and an attention mechanism. In this model, signals collected from multiple sensors are first concatenated to form a multi-channel signal, which is then used as input to the LSTM model. The LSTM model consists of three parts: an input gate, a forget gate, and an output gate, and can process time-series data. At each time step, the LSTM model updates the state of each gate based on the current input and the hidden state of the previous time step, and outputs the hidden state of the current time step.

[0240] To better assess the importance of signals from different channels, this model introduces an attention mechanism. This mechanism uses a weighted vector α to calculate the weights between different signal channels. The attention weights are calculated using a softmax function and then summed with the input signal to obtain the weighted input signal. This allows the LSTM model to better assess the importance of signals from different channels at each time step.

[0241] During training, the model updates its weight parameters via backpropagation, thereby automatically learning signal features and temporal information related to defects. During testing, for each new data point, the model inputs it into the LSTM network and calculates the activation value of the output layer. The activation value of the output layer is normalized using the Softmax function.

[0242] A well-trained attention-based Long Short-Term Memory (LSTM) network model can be used to predict whether new data contains defects. Specifically, for each new data point, it can be input into the LSTM network, and the activation value of the output layer can be calculated. The activation value of the output layer can be normalized using the Softmax function to obtain the probability distribution of the presence or absence of defects.

[0243] III. Real-time Early Warning System

[0244] For specific scenarios, such as automotive glass, existing methods cannot meet the needs of real-time detection. During vehicle operation, glass may develop defects due to impacts, abrasion, or other reasons. These defects can obstruct vision, reduce optical performance, and even pose safety hazards. Therefore, for applications such as automotive glass, a method is needed that can detect defects in real time during use, enabling timely warnings and intervention when problems are detected.

[0245] This section describes how to transmit detected glass defect information to users in real time so as to provide early warnings during use, and introduces the specific implementation method of the real-time warning system, as well as how to adapt it to different scenarios (such as automotive glass).

[0246] 1. Warning Level Classification

[0247] To better convey early warning information, the warning level is divided into three levels, each representing a different degree of urgency of the glass defect.

[0248] (1) Low risk (green): The defect is small and does not affect the normal use of the glass, but it needs to be monitored to prevent the problem from worsening.

[0249] (2) Medium risk (yellow): The defects are obvious and may affect the normal use of the glass. They need to be resolved in the short term.

[0250] (3) High risk (red): The defect is serious and may cause damage to the glass, requiring immediate action.

[0251] 2. Issue an early warning

[0252] The method of issuing warnings needs to be adjusted according to different scenarios and warning levels. Here's an example:

[0253] (1) Vehicle glass:

[0254] Low risk: A green icon is displayed on the car's dashboard to alert the driver;

[0255] Medium risk: A yellow icon appears on the car's dashboard, accompanied by a slight alarm sound, prompting the driver to resolve the issue as soon as possible;

[0256] High risk: A red icon appears on the car's dashboard, accompanied by a continuous emergency alarm sound, prompting the driver to stop immediately and check.

[0257] (2) Building glass:

[0258] Low risk: Send SMS or email notifications to relevant personnel through the building's property management system;

[0259] Medium risk: Send a text message or email to the relevant person in charge through the building's property management system and request that the problem be resolved within a certain period of time;

[0260] High risk: Send a text message or email to the relevant person in charge through the building's property management system, and request that professional personnel be organized immediately for inspection and repair.

[0261] 3. Implement early warning notification

[0262] Vehicle glass: The warning information is sent to the vehicle's dashboard via the vehicle's built-in communication module (such as vehicle Bluetooth, Wi-Fi (Wireless Fidelity), etc.) to display the warning level and prompt information in real time.

[0263] Building glass: The warning information is sent to the building property management system via wireless network (such as Wi-Fi, (Long Range Radio, LoRa) long-distance wireless power supply, etc.), and the system automatically sends the corresponding notification according to the warning level.

[0264] Through the design and implementation of the above real-time early warning system, appropriate early warning methods can be adopted according to different scenarios and early warning levels to ensure that early warning information is promptly delivered to users when glass defects occur, thereby improving users' safety awareness and reducing potential risks.

[0265] 4. Automatic updating and tracking of early warning information

[0266] To ensure that users are always aware of the latest status of glass defects, the real-time early warning system also needs to have automatic update and tracking functions.

[0267] Automatic updates: The system will periodically collect and process sensor signal data to update early warning information in real time. When the warning level changes, the system will automatically adjust the warning notification to ensure that users are always aware of the current risk situation.

[0268] Tracking Processing Progress: For glass defects for which warnings have been issued, the system can track the processing progress, helping users monitor the resolution status. For example, for building glass, the property management system can update the warning information and notify relevant personnel after the problem is resolved. For vehicle glass, users can view the processing progress through the car's dashboard and receive a notification after the problem is resolved.

[0269] 5. Adaptability to different environments and scenarios

[0270] A real-time early warning system should possess good adaptability to function properly in different environments and scenarios. Specifically, the system needs to consider the following factors:

[0271] (1) Transmission distance: The transmission distance of the warning information may vary depending on the scenario (such as car windows, building windows, etc.). The system should be able to support data transmission over different distances, such as through multiple communication methods such as Bluetooth and Wi-Fi.

[0272] (2) Environmental factors: The system needs to cope with various environmental factors, such as temperature, humidity, and light. For example, for outdoor building glass, the system should have waterproof and dustproof functions to ensure normal operation in harsh environments.

[0273] (3) Equipment compatibility: The real-time early warning system needs to be compatible with various devices, such as car dashboards and property management systems. In addition, different brands and models of equipment need to be considered to ensure that the early warning information can be displayed and transmitted correctly.

[0274] By taking the above factors into account, a highly adaptable real-time early warning system can be designed, providing users with timely and accurate early warning information in various scenarios and environments.

[0275] IV. System Integration and Optimization

[0276] 1. System Integration

[0277] System integration is the process of seamlessly integrating a glass defect detection system with existing equipment, such as automotive and building property management systems. This helps ensure that real-time monitoring and early warning functions operate correctly in practical applications. Here are some key integration steps and considerations:

[0278] ① Equipment Interface: To ensure compatibility with existing equipment, specific hardware and software interfaces need to be developed for the glass defect detection system. This includes communication interfaces with devices such as car dashboards and property management systems to enable real-time transmission of early warning information.

[0279] ② Installation and Layout: To ensure proper sensor installation and integration with existing equipment, the sensor's layout and installation method need to be considered. This may involve installation guidelines and technical support for different scenarios such as automotive glass and building glass.

[0280] ③ Power Consumption Management: Due to the limited power resources of existing equipment (such as automobiles), it is necessary to optimize the power consumption of the glass defect detection system. This can be achieved by reducing sensor power consumption and optimizing algorithm performance, ensuring the system operates normally under limited energy conditions.

[0281] ④ User Interface: To facilitate user access to and understanding of warning information, an intuitive and easy-to-use user interface needs to be designed. This may include warning displays on car dashboards, real-time monitoring interfaces in property management systems, etc.

[0282] 2. System Optimization

[0283] The goal of system optimization is to improve the performance of the glass defect detection system, including reducing false alarms and false negatives, and increasing detection speed and accuracy. The following are some areas for further development:

[0284] (1) Algorithm optimization: By continuously optimizing signal processing and defect detection algorithms, detection accuracy and speed can be improved. This may include using more efficient signal processing techniques and developing more powerful machine learning models.

[0285] Use more efficient signal processing techniques: multi-channel signal processing, wavelet transform, Fourier transform, and other methods can be employed to improve signal processing efficiency and accuracy.

[0286] Developing more powerful machine learning models: Deep learning, reinforcement learning, and other methods can be used to improve the accuracy and speed of defect detection. Furthermore, transfer learning, incremental learning, and other methods can be employed to address real-time changes in the environment and data.

[0287] (2) Sensor calibration: Regularly calibrate and maintain the sensor to ensure stable performance. This helps reduce false alarms and false negatives, and improves detection accuracy.

[0288] Regularly calibrate and maintain the sensor: This can be achieved by using standard calibration plates and regular maintenance to ensure the stability and accuracy of the sensor's performance.

[0289] Multi-sensor fusion: Multi-sensor fusion techniques, such as Kalman filtering, can be used to eliminate errors and uncertainties between sensors and improve detection accuracy.

[0290] (3) Dynamic threshold adjustment: The defect detection threshold is dynamically adjusted according to the actual application environment and conditions. This helps to reduce false alarms and false negatives and improve detection accuracy.

[0291] Based on the actual application environment and conditions, the defect detection threshold can be dynamically adjusted: adaptive algorithms, such as sliding window and dynamic threshold, can be used to improve detection accuracy and robustness.

[0292] Employing multi-threshold decision-making: Multi-threshold decision-making techniques, such as interval-based multi-threshold decision-making and confidence-based multi-threshold decision-making, can be used to further improve detection accuracy and robustness.

[0293] (4) Data fusion: By fusing data from multiple sensors, the accuracy and robustness of detection can be improved. This can be achieved through data fusion techniques such as weighted averaging and Kalman filtering.

[0294] Weighted average: Weighted average of data from multiple sensors to reduce noise and errors and improve detection accuracy.

[0295] Kalman filtering: The Kalman filtering method is used to fuse data from multiple sensors, eliminate sensor noise and errors, and improve detection accuracy and robustness.

[0296] Adaptive early warning strategy: Design adaptive early warning strategies for different scenarios and application needs. For example, during car driving, adjust the early warning level and notification method in real time based on factors such as vehicle speed and road conditions; in building property management scenarios, set appropriate urgency levels and notification strategies for defects based on information such as floor and area.

[0297] (6) Model Update and Online Learning: The defect detection model is continuously learned and updated online by collecting and analyzing new data. This helps improve the system's performance and adaptability, and reduces false positives and false negatives.

[0298] Establish a real-time data acquisition and analysis system to collect real-time data from sensors and process and analyze it.

[0299] By leveraging machine learning algorithms and model update techniques, new data can be used to learn and update the model in real time, thereby improving the model's accuracy and robustness.

[0300] For different application scenarios and devices, specific learning strategies and update methods can be customized, such as model update technologies based on incremental learning or online learning.

[0301] (7) Performance Monitoring and Feedback: Monitor system performance in real time and collect user feedback. Continuously optimize the system and improve user experience through analysis of performance indicators and user needs.

[0302] Collect user feedback and suggestions, use methods such as user surveys and user behavior analysis to analyze user needs and preferences, and continuously optimize the system based on user feedback.

[0303] Based on monitoring results and user feedback, design appropriate performance indicators and evaluation methods to assess system performance and user experience, so as to continuously improve system performance and user satisfaction.

[0304] In summary, the system integration and optimization phase requires consideration of compatibility with existing equipment, sensor layout, power management, and user interface. Simultaneously, improvements to the glass defect detection system's performance and user experience can be achieved through algorithm optimization, sensor calibration, dynamic threshold adjustment, data fusion, adaptive early warning strategies, model updates and online learning, and performance monitoring and feedback.

[0305] V. Adapting to Different Environmental Conditions

[0306] 1. Enhance the robustness and stability of the algorithm itself without adding components:

[0307] 1.1 Data Preprocessing: First, high-frequency noise is eliminated by using a smoothing filter, such as a median filter, mean filter, or Gaussian filter. Then, baseline drift correction is performed on the signal using the difference method or other methods.

[0308] Choosing different smoothing filters will have varying effects on the smoothness and spectral response of the signal, requiring the selection of an appropriate filter based on the specific circumstances. For example, a Gaussian filter can smooth a signal while preserving high-frequency components, while a median filter can remove noise without altering the signal's characteristics. Furthermore, the filter parameters need to be adjusted according to the signal's frequency and amplitude characteristics to achieve the best results.

[0309] 1.2 Outlier Detection and Removal: Tukey's fence method or Z-score method is used to detect and remove outliers from the signal. This reduces the impact of environmental factors on the signal and improves the algorithm's robustness.

[0310] Other outlier detection methods are available, such as Grubb's test, Dixon's Q test, and Hampelfilter. Afterward, an impact assessment of the data after outlier removal is necessary to determine the effectiveness of the outlier removal. Box plots and statistical graphs can be used to compare the distribution, mean, variance, and other statistical characteristics of the data before and after outlier removal, evaluating the reliability and stability of the data after outlier removal.

[0311] 1.3 Adaptive Threshold Adjustment: The defect detection threshold is adjusted in real time based on environmental parameters (such as light intensity and temperature). For example, a relationship model between the threshold and environmental parameters can be established through linear regression or other regression methods to dynamically adjust the threshold.

[0312] To implement adaptive threshold adjustment, it is necessary to collect and analyze data on environmental parameters (such as light intensity and temperature) and establish a relationship model between the threshold and these parameters. This relationship model can be established using methods such as linear regression, multinomial regression, and neural networks. Specifically, the following steps can be taken:

[0313] Collect environmental parameter data: In practical applications, environmental parameter data related to defect detection should be collected. For example, in glass defect detection, data such as light intensity and ambient temperature need to be collected. This data can be collected through sensors or obtained from other data sources.

[0314] Establishing a relationship model: Using the collected environmental parameter data and corresponding threshold labels as the training set, a regression model is used to establish the relationship between the thresholds and environmental parameters. Commonly used regression models include linear regression, multinomial regression, and neural networks.

[0315] Model parameter training: The parameters of the established relational model are trained to optimize the model's fitting ability. During training, data preprocessing is required, such as normalization and standardization, to avoid the impact of data scale differences on the model.

[0316] Threshold adjustment: The trained model is applied to a real-world environment, and the threshold is dynamically adjusted based on real-time collected environmental parameter data. Specifically, a model prediction method can be used to predict the corresponding threshold based on the current environmental parameter data, and then the detection threshold is dynamically adjusted.

[0317] Threshold adaptability testing: The adjusted threshold is tested to evaluate its performance and adaptability under different environments. For example, performance indicators such as accuracy and false alarm rate of the threshold are tested under different lighting and temperature conditions.

[0318] Parameter update: Based on test results and user feedback, update and optimize model parameters and thresholds to improve the robustness and performance of the algorithm.

[0319] In addition, it is necessary to consider how to collect environmental parameter data in real time, and how to preprocess and store it. Sensor data acquisition and processing can be implemented using microcontrollers or embedded systems, while data storage can be achieved using databases or file systems.

[0320] 1.4 Integrating Multiple Algorithms: Combining physics-based and learning-based methods, and through ensemble learning methods such as voting strategies, bootstrap aggregating (Bagging) algorithms, or boosting methods, the advantages of multiple algorithms are integrated to improve the stability and accuracy of the system under different environmental conditions.

[0321] 2. Possible solutions and analysis when adding components:

[0322] 2.1 Environmental Monitoring Sensors: Install light sensors (such as photoresistors, photodiodes, etc.) to measure ambient light intensity; install temperature sensors (such as NTC thermistors, thermocouples, etc.) to monitor ambient temperature. This data will be used to assist in adjusting algorithm parameters and improve the system's adaptability to environmental changes.

[0323] 2.2 Optical Filter: Based on the spectral response characteristics of the signal acquisition sensor, select a suitable narrowband optical filter to reduce the influence of ambient light on the sensor signal and improve signal quality.

[0324] 2.3 Sensor Heating or Cooling Device: Install heating elements or Peltier (thermal energy storage) coolers on the sensor to maintain its operation within a suitable temperature range. A closed-loop control system monitors the sensor temperature in real time and adjusts the power of the heating / cooling device to improve the system's stability and reliability under different temperature environments.

[0325] 2.4 Moisture-proofing: Methods such as using silica gel desiccant, moisture-proof coatings, or sealed boxes are employed to prevent sensors and circuits from getting damp. Moisture-proofing helps improve the stability and reliability of the system in humid environments.

[0326] 2.5 Redundancy Design: Multi-sensor redundancy is employed in key components, such as using dual sensors to simultaneously monitor the same area and then fusing their outputs. Redundancy design improves the system's reliability and anti-interference capability in harsh environments.

[0327] 2.6 High-Dynamic Range (HDR) Imaging: Installing an HDR module on camera-type sensors enables them to acquire clear images under high-contrast lighting conditions, reducing the impact of ambient light on image quality.

[0328] 2.7 Early Warning System Adapts to Environmental Changes: The early warning system should be able to dynamically adjust the warning level and notification method based on environmental parameters (such as light intensity and temperature). For example, under high light conditions, the warning level can be increased, and more obvious visual and auditory cues can be used to remind users; in low-temperature environments, warning information can be sent through the vehicle information system or mobile devices to prevent users from being exposed to cold environments for extended periods.

[0329] VI. Scalability and Applicability

[0330] 1. Adaptable to different types of glass

[0331] For different types of glass (such as architectural glass and household glass), wider applicability can be achieved by adjusting the sensor type and parameters. For example, for architectural glass, a sensor with higher resolution can be selected to detect finer cracks and defects; for household glass, a low-power sensor can be used to reduce energy consumption while achieving higher detection accuracy.

[0332] 2. Adjust sensor configuration

[0333] Depending on the requirements of different application scenarios, the number and layout of sensors can be adjusted to achieve the best detection results. For example, for large building glass, the number and density of sensors can be increased to cover a larger area;

[0334] For smaller pieces of glass in homes, the number of sensors can be reduced to lower costs and complexity.

[0335] 3. Customized detection methods and algorithms

[0336] Detection methods and algorithms can be customized for different scenarios and applications. For example, in architectural glass applications, detection algorithms can be developed for specific types of cracks and defects (such as stress cracks, thermal fatigue, etc.).

[0337] In home glass applications, it can detect common defects such as stains and scratches.

[0338] The above methods can achieve scalability and applicability of glass defect detection systems in different types of glass and application scenarios, thus providing broader application potential.

[0339] VII. Application

[0340] This embodiment expands upon various application areas and integrates with other related technologies.

[0341] 1. Technology Optimization

[0342] High-performance sensors: With the continuous development of sensor technology, higher resolution and sensitivity sensors can be used in the future to improve the accuracy and efficiency of defect detection.

[0343] Deep learning algorithms: By introducing deep learning algorithms, such as Convolutional Neural Networks (CNN) and Long Short-Term Memory Networks (LSTM), the accuracy and robustness of the algorithms can be further improved.

[0344] Adaptive Algorithms: Develop adaptive algorithms that can automatically adjust parameters according to environmental changes in order to improve system performance in complex environments.

[0345] 2. Expansion of application areas

[0346] New building materials: Apply glass defect detection technology to the detection of other new building materials, such as photovoltaic glass and high-performance glass fiber.

[0347] Industrial applications: In industrial production processes, glass defect detection technology can be used for real-time monitoring and early warning, improving production efficiency and quality.

[0348] In the security field: By combining technologies such as video surveillance, glass defect detection technology can be applied to the security field, such as monitoring the safety condition of glass in buildings or vehicles.

[0349] 3. Integration with other related technologies

[0350] Internet of Things (IoT): By combining glass defect detection systems with IoT technology, remote monitoring and management can be achieved, improving early warning efficiency.

[0351] Big data analytics: Utilizing big data analytics technology to collect and analyze large amounts of glass defect detection data, thereby providing a basis for optimizing algorithms and developing prevention strategies.

[0352] Artificial intelligence: Integrating glass defect detection technology with other artificial intelligence technologies (such as image recognition, natural language processing, etc.) to achieve a higher level of automation and intelligence.

[0353] This embodiment has wide applications in the following fields:

[0354] Lens-less and light-free glass defect monitoring: With the development of lens-less and light-free monitoring technologies, low-cost and high-efficiency monitoring of glass defects can be achieved, expanding its application potential in various application scenarios.

[0355] New building materials: Apply glass defect detection technology to the detection of other new building materials, such as photovoltaic glass and high-performance glass fiber.

[0356] Industrial applications: In industrial production processes, glass defect detection technology can be used for real-time monitoring and early warning, improving production efficiency and quality.

[0357] In the security field: By combining technologies such as video surveillance, glass defect detection technology can be applied to the security field, such as monitoring the safety condition of glass in buildings or vehicles to prevent potential dangers.

[0358] Intelligent Transportation: In intelligent transportation systems, glass defect detection technology can be used to monitor the condition of vehicle glass, improving driving safety and passenger comfort.

[0359] Smart Home Applications: Applying glass defect detection technology to the home sector allows for real-time monitoring of the condition of glass doors and windows, improving home safety and convenience.

[0360] The embodiments of the present invention have the following advantages:

[0361] 1. Real-time monitoring and early warning: The embodiments of the present invention can detect and warn of glass defects in real time, provide users with safety risk information in a timely manner, and reduce potential hazards.

[0362] 2. High accuracy and robustness: Advanced signal processing and machine learning algorithms are used to improve the accuracy of defect detection and the stability of the system in complex environments.

[0363] 3. Scalability and applicability: By adjusting the sensor configuration and parameters, the embodiments of the present invention can be applied to different types of glass and scenarios, and have wide applicability.

[0364] 4. Integration with existing equipment: The embodiments of the present invention can be easily integrated with existing equipment (such as automobiles, buildings, etc.) to realize real-time monitoring and early warning in practical applications.

[0365] 5. Lens-free and passive monitoring: The embodiments of the present invention adopt a lens-free and passive method, which reduces system complexity and cost and improves practicality.

[0366] The embodiments of the present invention have the following characteristics:

[0367] 1. Multi-sensor fusion: The embodiments of the present invention employ multiple types of sensors to effectively capture and fuse signals from different dimensions, thereby improving the performance of defect detection.

[0368] 2. Adaptive Algorithm: By developing an adaptive algorithm that can automatically adjust parameters according to environmental changes, the system's performance in complex environments has been improved.

[0369] 3. Environmental adaptability: Corresponding solutions are proposed for environmental factors such as changes in light intensity and temperature fluctuations to ensure the stability and reliability of the system under various environmental conditions.

[0370] This invention proposes a deep learning-based glass defect monitoring method, comprising a graph neural network (MT-GNN) for extracting data from different sensors and an attention-based LSTM model for defect detection based on sensor temporal information analysis. The glass defect monitoring method provided by this invention can simultaneously process data from different sensors and obtain detailed defect detection information. Furthermore, this invention introduces an attention mechanism to focus on key information, thereby improving the accuracy of defect detection.

[0371] The deep learning-based defect detection method of this invention, through multi-sensor data fusion and attention mechanisms, effectively improves the accuracy and efficiency of glass defect detection, and has broad application prospects and market potential. This invention is not only applicable to defect detection in specific fields such as automotive glass, but can also be extended to defect detection in other fields, such as architectural glass and electronic product glass. Furthermore, the method of this invention possesses certain advanced features and can provide strong technical support for research and development in related fields.

[0372] The present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described above.

[0373] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. A method for monitoring glass defects based on deep learning, characterized in that, Includes the following steps: S1. Acquire detection signals from the glass using two or more sensors; S2. A trained attention-based Long Short-Term Memory (LSTM) network model is used, wherein the signals collected by two or more sensors are fused and concatenated to form a multi-channel signal, which is then fed into the LSTM model to predict whether there are defects in the glass. Step S2 further includes extracting data from different sensors using a multi-task graph neural network (MT-GNN); wherein the glass surface is divided into a set of grids, and each grid cell is regarded as a node in a graph, the connection between nodes represents the relationship between adjacent grid cells, and a feature vector is assigned to each node, which contains feature data from the sensors, thereby constructing a graph structure of the glass surface; wherein the graph neural network has an input layer, several graph convolutional layers, and an output layer, the input layer receives the feature vector obtained according to the detection signal, the graph convolutional layers are used to learn the relationship between nodes and propagate information between nodes, and the output layer outputs data about the defect prediction task; The graph neural network has an output layer at the top for performing the following four specific tasks: Defect output layer: Perform binary classification on each node to determine whether the node has a defect; Defect location output layer: Perform regression on each node to predict the relative location of the defect within that node; Defect type classification output layer: Performs multiple classifications on each node to identify the type of defect within that node; Detailed Defect Location Output Layer: Performs dense regression on each node to predict the detailed location of defects within that node.

2. The glass defect monitoring method as described in claim 1, characterized in that, The LSTM model includes an input gate, a forget gate, and an output gate. At each time step in processing time series data, the LSTM model updates the state of each gate based on the current input and the hidden state of the previous time step, and outputs the hidden state of the current time step. The attention mechanism introduces a weighted vector to determine the weights between different signal channels, so that the LSTM model focuses on the importance of different channel signals at each time step.

3. The glass defect monitoring method as described in claim 2, characterized in that, The attention weights are calculated using the Softmax function, and then weighted and summed with the input signal to obtain the weighted input signal.

4. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, During training, the LSTM model updates its weight parameters through backpropagation, thereby automatically learning the signal features and timing information related to defects. During the testing phase, for each new input data point, the LSTM model calculates the activation value of the output layer and normalizes it using the Softmax function to obtain the probability distribution of the presence or absence of defects.

5. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, In step S2, the detection signal is first preprocessed, specifically including one or more of the following: Adaptive filtering removes noise while preserving important signal information. Empirical mode decomposition (EMD) decomposes the original signal into different local signal features in order to extract defect information from the signal and reduce interference.

6. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that, The two or more sensors include two types of sensors disposed on the side of the glass, one of which is an optical sensor, and the others include one or more of infrared sensors, electromagnetic wave sensors, acoustic sensors, and capacitive sensors, with one or more of each type of sensor; it is capable of detecting defects in real time during the use of the glass.

7. The glass defect monitoring method according to any one of claims 1 to 3, characterized in that; Step S1 includes acquiring a spectral image of the glass using a spectral image sensor; in step S2, the physical properties of the glass surface and interior are identified based on the spectral distribution of light reflected, absorbed, or transmitted by the glass, using the characteristics of the spectral image, thereby detecting defects in the glass. In step S1, a set of multispectral cameras is set up so that they can capture spectral images of the glass from different angles and distances; the multispectral cameras scan along multiple different directions, thereby enabling comprehensive detection of the surface and interior of the glass.

8. The glass defect monitoring method as described in claim 7, characterized in that, Step S2 further includes: using an adaptive deep convolutional neural network (ACNN) to extract and classify spectral image features to achieve defect identification; the adaptive deep convolutional neural network includes a series of convolutional layers and fully connected layers, wherein the convolutional layers are used to extract image features, and the fully connected layers are used to integrate the features for classification; the adaptive deep convolutional neural network has adaptive convolutional layers, which can automatically adjust the size and stride of their convolutional kernels according to the characteristics of the input image to extract important features in the image.

9. The glass defect monitoring method as described in claim 7, characterized in that, Step S2 further includes: processing the feature sequences of the spectral images using a temporal convolutional neural network (TCNN) to extract temporal information reflecting the defect development process; firstly, converting continuous spectral images into feature sequences, and then inputting these sequences into the temporal convolutional neural network; the temporal convolutional neural network uses multiple convolutional layers and recurrent layers to process the input feature sequences, thereby capturing the temporal information in the sequences; the training of the TCNN employs backpropagation and stochastic gradient descent, and the network parameters are updated by optimizing the loss function, so that the network can better identify and predict the development of defects.

Images