A glass defect monitoring apparatus

Abstract

The application discloses a kind of glass defect monitoring devices, including sensor and detection signal processing device, the sensor is used to collect the detection signal of glass, the detection signal processing device is judged according to the detection signal whether there is defect by trained neural network, and when judging defect, the type and position of the type of defect are determined;Wherein, the sensor includes two or more sensors arranged at the side of glass, one of which is an optical sensor, and the rest includes one or more of infrared sensor, electromagnetic wave sensor, acoustic sensor, capacitive sensor, and the number of each sensor is one or more.The application can get rid of the dependence on light source and imaging equipment;Real-time monitoring can be carried out on glass and monitoring cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of glass monitoring, and in particular to a glass defect monitoring device. BACKGROUND

[0002] Glass is a widely used material, and its quality is crucial for many application scenarios. Glass defects can lead to structural weakness, reduced optical performance, and aesthetic problems, so it is important to detect defects in glass. Existing glass defect detection methods mainly rely on traditional optical imaging technology and machine vision algorithms. These methods usually require the use of lenses for imaging and rely on external light sources for illumination. However, these methods have the following problems.

[0003] Dependence on light sources and imaging equipment: Traditional detection methods require dependence on external light sources and imaging equipment, which increases the complexity and cost of the system. At the same time, this dependence leads to sensitivity to changes in environmental lighting, which affects the detection effect (sensitive to lighting conditions, expensive equipment).

[0004] Limited detection dimensions: Existing methods can only detect whether there are defects in the glass, but cannot provide multi-dimensional information about the location and type of defects. This limits the effectiveness of the detection method in practical applications.

[0005] Poor adaptability to environments and scenarios: Due to the dependence on light sources and imaging equipment, existing methods often require parameter adjustment and equipment replacement when faced with different environments and application scenarios, resulting in poor system adaptability. SUMMARY

[0006] The purpose of the present application is to solve at least one of the above problems existing in the prior art and provide a glass defect monitoring device.

[0007] To achieve the above purpose, the present application adopts the following technical solutions:

[0008] A glass defect monitoring device, comprising a sensor and a detection signal processing device, the sensor is used to collect the detection signal of the glass, and the detection signal processing device determines whether there is a defect according to the detection signal through a trained neural network, and determines the type and location of the defect type when a defect is determined; wherein the sensor includes two or more sensors arranged on the side of the glass, one of which is an optical sensor, and the rest includes one or more of an infrared sensor, an electromagnetic wave sensor, an acoustic sensor, and a capacitive sensor, and the number of each sensor is one or more.

[0009] In some embodiments of the present application, the optical sensor detects defect-related reflection, transmission, scattering, refraction optical information of the glass surface; the infrared sensor detects defect-related thermal feature information of the glass surface; the acoustic sensor detects defect-related acoustic wave reflection feature information of the glass surface; the electromagnetic wave sensor detects defect-related electromagnetic wave reflection feature information of the glass surface; and the capacitance sensor detects defect-related capacitance feature information of the glass surface.

[0010] In some embodiments of the present application, a pre-warning module is further included for pre-warning operation according to the judgment result of the detection signal processing device, specifically including: receiving the defect prediction result from the neural network, and integrating to form a defect information list; traversing the defect information list, and judging whether to trigger pre-warning according to a preset pre-warning threshold and a confidence degree of the defect; and when pre-warning is triggered, performing one or more operations of displaying, sending, and recording pre-warning information.

[0011] In some embodiments of the present application, a communication module is further included for sending the pre-warning information to a designated monitoring device.

[0012] In some embodiments of the present application, the number of each sensor is one or more, and the detection signal processing device performs defect detection by fusing detection signals of multiple sensors, which include multiple detection signals of the same type or different types.

[0013] In some embodiments of the present application, the glass includes glass of a vehicle, a ship, an aircraft, a building, industrial equipment, a photovoltaic device, or a household device, and can detect defects in real time during use of the glass.

[0014] In some embodiments of the present application, the neural network is a multi-task graph neural network MT-GNN; wherein the glass surface is divided into a group of grids, and each grid unit is regarded as a node in the graph, the connection between the nodes represents the relationship between the adjacent grid units, and a feature vector is assigned to each node, which contains feature data from the sensor, thereby constructing a graph structure of the glass surface; wherein the graph neural network has an input layer, a plurality of graph convolution layers, and an output layer, the input layer receives a feature vector obtained according to the physical signal, the graph convolution layer is used to learn the relationship between the nodes and propagate the information between the nodes, and the output layer is used to process the output of the defect prediction task.

[0015] In some embodiments of the present application, the graph neural network includes output layers of the following four tasks:

[0016] A defect presence / absence output layer, which performs binary classification on each node to determine whether the node has a defect;

[0017] a defect rough position output layer, which performs regression on each node to predict the relative position of the defect within the node;

[0018] a defect type classification output layer, which performs multi-classification on each node to identify the type of the defect within the node;

[0019] a defect detailed position output layer, which performs dense regression on each node to predict the detailed position of the defect within the node;

[0020] Preferably, each task has a corresponding loss function, and the total loss function is:

[0021] L_total = λ1 L1 + λ2 L2 + λ3 L3 + λ4 L4

[0022] where L_total is the total loss function, L1, L2, L3, L4 are the loss functions of the four tasks respectively, and λ1, λ2, λ3, λ4 are the weights of the loss functions of each task;

[0023] Calculate the loss for each task separately and combine them to get the total loss.

[0024] More preferably, one or more of the following loss function settings are used:

[0025] Defect presence prediction uses binary cross-entropy loss function BCELoss:

[0026] L1 = BCELoss= -Σ [y log(p) + (1 - y) log(1 - p)]

[0027] where y represents the actual label (0 or 1) and p represents the probability predicted by the model;

[0028] Defect rough position prediction uses mean squared error loss function MSELoss:

[0029] L2 = MSELoss= Σ (y - p)^2 / N

[0030] where y represents the actual position, p represents the predicted position by the model, and N represents the number of samples;

[0031] Defect type classification prediction uses class cross-entropy loss function CCELoss:

[0032] L3 = CCELoss = -∑∑ y_i log(p_i)

[0033] Where y_i represents the actual category one-hot encoding, p_i represents the probability of model prediction;

[0034] Defect detailed location prediction, using mean square error loss MSELoss:

[0035] L4 = MSELoss = ∑ (y1 - p1)^2 / N

[0036] Where y1 represents the actual detailed location, p1 represents the detailed location predicted by the model, and N represents the sample quantity.

[0037] In some embodiments of the application, each layer of graph convolutional layer is according to the following formula:

[0038] X^(l+1) = sigma(AX^(l)W^(l))

[0039] Where X^(l) represents the node feature matrix of the lth layer, A represents the adjacency matrix, W^(l) represents the weight matrix of the lth layer, and sigma represents the activation function.

[0040] In some embodiments of the application, the detection signal processing device further adaptively adjusts the defect detection threshold of the detection signal according to the environmental parameter data, specifically comprising:

[0041] Using environmental parameter data and corresponding detection threshold label as a training set, training the relationship model between detection threshold and environmental parameter;

[0042] Through the relationship model, the corresponding threshold is predicted according to the current environmental parameter data, and the defect detection threshold is dynamically adjusted.

[0043] The application has the following beneficial effects:

[0044] ​The glass defect monitoring device provided by the application comprises a sensor and a detection signal processing device, the detection signal of the glass is collected through the sensor, and then the detection signal processing device is used to determine whether there is a defect according to the detection signal and determine the type and position of the defect type when it is determined that there is a defect; wherein the sensor is arranged on the side of the glass and comprises one or more of an optical sensor, an infrared sensor, an electromagnetic wave sensor, an acoustic sensor and a capacitive sensor, which can accurately identify and locate the glass defect, and can break away from the dependence on light sources and imaging equipment in the monitoring process; and the glass can be monitored in real time, and the glass defect can be discovered and identified in time, so that the influence and safety hazards caused by the defect on the use can be effectively avoided. The sensor is installed on the side of the glass on various devices to monitor the glass in situ and in real time, and no additional lens, light source and other equipment are needed, so that the monitoring cost can be greatly reduced and the real-time performance can be improved, therefore, the application has the advantages of low cost, high real-time performance and easy implementation. For example, when the application is applied to a vehicle, the demand for real-time monitoring of whether the glass has a defect can be met, and the safety during driving of the vehicle can be improved. However, the application can be widely applied to various installed glasses, and is not limited to vehicles.

[0045] Other beneficial effects in the embodiments of the application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a structural diagram of the glass defect monitoring device in the embodiments of the application;

[0047] Figure 2 is a structural diagram of the glass defect monitoring device in another embodiment of the application;

[0048] Figure 3 is an installation schematic diagram of the sensor of the glass defect monitoring device in the embodiments of the application on the side of the glass.

[0049] The reference signs are as follows:

[0050] 1 is a glass, and 2 is a sensor. DETAILED DESCRIPTION

[0051] The application will be further described below by comparing the drawings and combining the preferred embodiments. It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0052] It should be noted that the left, right, up, down, top and bottom orientation terms in the embodiments are only relative concepts or are referenced to the normal use state of the product, and should not be considered to be limiting.

[0053] The following embodiments of the application propose a glass defect monitoring device, as shown in Figure 1 ,Figure 3 , including a sensor 2 for collecting detection signals of the glass 1 and a detection signal processing device for determining the presence or absence of defects and determining the type and location of the defect type through a trained neural network according to the detection signals; wherein the sensor 2 includes two or more sensors arranged on the side of the glass 1, one of which is an optical sensor, and the rest includes one or more of an infrared sensor, an electromagnetic wave sensor, an acoustic sensor, and a capacitive sensor, and each sensor 2 has one or more.

[0054] In a preferred embodiment, the optical sensor detects the reflection, transmission, scattering, and refraction of the optical information related to the defect on the surface of the glass 1; the infrared sensor detects the thermal characteristic information related to the defect on the surface of the glass 1; the acoustic sensor detects the acoustic wave reflection characteristic information related to the defect on the surface of the glass 1; the electromagnetic wave sensor detects the electromagnetic wave reflection characteristic information related to the defect on the surface of the glass 1; and the capacitive sensor detects the capacitive characteristic information related to the defect on the surface of the glass 1.

[0055] In a preferred embodiment, the glass 1 includes a glass of a vehicle, a ship, an aircraft, a building, industrial equipment, a photovoltaic device, or a household device, which can detect defects in real time during use of the glass 1.

[0056] A glass defect monitoring device according to an embodiment of the present application is described in detail as follows:

[0057] I. Sensor

[0058] The sensor 2 is the core part of the entire glass defect monitoring device, responsible for collecting detection signals on the surface of the glass 1. The sensor 2 needs to maintain a certain distance from the surface of the glass 1 to obtain the best signal quality. The output signal of the sensor 2 is usually an analog signal.

[0059] Referring to Figure 3 , the embodiment of the present application uses multiple sensors 2 for data acquisition to achieve multi-dimensional glass defect detection. According to the above scheme, at least two sensors 2 are required, one of which is an optical sensor for capturing optical information on the surface of the glass; the other varies according to the scheme and can be selected from an infrared sensor, an acoustic sensor, an electromagnetic wave sensor, or a capacitive sensor, etc. for detecting thermal, acoustic wave, electromagnetic wave, or capacitive distribution information on the surface of the glass, thereby obtaining multi-dimensional characteristic information related to defects. The type of sensor 2 can be selected according to actual needs.

[0060] The functions of different sensors are as follows:

[0061] Optical sensor: Collects light signals in the glass and converts them into electrical signals. Here, the light signal is not just intensity, but also includes spectral distribution, phase distribution, and polarization. Optical sensor: Captures optical information of the glass surface, such as surface reflection, transmission, scattering, etc., which can be used to detect surface defects, bubbles, foreign matter, etc.

[0062] Infrared sensor: Detects the temperature distribution of the glass surface, which can obtain thermal characteristic information related to defects, such as thermal cracks and hot spots.

[0063] Acoustic sensor: Can be used to detect the sound reflection of the glass surface. By analyzing the acoustic reflection characteristics of the glass surface, the presence and approximate location of defects can be determined, such as glass cracks and internal bubbles.

[0064] Electromagnetic wave sensor: Analyzes the electromagnetic wave reflection characteristics of the glass surface to determine the presence and approximate location of defects, such as scratches and micro-cracks on the glass surface.

[0065] Capacitance sensor: Detects the capacitance distribution of the glass surface to obtain electrical characteristic information related to defects, such as defects and foreign matter on the glass surface.

[0066] Magnetic sensor: Can detect the magnetic field distribution in the glass to determine whether there are defects.

[0067] Thermal sensor: Can detect the temperature change of the glass surface to determine whether there are defects.

[0068] Pressure sensor: Can be used to detect the pressure change of the glass surface to determine whether there are defects.

[0069] Electromagnetic sensor: Can detect the electromagnetic field distribution in the glass to determine whether there are defects.

[0070] In specific embodiments, not limited to the above-mentioned sensors, even the optical sensor has different types, these sensors can realize defect detection by converting different properties of light into electrical signals: such as

[0071] Photodiode: Converts light intensity into current or voltage signals;

[0072] Photoelectric cell: Converts the energy of light into electrical energy to generate voltage;

[0073] Photomultiplier tube: Converts photons into electrons, and enhances the light signal through electron multiplication;

[0074] Optical Fiber Sensor: Utilizes the transmission characteristics of light in optical fibers to convert external physical quantities (such as temperature, pressure, etc.) into parameters such as light intensity, frequency, or phase, and then converts them into electrical signals.

[0075] Grating Sensor: Converts the wavelength or frequency of light into an electrical signal.

[0076] CCD (Charge-Coupled Device) Sensor: Converts photons into electric charges, and then converts them into electrical signals through charge transfer and amplification.

[0077] In the embodiments of the present application, in order to realize multi-dimensional glass defect detection, the following are several possible sensor 2 configuration schemes. Please note that this is just an example and does not represent that the sensor 2 configuration of the embodiments of the present application only includes these:

[0078] Example One:

[0079] Optical Sensor: Adopting a lensless imaging optical sensor can capture the optical information of the glass surface. Lensless imaging can reduce system complexity and cost, while improving the robustness to changes in environmental light.

[0080] Infrared Sensor: Infrared sensors can detect the temperature distribution of the glass surface, thereby obtaining thermal characteristic information related to defects.

[0081] Acoustic Sensor: By analyzing the acoustic wave reflection characteristics of the glass surface, the presence and approximate location of defects can be determined.

[0082] Scheme One can obtain optical, thermal, and acoustic wave characteristics of the glass surface through the combination of optical, infrared, and acoustic sensors. These characteristics help to realize multi-dimensional glass defect detection.

[0083] Example Two:

[0084] Optical Sensor: Adopting a lensless imaging optical sensor can capture the optical information of the glass surface.

[0085] Infrared Sensor: Infrared sensors can detect the temperature distribution of the glass surface, thereby obtaining thermal characteristic information related to defects.

[0086] Electromagnetic Wave Sensor: By analyzing the electromagnetic wave reflection characteristics of the glass surface, the presence and approximate location of defects can be determined.

[0087] Scheme Two can obtain optical, thermal, and electromagnetic wave characteristics of the glass surface through the combination of optical, infrared, and electromagnetic wave sensors. These characteristics help to realize multi-dimensional glass defect detection.

[0088] Example Three:

[0089] Optical Sensor: Optical sensors with lensless imaging can capture optical information of the glass surface.

[0090] Capacitive Sensor: Capacitive sensors can detect the capacitance distribution of the glass surface, thereby obtaining electrical characteristic information related to defects.

[0091] Acoustic Sensor: By analyzing the acoustic wave reflection characteristics of the glass surface, the presence and approximate location of defects can be determined.

[0092] Example Three combines optical, capacitive, and acoustic sensors to obtain optical, capacitive, and acoustic characteristics of the glass surface, which helps achieve multi-dimensional glass defect detection.

[0093] These three examples are just examples, and there are other combinations for the following reasons:

[0094] Variety of Sensor Technologies: There are many types of sensors on the market, each with different working principles and characteristics. These sensors can be combined to capture multiple characteristics of the glass surface, thereby achieving multi-dimensional defect detection.

[0095] Differences in Application Scenarios: Different application scenarios may have different requirements for glass defect detection. For example, some scenarios may require higher detection accuracy, while others may require higher real-time performance. Different types of sensor combinations can be used to achieve the best detection results for different needs.

[0096] Dynamic Nature of Technology Development: As science and technology continue to advance, new sensors and detection methods are constantly emerging. These new technologies can be combined with existing solutions to improve the performance of multi-dimensional glass defect detection.

[0097] II. Detection Signal Processing Device

[0098] The detection signal processing device uses a trained neural network to determine the presence or absence of defects based on the detection signal, and to determine the type and location of defects when a defect is detected.

[0099] The detection signal processing device is also responsible for converting the detection signal output by the sensor into a digital signal and processing it. The detection signal processing device internally includes an analog-to-digital converter (ADC) that converts analog signals into digital signals that can be processed by the processing module. In addition, it is responsible for synchronizing the signals of each sensor to ensure signal consistency.

[0100] The defect detection signal processing device also includes a microprocessor or embedded processor responsible for executing defect detection algorithms, including signal preprocessing, feature extraction, defect recognition, etc. The output of the processing module is defect information, including the approximate location, type, and precise location of the defect.

[0101] Signal processing involves processing the collected and converted electrical signals to extract information about glass defects. This part can use digital signal processing technology (Digital Signal Processing, abbreviated as DSP) to improve the accuracy and efficiency of signal processing.

[0102] Defect detection algorithm: Based on the information extracted by the signal processing module, use machine learning or other methods for recognition to determine whether there is a glass defect.

[0103] The glass defect monitoring device provided by the embodiment of the present application fuses the detection signals of the preprocessed and feature-extracted data of different sensors based on the graph neural network (GNN), and can simultaneously realize the tasks of judging whether there is a defect, determining the approximate location of the defect, classifying the type of the defect, and determining the detailed location of the defect. Specifically, the following contents are included:

[0104] 1. Data preprocessing and feature extraction: For the detection signals collected by the above-mentioned multiple sensors, the embodiment of the present application can perform data preprocessing and feature extraction, combine these features into a feature vector, and input the feature vector into the graph neural network.

[0105] 2. Constructing a graph structure: The glass surface is divided into a set of regular grids, and each grid cell is regarded as a node in the graph. The connection between nodes represents the relationship between adjacent grid cells. Assign a feature vector to each node, which contains feature data from all sensors, thereby constructing the graph structure of the glass surface.

[0106] The specific method is as follows:

[0107] 2.1 Grid division: The glass surface is divided into regular grids. The surface can be divided into rectangular or square grids, or divided according to specific geometric shapes such as triangles or hexagons, etc. Ensure that the size and density of the grid are appropriate to fully capture the features and defects of the glass surface.

[0108] 2.2 Node definition: Each grid cell is regarded as a node in the graph. Each node represents a local area on the glass surface. The node can be represented by the coordinates of the grid cell it is in, such as (x, y). The node can also have other attributes, such as the index of the node, the list of neighbor nodes, etc.

[0109] 2.3 Node Feature Vector: Assign a feature vector to each node, which contains feature data from all sensors. The feature data of each sensor can be pre-processed and standardized. Then, the feature data from different sensors is fused into the feature vector of the node. A simple feature concatenation method can be used to connect the features of each sensor in sequence to form a comprehensive feature vector.

[0110] 2.4 Feature Fusion Method: To better fuse the feature data of different sensors, a more complex feature fusion method can be used. For example, an attention mechanism or graph attention network can be used to adaptively learn the weights or importance of different sensor features. This way, the network can pay more attention to features that contribute more to the defect detection task.

[0111] 2.5 Node Connection: According to the position and distance between grid cells, establish the connection relationship between nodes. The common method is to connect each node with the nodes corresponding to its adjacent grid cells. Different neighbor connection methods can be considered, such as 4-neighbor connection or 8-neighbor connection, which is determined according to the characteristics and requirements of the problem.

[0112] 3、Design of Graph Neural Network: Design a multi-task graph neural network (MT-GNN) that includes an input layer, several layers of graph convolution, and an output layer. The input layer receives the feature vectors obtained from the physical signals, the graph convolution layer learns the relationship between nodes and propagates information, and the output layer processes the output of the defect prediction task.

[0113] Preferably, the embodiment of the present application adds four task-specific output layers at the top of the graph neural network to implement different tasks:

[0114] 3.1 Defect Existence Output Layer: Binary classification for each node to determine whether the node has a defect.

[0115] 3.2 Defect Approximate Position Output Layer: Regression for each node to predict the relative position of the defect within the node (relative to the node's coordinates).

[0116] 3.3 Defect Type Classification Output Layer: Multi-classification for each node to identify the type of defect within the node.

[0117] 3.4 Defect Detailed Position Output Layer: Dense regression for each node to predict the detailed position of the defect within the node.

[0118] The network structure, calculation process and calculation formula of the specific multi-task graph neural network (MT-GNN), including the design of the objective function and the loss function, are as follows:

[0119] 1、Network structure:

[0120] Input layer: Receives feature vectors for each node, which contain pre-processed and feature-extracted data from five sensors.

[0121] Graph convolutional layer: Contains several layers of graph convolutional operations to learn and propagate information between nodes. Each layer of graph convolution follows the following formula:

[0122] X^(l+1) = σ(AX^(l)W^(l))

[0123] where X^(l) represents the node feature matrix of the l-th layer, A represents the adjacency matrix, W^(l) represents the weight matrix of the l-th layer, and σ represents the activation function (such as ReLU).

[0124] Task-specific output layer: Add four task-specific output layers on top of the graph neural network, respectively for defect presence / absence, defect approximate location, defect type, and defect detailed location. These output layers can be fully connected layers, respectively for processing the output of different tasks.

[0125] 2、The calculation process is as follows:

[0126] 2.1 The input layer receives node feature vectors as input;

[0127] 2.2 Learn and propagate information between nodes through the graph convolutional layer;

[0128] 2.3 Task-specific output layers process the output of different tasks respectively.

[0129] where "information between nodes" refers to the relationship and transmission of information between the feature vectors of each node and its surrounding nodes. These feature vectors include pre-processed and feature-extracted data from five sensors, while the transmitted information is learned and propagated by the graph convolutional layer, including the adjacency relationship and feature similarity between nodes.

[0130] 3、Objective function and loss function:

[0131] In MT-GNN, each task has a corresponding loss function. The embodiment of the present invention can calculate the loss for each task separately and combine them to get the total loss. The following is the detailed formula of the loss function for each task:

[0132] 3.1 Defect presence / absence prediction loss function:

[0133] This is a binary classification problem, using binary cross-entropy loss (Binary Cross-Entropy Loss) as the loss function. It can effectively measure the difference between the model's predicted probability and the actual label, and make the model pay attention to misclassified samples during optimization.

[0134] L1 = Binary Cross-Entropy Loss (BCELoss) = -Σ [y log(p) + (1 - y) log(1 - p)]

[0135] where y represents the actual label (0 or 1), and p represents the probability predicted by the model.

[0136] 3.2 Defect Approximate Location Prediction Loss Function:

[0137] This is a regression problem, and the embodiments of the present application need to predict the approximate location of defects. Therefore, the Mean Squared Error Loss is used to measure the difference between the predicted value and the actual value. This loss function can effectively optimize the performance of the model in location prediction.

[0138] L2 = Mean Squared Error Loss (MSELoss) = Σ (y - p)^2 / N

[0139] where y represents the actual location, p represents the predicted location of the model, and N represents the number of samples.

[0140] 3.3 Defect Type Classification Prediction Loss Function:

[0141] This is a multi-classification problem, and the embodiments of the present application need to classify the defect type. Therefore, the Categorical Cross-Entropy Loss is used as the loss function. It can measure the difference between the predicted probability of the model in multiple categories and the actual label, and help to improve the performance of the model in classification tasks.

[0142] L3 = Categorical Cross-Entropy Loss = -Σ Σ y_i log(p_i)

[0143] where y_i represents the one-hot encoding of the actual category, and p_i represents the probability predicted by the model.

[0144] 3.4 Defect Detailed Location Prediction Loss Function:

[0145] This is a regression problem, and it needs to predict the detailed location of the defect. Therefore, the Mean Squared Error Loss is used to measure the difference between the predicted value and the actual value. This loss function can also effectively optimize the performance of the model in location prediction.

[0146] L4 = Mean Squared Error Loss (MSELoss) = Σ (y - p)^2 / N

[0147] where y represents the actual detailed location, p represents the detailed location predicted by the model, and N represents the number of samples.

[0148] 3.5 Combine the loss functions of the above four tasks to get the total loss function:

[0149] L_total = λ1 L1 + λ2 L2 + λ3 L3 + λ4 L4

[0150] where λ1, λ2, λ3 and λ4 are the weights of each task loss function, which can be adjusted according to the actual problem and data.

[0151] Setting the loss function as above has the following benefits:

[0152] (1) By calculating the loss for each task separately, it can ensure that the model contributes to learning each task. This helps improve the model's performance on each task.

[0153] (2) Using the total loss function, the importance of different tasks can be balanced. By adjusting the weights of each task loss function, the performance of the model on different tasks can be optimized.

[0154] (3) The total loss function helps to realize end-to-end training, so that the model learns multiple tasks in a unified framework, improving the efficiency of training and prediction.

[0155] 4、Training and prediction:

[0156] Use the labeled dataset to train MT-GNN, minimizing the total loss function.

[0157] In the prediction phase, input the sensor feature vector of the glass to be detected, and get the prediction results of each task through MT-GNN:

[0158] a) Defect presence prediction: output the binary classification probability of each node, and set a threshold (such as 0.5) to determine whether there is a defect.

[0159] b) Defect approximate location prediction: output the relative position of the defect within each node (relative to the node's coordinates).

[0160] c) Defect type classification prediction: output the multi-classification probability of each node, and determine the defect type according to the class with the highest probability.

[0161] d) Defect detailed location prediction: output the detailed location of the defect within each node.

[0162] Analyze the prediction results:

[0163] According to the defect presence prediction results, determine the nodes on the glass where defects exist.

[0164] For the defective nodes, use the defect approximate location and defect detailed location prediction results to determine the actual location of the defect on the glass.

[0165] In specific embodiments, as shown in Figure 2 , the following can also be included:

[0166] III. Early warning module

[0167] The embodiments of the present application include an early warning module for performing early warning operations according to the judgment results of the detection signal processing device, specifically including: receiving defect prediction results from the multi-task graph neural network, and integrating to form a defect information list; traversing the defect information list, and determining whether to trigger early warning according to a preset early warning threshold and the confidence of the defect; when early warning is triggered, performing one or more operations of displaying, sending, and recording early warning information.

[0168] For the specific scheme of the early warning operation, the following steps can be used to implement it:

[0169] (1) Set the early warning threshold: set an early warning threshold for each defect type according to historical data and experience, such as a confidence level greater than 0.8 for general early warning and a confidence level greater than 0.9 for serious early warning.

[0170] (2) Defect information integration: the early warning module receives defect prediction results from the multi-task graph neural network (MT-GNN). Integration includes defect type, location, confidence, and other information to form a defect information list.

[0171] (3) Design early warning strategy: traverse the defect information list and determine whether to trigger early warning according to the preset threshold. For example, if the confidence of a certain defect is greater than 0.8, a general early warning is triggered; if the confidence is greater than 0.9, a serious early warning is triggered.

[0172] (4) Early warning information presentation: design a real-time monitoring interface to display defect information. A table can be used to list defect types, locations, confidence levels, etc., and colors or icons can be used to represent different levels of early warning.

[0173] (5) Early warning notification: When the early warning is triggered, notify the relevant personnel in time. You can use SMS gateway, email server or mobile APP push to send early warning information. At the same time, record the early warning information in the database or log file for subsequent analysis.

[0174] For specific scenarios such as automotive glass, existing methods cannot meet the real-time detection requirements. During the driving process, the glass may have defects due to impact, wear or other reasons. These defects may cause vision obstruction, reduced optical performance, or even safety hazards. Therefore, for automotive glass and other application scenarios, a method is needed to detect defects in real time during use, so that timely warning and handling can be performed when problems are found.

[0175] In this part, it will also describe how to transmit the detected glass defect information to the user in real time for early warning at any time during use, and introduce the specific implementation method of the real-time early warning system and how to adapt it in different scenarios (such as automotive glass).

[0176] 1、Early warning level division

[0177] In order to better deliver early warning information, the early warning level is divided into three levels, representing different levels of glass defects.

[0178] (1) Low risk (green): The defect is small and does not affect the normal use of the glass, but needs to be paid attention to in case the problem worsens.

[0179] (2) Medium risk (yellow): The defect is obvious and may affect the normal use of the glass, which needs to be resolved in the short term.

[0180] (3) High risk (red): The defect is serious and may cause damage to the glass, which needs to be addressed immediately.

[0181] 2、Early warning

[0182] According to different scenarios and early warning levels, the way to issue early warnings needs to be adjusted. The following is an example:

[0183] (1) Vehicle-mounted glass:

[0184] Low risk: Display a green icon on the car's instrument panel to prompt the driver to pay attention;

[0185] Medium risk: Display a yellow icon on the car's instrument panel with a slight alarm sound to prompt the driver to resolve the problem as soon as possible;

[0186] High risk: Display a red icon on the car's instrument panel with a continuous emergency alarm sound to prompt the driver to stop and check immediately.

[0187] (2) Building glass:

[0188] Low risk: Send a short message or email notification to the responsible person through the building property management system;

[0189] Medium risk: Send a short message or email notification to the responsible person through the building property management system, and require the problem to be solved within a certain time;

[0190] High risk: Send a short message or email notification to the responsible person through the building property management system, and require immediate organization of professional personnel for inspection and repair.

[0191] 3. Realize early warning notification

[0192] Vehicle glass: Send early warning information to the car dashboard through the built-in communication module of the car (such as car Bluetooth, Wi-Fi (Wireless Fidelity, Wi-Fi) wireless fidelity, etc.), and display the early warning level and prompt information in real time.

[0193] Building glass: Send early warning information to the building property management system through wireless network (such as Wi-Fi, (Long Range Radio, LoRa) long distance wireless power supply, etc.), and the system automatically sends the corresponding notification according to the early warning level.

[0194] Through the design and implementation of the above real-time early warning system, appropriate early warning methods can be taken according to different scenarios and early warning levels to ensure that early warning information is delivered to users in a timely manner when glass defects occur, improve users' safety awareness and reduce potential risks.

[0195] 4. Automatic update and tracking of early warning information

[0196] In order to ensure that users always know the latest status of glass defects, the real-time early warning system also needs to have the function of automatic update and tracking.

[0197] Automatic update: The system will regularly collect signal data of sensors and process them to update early warning information in real time. When the early warning level changes, the system will automatically adjust the early warning notification to ensure that users always know the current risk situation.

[0198] Tracking processing progress: For glass defects that have issued early warning, the system can track the processing progress to help users monitor the problem solving situation. For example, for building glass, the property management system can update the early warning information and notify the responsible person after processing. For vehicle glass, users can check the processing progress through the car dashboard and receive notification after the problem is solved.

[0199] 5. Adaptability to different environments and scenarios

[0200] The real-time warning system should have good adaptability to work normally in different environments and scenarios. Specifically, the system needs to consider the following factors:

[0201] (1) Transmission distance: According to different scenarios (such as car glass, building glass, etc.), the transmission distance of warning information may be different. The system should be able to support data transmission of different distances, such as through Bluetooth, Wi-Fi, etc.

[0202] (2) Environmental factors: The system needs to cope with various environmental factors such as temperature, humidity, light, etc. For example, for outdoor building glass, the system should have waterproof, dustproof, etc. Function to ensure normal operation in harsh environments.

[0203] (3) Device compatibility: The real-time warning system needs to be compatible with various devices such as car instrument panel, property management system, etc. In addition, it needs to consider different brands and models of devices to ensure that the warning information can be correctly displayed and transmitted.

[0204] By considering the above factors, a real-time warning system with strong adaptability can be designed, which can provide timely and accurate warning information to users in various scenarios and environments.

[0205] Four, communication module

[0206] As shown in Figure 2 , the communication module is responsible for sending the defect information output by the processing module to the host computer or other devices. The communication module can use wired or wireless methods such as serial communication, Ethernet, Wi-Fi or Bluetooth, etc. The communication module needs to establish a stable communication connection between the detection signal processing device and the host computer or other devices.

[0207] Five, power module

[0208] As shown in Figure 2 , the power module is responsible for providing stable power supply for the entire system. This module needs to be designed according to the power requirements of each sub-module, including voltage, current and stability, etc.

[0209] The following embodiments are further described in combination with the following embodiments. It should be emphasized that the following embodiments are only exemplary embodiments of the present application, and the present application is not limited to only this embodiment.

[0210] Example 1

[0211] The glass defect detection method in the embodiment of the present application can be used for various types of glass, for example:

[0212] 1. Building glass: used in building exterior walls, curtain walls, etc., need to detect whether the surface is flat, whether there are bubbles, cracks and other defects.

[0213] 2. Optical glass: used to manufacture various optical devices, such as lenses, prisms, filters, etc., need to detect whether the surface is smooth, whether there are scratches, bumps and other defects.

[0214] 3. Intelligent glass: used in electronic displays, smart home, etc. scenarios, need to detect whether the surface is flat, whether there are scratches, stains and other defects.

[0215] The specific use process of the device is as follows:

[0216] S1. Choose one kind of optical sensor and another kind of sensor, this embodiment chooses resistance pressure sensor, respectively:

[0217] (1) Optical sensor: CMOS image sensor OV5640, mainly because this model has high resolution and sensitivity, and supports functions such as automatic exposure and automatic white balance, which can adapt to the shooting needs in different light environments. In addition, the size of this model is relatively small, which can be easily installed on the car glass.

[0218] (2) Resistance pressure sensor: FSR406, mainly because this model has high sensitivity and accuracy, and good stability, which can accurately measure the deformation of the car glass. In addition, the cost of this model is relatively low, which is suitable for large-scale application.

[0219] S2. Sensor installation position:

[0220] (1) The number of CMOS image sensors: 2, respectively installed on the left and right sides of the car window glass.

[0221] Installation position: the sensor is about 10mm away from the glass surface, which can effectively detect the defects on the glass surface.

[0222] (2) The number of resistance pressure sensors: 2, respectively installed on the left and right sides of the car window glass.

[0223] Installation position: the sensor is about 10mm away from the glass surface, which can effectively detect the deformation and curvature of the glass.

[0224] S3. Design of detection signal processing device:

[0225] A trained multi-task graph neural network (MT-GNN) model is used to comprehensively process data from a CMOS image sensor and a resistive pressure sensor. The CMOS image sensor is used to detect glass surface defects, and the resistive pressure sensor is used to detect glass deformation and bending. The input of the MT-GNN model is the data of the two sensors, and the output is information such as defect type, location, confidence, etc.

[0226] S4. Early warning module design:

[0227] (1) Early warning threshold setting: According to historical data and experience, set a warning threshold for each defect type, such as a general warning when the confidence is greater than 0.8, and a serious warning when the confidence is greater than 0.9.

[0228] (2) Defect information integration: The early warning module receives the defect prediction results from the MT-GNN. Integration includes defect type, location, confidence, etc., forming a defect information list.

[0229] (3) Design of early warning strategy: Traverse the defect information list, and determine whether to trigger the early warning according to the preset threshold. For example, if the confidence of a certain defect is greater than 0.8, a general warning is triggered; if the confidence is greater than 0.9, a serious warning is triggered.

[0230] (4) Early warning information presentation: Design a real-time monitoring interface to display defect information. You can list defect types, locations, confidence levels, etc. in table form, and use colors or icons to represent different levels of warning. The monitoring interface can display real-time images of the vehicle window glass to facilitate operators to view.

[0231] S5. Early warning notification: When the early warning is triggered, notify the relevant personnel in a timely manner. You can use SMS gateways, email servers, or mobile app push notifications to send early warning information. At the same time, record the early warning information in the database or log file for subsequent analysis.

[0232] Example 2

[0233] I. System integration and optimization

[0234] 1. System integration

[0235] System integration is the process of seamlessly integrating the glass defect detection system with existing equipment (such as cars, building property management systems, etc.). This helps to ensure that real-time monitoring and early warning functions can function normally in actual applications. Here are some key integration steps and considerations:

[0236] ① Device interface: To ensure compatibility with existing equipment, a specific hardware and software interface needs to be developed for the glass defect detection system. This includes communication interfaces with car instrument panels, property management systems, etc. to transmit early warning information in real time.

[0237] ②Installation and Layout: To ensure that the sensor can be properly installed and integrated with existing equipment, the layout and installation method of the sensor need to be considered. This may involve installation guidelines and technical support in different scenarios such as automotive glass, building glass, etc.

[0238] ③Power Consumption Management: Due to the limited power resources of existing equipment (such as cars), it is necessary to optimize the power consumption of the glass defect detection system. This can be achieved by reducing sensor power consumption, optimizing algorithm performance, etc., to ensure that the system works normally under limited energy conditions.

[0239] ④User Interface: In order to facilitate user use and understanding of early warning information, an intuitive and easy-to-use user interface needs to be designed. This may include early warning display on the car dashboard, real-time monitoring interface in property management system, etc.

[0240] 2、System Optimization

[0241] The goal of system optimization is to improve the performance of the glass defect detection system, including reducing false positives and false negatives, improving detection speed and accuracy, etc. The following are some expandable directions:

[0242] (1) Optimize the algorithm: Through continuous optimization of signal processing and defect detection algorithms, detection accuracy and speed can be improved. This may include using more efficient signal processing techniques, developing more powerful machine learning models, etc.

[0243] Use more efficient signal processing techniques: Methods such as multi-channel signal processing, wavelet transform, Fourier transform, etc. can be used to improve signal processing efficiency and accuracy.

[0244] Develop more powerful machine learning models: Methods such as deep learning, reinforcement learning, etc. can be used to improve the accuracy and speed of defect detection. In addition, methods such as transfer learning, incremental learning, etc. can be used to cope with real-time changes in the environment and data.

[0245] (2) Sensor Calibration: Regularly calibrate and maintain the sensor to ensure its stable performance. This helps to reduce false positives and false negatives, and improve detection accuracy.

[0246] Regularly calibrate and maintain the sensor: Methods such as standard calibration board, regular maintenance, etc. can be used to ensure the stability and accuracy of the sensor performance.

[0247] Use multi-sensor fusion: Methods such as Kalman filtering, etc. can be used to eliminate errors and uncertainties between sensors, and improve detection accuracy.

[0248] (3) Dynamic threshold adjustment: dynamically adjust the defect detection threshold based on the actual application environment and conditions. This helps to reduce false positives and false negatives, improving detection accuracy.

[0249] Dynamic adjustment of defect detection threshold based on actual application environment and conditions: adaptive algorithms such as sliding window, dynamic threshold, etc. can be used to improve detection accuracy and robustness.

[0250] Multi-threshold decision: multi-threshold decision technology such as interval-based multi-threshold decision, confidence-based multi-threshold decision, etc. can be used to further improve detection accuracy and robustness.

[0251] (4) Data fusion: by fusing data from multiple sensors, the accuracy and robustness of detection can be improved. This can be achieved through weighted averaging, Kalman filtering, etc. data fusion technology

[0252] Weighted average: weighted average of multiple sensor data to reduce noise and error and improve detection accuracy.

[0253] Kalman filtering: use Kalman filtering method to fuse multi-sensor data, eliminate sensor noise and error, and improve detection accuracy and robustness.

[0254] Adaptive warning strategy: design adaptive warning strategies for different scenarios and application requirements. For example, during vehicle driving, adjust the warning level and notification method in real time according to vehicle speed, road conditions, etc.; in the context of building property management, set appropriate urgency and notification strategies for defects based on floor, area, etc. information.

[0255] (6) Model update and online learning: continuously collect and analyze new data to perform online learning and update of the defect detection model. This helps to improve system performance and adaptability, reduce false positives and false negatives.

[0256] Establish a real-time data acquisition and analysis system to collect real-time data from sensors and process and analyze them.

[0257] Use machine learning algorithms and model update techniques to learn and update the model in real time with new data, improving model accuracy and robustness.

[0258] Customize specific learning strategies and update methods for different application scenarios and devices, such as incremental learning or online learning-based model update techniques.

[0259] (7) Performance monitoring and feedback: monitor system performance in real time and collect user feedback. Through the analysis of performance indicators and user needs, continuously optimize the system and improve user experience.

[0260] Collect user feedback and suggestions, use user surveys, user behavior analysis, etc. to analyze user needs and preferences, and continuously optimize the system based on user feedback.

[0261] Based on the monitoring results and user feedback, design appropriate performance indicators and evaluation methods to evaluate system performance and user experience to continuously improve system performance and user satisfaction.

[0262] In summary, in the system integration and optimization phase, we need to consider compatibility with existing equipment, sensor layout, power management, user interface, etc. At the same time, through optimization algorithms, sensor calibration, dynamic threshold adjustment, data fusion, adaptive early warning strategy, model update and online learning, performance monitoring and feedback, etc. to improve the performance and user experience of the glass defect detection system.

[0263] II. Responding to different environmental conditions

[0264] 1. Enhance the robustness and stability of the algorithm itself without adding components:

[0265] 1.1 Data preprocessing: First, use a smoothing filter such as a median filter, mean filter or Gaussian filter to eliminate high-frequency noise. Next, use difference method or other methods to correct the baseline drift of the signal.

[0266] Different smoothing filters will have different effects on the smoothness and frequency response of the signal, and the appropriate filter needs to be selected according to the actual situation. For example, the Gaussian filter can smooth the signal and retain the high-frequency components, while the median filter can remove noise without changing the characteristics of the signal. At the same time, the parameters of the filter need to be adjusted according to the frequency and amplitude characteristics of the signal, etc. to obtain the best effect.

[0267] 1.2 Outlier detection and elimination: Use Tukey's fence method or Z-score method to detect and eliminate outliers in the signal. This can reduce the impact of environmental factors on the signal and improve the robustness of the algorithm.

[0268] There are other methods for detecting outliers, such as Grubb's test, Dixon's Q test, Hampel filter, etc. After removing outliers, the effect of removing outliers needs to be evaluated to determine the effect of removing outliers. Box plot, statistical chart, etc. can be used to compare the distribution, mean, variance, etc. of the data before and after removing outliers to evaluate the reliability and stability of the data after removing outliers.

[0269] 1.3 Adaptive threshold adjustment: Adjust the defect detection threshold in real-time according to environmental parameters (such as light intensity, temperature, etc.). For example, establish a relationship model between the threshold and environmental parameters through linear regression or other regression methods, and dynamically adjust the threshold.

[0270] In the implementation of adaptive threshold adjustment, it is necessary to collect and analyze the data of environmental parameters (such as light, temperature, etc.) and establish a relationship model between the threshold and environmental parameters. This relationship model can be established by linear regression, polynomial regression, neural network, etc. Specifically, the following steps can be taken:

[0271] Collect environmental parameter data: In practical application scenarios, environmental parameter data related to defect detection should be collected. For example, in glass defect detection, light intensity and environmental temperature data should be collected. These data can be collected by sensors or obtained from other data sources.

[0272] Establish a relationship model: Use the collected environmental parameter data and corresponding threshold labels as a training set to establish a relationship model between the threshold and environmental parameters through a regression model. Common regression models include linear regression, polynomial regression, neural network, etc.

[0273] Model parameter training: Train the parameters of the established relationship model to optimize the fitting ability of the model. During training, data preprocessing such as normalization and standardization should be performed to avoid the influence of data scale differences on the model.

[0274] Threshold adjustment: Apply the trained model to the actual environment and dynamically adjust the threshold according to the real-time collected environmental parameter data. Specifically, the model prediction method can be used to predict the corresponding threshold according to the current environmental parameter data and dynamically adjust the detection threshold.

[0275] Threshold adaptability test: Test the adjusted threshold to evaluate its performance and adaptability in different environments. For example, test the accuracy and false alarm rate of the threshold under different light and temperature conditions.

[0276] Parameter update: Update and optimize the model parameters and thresholds according to the test results and user feedback to improve the robustness and performance of the algorithm.

[0277] In addition, it is also necessary to consider how to collect environmental parameter data in real time and perform preprocessing and storage. Sensor data collection and processing can be implemented using a single-chip microcomputer or an embedded system, while data storage can be achieved using a database or file system, etc.

[0278] 1.4 Integration of multiple algorithms: Combine physical-based methods and learning-based methods through ensemble learning methods such as voting strategy, Bootstrap aggregating (Bagging), or Boosting method, to integrate the advantages of multiple algorithms and improve the stability and accuracy of the system under different environmental conditions.

[0279] 2. Possible solutions and analysis in the case of adding components:

[0280] 2.1 Environmental monitoring sensors: Install light sensors (such as photoresistors, photodiodes, etc.) to measure ambient light intensity; install temperature sensors (such as Negative Temperature Coefficient (NTC) thermistors, thermocouples, etc.) to monitor ambient temperature. These data will be used to assist in adjusting algorithm parameters and improve the adaptability of the system to environmental changes.

[0281] 2.2 Optical filter: According to the spectral response characteristics of the signal acquisition sensor, select appropriate narrowband optical filters to reduce the influence of ambient light on the sensor signal and improve signal quality.

[0282] 2.3 Sensor heating or refrigeration device: Install a heating element or Peltier (semiconductor refrigeration plate) refrigerator for the sensor to maintain its operation within the appropriate temperature range. Through a closed-loop control system, the sensor temperature is monitored in real time and the power of the heating / cooling device is adjusted to improve the stability and reliability of the system under different temperature environments.

[0283] 2.4 Moisture-proof treatment: Use silica gel desiccant, moisture-proof coating, or sealed box, etc. to prevent the sensor and circuit from being damp. Moisture-proof treatment helps to improve the stability and reliability of the system in humid environments.

[0284] 2.5 Redundant design: Use multiple sensor redundancy design on key components, such as using dual sensors to monitor the same area simultaneously, and then fusing their outputs. Redundant design can improve the reliability and anti-interference ability of the system in harsh environments.

[0285] 2.6 High-Dynamic Range (HDR) imaging: Install an HDR module for camera-type sensors to enable them to acquire clear images under high-contrast lighting conditions, reducing the impact of ambient light on image quality.

[0286] 2.7 Warning system adapts to environmental changes: The warning system should be able to dynamically adjust the warning level and notification method according to environmental parameters (such as light, temperature, etc.). For example, in high light conditions, the warning level can be increased, and more obvious visual and auditory prompts can be used to remind users; in low temperature environment, warning information can be sent through vehicle information system or mobile device to avoid long-term exposure of users in cold environment.

[0287] III. Scalability and applicability

[0288] 1. Adapt to different types of glass

[0289] For different types of glass (such as architectural glass, home glass, etc.), a wider range of applicability can be achieved by adjusting the sensor type and parameters. For example: for architectural glass, sensors with higher resolution can be selected to detect finer cracks and defects; for home glass, low-power sensors can be used to reduce energy consumption while achieving high detection accuracy.

[0290] 2. Adjust sensor configuration

[0291] According to the needs of different application scenarios, the number and layout of sensors can be adjusted to achieve the best detection effect. For example: for large architectural glass, the number and density of sensors can be increased to cover a larger area;

[0292] For smaller home glass, the number of sensors can be reduced to reduce cost and complexity.

[0293] 3. Customize detection methods and algorithms

[0294] For different scenarios and applications, detection methods and algorithms can be customized. For example: in architectural glass applications, detection algorithms can be developed for specific types of cracks and defects (such as stress cracks, thermal fatigue, etc.);

[0295] In home glass applications, detection can be performed for common stains and scratches and other defects.

[0296] Through the above methods, the scalability and applicability of the glass defect detection system in different types of glass and application scenarios can be achieved, thereby providing wider application potential.

[0297] IV. Application

[0298] This embodiment has multiple application fields and can be integrated with other related technologies.

[0299] 1. Technical optimization

[0300] High-performance sensors: With the continuous development of sensor technology, higher resolution and sensitivity sensors can be adopted in the future to improve the accuracy and efficiency of defect detection.

[0301] Deep learning algorithms: By introducing deep learning algorithms such as Convolutional Neural Network (CNN) and Long Short-Term Memory Network (LSTM), the accuracy and robustness of the algorithm can be further improved.

[0302] Adaptive algorithms: Develop adaptive algorithms that can automatically adjust parameters according to environmental changes to improve system performance in complex environments.

[0303] 2、Application field expansion

[0304] New building materials: Apply glass defect detection technology to the detection of other new building materials, such as photovoltaic glass, high-performance glass fiber, etc.

[0305] Industrial field: In the industrial production process, glass defect detection technology can be used for real-time monitoring and early warning to improve production efficiency and quality.

[0306] Security field: Combine video monitoring and other technologies to apply glass defect detection technology to the security field, such as monitoring the safety of building or vehicle glass.

[0307] 3、Integration with other related technologies

[0308] Internet of Things: By combining glass defect detection systems with Internet of Things technology, remote monitoring and management can be achieved, improving early warning efficiency.

[0309] Big data analysis: Use big data analysis technology to collect and analyze a large amount of glass defect detection data, providing a basis for optimizing algorithms and developing prevention strategies.

[0310] Artificial intelligence: Integrate glass defect detection technology with other artificial intelligence technologies (such as image recognition, natural language processing, etc.) to achieve higher levels of automation and intelligence.

[0311] This embodiment has wide application in the following fields:

[0312] Lensless and light source-free glass defect monitoring: With the development of lensless and light source-free monitoring technology, low-cost and high-efficiency monitoring of glass defects can be achieved, expanding the application potential in various application scenarios.

[0313] New building materials: Apply glass defect detection technology to the detection of other new building materials, such as photovoltaic glass, high-performance glass fiber, etc.

[0314] Industrial field: In industrial production processes, glass defect detection technology can be used for real-time monitoring and early warning, improving production efficiency and quality.

[0315] Security field: Combined with video monitoring and other technologies, glass defect detection technology can be applied to the security field, such as monitoring the safety of building or vehicle glass, preventing potential dangers.

[0316] Intelligent transportation: In intelligent transportation systems, glass defect detection technology can be used to monitor the condition of vehicle glass, improve driving safety and passenger comfort.

[0317] Smart home: Applying glass defect detection technology to the home field can monitor the condition of home glass doors and windows in real time, improving home safety and convenience.

[0318] The embodiments of the present application have the following advantages:

[0319] 1. Real-time monitoring and early warning: The embodiments of the present application can detect and warn glass defects in real time, provide safety risk information to users in time, and reduce potential harm.

[0320] 2. High accuracy and robustness: Advanced signal processing and machine learning algorithms are used to improve the accuracy of defect detection and the stability of the system in complex environments.

[0321] 3. Scalability and applicability: By adjusting the sensor configuration and parameters, the embodiments of the present application can be applied to different types of glass and scenes, and have wide applicability.

[0322] 4. Integration with existing equipment: The embodiments of the present application can be easily integrated with existing equipment (such as cars, buildings, etc.), realizing real-time monitoring and early warning in practical applications.

[0323] 5. No lens and passive monitoring: The embodiments of the present application use a lens-free and passive method, reducing system complexity and cost, and improving practicality.

[0324] The embodiments of the present application also have the following characteristics:

[0325] 1. Multi-sensor fusion: The embodiments of the present application use multiple types of sensors to effectively capture and fuse signals of different dimensions, improving the performance of defect detection.

[0326] 2. Adaptive algorithm: By developing an adaptive algorithm that can automatically adjust parameters according to environmental changes, the performance of the system in complex environments is improved.

[0327] 3. Environmental adaptability: Solutions are proposed for environmental factors such as light changes and temperature fluctuations to ensure the stability and reliability of the system under various environmental conditions.

[0328] The glass defect monitoring device proposed by the embodiment of the present application does not use lenses and external light sources, uses the side-mounted sensor to detect defects in real time during the use of the glass, and through the organic combination of the sensor module, the signal processing module, the defect detection algorithm module and the real-time early warning module, the device proposed by the embodiment of the present application can effectively detect defects without using lenses and light sources, and provides a feasible and innovative solution for specific scenarios (such as automobile glass).

[0329] The embodiment of the present application can be applied to the fields of automobiles, building glass and the like, and has a wide application prospect. The traditional glass defect detection method needs to use lenses and external light sources, while the method proposed by the embodiment of the present application not only can avoid these cost and technical limitations, but also can improve the detection precision and efficiency. The glass defect monitoring method proposed by the embodiment of the present application has innovation and practicality, and can improve the production efficiency and quality control capability of the glass manufacturing industry.

[0330] The above is a further detailed description of the present application in combination with a specific preferred embodiment, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A glass defect monitoring apparatus, characterized by, The method comprises a sensor for collecting a detection signal of the glass and a detection signal processing device for determining whether there is a defect on the glass surface according to the detection signal through a trained neural network, and determining the type and location of the defect when a defect is determined; wherein the sensor comprises two or more sensors arranged on the side of the glass, one of which is an optical sensor, and the others include one or more of an infrared sensor, an electromagnetic wave sensor, an acoustic sensor, and a capacitive sensor; the number of each sensor is one or more; The neural network is a multi-task graph neural network (MT-GNN); wherein the glass surface is divided into a set of grids, and each grid cell is regarded as a node in the graph, the connection between the nodes represents the relationship between the adjacent grid cells, and each node is assigned a feature vector containing feature data from the sensor, thereby constructing a graph structure of the glass surface; wherein the graph neural network has an input layer, a plurality of graph convolution layers, and an output layer, the input layer receives a feature vector obtained from a physical signal, the graph convolution layer is used to learn the relationship between the nodes and propagate the information between the nodes, and the output layer is used to process the output of the defect prediction task; the graph neural network includes the output layers of the following four tasks: a defect existence output layer, which performs binary classification on each node to determine whether the node has a defect; a defect approximate location output layer, which performs regression on each node to predict the relative location of the defect in the node; a defect type classification output layer, which performs multi-classification on each node to identify the type of defect in the node; and a defect detailed location output layer, which performs dense regression on each node to predict the detailed location of the defect in the node.

2. The glass defect monitoring apparatus of claim 1, wherein, The optical sensor detects the reflection, transmission, scattering, and refraction optical information related to the defect on the glass surface; the infrared sensor detects the thermal characteristic information related to the defect on the glass surface; The acoustic sensor detects the acoustic wave reflection characteristic information related to the defect on the glass surface; the electromagnetic wave sensor detects the electromagnetic wave reflection characteristic information related to the defect on the glass surface; and the capacitive sensor detects the capacitive characteristic information related to the defect on the glass surface.

3. The glass defect monitoring apparatus of claim 1, wherein, The method further comprises a pre-warning module for performing a pre-warning operation according to the judgment result of the detection signal processing device, specifically including: receiving the defect prediction result from the neural network, and integrating to form a defect information list; traversing the defect information list, and determining whether to trigger a pre-warning according to a pre-set pre-warning threshold and a confidence level of the defect; when the pre-warning is triggered, performing one or more operations of displaying, sending, and recording the pre-warning information.

4. The glass defect monitoring apparatus of claim 3, wherein, The method further comprises a communication module for sending the pre-warning information to a designated monitoring device.

5. The glass defect monitoring apparatus of claim 1, wherein, The number of each sensor is one or more, the detection signal processing device performs defect detection by fusing the detection signals of multiple sensors, and the detection signals of the multiple sensors include multiple detection signals of the same type or different types.

6. The glass defect monitoring apparatus of any of claims 1 to 5, wherein, The glass includes glass of a vehicle, a ship, an aircraft, a building, industrial equipment, a photovoltaic device or a household device, and the glass defect monitoring device can detect defects in real time during use of the glass.

7. The glass defect monitoring apparatus of any of claims 1 to 5, wherein, Each layer of graph convolution layer performs feature processing operation according to the following formula: X^(l+1) = sigma (AX^(l)W^(l)) Wherein, X^(l) represents the node feature matrix of the lth layer, X^(l+1) represents the node feature matrix of the l+1th layer, A represents the adjacency matrix, W^(l) represents the weight matrix of the lth layer, and sigma represents the activation function.

8. The glass defect monitoring apparatus of any of claims 1 to 5, wherein, The detection signal processing device further adaptively adjusts the defect detection threshold of the detection signal according to the environmental parameter data, and specifically comprises: Using the environmental parameter data and the corresponding detection threshold label as a training set, training a relationship model between the detection threshold and the environmental parameter; Through the relationship model, the corresponding threshold is predicted according to the current environmental parameter data, and the defect detection threshold is dynamically adjusted.

Images