Image Recognition Method Based on Multi-Branch Reverse Residual Network Model and its Application in Glass Inspection
By combining a multi-branch inverse residual network model with ultrasonic measurement, the problems of large model parameters and insufficient accuracy in glass testing are solved, enabling rapid and accurate assessment of glass stress and health status, applicable to the testing of transportation vehicles and architectural glass.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHAANXI NORMAL UNIV
- Filing Date
- 2023-06-16
- Publication Date
- 2026-05-19
AI Technical Summary
In existing glass testing technologies, high-performance network models have large parameters, require high-performance computing equipment, and have long inference times, while lightweight models have insufficient accuracy and cannot effectively assess the health status of glass during use.
A lightweight image recognition method based on a multi-branch inverse residual network model is adopted. By enhancing processing, multi-branch inverse residual structure and attention mechanism, glass features are extracted. Combined with ultrasonic measurement of glass stress, a fast and accurate glass stress detection and health status assessment can be achieved.
While reducing the number of model parameters and computational load, it improves the accuracy and speed of glass inspection, enabling non-destructive and continuous inspection of glass stress and health status, and is suitable for the regular maintenance of glass in transportation vehicles and buildings.
Smart Images
Figure CN116681988B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image recognition technology, and relates to an image recognition method based on a multi-branch inverse residual network model and its application in glass detection. Background Technology
[0002] In our daily lives, glass windows, glass doors, and glass walls are ubiquitous, mostly made of glass. Because glass is a transparent surface, glass inspection is largely focused on quality control during the production process. For example, document 202310172528.6 describes a glass inspection device and method. This method involves cameras at different positions automatically scanning the glass from different angles as it moves at a constant speed on a conveyor belt. The scan results are transmitted to a computer, and operators then analyze the scanned images to determine if the four edges of the glass are damaged. However, this method only addresses quality control during the production stage and cannot assess the health condition of the glass during use. There is also a glass detection method based on deep learning and ghosting phenomenon (202310128767.1), which mainly addresses the problem that computer vision systems such as robots and drones often overlook the glass surface, frequently detecting areas that are not the glass surface but rather scenes penetrating the glass, thus affecting detection accuracy. This method primarily performs glass detection on a single original input image (an RGB image). Based on the original input image, a deep learning method using a backbone network is used to extract ghosting features and obtain a predicted ghosting region map. After connecting the predicted ghosting region map with the original input image channels, glass features are extracted based on the backbone network under the guidance of ghosting cues. Then, a convolutional neural network is used to decode the glass features and segment the glass region, outputting a predicted glass region map. Compared to existing technologies, this application performs glass detection based on a single image, making it applicable to a wider range of situations. Using a backbone network allows for more accurate and efficient extraction of ghosting features and glass region features, while utilizing the ghosting phenomenon enables more accurate localization of the glass region, obtaining a high-quality predicted glass region map. However, it is merely a ghosting processing technique for the glass and background images during the detection process and cannot directly assess the quality of the glass.
[0003] Image recognition, as an important application area of computer vision, is of great significance to the innovative development of artificial intelligence algorithms. With the iteration of deep learning theory and computing devices, more and more high-performance models are being applied in fields such as face recognition, fingerprint recognition, vehicle recognition, satellite cloud image recognition and clinical medical diagnosis.
[0004] Deep learning has evolved from LeNet to AlexNet, and later to VGG, GoogleNet, ResNet, DenseNet, and so on. With the continuous development of these iconic models, the accuracy of image recognition has increased, and their applications have become more and more widespread.
[0005] However, the following problems still exist in the image processing of glass detection technology: (1) High-performance network models have large parameters, high requirements for computing equipment, and long inference time; (2) Networks with small model parameters are easy to deploy, but the accuracy is insufficient.
[0006] Therefore, the technology of applying deep learning network models to glass detection is still in the exploratory stage and there are still shortcomings in its application. Summary of the Invention
[0007] To address the technical problems of existing image recognition network models used in glass inspection, such as large number of model parameters, insufficient accuracy of lightweight models, and long training time, this invention provides an image recognition method based on a multi-branch inverse residual network model, which features high accuracy, small number of model parameters, and short training time.
[0008] This invention also provides an image recognition method for the multi-branch inverse residual network model, which can be applied to glass stress detection and glass health status assessment, and can accurately and quickly detect the stress value and health status of the glass.
[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0010] An image recognition and classification method based on a lightweight multi-branch inverse residual network model includes the following steps:
[0011] S1) Enhance the image to obtain the original matrix W;
[0012] S2) Input the original matrix W into the first preprocessing module, and after denoising by 5*5 convolution and 3*3 pooling, the feature matrix W0 is obtained.
[0013] S3) Input the feature matrix W0 into the second feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W1.
[0014] S4) Input the feature matrix W1 into the third feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W2.
[0015] S5) Input the feature matrix W2 into the fourth feature extraction module. After convolution and pooling through two stacked multi-branch inverse residual structures, extract the matrix information to obtain the inverse residual feature W3.
[0016] S6) Input the feature matrix W3 into the fifth feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W4.
[0017] S7) Input the inverse residual feature W4 into the adaptive average pooling layer, and then into the fully connected layer. Combine the extracted distribution features to form a one-dimensional feature vector. Then, perform a weighted summation on the feature vector and output it to complete the image recognition and classification.
[0018] The output characteristics of the multi-branch inverse residual structure in steps (S3), (S4), (S5), and (S6) are as follows:
[0019]
[0020] Among them, W i (x) represents the feature output by the i-th layer, i = 1, 2, 3, 4; I is the IremulbNet network, Mish is the activation function, and F... i The diagram illustrates the convolutional relationship of the intermediate hidden layers. w05 and w06 represent the results of concatenating and summing the dimensions of the feature matrix during feature extraction, respectively.
[0021] Further specifying, step S3) specifically includes:
[0022] S3.1) Input the feature matrix W0 into the second feature extraction module, and first perform dimensionality operation through a 1*1 first convolution to obtain matrix W01;
[0023] S3.2) Matrix W01 is input to the first branch, the second branch, and the third branch respectively. In the first branch, it undergoes a 3*3 depthwise convolution and then a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the main feature matrix W02. In the second branch, it undergoes a 5*5 depthwise convolution and then a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the large receptive field matrix W03. In the third branch, it undergoes a 3*3 pooling process or is directly passed to the SE attention module, outputting matrix W04.
[0024] S3.3) Merge the main feature matrix W02, the large receptive field matrix W03, and the matrix W04 extracted in step S3.2) along the channel dimension to obtain the merged matrix W05; at the same time, sum W02, W03, and W04 to obtain W06; concatenate W05 and W06 along the channel, and perform a 1*1 second convolution pointwise to obtain the multi-branch information matrix W07;
[0025] S3.4) Summing the multi-branch information matrix W07 output from step S3.3) with the feature matrix W0 after dimensional transformation, we obtain the inverse residual feature W1.
[0026] This invention also provides a glass stress detection method based on a lightweight multi-branch inverse residual network model, which includes the following steps:
[0027] 1) The change of residual stress in glass is measured by comparing the ultrasonic wave with the nonlinear coefficient, and a time-domain image of glass stress is obtained.
[0028] 2) Perform short-time Fourier transform on the time-domain image of glass stress to obtain the time-frequency domain image, and establish a dataset;
[0029] 3) Enhance the image data in the dataset from step 2) to obtain the original matrix W;
[0030] 4) Input the original matrix W from step 3) into the lightweight multi-branch inverse residual network model and train it according to the method of claim 1 to extract the features of the image and obtain the weight model file;
[0031] 5) Load the weight model file trained in step 4) into the prediction program, input the time domain data of the stress of the glass to be tested, and obtain the stress value of the glass to be tested by model recognition and prediction according to the methods in steps 1) to 4).
[0032] Further specifying, step 1) specifically includes:
[0033] 1.1) Install the transmitting end and receiving end of the ultrasonic transmitting and receiving device on both sides of the glass, respectively;
[0034] 1.2) A periodic pulsed sinusoidal signal generated by the signal processing unit is input to the transmitting end to emit pulsed ultrasonic waves, which are received by the receiving end after passing through the glass 8;
[0035] 1.3) The received ultrasonic signal is input into the signal processing unit, and the relative nonlinear coefficient of the glass is obtained by taking into account that the relative nonlinear coefficient value is proportional to the stress of the stress-free standard glass.
[0036] 1.4) Signal processing unit 4 correlates the nonlinear coefficient with its corresponding stress, plots the curve of the nonlinear coefficient as a function of stress, and obtains the time-domain image of glass stress.
[0037] Further specifying, step 4) specifically includes:
[0038] 4.1) Input the original matrix W from step 3) into the first preprocessing module, and after denoising by 5*5 convolution and 3*3 pooling, obtain the feature matrix W0;
[0039] 4.2) Input the feature matrix W0 into the second feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W1.
[0040] 4.3) Input the inverse residual feature W1 into the third feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W2.
[0041] 4.4) Input the inverse residual feature W2 into the fourth feature extraction module. After passing through two stacked multi-branch inverse residual structures for convolution and pooling, extract the matrix information to obtain the inverse residual feature W3.
[0042] 4.5) Input the inverse residual feature W3 into the fifth feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W4.
[0043] 4.6) Input the inverse residual feature W4 into the adaptive average pooling layer, and then into the fully connected layer. Combine the extracted distribution features to form a one-dimensional feature vector. Then, perform a weighted summation on the feature vector and output the weight model file.
[0044] The output characteristics of the multi-branch inverse residual structure in steps (4.2), (4.3), (4.4), and (4.5) are as follows:
[0045]
[0046] Among them, W i (x) represents the feature output by the i-th layer, i = 1, 2, 3, 4; I is the IremulbNet network, Mish is the activation function, and F... i The diagram illustrates the convolutional relationship of the intermediate hidden layers. w05 and w06 represent the results of concatenating and summing the dimensions of the feature matrix during feature extraction, respectively.
[0047] Further specifying, step 4.2) specifically includes the following training process:
[0048] 4.2.1) Input the feature matrix W0 into the second feature extraction module, and first perform dimensionality operation through a 1*1 first convolution to obtain matrix W01;
[0049] 4.2.2) Matrix W01 is input to the first branch, the second branch, and the third branch respectively. In the first branch, it is processed by a 3*3 depthwise convolution and then by a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the main feature matrix W02. In the second branch, it is processed by a 5*5 depthwise convolution and then by a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the large receptive field matrix W03. In the third branch, it is processed by a 3*3 pooling process or directly passed to the SE attention module, outputting the third branch matrix W04.
[0050] 4.2.3) Merge the main feature matrix W02, the large receptive field matrix W03, and the third branch matrix W04 extracted in step 4.2.2) along the channel dimension to obtain the merged matrix W05; at the same time, sum W02, W03, and W04 to obtain W06; concatenate W05 and W06 along the channel, and perform a 1*1 second convolution pointwise to obtain the multi-branch information matrix W07;
[0051] 4.2.4) Summing the multi-branch information matrix W07 output from step 4.2.3) with the feature matrix W0 after dimensional transformation, we obtain the inverse residual feature W1.
[0052] This invention also provides a glass health assessment method based on a lightweight multi-branch inverse residual network model, which includes the following steps:
[0053] (1) The stress value of the glass is detected by the glass stress detection method based on the lightweight multi-branch inverse residual network model according to any one of claims 3 to 6;
[0054] (2) If the stress value is less than 60% of the maximum stress value, the glass is in good condition; if the stress value is 60% to 80% of the maximum stress value, it is a yellow warning, and the glass is at risk of breaking; if the stress value is less than 80% of the maximum stress value, it is a red warning, and the glass is at greater risk of breaking.
[0055] The present invention also provides a glass stress detection system based on a lightweight multi-branch inverse residual network model, which includes an ultrasonic transmitting and receiving device and a signal processing unit.
[0056] An ultrasonic transmitting and receiving device is set on both sides of the glass to be tested, including a transmitting end and a receiving end. The transmitting end emits pulsed ultrasonic signals, which pass through the glass and are received by the receiving end, which then transmits the received signals to the signal processing unit.
[0057] The signal processing unit is used to send periodic pulse sinusoidal signals to the transmitting end of the ultrasonic transmitting and receiving device, and simultaneously receive ultrasonic signals from the receiving end of the ultrasonic transmitting and receiving device. The signal is processed using the glass stress detection method based on the lightweight multi-branch inverse residual network model as described in claim 3, and the glass state status is output.
[0058] Further specified, the signal processing unit includes a central processing module, a signal preprocessing module, and a multi-branch inverted residual signal processing module;
[0059] Central processing module: Used to emit periodic pulse sinusoidal signals and receive ultrasonic signals, transmit the received ultrasonic signals to the signal preprocessing module, and receive the glass stress value transmitted by the multi-branch inverse residual signal processing module and output it.
[0060] Signal preprocessing module: Receives the ultrasound signal sent by the central processing module, performs short-time Fourier transform on the time-domain data to obtain a time-frequency domain image, extracts the feature information of the time-frequency domain image, and forwards the information to the multi-branch inverse residual signal processing module;
[0061] Multi-branch inverse residual signal processing module: The enhanced image information is trained in a multi-branch inverse residual network model, the weight model file is extracted, and the weight model file is loaded into the prediction program. It is compared with the information of the glass stress to be detected, and the glass stress value obtained by comparison is transmitted to the central processing module.
[0062] Further defined, the multi-branch inverse residual signal processing module includes a first preprocessing module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, and a sixth classification module connected in sequence;
[0063] The first preprocessing module is used to denoise the original matrix W by passing it through a 5*5 convolutional layer and a 3*3 pooling layer in sequence, and output the feature matrix W0.
[0064] The second feature extraction module is used to extract matrix information by performing convolution and pooling on the feature matrix W0 through a multi-branch inverse residual structure, and output the inverse residual feature W1.
[0065] The third feature extraction module is used to extract matrix information by performing convolution and pooling on the inverse residual feature W1 through a multi-branch inverse residual structure, and output the inverse residual feature W2.
[0066] The fourth feature extraction module is used to extract matrix information by convolution and pooling two consecutive stacked multi-branch inverse residual structures of the inverse residual feature W2, and output the inverse residual feature W3.
[0067] The fifth feature extraction module is used to extract matrix information by performing convolution and pooling on the inverse residual feature W3 through a multi-branch inverse residual structure, and output the inverse residual feature W4.
[0068] The sixth classification module includes an adaptive average pooling layer and a fully connected layer. It is used to process the inverse residual feature W4 through the average pooling layer and the fully connected layer in sequence to combine the distribution features of the extracted image to form a one-dimensional feature vector. The feature vector is then weighted and summed to output a weight model file.
[0069] Further defined, the multi-branch inverted residual structure includes a 1*1 first convolutional layer, a novel feature fusion layer, and a 1*1 second convolutional layer connected in sequence;
[0070] The first 1x1 convolutional layer is used to perform dimensionality operations on the feature matrix by passing it through the first 1x1 convolution.
[0071] The novel feature fusion method comprises three parallel branches: a first branch, a second branch, and a third branch. Within the first branch, the feature is processed by a 3x3 depthwise convolution followed by 3x3 pooling before being passed to the SE attention module (or directly to the SE attention module) for channel feature enhancement, outputting the main feature matrix. Within the second branch, the feature is processed by a 5x5 depthwise convolution followed by 3x3 pooling before being passed to the SE attention module (or directly to the SE attention module) for channel feature enhancement, outputting the large receptive field matrix. Within the third branch, the feature is processed by 3x3 pooling or directly passed to the SE attention module, outputting the third branch matrix. The main feature matrix, the large receptive field matrix, and the third branch matrix are then merged along the channel dimension to output a merged matrix. Simultaneously, the main feature matrix, the large receptive field matrix, and the third branch matrix are summed to obtain a summation matrix. Finally, the merged matrix and the summation matrix are concatenated along the channel dimension to complete the novel feature fusion.
[0072] The 1*1 second convolutional layer processes the matrix after the fusion of new features through pointwise convolution, and outputs a multi-branch information matrix.
[0073] The inverse residual feature is obtained by summing the output multi-branch information matrix with the feature matrix after dimensional transformation.
[0074] Compared with the prior art, the beneficial effects of the present invention are:
[0075] 1) The lightweight multi-branch inverse residual network model used in this invention reconstructs an inverse residual structure, which combines Inception structure, inverse residual structure and attention mechanism and other technologies to improve classification accuracy while greatly reducing the number of model parameters and computation.
[0076] 2) This invention reduces the number of channels by utilizing the output of the inverted residual structure, and uses depthwise separation convolution in multi-branch convolution. The operation of convolution with different sizes can increase the diversity of features. At the same time, the feature fusion stage changes the width of the network, which effectively reduces the number of model parameters and computational cost.
[0077] 3) In the feature extraction stage, this invention uses a special feature fusion method. The feature information obtained from the three branches is concatenated and summed. Then, the two results are concatenated along the channel dimension. The output feature matrix contains richer and more complex feature information, which effectively enhances the feature extraction capability and can better propagate errors during backpropagation.
[0078] 4) Compared with existing stacked residual and inverse residual structures, the multi-branch structure of this invention can mix feature information from multiple different receptive fields during the feature extraction stage, which can enhance the diversity of features. Moreover, it can make different adjustments based on whether downsampling is used to effectively transmit feature information. The SE module supplements the lack of attention to channel information, thus having better feature extraction capabilities, which is conducive to improving the network's learning ability and significantly improving the network's performance.
[0079] 5) The lightweight multi-branch inverse residual network model of the present invention is applied to stress detection of glass, which can realize in-service, non-destructive and continuous detection of glass. Moreover, the detection speed is fast and the results are accurate. It can be applied to the periodic maintenance of transportation vehicles such as aircraft and high-speed trains or the periodic inspection and evaluation of glass curtain walls, which can greatly improve the detection efficiency and ensure the detection results, and greatly reduce the quality and safety accidents of in-service glass. Attached Figure Description
[0080] Figure 1 This is a schematic diagram of the lightweight multi-branch inverse residual network IremulbNet.
[0081] Figure 2 This is a schematic diagram of a multi-branch inverted residual module;
[0082] Figure 3 This is a schematic diagram of the SE module;
[0083] Figure 4 This is a schematic diagram of the principle of glass internal stress detection.
[0084] Figure 5A This is a time-domain image of the internal stress in the glass;
[0085] Figure 5B This is a frequency domain image of the internal stress of the glass;
[0086] Figure 5C This is a time-frequency domain image of the internal stress in the glass.
[0087] Figure 6AThis is a comparison chart of the parameter quantities of each network model in Example 1;
[0088] Figure 6B This is a comparison chart of the computational complexity of each network model in Example 1; Detailed Implementation
[0089] The invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0090] The experimental environment for this invention is a server running CentOS with an Intel(R) Xeon(R) E5-2640 v3 processor with a clock speed of 2.6GHz and 125.65GB of memory. The graphics card is a GeForce RTX 3090 with 24GB of video memory. The program environment is Python version 3.9.1, PyTorch version 1.10.0, and CUDA version 11.3.
[0091] The technical idea of this invention is to establish a lightweight multi-branch inverse residual network model (hereinafter referred to as IremulbNet) for image recognition and classification. The reconstructed inverse residual structure is composed of a multi-branch structure, an attention mechanism, and the inverse residual module of the Mobilenet v2 network. It can improve network performance and increase computational efficiency while significantly reducing the number of parameters and computational load.
[0092] The lightweight multi-branch inverse residual network model IremulbNet provided by this invention includes a first preprocessing module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, and a sixth classification module connected in sequence; the second, third, and fifth modules are composed of a multi-branch inverse residual module; the fourth feature extraction module is composed of two multi-branch inverse residual modules stacked together.
[0093] Furthermore, the following section details the reconstruction of the inverse residual structure in the lightweight multi-branch inverse residual network IremulbNet of this invention, focusing on multi-branch feature extraction and novel feature fusion methods. (See also...) Figure 1 .
[0094] In the feature extraction stage, the feature information w0 obtained after a 1x1 dimensionality transformation is processed using a three-branch structure. Branch one first extracts features using a 3x3 depthwise convolution. Then, if the stride is 2, it performs downsampling using 3x3 max pooling, reducing computation while preserving the main features. If the stride is 1, the features are directly passed on. Next, the SE attention mechanism is used to assign channel weights to the resulting feature matrix. Branch two is similar to branch one, except that it uses a 5x5 depthwise convolution. Branch three determines whether to perform 3x3 pooling downsampling on w0 or pass it directly based on the stride information.
[0095] Compared to stacked residual and inverse residual structures, the multi-branch structure of this invention can mix feature information from multiple different receptive fields during the feature extraction stage, which can enhance feature diversity. Moreover, it can make different adjustments based on whether downsampling is used to effectively transmit feature information. The SE module supplements the lack of attention to channel information, thus having better feature extraction capabilities, which is conducive to improving the network's learning ability and significantly improving the network's performance.
[0096] Depthwise separable convolution plays a crucial role in reducing the number of parameters and computational cost of the model. It consists of depthwise convolution and pointwise convolution. Depthwise convolution is a Dk×Dk convolution with M channels, where each convolution kernel corresponds to one channel. Pointwise convolution is a 1×1 M-channel convolution.
[0097] For an input feature map of size Df×Df with M channels, and an output feature map of size Df×Df with N channels, the ratio of the number of parameters in a depthwise separable convolution to that in a standard convolution is shown in Equation (1), and the ratio of its computational complexity is shown in Equation (2).
[0098]
[0099]
[0100] Where Df is the feature map size, Dk is the convolution kernel size, M is the number of input channels, and N is the number of output channels.
[0101] In the IremulbNet network, multi-branch feature extraction uses 5x5 and 3x3 convolutional kernels, i.e., Dk=5 and Dk=3. Compared to the traditional convolution process, the number of parameters in DSC can be significantly reduced, and the number of output channels of the multi-branch inverse residual structure is also significantly reduced. Therefore, the IremulbNet model of this invention uses a multi-branch inverse residual structure, which greatly reduces the number of parameters in the model.
[0102] The SE module consists of two steps: squeezing and excitation. Squeeze compresses the spatial information of the image to a 1×1 shape, retaining only the channel information. Excitation predicts the importance of each channel and multiplies these predictions by the original feature map. The most significant feature of the SE module is its channel-level attention mechanism, and its structure is as follows: Figure 3 As shown, for an image with C channels and a size of W×H, it first uses global pooling and fully connected layers to preserve the image channel information, resulting in a 1×1 feature matrix wc with C channels. Then, it multiplies this feature matrix with the original image information and assigns weights according to the channels. As a lightweight module, adding it to an existing network only increases computational cost and can significantly improve performance.
[0103] There are two common methods for feature fusion: one is the dimensional concatenation of feature matrices, exemplified by the Inception structure; the other is the addition of feature data of the same dimension in the residual structure and the inverse residual structure. The feature fusion method used in the model of this invention combines both methods. The results out1, out2, and out3 processed in each of the three branches are concatenated dimensionally to obtain w1. Simultaneously, the results of the three are summed to obtain w2. Finally, w1 and w2 are concatenated as the final result and input into the next convolutional layer.
[0104] The features output by the i-th layer in existing ResNet and MobileNet models are:
[0105]
[0106] The characteristics of the output of the i-th layer in the inverse residual model of this invention are:
[0107]
[0108] Among them, W i (x) represents the feature output by the i-th layer, I is the IremulbNet network, Mish is the activation function, and F... i The diagram illustrates the convolutional relationship of the intermediate hidden layers. w05 and w06 represent the results of concatenating and summing the dimensions of the feature matrix during feature extraction, respectively.
[0109] From formulas (4) and (5), it can be seen that the IremulbNet of the present invention contains richer and more complex feature information in its output feature matrix compared with the existing ResNet and MobileNet models, which effectively enhances the feature extraction capability and can better propagate errors in backpropagation.
[0110] The max pooling layer and adaptive average pooling layer of this invention are known in the art. Max pooling means that the maximum value in the receptive field is preserved after pooling, and average pooling means that the average value of all data in the receptive field is preserved after pooling.
[0111] The reconstructed multi-branch inverse residual network model described above is then used for image recognition and further applications.
[0112] Specifically, the image recognition and classification method of the present invention based on a lightweight multi-branch inverse residual network model, see [link to relevant documentation]. Figures 2-3 This includes the following steps:
[0113] S1) Enhance the image to obtain the original matrix W;
[0114] S2) Input the original matrix W into the first preprocessing module, and after denoising by 5*5 convolution and 3*3 pooling, the feature matrix W0 is obtained.
[0115] S3) Input the feature matrix W0 into the second feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W1; specifically:
[0116] S3.1) Input the feature matrix W0 into the second feature extraction module, and first perform dimensionality operation through a 1*1 first convolution to obtain matrix W01;
[0117] S3.2) Matrix W01 is input to the first branch, the second branch, and the third branch respectively. In the first branch, it undergoes a 3*3 depthwise convolution and then a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the main feature matrix W02. In the second branch, it undergoes a 5*5 depthwise convolution and then a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the large receptive field matrix W03. In the third branch, it undergoes a 3*3 pooling process or is directly passed to the SE attention module, outputting matrix W04.
[0118] S3.3) Merge the main feature matrix W02, the large receptive field matrix W03, and the matrix W04 extracted in step S3.2) along the channel dimension to obtain the merged matrix W05; at the same time, sum W02, W03, and W04 to obtain W06; concatenate W05 and W06 along the channel, and perform a 1*1 second convolution pointwise to obtain the multi-branch information matrix W07;
[0119] S3.4) Summing the multi-branch information matrix W07 output from step S3.3) with the feature matrix W0 after dimensional transformation, we obtain the inverse residual feature W1.
[0120] S4) Input the feature matrix W1 into the third feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W2.
[0121] S5) Input the feature matrix W2 into the fourth feature extraction module. After convolution and pooling through two stacked multi-branch inverse residual structures, extract the matrix information to obtain the inverse residual feature W3.
[0122] S6) Input the feature matrix W3 into the fifth feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W4.
[0123] S7) Input the inverse residual feature W4 into the adaptive average pooling layer, and then into the fully connected layer. Combine the extracted distribution features to form a one-dimensional feature vector. Then, perform a weighted summation on the feature vector and output the weight model file to complete the image recognition and classification.
[0124] The output characteristics of the multi-branch inverse residual structure in steps (S3), (S4), (S5), and (S6) are as follows:
[0125]
[0126] Among them, W i (x) represents the feature output by the i-th layer, i = 1, 2, 3, 4; I is the IremulbNet network, Mish is the activation function, and F... i The diagram illustrates the convolutional relationship of the intermediate hidden layers. w05 and w06 represent the results of concatenating and summing the dimensions of the feature matrix during feature extraction, respectively.
[0127] Furthermore, such as Figure 2 As shown, the multi-branch inverse residual module includes a 1*1 first convolutional layer, a novel feature fusion multi-branch inverse residual structure, and a 1*1 second convolutional layer connected in sequence.
[0128] Furthermore, the lightweight multi-branch inverse residual network model is characterized in that the novel feature fusion multi-branch inverse residual structure includes a first branch, a second branch, and a third branch in parallel; the first branch, the second branch, and the third branch are simultaneously connected after the 1*1 first convolutional layer; and the branches are connected to the 1*1 second convolutional layer after novel feature fusion.
[0129] Furthermore, the lightweight multi-branch inverse residual network model is characterized in that the first branch includes sequentially connected 3*3 depthwise convolutions, which pass through a 3*3 pooling layer when the stride is 2, and are directly passed through when the stride is 1, and then pass through the SE attention module; the second branch includes sequentially connected 5*5 depthwise convolutions, which pass through a 3*3 pooling layer when the stride is 2, and are directly passed through when the stride is 1, and then pass through the SE attention module; the third branch similarly passes through a 3*3 pooling layer when the stride is 2, and is directly passed through when the stride is 1.
[0130] Furthermore, the lightweight multi-branch inverse residual network model is characterized in that the first preprocessing module sequentially includes a 5*5 convolutional layer and a 3*3 pooling layer.
[0131] Furthermore, the second, third, fourth, and fifth feature extraction modules sequentially include a novel feature fusion multi-branch inverted residual structure.
[0132] Furthermore, the sixth classification module sequentially includes an adaptive average pooling layer and a fully connected layer.
[0133] This invention designs a lightweight CNN model with multi-branch structure, attention mechanism, and inverse residual structure as its core, namely the lightweight multi-branch inverse residual network model IremulbNet. In the multi-branch feature extraction stage, multiple branches are used for different operations to increase feature diversity. A 5x5 convolutional layer is used to expand the receptive field. Simultaneously, an SE attention mechanism is used after convolution to better configure information according to channels. To reduce the number of model parameters, the inverse residual structure and depthwise separable convolution are introduced into the model design. In the classification stage, an adaptive average pooling layer is used to compress the size of the output feature map, preserving invariant feature information. A fully connected layer is then designed to combine distributed features and achieve classification.
[0134] This image recognition and classification method based on a lightweight multi-branch inverse residual network model is applied to glass inspection, involving a glass stress detection method based on a lightweight multi-branch inverse residual network model. (See [link to relevant documentation]). Figure 4 As shown in Figure 5, this method is implemented by the following steps:
[0135] 1) The change in residual stress within the glass was measured using an ultrasonic contrast nonlinear coefficient method to obtain time-domain data of the stress within the glass; see [link to relevant documentation]. Figure 5A and Figure 5B Specifically:
[0136] 1.1) The transmitting end 1 and the receiving end 2 of the ultrasonic transmitting and receiving device are mounted in pairs on multiple metal guide rails 6 with adjustable spacing. The parallel metal guide rails 6 form an ultrasonic transducer matrix with a certain number and spacing. The ultrasonic transmitting end and the ultrasonic receiving end are controlled by corresponding servo motors 7 and can move independently along the X, Y and Z coordinates. It is determined whether the ultrasonic transmitting end 1 and the ultrasonic receiving end 2 are loaded onto the test point of the glass. If not, the position is readjusted. If yes, proceed to the next step.
[0137] 1.2) The periodic pulse sinusoidal signal generated by the ultrasonic image processing unit 4 is input to the transmitting device 1 to emit pulsed ultrasonic waves, which are received by the receiving device 2 after passing through the glass 8;
[0138] 1.3) The received ultrasonic signal is input into system 4. Based on the fact that the relative nonlinear coefficient value is proportional to the stress of the stress-free standard glass, the relative nonlinear coefficient corresponding to the glass is obtained.
[0139] 1.4) Signal processing unit 4 correlates the nonlinear coefficient with its corresponding stress, plots the curve of the nonlinear coefficient with stress, obtains the image of the residual stress change in the glass, and then obtains the time domain image of the glass stress.
[0140] 2) Perform a short-time Fourier transform (STFT) on the time-domain image of glass stress to obtain a time-frequency domain image, such as... Figure 5C Create a dataset;
[0141] 3) Enhance the image data in the dataset from step 2) to obtain the original matrix W;
[0142] 4) Input the original matrix W from step 3) into the lightweight multi-branch inverse residual network model and train it according to the method of claim 1 to extract image features and obtain the weight model file; specifically:
[0143] 4.1) Input the original matrix W from step 3) into the first preprocessing module, and after denoising by 5*5 convolution and 3*3 pooling, obtain the feature matrix W0;
[0144] 4.2) Input the feature matrix W0 into the second feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W1.
[0145] 4.2.1) Input the feature matrix W0 into the second feature extraction module, and first perform dimensionality operation through a 1*1 first convolution to obtain matrix W01;
[0146] 4.2.2) Matrix W01 is input to the first branch, the second branch, and the third branch respectively. In the first branch, it is processed by a 3*3 depthwise convolution and then by a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the main feature matrix W02. In the second branch, it is processed by a 5*5 depthwise convolution and then by a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the large receptive field matrix W03. In the third branch, it is processed by a 3*3 pooling process or directly passed to the SE attention module, outputting the third branch matrix W04.
[0147] 4.2.3) Merge the main feature matrix W02, the large receptive field matrix W03, and the third branch matrix W04 extracted in step 4.2.2) along the channel dimension to obtain the merged matrix W05; at the same time, sum W02, W03, and W04 to obtain W06; concatenate W05 and W06 along the channel, and perform a 1*1 second convolution pointwise to obtain the multi-branch information matrix W07;
[0148] 4.2.4) Summing the multi-branch information matrix W07 output from step 4.2.3) with the feature matrix W0 after dimensional transformation, we obtain the inverse residual feature W1.
[0149] 4.3) Input the inverse residual feature W1 into the third feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W2.
[0150] 4.4) Input the inverse residual feature W2 into the fourth feature extraction module. After convolution and pooling through two stacked multi-branch inverse residual structures, the matrix information is extracted. That is, the second processing takes the result of the first processing as the input value and performs convolution and pooling through two stacked multi-branch inverse residual structures to extract the matrix information, thus obtaining the inverse residual feature W3.
[0151] 4.5) Input the inverse residual feature W3 into the fifth feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W4.
[0152] 4.6) Input the inverse residual feature W4 into the adaptive average pooling layer, and then into the fully connected layer. Combine the extracted distribution features to form a one-dimensional feature vector. Then, perform a weighted summation on the feature vector and output the weight model file.
[0153] The output characteristics of the multi-branch inverse residual structure in steps (4.2), (4.3), (4.4), and (4.5) above are as follows:
[0154]
[0155] Among them, W i (x) represents the feature output by the i-th layer, i = 1, 2, 3, 4; I is the IremulbNet network, Mish is the activation function, and F... i The diagram illustrates the convolutional relationship of the intermediate hidden layers. w05 and w06 represent the results of concatenating and summing the dimensions of the feature matrix during feature extraction, respectively.
[0156] 5) Load the weight model file trained in step 4) into the prediction program, input the time domain data of the stress of the glass to be tested, and obtain the stress value of the glass to be tested by model recognition and prediction according to the methods in steps 1) to 4).
[0157] After testing, the health condition of the glass is assessed based on the stress value. If the stress value is less than 60% of the maximum stress value, the glass is in good health; if the stress value is between 60% and 80% of the maximum stress value, a yellow warning is issued, indicating a risk of breakage, and replacement is recommended; if the stress value is below 80% of the maximum stress value, a red warning is issued, indicating a higher risk of breakage, and replacement is necessary.
[0158] The glass stress detection system based on a lightweight multi-branch inverse residual network model involved in this invention includes an ultrasonic transmitting and receiving device and a signal processing unit.
[0159] An ultrasonic transmitting and receiving device is set on both sides of the glass to be tested, including a transmitting end 1 and a receiving end 2. The transmitting end 1 emits a pulse ultrasonic signal, which passes through the glass and is received by the receiving end, and the received signal is transmitted to the signal processing unit.
[0160] The signal processing unit is used to send periodic pulse sinusoidal signals to the transmitting end 1 of the ultrasonic transmitting and receiving device, and simultaneously receive ultrasonic signals from the receiving end of the ultrasonic transmitting and receiving device. The signal is processed using the glass stress detection method based on the lightweight multi-branch inverse residual network model as described in claim 3, and the glass state status is output.
[0161] To further explain, the signal processing unit includes a central processing module, a signal preprocessing module, and a multi-branch inverted residual signal processing module;
[0162] Central processing module: Used to emit periodic pulse sinusoidal signals and receive ultrasonic signals, transmit the received ultrasonic signals to the signal preprocessing module, and receive the glass stress value transmitted by the multi-branch inverse residual signal processing module and output it.
[0163] Signal preprocessing module: Receives the ultrasound signal sent by the central processing module, performs short-time Fourier transform on the time-domain data to obtain a time-frequency domain image, extracts the feature information of the time-frequency domain image, and forwards the information to the multi-branch inverse residual signal processing module;
[0164] Multi-branch inverse residual signal processing module: The enhanced image information is trained in a multi-branch inverse residual network model, the weight model file is extracted, and the weight model file is loaded into the prediction program. It is compared with the information of the glass stress to be detected, and the glass stress value obtained by comparison is transmitted to the central processing module.
[0165] For further explanation, see [link to documentation]. Figure 1 The multi-branch inverse residual signal processing module includes a first preprocessing module (corresponding to) connected in sequence. Figure 1 The preprocessing module, the second feature extraction module (BIock*1), the third feature extraction module (BIock*1), the fourth feature extraction module (BIock*2), the fifth feature extraction module (BIock*1), and the sixth classification module (corresponding to...) Figure 1 (Classification module);
[0166] The first preprocessing module is used to denoise the original matrix W by passing it through a 5*5 convolutional layer and a 3*3 pooling layer in sequence, and output the feature matrix W0.
[0167] The second feature extraction module is used to extract matrix information by performing convolution and pooling on the feature matrix W0 through a multi-branch inverse residual structure, and output the inverse residual feature W1.
[0168] The third feature extraction module is used to extract matrix information by performing convolution and pooling on the inverse residual feature W1 through a multi-branch inverse residual structure, and output the inverse residual feature W2.
[0169] The fourth feature extraction module is used to extract matrix information by convolving and pooling two consecutive stacked multi-branch inverse residual structures of the inverse residual feature W2. That is, after the inverse residual feature W2 is convolved and pooled by a multi-branch inverse residual structure to extract matrix information, the matrix information is used as the input value to be convolved and pooled by a multi-branch inverse residual structure again to extract matrix information, and finally the inverse residual feature W3 is output.
[0170] The fifth feature extraction module is used to extract matrix information by performing convolution and pooling on the inverse residual feature W3 through a multi-branch inverse residual structure, and output the inverse residual feature W4.
[0171] The sixth classification module includes an adaptive average pooling layer and a fully connected layer. It is used to process the inverse residual feature W4 through the average pooling layer and the fully connected layer in sequence to combine the distribution features of the extracted image to form a one-dimensional feature vector. The feature vector is then weighted and summed to output a weight model file.
[0172] The multi-branch inverse residual structure includes a 1*1 first convolutional layer, a novel feature fusion layer, and a 1*1 second convolutional layer connected in sequence; the 1*1 first convolutional layer is used to perform dimensionality operations on the feature matrix through the 1*1 first convolution.
[0173] The novel feature fusion method comprises three parallel branches: a first branch, a second branch, and a third branch. Within the first branch, the feature is processed by a 3x3 depthwise convolution followed by 3x3 pooling before being passed to the SE attention module (or directly to the SE attention module) for channel feature enhancement, outputting the main feature matrix. Within the second branch, the feature is processed by a 5x5 depthwise convolution followed by 3x3 pooling before being passed to the SE attention module (or directly to the SE attention module) for channel feature enhancement, outputting the large receptive field matrix. Within the third branch, the feature is processed by 3x3 pooling or directly passed to the SE attention module, outputting the third branch matrix. The main feature matrix, the large receptive field matrix, and the third branch matrix are then merged along the channel dimension to output a merged matrix. Simultaneously, the main feature matrix, the large receptive field matrix, and the third branch matrix are summed to obtain a summation matrix. Finally, the merged matrix and the summation matrix are concatenated along the channel dimension to complete the novel feature fusion.
[0174] The 1*1 second convolutional layer processes the matrix after the fusion of new features through pointwise convolution, and outputs a multi-branch information matrix.
[0175] The inverse residual feature is obtained by summing the output multi-branch information matrix with the feature matrix after dimensional transformation.
[0176] The advantages of the multi-branch inverse residual network model of the present invention are further compared with several existing common network models, and the results are as follows:
[0177] Lightweight Analysis of Experiment 1 Model
[0178] The number of parameters and computational cost of the model are evaluated, and the parameters involved in the IremulbNet model of this invention, such as convolution, activation, normalization and fully connected layers, are statistically analyzed.
[0179] The IremulbNet of this invention is compared with seven existing network models in terms of parameter count, computational cost, and training time. The seven network models are ResNet50, DenseNet121, MobleNetV2, MobleNetV3, ShuffleNetV2, RegNet, and PeleeNet.
[0180] During the evaluation process, the network input consisted of 3-channel 224*224 color images, and the performance analysis tool Profile was used to analyze the parameter count and model scaling of the seven models. The results are shown in Figure 6.
[0181] from Figure 6A As can be seen from the model, the parameter count of IremulbNet in this invention is 2.691 MB, while the classic ResNet50 has 25.56 MB, and DenseNet121 has 7.979 MB. Meanwhile, the parameter counts of lightweight models MobleNetV2, MobleNetV3, ShuffleNetV2, RegNet, and PeleeNet are 3.505 MB, 5.483 MB, 2.279 MB, 2.685 MB, and 2.802 MB, respectively. Compared to the ResNet50 model, IremulbNet reduces the parameter count by 89.47%, and compared to the DenseNet121 model, it reduces the parameter count by 66.27%. Its parameter count is not significantly different from RegNet and PeleeNet.
[0182] exist Figure 6BIn the test, using a 224x224 color image as input, IremulbNet's computational cost was 481.58 MB, while ResNet50, DenseNet121, MobleNetV2, MobleNetV3, ShuffleNetV2, RegNet, and PeleeNet had computational costs of 4210.688 MB, 2934.784 MB, 314.149 MB, 225.442 MB, 148.808 MB, 203.384 MB, and 513.876 MB, respectively. Compared to ResNet50 and DenseNet121, IremulbNet reduced computational cost by 88.56% and 83.6%, respectively.
[0183] Experiment 2
[0184] This embodiment uses glass health assessment as an example. The time-frequency domain images of glass stress obtained are from experiments. Ultrasonic comparison of nonlinear coefficients is used to measure the changes in residual stress within the glass, obtaining experimental data for glass under different stresses. Next, short-time Fourier transform (STFT) is used to process the time-domain glass signal to obtain time-frequency domain image data. In the experiment, the internal stress range of the glass was 30MPa-105MPa. The dataset consists of 1110 images classified into two categories, including 600 images of good health (stress value less than 70MPa) and 510 images of poor health (stress value greater than 70MPa).
[0185] The time-frequency domain image data of the glass was divided into training, validation, and test sets in an 8:1:1 ratio using a segmentation procedure. The acquired data was preprocessed by cropping and standardizing the data according to its original size and random size. Then, the IremulbNet model provided in this invention was used to input the obtained image information into the network model to complete the classification. The number of model parameters, processing time, and accuracy were recorded, and the results are shown in Table 1.
[0186] Meanwhile, seven classic CNN models, namely Resnet50, DenseNet121, MobleNetV2, MobleNetV3, ShuffleNetV2, RegNet, and PeleeNet, were used to identify time-frequency domain images of glass with different stresses. The number of model parameters, computational cost, and accuracy were recorded and compared. The results are shown in Table 1.
[0187] Table 1. Recognition performance of glass time-frequency domain images (2-class classification)
[0188]
[0189]
[0190] Table 1 shows that the IremulbNet network of this application exhibits good performance even when the differences between datasets are minimal. Its classification accuracy is 97.8%, the same as that of classic models ResNet50 and DenseNet121. However, in terms of parameter count and computational cost, the computational cost of IremulbNet is reduced by 88.56% compared to ResNet50 and by 83.6% compared to DenseNet121; the number of parameters is reduced by 89.47% compared to ResNet50 and by 66.27% compared to DenseNet121. In terms of classification accuracy, it outperforms the lightweight models MobileNet V2, ShuffleNet V2, and RegNet by 3.1%, 4.1%, and 5.2%, respectively.
[0191] The IremulbNet model provided in this invention demonstrates that the reconstructed inverse residual structure and the feature fusion method in this model have a certain impact on the network performance. The designed depth and width parameters are reasonable, making the model lightweight, efficient, and highly accurate.
[0192] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. An image recognition method based on a multi-branch inverse residual network model, characterized in that, Includes the following steps: S1) Enhance the image to obtain the original matrix W; S2) Input the original matrix W into the first preprocessing module, and after denoising by 5*5 convolution and 3*3 pooling, obtain the feature matrix W0; S3) Input the feature matrix W0 into the second feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W1. S4) Input the inverse residual feature W1 into the third feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W2. S5) Input the inverse residual feature W2 into the fourth feature extraction module. After convolution and pooling through two stacked multi-branch inverse residual structures, extract the matrix information to obtain the inverse residual feature W3. S6) Input the inverse residual feature matrix W3 into the fifth feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W4. S7) Input the inverse residual feature W4 into the adaptive average pooling layer, and then into the fully connected layer. Combine the extracted distribution features to form a one-dimensional feature vector. Then, perform a weighted summation on the feature vector and output the weight model file to complete the image recognition and classification. The output characteristics of the multi-branch inverse residual structure in steps S3), S4), S5), and S6) are as follows: Among them, W i ( x ) represents the feature output by the i-th layer. i =1, 2, 3, 4; I is a multi-branch inverse residual network, Mish is the activation function, F i The diagram illustrates the convolutional relationship of the intermediate hidden layers. w05 and w06 represent the results of concatenating and summing the dimensions of the feature matrix during feature extraction, respectively.
2. The image recognition method based on a multi-branch inverse residual network model according to claim 1, characterized in that, Step S3) specifically includes: S3.1) Input the feature matrix W0 into the second feature extraction module, and first perform dimensionality operation through a 1*1 first convolution to obtain matrix W01; S3.2) Matrix W01 is input to the first branch, the second branch, and the third branch respectively. In the first branch, it undergoes a 3*3 depthwise convolution and then a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the main feature matrix W02. In the second branch, it undergoes a 5*5 depthwise convolution and then a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the large receptive field matrix W03. In the third branch, it undergoes a 3*3 pooling process or is directly passed to the SE attention module, outputting matrix W04. S3.3) Merge the main feature matrix W02, the large receptive field matrix W03, and the matrix W04 extracted in step S3.2) along the channel dimension to obtain the merged matrix W05; at the same time, sum W02, W03, and W04 to obtain W06; concatenate W05 and W06 along the channels, and perform a 1*1 second convolution pointwise convolution to obtain the multi-branch information matrix W07; S3.4) Summing the multi-branch information matrix W07 output from step S3.3) with the feature matrix W0 after dimensional transformation, we obtain the inverse residual feature W1.
3. A glass stress detection method based on a multi-branch inverse residual network model, characterized in that, Includes the following steps: 1) The change of residual stress in glass is measured by comparing the ultrasonic wave with the nonlinear coefficient, and a time-domain image of glass stress is obtained. 2) Perform short-time Fourier transform on the time-domain image of glass stress to obtain the time-frequency domain image, and establish a dataset; 3) Enhance the image data in the dataset from step 2) to obtain the original matrix W; 4) Input the original matrix W from step 3) into the lightweight multi-branch inverse residual network model and train it according to the method of claim 1 to extract the features of the image and obtain the weight model file; 5) Load the weight model file trained in step 4) into the prediction program, input the time domain data of the stress of the glass to be tested, and obtain the stress value of the glass to be tested by model recognition and prediction according to the methods in steps 1) to 4).
4. The glass stress detection method based on a multi-branch inverse residual network model according to claim 3, characterized in that, Step 1) specifically refers to: 1.1) Install the transmitting and receiving ends of the ultrasonic transmitting and receiving device on both sides of the glass, respectively; 1.2) A periodic pulsed sinusoidal signal is generated by the signal processing unit and input to the transmitting end to emit pulsed ultrasonic waves, which are received by the receiving end after passing through the glass; 1.3) The received ultrasonic signal is input into the signal processing unit. Based on the fact that the relative nonlinear coefficient value is proportional to the stress of the stress-free standard glass, the corresponding relative nonlinear coefficient of the glass is obtained. 1.4) The signal processing unit correlates the nonlinear coefficient with its corresponding stress, plots the curve of the nonlinear coefficient as a function of stress, and obtains the time-domain image of the glass stress.
5. The glass stress detection method based on a multi-branch inverse residual network model according to claim 4, characterized in that, Step 4) specifically involves: 4.1) Input the original matrix W from step 3) into the first preprocessing module, and after denoising by 5*5 convolution and 3*3 pooling, obtain the feature matrix W0; 4.2) Input the feature matrix W0 into the second feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W1. 4.3) Input the inverse residual feature W1 into the third feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W2. 4.4) Input the inverse residual feature W2 into the fourth feature extraction module. After passing through two stacked multi-branch inverse residual structures for convolution and pooling, the matrix information is extracted to obtain the inverse residual feature W3. 4.5) Input the inverse residual feature W3 into the fifth feature extraction module. After convolution and pooling through a multi-branch inverse residual structure, extract the matrix information to obtain the inverse residual feature W4. 4.6) Input the inverse residual feature W4 into the adaptive average pooling layer, and then into the fully connected layer. Combine the extracted distribution features to form a one-dimensional feature vector. Then, perform a weighted summation on the feature vector and output the weighted model file. The output characteristics of the multi-branch inverse residual structure in steps 4.2), 4.3), 4.4), and 4.5) are as follows: Among them, W i ( x ) represents the feature output by the i-th layer. i =1, 2, 3, 4; I is a multi-branch inverse residual network, Mish is the activation function, F i The diagram illustrates the convolutional relationship of the intermediate hidden layers. w05 and w06 represent the results of concatenating and summing the dimensions of the feature matrix during feature extraction, respectively.
6. The glass stress detection method based on a multi-branch inverse residual network model according to claim 5, characterized in that, The specific training process in step 4.2) includes: 4.2.1) Input the feature matrix W0 into the second feature extraction module, and first perform dimensionality operation through a 1*1 first convolution to obtain matrix W01; 4.2.2) Matrix W01 is input to the first branch, the second branch, and the third branch respectively. In the first branch, it undergoes a 3*3 depthwise convolution and then a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the main feature matrix W02. In the second branch, it undergoes a 5*5 depthwise convolution and then a 3*3 pooling process before being passed to the SE attention module or directly to the SE attention module for channel feature enhancement, outputting the large receptive field matrix W03. In the third branch, it undergoes a 3*3 pooling process or is directly passed to the third branch matrix W04. 4.2.3) Merge the main feature matrix W02, the large receptive field matrix W03, and the third branch matrix W04 extracted in step 4.2.2) along the channel dimension to obtain the merged matrix W05; at the same time, sum W02, W03, and W04 to obtain W06; concatenate W05 and W06 along the channel, and perform a 1*1 second convolution pointwise convolution to obtain the multi-branch information matrix W07; 4.2.4) Summing the multi-branch information matrix W07 output from step 4.2.3) with the feature matrix W0 after dimensional transformation, we obtain the inverse residual feature W1.
7. A glass health assessment method based on a multi-branch inverse residual network model, characterized in that, Includes the following steps: (1) The stress value of the glass is detected by the glass stress detection method based on the multi-branch inverse residual network model according to any one of claims 3 to 6; (2) If the stress value is less than 60% of the maximum stress value, the glass is in good condition; if the stress value is 60% to 80% of the maximum stress value, it is a yellow warning, and the glass is at risk of breaking; if the stress value is less than 80% of the maximum stress value, it is a red warning, and the glass is at greater risk of breaking.
8. A glass stress detection system based on a multi-branch inverse residual network model, characterized in that, Includes ultrasonic transmitting and receiving devices and signal processing units; An ultrasonic transmitting and receiving device is set on both sides of the glass to be tested, including a transmitting end and a receiving end. The transmitting end emits pulsed ultrasonic signals, which pass through the glass and are received by the receiving end, which then transmits the received signals to the signal processing unit. The signal processing unit is used to send periodic pulse sinusoidal signals to the transmitting end of the ultrasonic transmitting and receiving device, and simultaneously receive ultrasonic signals from the receiving end of the ultrasonic transmitting and receiving device. The signal is processed using the glass stress detection method based on the multi-branch inverse residual network model as described in claim 3, and the glass state status is output.
9. The glass stress detection system based on a multi-branch inverse residual network model according to claim 8, characterized in that, The signal processing unit includes a central processing module, a signal preprocessing module, and a multi-branch inverted residual signal processing module; Central processing module: Used to emit periodic pulse sinusoidal signals and receive ultrasonic signals, transmit the received ultrasonic signals to the signal preprocessing module, and receive the glass stress value transmitted by the multi-branch inverse residual signal processing module and output it. Signal preprocessing module: Receives the ultrasound signal sent by the central processing module, performs short-time Fourier transform on the time-domain data to obtain a time-frequency domain image, extracts the feature information of the time-frequency domain image, and forwards the information to the multi-branch inverse residual signal processing module; Multi-branch inverse residual signal processing module: The enhanced image information is trained in a multi-branch inverse residual network model, the weight model file is extracted, and the weight model file is loaded into the prediction program. It is compared with the information of the glass stress to be detected, and the glass stress value obtained by comparison is transmitted to the central processing module.
10. The glass stress detection system based on a multi-branch inverse residual network model according to claim 9, characterized in that, The multi-branch inverse residual signal processing module includes a first preprocessing module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, and a sixth classification module connected in sequence. The first preprocessing module is used to denoise the original matrix W by passing it through a 5*5 convolutional layer and a 3*3 pooling layer in sequence, and output the feature matrix W0. The second feature extraction module is used to extract matrix information by performing convolution and pooling on the feature matrix W0 through a multi-branch inverse residual structure, and output the inverse residual feature W1. The third feature extraction module is used to extract matrix information by performing convolution and pooling on the inverse residual feature W1 through a multi-branch inverse residual structure, and output the inverse residual feature W2. The fourth feature extraction module is used to extract matrix information by convolution and pooling two consecutive stacked multi-branch inverse residual structures of the inverse residual feature W2, and output the inverse residual feature W3. The fifth feature extraction module is used to extract matrix information by performing convolution and pooling on the inverse residual feature W3 through a multi-branch inverse residual structure, and output the inverse residual feature W4. The sixth classification module includes an adaptive average pooling layer and a fully connected layer. It is used to process the inverse residual feature W4 through the average pooling layer and the fully connected layer in sequence to combine the distribution features of the extracted image to form a one-dimensional feature vector. The feature vector is then weighted and summed to output a weight model file.
11. The glass stress detection system based on a multi-branch inverse residual network model according to claim 10, characterized in that, The multi-branch inverse residual structure includes a 1*1 first convolutional layer, a novel feature fusion layer, and a 1*1 second convolutional layer connected in sequence. The first 1x1 convolutional layer is used to perform dimensionality operations on the feature matrix by passing it through the first 1x1 convolution. The novel feature fusion method comprises three parallel branches: a first branch, a second branch, and a third branch. Within the first branch, the feature is processed by a 3x3 depthwise convolution followed by 3x3 pooling before being passed to the SE attention module (or directly to the SE attention module) for channel feature enhancement, outputting the main feature matrix. Within the second branch, the feature is processed by a 5x5 depthwise convolution followed by 3x3 pooling before being passed to the SE attention module (or directly to the SE attention module) for channel feature enhancement, outputting the large receptive field matrix. Within the third branch, the feature is processed by 3x3 pooling or directly passed to the SE attention module, outputting the third branch matrix. The main feature matrix, the large receptive field matrix, and the third branch matrix are then merged along the channel dimension to output a merged matrix. Simultaneously, the main feature matrix, the large receptive field matrix, and the third branch matrix are summed to obtain a summation matrix. Finally, the merged matrix and the summation matrix are concatenated along the channel dimension to complete the novel feature fusion. The 1*1 second convolutional layer processes the matrix after the fusion of new features through pointwise convolution, and outputs a multi-branch information matrix. The inverse residual feature is obtained by summing the output multi-branch information matrix with the feature matrix after dimensional transformation.