A dual-modal industrial defect detection method, device, and medium based on SqueezeNet network

Abstract

This application relates to a bimodal industrial defect detection method, apparatus, and medium based on a SqueezeNet network. The bimodal industrial defect detection method based on a SqueezeNet network includes: acquiring a first image set and a second image set; inputting the first image set and the second image set into a SqueezeNet network for training to obtain a defect detection model. The SqueezeNet network includes a first convolutional layer, a bimodal Fire unit, a second convolutional layer, and a classifier connected in sequence. The bimodal Fire unit is used to dynamically fuse bimodal image features. A third image and a fourth image of the workpiece to be detected, acquired under two different lighting conditions, are input into the defect detection model to obtain the defect type of the workpiece to be detected, thereby improving the detection accuracy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a dual-modal industrial defect detection method, apparatus, and medium based on the SqueezeNet network. Background Technology

[0002] The accuracy of industrial defect detection is highly dependent on the compatibility between the physical characteristics of the defect and the lighting scheme. Different lighting schemes can enhance the imaging characteristics of specific types of defects through optical path design (such as incident angle and light intensity distribution).

[0003] However, traditional lightweight networks (such as SqueezeNet and MobileNet) only support single-source image input. To cover diverse defects in complex industrial scenarios (such as scratches on metal parts, poor welding, coating peeling, etc.), it is necessary to independently construct training datasets, annotation systems, and dedicated detection models for each lighting scheme (bright field, dark field, polarized light, etc.), and there is no parameter sharing or feature interaction between models.

[0004] Existing technologies suffer from the inability to collaboratively optimize cross-modal features due to the independent training of detection models. For example, minute scratches on the edge of a metal part appear blurred under bright field conditions due to reflective interference, but their clear outline features in dark field conditions are not utilized in combination, resulting in low detection accuracy of a single model. Summary of the Invention

[0005] Therefore, it is necessary to provide a dual-modal industrial defect detection method, device, and medium based on SqueezeNet network to address the aforementioned technical problems.

[0006] In a first aspect, embodiments of this application provide a dual-modal industrial defect detection method based on the SqueezeNet network, the method comprising:

[0007] A first image set and a second image set are acquired from multiple workpieces under two different lighting conditions; wherein, the first image set includes multiple first images, and the second image set includes multiple second images;

[0008] The first image set and the second image set are input into the SqueezeNet network for training to obtain a defect detection model. The SqueezeNet network includes a first convolutional layer, a bimodal Fire unit, a second convolutional layer, and a classifier connected in sequence. The first convolutional layer is used to perform convolution operations on each of the first image and each of the second image to obtain each of the first feature maps and each of the second feature maps. The bimodal Fire unit is used to generate each of the first weight maps and each of the second weight maps through adaptive hybrid attention based on each of the first feature maps and each of the first weight maps, and to obtain each of the third feature maps and each of the second weight maps based on each of the second feature maps. The second convolutional layer is used to perform convolution operations on each of the third feature maps and each of the fourth feature maps and then input them into the classifier to obtain the defect type of each workpiece.

[0009] The third and fourth images of the workpiece to be inspected, acquired under the two lighting conditions respectively, are input into the defect detection model to obtain the defect type of the workpiece to be inspected.

[0010] In one embodiment, the bimodal Fire unit includes a plurality of bimodal Fire modules connected in sequence, wherein the output of the current bimodal Fire module is used as the input of the next bimodal Fire module, and the bimodal Fire module includes an independent compression layer, a cross-attention fusion layer, and a multi-scale expansion layer;

[0011] The independent compression layer is used to extract low-redundancy features from the input first feature map and second feature map respectively to obtain the fifth feature map and the sixth feature map;

[0012] The cross-attention fusion layer is used to concatenate the fifth feature map and the sixth feature map to obtain a concatenated feature map, and generate a first weight map and a second weight map based on the concatenated feature map through adaptive hybrid attention; and to obtain a seventh feature map based on the fifth feature map and the first weight map, and to obtain an eighth feature map based on the sixth feature map and the second weight map;

[0013] The multi-scale extension layer is used to extract features based on the seventh feature map of the convolution kernel to obtain the third feature map, and to extract features based on the eighth feature map of the convolution kernel to obtain the fourth feature map.

[0014] In one embodiment, the cross-attention fusion layer includes a channel attention module, a spatial attention module, and a gated fusion network.

[0015] The channel attention module is used to obtain channel attention features based on the stitched feature map through a channel attention mechanism;

[0016] The spatial attention module is used to obtain spatial attention features based on the stitched feature map through a spatial attention mechanism;

[0017] The gated fusion network is used to obtain a first fusion weight map and a second fusion weight map based on the stitched feature map, and to combine the channel attention feature and the spatial attention feature to obtain the first weight map and the second weight map.

[0018] In one embodiment, in the channel attention module, the channel attention feature M C (X) is calculated using the following formula:

[0019]

[0020] In the spatial attention module, the spatial attention feature M S (X) is calculated using the following formula:

[0021]

[0022] In the gated fusion network, the first weight map A1 and the second weight map A2 are calculated using the following formula:

[0023]

[0024] in: For learnable parameters, It is the ReLU activation function. Here, X is the sigmoid function, X is the concatenated feature map, GAP is the global average pooling operation, Conv is the convolution operation, and W is the sigmoid function. C W S These are the first fusion weight map and the second fusion weight map, respectively.

[0025] In one embodiment, in the independent compression layer, the fifth feature map S1 and the sixth feature map S2 are calculated using the following formula:

[0026]

[0027] In the cross-attention fusion layer, the seventh feature map and the eighth feature map Calculated using the following formula:

[0028]

[0029] In the multi-scale expansion layer, the third feature map Expand(F1) and the fourth feature map Expand(F2) are calculated using the following formulas:

[0030]

[0031] Among them, F m and F d These are the first feature map and the second feature map, respectively.

[0032] In one embodiment, the SqueezeNet network is trained using a total loss function, which consists of multiple levels of orthogonal loss functions. and cross-entropy loss function The weighted composition and the formula for calculating the total loss function are as follows:

[0033]

[0034] Among them, the The cross-entropy loss function is... This is a multi-level orthogonality loss function.

[0035] In one embodiment, the multi-level orthogonality loss function is calculated as follows:

[0036] The dual-modal Fire unit is divided into three levels, and feature pairs output by each level are obtained respectively;

[0037] Based on the feature pairs output from each level, the orthogonal loss corresponding to each level is calculated. The calculation formula is as follows:

[0038]

[0039] Where F1 and F2 are the feature pairs output at each level, and B is the number of batches trained by the model;

[0040] The multi-level orthogonality loss function is obtained by weighting and summing the weights of each level and the corresponding orthogonal losses.

[0041] Secondly, embodiments of this application also provide a dual-modal industrial defect detection device based on the SqueezeNet network, the device comprising:

[0042] The acquisition module is used to acquire a first image set and a second image set of multiple workpieces under two lighting conditions respectively; wherein, the first image set includes multiple first images, and the second image set includes multiple second images;

[0043] The training module is used to input the first image set and the second image set into the SqueezeNet network for training to obtain a defect detection model. The SqueezeNet network includes a first convolutional layer, a bimodal Fire unit, a second convolutional layer, and a classifier connected in sequence. The first convolutional layer is used to perform convolution operations on each of the first image and each of the second image to obtain each of the first feature maps and each of the second feature maps. The bimodal Fire unit is used to generate each of the first weight maps and each of the second weight maps based on each of the first feature maps and each of the second feature maps through adaptive hybrid attention, and to obtain each of the third feature maps and each of the first weight maps, and to obtain each of the fourth feature maps based on each of the second feature maps and each of the second weight maps. The second convolutional layer is used to perform convolution operations on each of the third feature maps and each of the fourth feature maps and then input them into the classifier to obtain the defect type of each workpiece.

[0044] The detection module is used to input the third and fourth images of the workpiece to be inspected, which are respectively acquired under the two lighting conditions, into the defect detection model to obtain the defect type of the workpiece to be inspected.

[0045] Thirdly, embodiments of this application also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method described in the first aspect above.

[0046] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect above.

[0047] The aforementioned dual-modal industrial defect detection method, device, and medium based on the SqueezeNet network propose a dual-modal dynamic fusion network structure based on a lightweight SqueezeNet network architecture for defect images acquired under different lighting schemes (such as bright field, dark field, and multi-angle polarized light). This structure can efficiently fuse multimodal features and improve detection accuracy, making it suitable for online defect detection in industrial scenarios such as metal processing parts.

[0048] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 This is a hardware structure block diagram of a terminal device for a dual-modal industrial defect detection method based on the SqueezeNet network in one embodiment.

[0051] Figure 2 This is a flowchart illustrating a dual-modal industrial defect detection method based on the SqueezeNet network in one embodiment.

[0052] Figure 3 This is a comparison image of different lighting schemes in one embodiment;

[0053] Figure 4 This is a comparison image of different lighting schemes in another embodiment;

[0054] Figure 5 A diagram of the SqueezeNet network structure in one embodiment;

[0055] Figure 6 This is a network structure diagram of a dual-modal Fire module in one embodiment;

[0056] Figure 7 This is a network architecture diagram of the cross-attention fusion layer in one embodiment;

[0057] Figure 8 This is a structural block diagram of a dual-modal industrial defect detection device based on the SqueezeNet network in one embodiment;

[0058] Figure 9 This is a schematic diagram of the computer device structure in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0060] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal of the dual-modal industrial defect detection method based on the SqueezeNet network in this embodiment. Figure 1 As shown, a terminal may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0061] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the dual-modal industrial defect detection method based on the SqueezeNet network in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module for wireless communication with the Internet.

[0063] This application provides a dual-modal industrial defect detection method based on the SqueezeNet network, which is applied to... Figure 1 Taking the terminal in the example of this, for example... Figure 2 As shown, the method includes the following steps:

[0064] Step 201: Obtain a first image set and a second image set of multiple workpieces under two lighting conditions respectively; wherein, the first image set includes multiple first images, and the second image set includes multiple second images.

[0065] Images acquired under different lighting conditions contain complementary physical information, such as the global texture of the bright field and the local morphology of the dark field. For example... Figure 3 and Figure 4 Comparison images of different lighting schemes, such as Figure 3 As shown, the left side is dark field lighting (low-angle directional light source), and the right side is bright field lighting (high-angle uniform light source). Figure 3 As can be seen, the pressure marks in the red area are more noticeable under dark lighting; for example... Figure 4 As shown, the left side is bright field lighting (high-angle uniform light source), and the right side is dark field lighting (low-angle directional light source). Figure 4 As can be seen, the bumps and knocks in the red area are more noticeable under bright lighting.

[0066] For example, the two lighting conditions in this application are bright field lighting and dark field lighting. The first image set consists of multiple images acquired under bright field lighting conditions, and the second image set consists of multiple images acquired under dark field lighting conditions.

[0067] Step 202: Input the first image set and the second image set into the SqueezeNet network for training to obtain a defect detection model. Figure 5 This is the network structure diagram of SqueezeNet in this application, as shown below. Figure 5 As shown, the SqueezeNet network includes a first convolutional layer, a bimodal Fire unit 200, a second convolutional layer, and a classifier connected in sequence. The first convolutional layer is used to perform convolution operations on each first image and each second image to obtain each first feature map and each second feature map. The bimodal Fire unit is used to generate each first weight map and each second weight map based on each first feature map and each second feature map through adaptive hybrid attention, and to obtain each third feature map based on each first feature map and each first weight map, and to obtain each fourth feature map based on each second feature map and each second weight map. The second convolutional layer is used to perform convolution operations on each third feature map and each fourth feature map and then input them into the classifier to obtain the defect type of each workpiece.

[0068] Among them, the network architecture of SqueezeNet in this application is an improvement on the existing SqueezeNet network architecture. It constructs a bimodal Fire unit, which is the core unit for dynamic interaction and fusion of bimodal features acquired under two lighting conditions. This solves the problem of insufficient utilization of complementary features across light sources in the traditional single-modal architecture of SqueezeNet network.

[0069] Step 203: Input the third and fourth images of the workpiece to be inspected, acquired under the two lighting conditions respectively, into the defect detection model to obtain the defect type of the workpiece to be inspected.

[0070] In steps S201 to S203 above, for defect images acquired under different lighting schemes (such as bright field, dark field, and multi-angle polarized light), a dual-modal dynamic fusion network structure based on the lightweight SqueezeNet network architecture is proposed. This structure can efficiently fuse multimodal features and improve detection accuracy, and is suitable for online defect detection in industrial scenarios such as metal processing parts.

[0071] Specifically, the dual-mode Fire unit includes multiple dual-mode Fire modules connected in sequence, and the output of the current dual-mode Fire module serves as the input of the next dual-mode Fire module. Figure 6 This is a network structure diagram of the DualModalFire module in this application, which includes an independent compression layer, a cross-attention fusion layer, and a multi-scale expansion layer.

[0072] The dual-squeeze compression layer is used to compress the input first feature map F. m and the second feature map F d Low-redundancy feature extraction is performed separately to obtain the fifth feature map S1 and the sixth feature map S2.

[0073] Mathematical expression:

[0074]

[0075] Among them, the 1×1 convolution kernel compresses the number of channels to half of the original, F m and F d S1 and S2 are the first and second feature maps, respectively, and S1 and S2 are the fifth and sixth feature maps, respectively.

[0076] The independent compression layer of this application is designed for input under two lighting conditions (first feature map ( ) and the second feature map ( Low-redundancy feature extraction is performed separately to avoid feature confusion between modes, which has the following advantages: independent processing of modal differences (such as the high dynamic range of bright field and the low signal-to-noise ratio of dark field) and preservation of the physical characteristics of light source; lightweight compression to avoid interference from redundant features.

[0077] The Cross-Attention Fusion layer is used to concatenate the fifth feature map S1 and the sixth feature map S2 to obtain a concatenated feature map, and to generate a first weight map A1 and a second weight map A2 based on the concatenated feature map through adaptive hybrid attention; and to obtain a seventh feature map based on the fifth feature map S1 and the first weight map A1. And based on the sixth feature map S2 and the second weight map A2, an eighth feature map is obtained. .

[0078] Mathematical expression:

[0079] (1) Feature concatenation: First, the two features obtained from the independent compression layers are concatenated. and The images are stitched together to obtain a stitched feature map. .

[0080]

[0081] (2) Attention weight generation: The concatenated features Weight maps are generated by adaptive hybrid attention, resulting in a first weight map A1 and a second weight map A2.

[0082]

[0083] (3) Residual symmetric fusion: The resulting weight map Two features obtained through the formula below and independent compression layers and The fusion process yields the seventh feature map. and the eighth feature map .

[0084]

[0085]

[0086] The cross-attention fusion layer in this application dynamically learns the correlation between features in different modalities (bright field and dark field) through an adaptive hybrid attention mechanism, thereby achieving defect type-driven weight allocation. Adaptive adjustment based on defect type, residual connections preserve the original feature distribution, and training convergence speed is improved.

[0087] The multi-scale expansion layer (Expand) is used to apply the seventh feature map based on the convolution kernel. Feature extraction is performed to obtain the third feature map Expand(F1), and the eighth feature map is processed based on the convolution kernel. Feature extraction is performed to obtain the fourth feature map Expand(F2).

[0088] Mathematical expression:

[0089]

[0090]

[0091] This application's multi-scale extension layer, through multi-scale convolutional kernels, fuses global semantics and local details to enhance the representation capability of defect features. (The text then repeats the process of using...) convolution kernel size and Size of convolution kernel pairs and Feature extraction is performed separately, using different receptive fields. Specifically, 1×1 convolutions extract global semantic features, while 3×3 convolutions capture local texture details. The number of output channels is restored to four times that before compression, maintaining compatibility with the original SqueezeNet network.

[0092] The bimodal Fire module of this application achieves the following breakthroughs through a three-level architecture of "independent compression-dynamic fusion-multi-scale expansion": 1. Lightweight and precision balance: Supports bimodal dynamic interaction with only a 53.8% increase in parameters, solving the problem of lack of support for multi-source input in traditional lightweight networks; 2. Physical characteristic-driven dynamic fusion: Encodes the physical characteristics of the light source (such as high resolution in bright field and edge enhancement in dark field) into weight vectors through a cross-attention mechanism, realizing defect-adaptive feature fusion; 3. High compatibility with industrial scenarios: The module's output channel is aligned with the original SqueezeNet network, supporting plug-and-play replacement, which is convenient for migration to existing industrial inspection systems.

[0093] In industrial quality inspection scenarios, traditional attention mechanisms face two major challenges: First, single-dimensional attention (such as channel-only or spatial-only) struggles to handle the feature differences of multi-morphological defects. For example, scratches on metal surfaces require feature enhancement in the channel dimension, while the positional offset of electronic components requires precise spatial positioning. Second, fixed-weight multi-attention fusion methods lack adaptability to complex backgrounds and are prone to false activations in scenarios with uneven lighting or texture interference. To address these challenges, this application's cross-attention fusion layer employs a two-layer dynamically coupled attention architecture. Figure 7 The network architecture of the cross-attention fusion layer is presented. Its core innovation lies in the spatially adaptive weight allocation mechanism. Unlike the fixed weight addition of traditional methods (such as CBAM), this application generates pixel-level fusion weights through a gating network, so that each spatial location can autonomously select the optimal attention combination mode according to the local feature complexity.

[0094] like Figure 7As shown, the cross-attention fusion layer includes a channel attention module, a spatial attention module, and a gating fusion network.

[0095] The channel attention module is used to obtain channel attention features based on the stitched feature map through a channel attention mechanism.

[0096] Mathematical expression:

[0097]

[0098] in: For learnable parameters, It is the ReLU activation function. Let X be the sigmoid function and X be the concatenated feature map. GAP stands for Global Average Pooling.

[0099] The channel attention module of this application enables the weight calculation of each channel to perceive the statistical characteristics of the entire image region through global average pooling (GAP) operation; and effectively captures the nonlinear dependencies between channels through the dimensionality reduction-dimensionality increase structure constructed by two-level fully connected layers.

[0100] The spatial attention module is used to obtain spatial attention features based on the stitched feature map through a spatial attention mechanism.

[0101] Mathematical expression:

[0102]

[0103] Where: Conv represents the convolution operation.

[0104] The spatial attention module of this application uses a 3×3 convolution kernel instead of a larger receptive field to preserve the fine structure of defect edges; the single-channel output design allows the module to focus on the salience of spatial location rather than the combination of channel dimensions; it has a minimal number of parameters, containing only C×1×3×3=9C parameters, and only 4,608 parameters are needed when C=512, making it suitable for deployment on edge computing devices.

[0105] The gated fusion network is used to obtain a first fusion weight map and a second fusion weight map based on the stitched feature map, and to combine the channel attention feature and the spatial attention feature to obtain the first weight map and the second weight map.

[0106] Mathematical expression:

[0107]

[0108] in, These are the first fusion weight map and the second fusion weight map, respectively.

[0109] The gated fusion network in this application has spatial adaptability: the gated network generates independent fusion weights for each pixel. This enables fine-grained attention combination control; it is noise robust: through The probability normalization property of the function allows the gated network to maintain the stability of the weight distribution under noise interference; it also has scene awareness capabilities: the convolutional layers of the gated network can implicitly learn the feature patterns of different material surfaces.

[0110] In industrial defect detection, defects of the same class exhibit high intra-class differences due to factors such as deformation and illumination (e.g., changes in the length and direction of metal scratches), while defects of different classes show local similarities (e.g., the local textures of cracks and scratches are similar), leading to confusing regions in the feature space. Traditional cross-entropy loss only focuses on class separability and lacks explicit constraints on the geometric structure of the feature space.

[0111] This application introduces a loss function based on matrix orthogonality, constraining the feature space from both algebraic and geometric perspectives. Algebraic perspective: It forces the covariance matrices of feature vectors from different classes to approach orthogonality in the embedding space, maximizing the discriminability between classes. Geometric perspective: Through hyperspherical mapping, it ensures that feature vectors of the same class are compactly distributed on a sphere, maximizing the angle between feature vectors of different classes.

[0112] In one embodiment, the total loss function consists of multiple levels of orthogonal loss functions. and cross-entropy loss function The weighted structure is used to train the SqueezeNet network using the total loss function.

[0113] Among them, the multi-level orthogonality loss function The calculation process is as follows:

[0114] The bimodal Fire units in the SqueezeNet network are divided into three levels, such as... Figure 5 As shown, layer 0 210 includes bimodal Fire module 2, bimodal Fire module 3, and bimodal Fire module 4; layer 1 220 includes bimodal Fire module 5, bimodal Fire module 6, bimodal Fire module 7, and bimodal Fire module 8; layer 2 230 includes bimodal Fire module 9. Then, the feature pairs (F1 and F2) output from each layer are obtained respectively. Through the multi-stage features of the SqueezeNet network, the orthogonality of shallow (edge / texture) and deep (semantic) features is constrained at the same time.

[0115] The orthogonality calculation process is as follows:

[0116] Matrix expansion: The 4D feature tensor [B,C,H,W] is flattened into a 2D matrix [B, CHW], where B is the number of batches trained on the model, C is the number of channels, H is the height of the feature map, and W is the width of the feature map.

[0117] Correlation matrix calculation: Calculate the correlation of features between samples using matrix multiplication.

[0118]

[0119] Orthogonal loss calculation: Using the Frobenius norm to measure the strength of the off-diagonal elements of the correlation matrix:

[0120]

[0121] The orthogonality loss at each level is calculated using the method described above. Then, the orthogonality loss of each level is summed with its corresponding weight to obtain the multi-level orthogonality loss. .

[0122] The weights at each level employ a hierarchical decay mechanism, which can emphasize the decoupling of shallow features (such as edges / textures) and weaken the orthogonal constraints of deep semantic features.

[0123] An exponential decay mechanism is employed, with loss weights at different levels adjusted accordingly. Decreasing, for example:

[0124] Layer 0 weight: 0.8^0 = 1.0;

[0125] The weight of the first layer is 0.8^1 = 0.8;

[0126] Second layer weight: 0.8^2 = 0.64;

[0127] The combined training of the aforementioned multi-level orthogonality loss and cross-entropy loss results in a total loss function that is a linearly weighted sum of the cross-entropy loss and the multi-level orthogonal loss. The formula for calculating the total loss function is as follows:

[0128]

[0129] in: The standard cross-entropy loss is used as the main loss for classification. For multi-level orthogonality loss, This is the collaborative weighting coefficient (e.g., set to 0.1).

[0130] In one example embodiment, images of two lighting schemes were acquired simultaneously to construct a dataset of 2000 industrial defects. The dataset was trained using the SqueezeNet network proposed in this application, with joint optimization of cross-entropy loss and orthogonal loss. The training cycle consisted of 100 iterations. The experimental results are compared in Table 1.

[0131] Table 1

[0132]

[0133] This application, based on the improved bimodal Fire module of SqueezeNet and its dynamic fusion, adaptive attention (adaptive hybrid attention mechanism with channel-space decoupling) and orthogonal loss collaborative design, achieves a breakthrough of 98% accuracy in industrial defect detection while maintaining the lightweight advantage of SqueezeNet (only 1.40M parameters), providing a high-precision and low-cost intelligent detection solution for edge computing scenarios.

[0134] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0135] In one embodiment, such as Figure 8 As shown, a dual-modal industrial defect detection device based on the SqueezeNet network is provided, the device comprising:

[0136] The acquisition module 10 is used to acquire a first image set and a second image set of multiple workpieces under two lighting conditions respectively; wherein, the first image set includes multiple first images, and the second image set includes multiple second images;

[0137] Training module 20 is used to input the first image set and the second image set into the SqueezeNet network for training to obtain a defect detection model. The SqueezeNet network includes a first convolutional layer, a bimodal Fire module, a second convolutional layer, and a classifier connected in sequence. The first convolutional layer is used to perform convolution operations on each of the first image and each of the second image to obtain each of the first feature maps and each of the second feature maps. The bimodal Fire module is used to generate each of the first weight maps and each of the second weight maps based on each of the first feature maps and each of the second feature maps through adaptive hybrid attention, and to obtain each of the third feature maps and each of the first weight maps, and to obtain each of the fourth feature maps based on each of the second feature maps and each of the second weight maps. The second convolutional layer is used to perform convolution operations on each of the third feature maps and each of the fourth feature maps and then input them into the classifier to obtain the defect type of each workpiece.

[0138] The detection module 30 is used to input the third and fourth images of the workpiece to be detected, which are respectively acquired under the two lighting conditions, into the defect detection model to obtain the defect type of the workpiece to be detected.

[0139] In one embodiment, the bimodal Fire unit includes multiple sequentially connected bimodal Fire modules. The output of the current bimodal Fire module serves as the input of the next bimodal Fire module. Each bimodal Fire module includes an independent compression layer, a cross-attention fusion layer, and a multi-scale expansion layer. The independent compression layer extracts low-redundancy features from the input first and second feature maps to obtain a fifth and a sixth feature map, respectively. The cross-attention fusion layer concatenates the fifth and sixth feature maps to obtain a concatenated feature map, and generates a first and a second weight map based on the concatenated feature map using adaptive hybrid attention. It also obtains a seventh feature map based on the fifth and first weight maps, and an eighth feature map based on the sixth and second weight maps. The multi-scale expansion layer extracts features from the seventh feature map using convolutional kernels to obtain a third feature map, and extracts features from the eighth feature map using convolutional kernels to obtain a fourth feature map.

[0140] In one embodiment, the cross-attention fusion layer includes a channel attention module, a spatial attention module, and a gated fusion network. The channel attention module is used to obtain channel attention features based on the stitched feature map through a channel attention mechanism. The spatial attention module is used to obtain spatial attention features based on the stitched feature map through a spatial attention mechanism. The gated fusion network is used to obtain a first fusion weight map and a second fusion weight map based on the stitched feature map, and to combine the channel attention features and the spatial attention features to obtain the first weight map and the second weight map.

[0141] In one embodiment, in the channel attention module, the channel attention feature M C (X) is calculated using the following formula:

[0142]

[0143] In the spatial attention module, the spatial attention feature M S (X) is calculated using the following formula:

[0144]

[0145] In the gated fusion network, the first weight map A1 and the second weight map A2 are calculated using the following formula:

[0146]

[0147] in: For learnable parameters, It is the ReLU activation function. Here, X is the sigmoid function, X is the concatenated feature map, GAP is the global average pooling operation, and W is the sigmoid function. C W S These are the first fusion weight map and the second fusion weight map, respectively.

[0148] In one embodiment, in the independent compression layer, the fifth feature map S1 and the sixth feature map S2 are calculated using the following formula:

[0149]

[0150] In the cross-attention fusion layer, the seventh feature map and the eighth feature map Calculated using the following formula:

[0151]

[0152] In the multi-scale expansion layer, the third feature map Expand(F1) and the fourth feature map Expand(F2) are calculated using the following formulas:

[0153]

[0154] Among them, F m and F d These are the first feature map and the second feature map, respectively.

[0155] In one embodiment, the training module 20 further includes training the SqueezeNet network using a total loss function, the total loss function being composed of multiple levels of orthogonal loss functions. and cross-entropy loss function The weighted composition and the formula for calculating the total loss function are as follows:

[0156]

[0157] Among them, the The cross-entropy loss function is... This is a multi-level orthogonality loss function.

[0158] In one embodiment, the multi-level orthogonality loss function is calculated as follows:

[0159] The dual-modal Fire unit is divided into three levels, and feature pairs output by each level are obtained respectively;

[0160] Based on the feature pairs output from each level, the orthogonal loss corresponding to each level is calculated. The calculation formula is as follows:

[0161]

[0162] Where F1 and F2 are the feature pairs output at each level, and B is the number of batches trained by the model;

[0163] The multi-level orthogonality loss function is obtained by weighting and summing the weights of each level and the corresponding orthogonal losses.

[0164] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0165] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a dual-modal industrial defect detection method based on the SqueezeNet network.

[0166] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0167] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the embodiments of the dual-modal industrial defect detection method based on the SqueezeNet network described above.

[0168] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A dual-modal industrial defect detection method based on SqueezeNet network, characterized in that, The method includes: A first image set and a second image set are acquired from multiple workpieces under two different lighting conditions; wherein, the first image set includes multiple first images, and the second image set includes multiple second images; The first image set and the second image set are input into the SqueezeNet network for training to obtain a defect detection model. The SqueezeNet network includes a first convolutional layer, a bimodal Fire unit, a second convolutional layer, and a classifier connected in sequence. The first convolutional layer is used to perform convolution operations on each of the first image and each of the second image to obtain each of the first feature maps and each of the second feature maps. The bimodal Fire unit is used to generate each of the first weight maps and each of the second feature maps through adaptive hybrid attention based on each of the first feature maps and each of the second feature maps, and then... The system obtains third feature maps based on the first weight map and each second feature map, and fourth feature maps based on each second feature map and each second weight map; the second convolutional layer is used to perform convolution operations on each third feature map and each fourth feature map respectively, and then inputs them into the classifier to obtain the defect type of each workpiece; wherein: the bimodal Fire unit includes multiple bimodal Fire modules connected in sequence, the output of the current bimodal Fire module is used as the input of the next bimodal Fire module, and the bimodal Fire module includes an independent compression layer, a cross-attention fusion layer, and a multi-scale expansion layer; The independent compression layer is used to extract low-redundancy features from the input first feature map and second feature map respectively to obtain the fifth feature map and the sixth feature map; The cross-attention fusion layer is used to concatenate the fifth feature map and the sixth feature map to obtain a concatenated feature map, and generate a first weight map and a second weight map based on the concatenated feature map through adaptive hybrid attention; and to obtain a seventh feature map based on the fifth feature map and the first weight map, and to obtain an eighth feature map based on the sixth feature map and the second weight map; The multi-scale expansion layer is used to extract features based on the seventh feature map of the convolution kernel to obtain the third feature map, and to extract features based on the eighth feature map of the convolution kernel to obtain the fourth feature map; The third and fourth images of the workpiece to be inspected, acquired under the two lighting conditions respectively, are input into the defect detection model to obtain the defect type of the workpiece to be inspected.

2. The method according to claim 1, characterized in that, The cross-attention fusion layer includes a channel attention module, a spatial attention module, and a gated fusion network. The channel attention module is used to obtain channel attention features based on the stitched feature map through a channel attention mechanism; The spatial attention module is used to obtain spatial attention features based on the stitched feature map through a spatial attention mechanism; The gated fusion network is used to obtain a first fusion weight map and a second fusion weight map based on the stitched feature map, and to combine the channel attention feature and the spatial attention feature to obtain the first weight map and the second weight map.

3. The method according to claim 2, characterized in that, In the channel attention module, the channel attention feature M C (X) is calculated using the following formula: ； In the spatial attention module, the spatial attention feature M S (X) is calculated using the following formula: ； In the gated fusion network, the first weight map A1 and the second weight map A2 are calculated using the following formula: ； in: For learnable parameters, It is the ReLU activation function. Here, X is the sigmoid function, X is the concatenated feature map, GAP is the global average pooling operation, Conv is the convolution operation, and W is the sigmoid function. C ,W S These are the first fusion weight map and the second fusion weight map, respectively.

4. The method according to claim 2, characterized in that, In the independent compression layer, the fifth feature map S1 and the sixth feature map S2 are calculated using the following formula: ； In the cross-attention fusion layer, the seventh feature map and the eighth feature map Calculated using the following formula: ； In the multi-scale expansion layer, the third feature map Expand(F1) and the fourth feature map Expand(F2) are calculated using the following formulas: ； Among them, F m and F d These are the first feature map and the second feature map, respectively.

5. The method according to claim 1, wherein The SqueezeNet network is trained using a total loss function, which consists of multiple levels of orthogonal loss functions. and cross-entropy loss function The weighted composition and the formula for calculating the total loss function are as follows: ； in, Let cross-entropy be the loss function. This is a multi-level orthogonality loss function.

6. The method according to claim 5, characterized in that, The formula for calculating the multi-level orthogonality loss function is as follows: The dual-modal Fire unit is divided into three levels, and feature pairs output by each level are obtained respectively; Based on the feature pairs output from each level, the orthogonal loss corresponding to each level is calculated. The calculation formula is as follows: ； Where F1 and F2 are the feature pairs output at each level, and B is the number of batches trained by the model; The multi-level orthogonality loss function is obtained by weighting and summing the weights of each level and the corresponding orthogonal losses.

7. A dual-modal industrial defect detection device based on SqueezeNet network, characterized in that, The device includes: The acquisition module is used to acquire a first image set and a second image set of multiple workpieces under two lighting conditions respectively; wherein, the first image set includes multiple first images, and the second image set includes multiple second images; The training module is used to input the first image set and the second image set into the SqueezeNet network for training to obtain a defect detection model. The SqueezeNet network includes a first convolutional layer, a bimodal Fire unit, a second convolutional layer, and a classifier connected in sequence. The first convolutional layer is used to perform convolution operations on each of the first image and each of the second image to obtain each of the first feature maps and each of the second feature maps. The bimodal Fire unit is used to generate each of the first weight maps and each of the second feature maps through adaptive hybrid attention, and to generate each of the first weight maps and each of the second weight maps based on each of the first feature maps and each of the second feature maps. The method obtains third feature maps based on the first feature map and each of the first weight maps, and obtains fourth feature maps based on each of the second feature maps and each of the second weight maps; the second convolutional layer is used to perform convolution operations on each of the third feature maps and each of the fourth feature maps respectively, and then inputs them into the classifier to obtain the defect type of each workpiece; wherein: the bimodal Fire unit includes multiple bimodal Fire modules connected in sequence, the output of the current bimodal Fire module is used as the input of the next bimodal Fire module, and the bimodal Fire module includes an independent compression layer, a cross-attention fusion layer, and a multi-scale expansion layer; The independent compression layer is used to extract low-redundancy features from the input first feature map and second feature map respectively to obtain the fifth feature map and the sixth feature map; The cross-attention fusion layer is used to concatenate the fifth feature map and the sixth feature map to obtain a concatenated feature map, and generate a first weight map and a second weight map based on the concatenated feature map through adaptive hybrid attention; and to obtain a seventh feature map based on the fifth feature map and the first weight map, and to obtain an eighth feature map based on the sixth feature map and the second weight map; The multi-scale expansion layer is used to extract features based on the seventh feature map of the convolution kernel to obtain the third feature map, and to extract features based on the eighth feature map of the convolution kernel to obtain the fourth feature map; The detection module is used to input the third and fourth images of the workpiece to be inspected, which are respectively acquired under the two lighting conditions, into the defect detection model to obtain the defect type of the workpiece to be inspected.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images