Complex scene industrial defect recognition method combined with multi-modal sensor monitoring

Abstract

The present application relates to the field of electronic information technology, and discloses a kind of complex scene industrial defect identification method of combined multi-modal sensor monitoring, comprising the following steps: using industrial gray scale camera and CCD visible light camera to collect the image of SMT mainboard, respectively obtain gray image and RGB color image;Construct multi-modal fusion adaptation detection network, modal adaptation fusion is carried out to the gray image and RGB color image of SMT mainboard, and using detection head detects SMT mainboard defect;The present application uses spatial attention fusion to excavate the correlation between the multi-modal imaging data of SMT mainboard gray image and RGB color image in spatial position, and generates fusion feature map by spatial position weighting.Simultaneously, using modal adaptation and induced attention mechanism to guide detection network to focus on SMT mainboard defect feature area, realize the real-time accurate detection of SMT mainboard defect, improve SMT mainboard defect feature discrimination ability.

Description

Technical Field

[0001] This invention relates to the field of electronic information technology, and more specifically to a method for identifying industrial defects in complex scenarios using combined multimodal sensing and monitoring. Background Technology

[0002] With the rapid development of deep learning-based computer vision, deep learning-based object detection methods are being applied more and more extensively in industrial defect detection. Currently, sufficient solutions have been proposed for visualizing defects in most industrial scenarios. Meanwhile, for complex defects where multimodal data can be collected from multiple sensors, computers are typically used to process and fuse the different data before defect detection. This image fusion method possesses redundancy and complementarity, overcoming the limitations of the original image in terms of resolution, physical properties, and information content. It can also suppress noise and enhance the aggregation of image information. Ultimately, this can significantly improve the detection accuracy of complex defects with multimodal data in industrial scenarios.

[0003] However, existing image data fusion methods are computationally complex and difficult to apply to real-time industrial production line inspections. Furthermore, the fusion models lack robustness and cannot adapt to interference from lighting and shadows, making it difficult to effectively detect complex defect features. Therefore, there is an urgent need for a method for industrial defect identification in complex scenarios that combines multimodal sensing and monitoring. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for identifying industrial defects in complex scenarios using combined multimodal sensing and monitoring.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for identifying industrial defects in complex scenarios using combined multimodal sensing and monitoring includes the following steps:

[0007] Step 1: Use an industrial grayscale camera and a CCD visible light camera to acquire images of the SMT motherboard, obtaining grayscale images and RGB color images respectively;

[0008] Step 2: Construct a multimodal fusion and adaptation detection network to perform modal adaptation and fusion on the grayscale and RGB color images of the SMT motherboard, and use the detection head to detect defects on the SMT motherboard; the multimodal fusion and adaptation detection network includes:

[0009] A two-dimensional residual convolutional network is used to extract deep variation feature maps from grayscale and RGB color images of SMT motherboards, respectively. Where i represents the SMT motherboard image number;

[0010] The spatial attention fusion module utilizes deep variation feature maps. We obtain spatial location information feature maps M for two modalities, and then input the spatial location information feature maps M into a convolutional layer for interactive learning to obtain the spatial attention feature map Z. i ; through Z i Obtain the spatial attention weight map and calculate the fused feature map.

[0011] The modal adaptation and fusion module fuses feature maps. Convolution and max pooling operations are performed to obtain SMT defect feature maps.

[0012] The attention-inducing module will Encoded as keyword K The encoding is used to query Q and value V, and the SMT motherboard feature map after induced attention encoding in the i-th layer is calculated.

[0013] The inspection head fuses the feature map with the SMT motherboard feature map after inducing attention encoding. Detect defects in SMT motherboards.

[0014] Furthermore, step one specifically includes:

[0015] S11: Place the grayscale camera and the CCD color camera parallel to each other above the SMT motherboard to be inspected, and set them to the same sampling frequency;

[0016] S12: Acquires grayscale images of the SMT motherboard by setting the shooting trigger signal. and RGB color images n represents the image number of the SMT motherboard being photographed.

[0017] Furthermore, step two specifically includes:

[0018] S21: Design a two-dimensional residual convolutional network to extract grayscale images of SMT motherboards. Gray-scale deep layer variation feature map RGB color image Color deep layer variation feature map

[0019]

[0020] in These are the input and output of the l-th layer residual unit in a two-dimensional residual convolutional network, respectively, where f is the residual mapping function and θ is the input and output of the l-th layer residual unit. l These are the learning parameters;

[0021] S22: In the spatial attention fusion module, and Perform average pooling and max pooling operations along the channel direction respectively, and then... SMT motherboard RGB color imaging information and The grayscale imaging information of the SMT motherboard is aggregated in the channel direction and then stitched together in the channel direction to obtain the spatial position information feature maps M∈R of the two modes. 4 ×H×W H and W represent the height and width of the spatial location information feature map, respectively.

[0022] The spatial location information feature map M is input into the convolutional layer for interactive learning: Z i =Conv(M; θ); where Z i ∈R 2 ×H×W Here, θ represents the spatial attention feature map, Conv(·) represents the convolutional layer, and θ represents the interaction learning parameters.

[0023] Using softmax to extract the spatial attention feature map Z i Mapping the channels to the range [0,1] yields two spatial attention weight maps;

[0024] Calculate the fused feature map using the generated spatial attention weight map. Where c represents the number of channels in the fused feature map;

[0025] S23: Merge feature maps The input is fed into the modality adaptation and fusion module to fuse the feature maps. Convolution and max pooling operations are performed to obtain SMT defect feature maps.

[0026] S24: The induced attention module uses an induced attention encoding method to... Encode as key K∈R C×h×w , Encoding is query Q∈R C×H×W Sum of values ​​V∈R C×H×W ,Will and Adjust alignment in the channel direction;

[0027] Induced attention involves convolution between Q and K to obtain the matching matrix P:

[0028] The convolution of P at position (x,y) is calculated as follows: P(x,y)=∑ h,w Q(x+h,y+w)K(h,w);

[0029] Using convolution to Location matching of SMT defect feature regions in the data;

[0030] The matching matrix P is input into the softmax layer for spatial location calculation, resulting in the location weight information Att.

[0031]

[0032] in, It is a scaling factor;

[0033] Upsample the position weight information Att to obtain the result. Spatial weight regions Att of the same size up :

[0034] Att up =UP(Att);

[0035] Where UP(·) represents bilinear interpolation upsampling; finally, weights are assigned to the value V to induce attention:

[0036]

[0037] in, This represents the SMT motherboard feature map after the i-th layer undergoes induced attention encoding;

[0038] S25: Merge feature maps and SMT motherboard feature map after induced attention encoding The data is input into the detection head, and the output is the detection result of defects on the SMT motherboard.

[0039] Further, in step S22, the and Perform average pooling and max pooling operations along the channel direction respectively, and then... SMT motherboard RGB color imaging information and The grayscale imaging information of the SMT motherboard is aggregated in the channel direction, and then stitched together in the channel direction to obtain the spatial position information feature map M∈R of the two modes. 4×H×W hour:

[0040]

[0041] Where Cat(·) represents channel concatenation, AvgPool(·) represents average pooling, and MaxPool(·) represents max pooling. These represent the color depth variation feature maps after average pooling and after max pooling, respectively. These represent the gray-level depth variation feature maps after average pooling and max pooling, respectively.

[0042] Furthermore, in step S22, softmax is used to apply the spatial attention feature map Z. i Mapping the channels to the range [0,1] yields two spatial attention weight maps. The calculation of the fused feature map using these generated spatial attention weight maps specifically includes:

[0043] Will Defined as Z i , as the feature map of the first channel; Defined as Z i , as the feature map of the second channel;

[0044] At position (x,y) Corresponding grayscale space attention weights for:

[0045]

[0046] in, and These represent the positions at (x, y). element values ​​and The element value;

[0047] At position (x,y) Corresponding color space attention weights for:

[0048]

[0049] Finally, the fused feature map is calculated using the spatial attention weight map, which is composed of spatial attention weights.

[0050]

[0051] This indicates a convolution operation.

[0052] Furthermore, in step S23, the fused feature map is... Convolution and max pooling operations are performed to obtain SMT defect feature maps. hour:

[0053]

[0054] Where GMP(·) represents the maximum compression pooling operation.

[0055] Furthermore, in step S24, the induced attention encoding method is used to... Encode as key K∈R C×h×w , Encoding is query Q∈R C×H×W Sum of values ​​V∈R C×H×W hour:

[0056]

[0057] Among them, W Q ∈R 1×H×W W K ∈R 1×h×w and W V ∈R 1×H×W These represent the weight matrices for query Q, keyword K, and value V, respectively.

[0058] Compared with the prior art, the beneficial technical effects of the present invention are:

[0059] Spatial attention fusion is employed to uncover the spatial correlation between multimodal imaging data of SMT motherboard grayscale images and RGB color images, generating a fused feature map through spatial location weighting. Simultaneously, modal adaptation and induced attention mechanisms are used to guide the detection network to focus on defect feature regions of the SMT motherboard, achieving real-time and accurate defect detection and improving the defect feature discrimination capability of SMT motherboards. Attached Figure Description

[0060] Figure 1 This is a framework diagram of the multimodal fusion adaptation detection network of the present invention. Detailed Implementation

[0061] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0062] The proposed method for identifying industrial defects in complex scenarios using combined multimodal sensing and monitoring can effectively identify industrial defects in SMT (Surface Mount Technology) production.

[0063] The method for identifying industrial defects in complex scenarios using combined multimodal sensing monitoring in this invention includes the following steps:

[0064] Step 1: Use an industrial grayscale camera and a CCD visible light camera to acquire grayscale images and RGB color images of SMT motherboard components.

[0065] Step one specifically includes the following sub-steps:

[0066] S11: Place the grayscale camera and the CCD color camera parallel to each other above the SMT motherboard to be inspected, and set them to the same sampling frequency.

[0067] S12: Acquires a grayscale image of the SMT motherboard based on the set shooting trigger signal. and RGB color images n represents the sequence number of the SMT motherboard image captured.

[0068] Step Two: As Figure 1 As shown, a multimodal fusion and adaptation detection network is constructed to perform modal adaptation and fusion on grayscale images and RGB color images of SMT motherboard components, and to realize SMT motherboard defect detection using a detection head.

[0069] Step two specifically includes the following sub-steps:

[0070] S21: Design a 2D residual convolutional network to extract grayscale images of SMT motherboard components. and RGB color images Deep variation feature map It can be written as:

[0071]

[0072] in These are the input and output units of the l-th residual unit, respectively, where f is the residual mapping function and θ is the input and output units. l These are the learning parameters.

[0073] S22: In the spatial attention fusion module, and By performing averaging and max pooling operations along the channel direction respectively, we can achieve... SMT motherboard RGB color imaging information and In SMT motherboards, grayscale imaging information of components is aggregated along the channel direction, reducing the number of parameters required for subsequent network learning. Then, the data is concatenated along the channel direction to obtain spatial location information feature maps M∈R for two modalities. 4×H×W Specifically, it is expressed as follows:

[0074]

[0075]

[0076] Among them, F' avg ∈R 1×H×W and F' max ∈R 1×H×W These are represented as the feature maps after average pooling and max pooling, respectively. Finally, the feature maps are input into the convolutional layer for interactive learning. The convolutional layer consists of two convolutions, a ReLU function, and a softmax function. The first convolution has 128 3x3 kernels, and the second convolution has two kernels of the same size; represented as follows:

[0077] Zi =Conv(M; θ);

[0078] Among them, Z i ∈R 2×H×W The output is the spatial attention feature map, Conv(·) is the convolutional layer, and θ is the interaction learning parameter. Then, softmax is used to transform the convolutional output Z. i Mapping the channels to the range [0, 1] yields two spatial attention weight maps. Specifically, Defined as Z i This is the feature map of the first channel. Defined as Z i , is the feature map of the second channel, for the position (x,y) Spatial attention weights Represented as:

[0079]

[0080] in, and These represent the positions at (x, y). and Element value, This represents the spatial weight at position (x, y). For the position (x, y), Spatial attention weights Represented as:

[0081]

[0082] Finally, using the spatial attention weight map composed of the generated spatial attention weights, the fused feature map is calculated using the following formula.

[0083]

[0084] This indicates a convolution operation.

[0085] S23: Then, the fused feature map obtained by the spatial attention fusion module is... The input is fed into the modality adaptation and fusion module. The fused feature map is then processed. Max pooling is performed, and the fused feature maps are merged before max pooling compression. The input is convolved into a 3x3 kernel, and after each convolution, a ReLU activation function, LRN, and max pooling are applied. The height (h) and width (w) of the max pooled inputs into the three modality adaptation fusion modules are set to 10x10, 8x8, and 5x5, respectively, to obtain a more efficient representation. The formula is as follows:

[0086]

[0087] in, This is the SMT defect feature map obtained after being input into the modal adaptation and fusion module. It is extracted using GMP global max pooling. The wafer defect foreground region is learned using the Conv(*; θ) constraint. A new tensor W is also defined. MA ∈R C×L×L Its size is L×L, and The values ​​are the same, where C represents the number of weight channels and L represents the height and width of the weights.

[0088]

[0089] Where h and w represent respectively The height and width. Modal adaptation operations can be represented as follows:

[0090]

[0091] Where MAX(*) represents taking the maximum value of the input variable within a region of size h×w, and assigning 0 to all other variable values. From the above formula, it can be seen that W MA The vast majority of W was assigned a value of 0, leaving only the h*w size feature region. MA Inputting this into subsequent convolutional layers will significantly reduce the difficulty of network inference and enable more efficient learning of SMT defect features.

[0092] S24: Finally, a two-to-two induced attention coding method is adopted to... Encoded as K∈R C×h×w , Encoded as Q∈R C×H×W and V∈R C×H×W ,Will and Adjusting the alignment along the channel direction is shown below:

[0093]

[0094]

[0095]

[0096] Among them, W Q ∈R 1×H×W W K ∈R 1×h×w and W V ∈R 1×H×WLet Q represent the query, K, and the weight matrix of the value V, respectively. As shown in the formula above, the encoded result is a two-dimensional tensor, avoiding spatial location encoding. By... Size pooling compression The size of the matrix significantly reduces the number of encoding parameters and inference speed. Furthermore, to simplify the network and reduce the difficulty of inference, it is configured as a single-head inference method. Second, induced attention involves convolution between Q and K to obtain the matching matrix P, expressed as:

[0097]

[0098] in, This represents the convolution operation. The convolution calculation of P at position (x, y), taking a single channel as an example, is as follows:

[0099] P(x,y)=∑ h,w Q(x+h,y+w)K(h,w);

[0100] Using convolution can effectively... The SMT defect feature regions in the image are matched for location, with a convolution stride of 5 and padding of 2. Third, the matching matrix P is input into the softmax layer for spatial location calculation, yielding the location weight information Att, as shown below:

[0101]

[0102] in, The scaling factor is used to prevent excessively large multiplications during convolution, causing softmax to converge to regions with minimal gradients. Experiments show that D = c*h performs well. Fourth, the positional weight information Att is upsampled to obtain... The same size spatial weight region Att up , means as follows:

[0103] Att up =UP(Att);

[0104] Here, UP(*) represents bilinear interpolation upsampling. Finally, weights are applied to V to induce attention, as shown below:

[0105]

[0106] in, This represents the SMT motherboard feature map after induced attention encoding in the i-th layer. It is activated by two consecutive convolution operations and the LeakyReLU function, with a 3x3 kernel size. High attention to defect feature regions in the SMT motherboard feature map can be achieved by continuously stacking induced attention modules.

[0107] S25: Feeds back the fused features and adjusted SMT motherboard features to the inspection head, thereby achieving high-precision inspection of SMT motherboard defects.

[0108] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0109] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for identifying industrial defects in complex scenarios using combined multimodal sensing and monitoring, comprising the following steps: Step 1: Use an industrial grayscale camera and a CCD visible light camera to acquire images of the SMT motherboard, obtaining grayscale images and RGB color images respectively; Step 2: Construct a multimodal fusion and adaptation detection network to perform modal adaptation and fusion on the grayscale and RGB color images of the SMT motherboard, and use the detection head to detect defects on the SMT motherboard; the multimodal fusion and adaptation detection network includes: A two-dimensional residual convolutional network is used to extract deep variation feature maps from grayscale and RGB color images of SMT motherboards, respectively. , Where i represents the SMT motherboard image number; The spatial attention fusion module utilizes deep variation feature maps. , The spatial location information feature maps of the two modes are obtained. Spatial location information feature map The input is interactively learned in a convolutional layer to obtain a spatial attention feature map. ;pass Obtain the spatial attention weight map and calculate the fused feature map. ; The modal adaptation and fusion module fuses feature maps. Convolution and max pooling operations are performed to obtain SMT defect feature maps. ; The attention-inducing module will Encoding as keywords , Encoding for query Sum The feature map of the SMT motherboard after induced attention encoding in the i-th layer is calculated. Specifically, this includes: the induced attention module employs an induced attention encoding method, which... Encoding as keywords , Encoding for query Sum ,Will and Adjust alignment in the channel direction; Induced attention involves convolution between Q and K to obtain the matching matrix P: ; P in The convolution at a given position is calculated as follows: ; Using convolution to Location matching of SMT defect feature regions in the data; The matching matrix P is input into the softmax layer to calculate the spatial location and obtain the location weight information. : ； in, It is a scaling factor; Location weight information Perform upsampling to obtain the same as Spatial weight regions of the same size : ； in, This indicates bilinear interpolation upsampling; finally, the value... By assigning weights, attention can be induced: ； in, This represents the SMT motherboard feature map after the i-th layer undergoes induced attention encoding; The inspection head fuses the feature map with the SMT motherboard feature map after inducing attention encoding. To detect defects in SMT motherboards.

2. The method for identifying industrial defects in complex scenarios using combined multimodal sensing monitoring as described in claim 1, characterized in that, Step one specifically includes: S11: Place the grayscale camera and the CCD color camera parallel to each other above the SMT motherboard to be inspected, and set them to the same sampling frequency; S12: Acquires grayscale images of the SMT motherboard by setting the shooting trigger signal. and RGB color images , where n represents the image number of the SMT motherboard being photographed.

3. The method for identifying industrial defects in complex scenarios using combined multimodal sensing monitoring as described in claim 1, characterized in that, Step two specifically includes: S21: Design a two-dimensional residual convolutional network to extract grayscale images of SMT motherboards. Gray-scale deep layer variation feature map RGB color images Color deep layer variation feature map : ； in , These are the input and output of the l-th layer residual unit in the two-dimensional residual convolutional network, respectively, and f is the residual mapping function. These are the learning parameters; S22: In the spatial attention fusion module, and Perform average pooling and max pooling operations along the channel direction respectively, and then... SMT motherboard RGB color imaging information and The grayscale imaging information of the SMT motherboard is aggregated in the channel direction and then stitched together in the channel direction to obtain the spatial position information feature maps of the two modes. : and These are the height and width of the spatial location information feature map, respectively. Spatial location information feature map Interactive learning through input into convolutional layers: ;in, This is a spatial attention feature map. It is a convolutional layer. For interactive learning parameters; Use softmax to extract spatial attention feature maps Mapped on the channel Between these, two spatial attention weight maps are obtained; Calculate the fused feature map using the generated spatial attention weight map. Where c represents the number of channels in the fused feature map; S23: Merge feature maps The input is fed into the modality adaptation and fusion module to fuse the feature maps. Convolution and max pooling operations are performed to obtain SMT defect feature maps. ; S24: The induced attention module employs an induced attention encoding method, based on... and get ; S25: Merge feature maps and SMT motherboard feature map after induced attention encoding The data is input into the detection head, and the output is the detection result of defects on the SMT motherboard.

4. The method for identifying industrial defects in complex scenarios using combined multimodal sensing monitoring according to claim 3, characterized in that, In step S22, and Perform average pooling and max pooling operations along the channel direction respectively, and then... SMT motherboard RGB color imaging information and The grayscale imaging information of the SMT motherboard is aggregated in the channel direction, and then stitched together in the channel direction to obtain the spatial position information feature map of the two modes. hour: ； in, Indicates channel splicing. This indicates the average pooling operation. This indicates a max pooling operation. These represent the color depth variation feature maps after average pooling and after max pooling, respectively. These represent the gray-level depth variation feature maps after average pooling and max pooling, respectively.

5. The method for identifying industrial defects in complex scenarios using combined multimodal sensing monitoring according to claim 3, characterized in that, In step S22, the spatial attention feature map is processed using softmax. Mapped on the channel Between these two spatial attention weight maps, two spatial attention weight maps are obtained. When calculating the fused feature map using the generated spatial attention weight maps, the specific steps include: Will Defined as , as the feature map of the first channel; Defined as , as the feature map of the second channel; exist Location Corresponding grayscale space attention weights for: ； in, and They represent in Location element values ​​and The element value; exist Location Corresponding color space attention weights for: ； Finally, the fused feature map is calculated using the spatial attention weight map, which is composed of spatial attention weights. : ； This indicates a convolution operation.

6. The method for identifying industrial defects in complex scenarios using combined multimodal sensing monitoring according to claim 3, characterized in that, In step S23, the fused feature map is processed. Convolution and max pooling operations are performed to obtain SMT defect feature maps. hour: ； in, This indicates the maximum compression pooling operation.

7. The method for identifying industrial defects in complex scenarios using combined multimodal sensing monitoring according to claim 3, characterized in that, In step S24, the induced attention coding method is used to... Encoding as keywords , Encoding for query Sum hour: ； ； ； in, , and These represent the weight matrices for query Q, keyword K, and value V, respectively.

Images