Microscopic inspection device and method for strain gauge patch based on multi-modal fusion

Abstract

The invention relates to a strain gauge patch microscopic inspection device and method based on multi-modal fusion, the device comprises an imaging device, an illumination and optical assembly and an auxiliary module, and the imaging device comprises a visible light camera, a polarization camera and an infrared camera. According to the invention, high-precision measurement requirements and challenges of complex industrial scenes are fully considered, and the method has the characteristics of multi-modal fusion, high-precision identification and dynamic self-adaption; a visible light-polarized light-thermal imaging multi-mode cooperative detection technology based on deep learning is used as a core method, and position deviation, adhesive layer defects and a curing state of a strain gauge are used as core monitoring indexes, so that the comprehensiveness and reliability of detection are remarkably improved; cross-modal feature fusion is realized through a cross attention mechanism and LSTM time sequence modeling, adaptive polarization illumination and online feedback control are combined, the method can adapt to various materials, deployment is simple and convenient, real-time detection is supported, and a guarantee is provided for high-reliability measurement in the field of precision manufacturing.

Description

Technical Field

[0001] The present invention relates to a non-destructive testing of a strain measurement device, and in particular to a strain gauge patch microscopic testing device and method based on multimodal fusion. Background Art

[0002] Strain gauges are a widely used technology in engineering measurement and control systems. Based on the strain gauge's resistance strain effect, they convert mechanical strain into an electrical signal, enabling precise strain measurement of an object. The operating principle of strain gauges is based on the resistance strain effect. When the test object deforms due to an external force, the strain gauge attached to the surface also deforms. Because the strain gauge's sensitive grid is made of conductive material, its resistance changes with deformation. This resistance change can be detected and recorded by a measurement circuit, resulting in strain data on the test object.

[0003] In the fields of industrial measurement and structural health monitoring, strain gauge patches are widely used for the precise measurement of mechanical parameters such as stress and strain. The measurement accuracy is directly dependent on the quality of the patch process, including the uniformity of the adhesive layer, position accuracy, and defect control. However, traditional detection methods rely on manual visual inspection and observation with a microscope or magnifying glass, which is inefficient and highly subjective, making it difficult to detect micron-level defects. They rely on contact measurement, using probes to measure resistance or ultrasonic testing, which may damage the strain gauge or the substrate material. In recent years, with the rapid development of the Internet of Things and artificial intelligence technologies, single-modal machine vision uses visible light imaging. Directly using machine vision to check for defects cannot identify defects within the transparent adhesive layer or interference from the reflection of the metal substrate.

[0004] To address the above issues, the present invention proposes a strain gauge patch microscopic inspection device and method based on multimodal fusion, which aims to meet the non-destructive detection of internal defects in the transparent adhesive layer of the strain gauge and realize the integrity detection of the metal foil grid line. Summary of the Invention

[0005] In light of current technological deficiencies, the present invention provides a strain gauge patch microscopic inspection device and method based on multimodal fusion. When used to inspect the installation quality of patch-type strain gauges, the present invention effectively addresses issues such as the inability of ordinary visible light cameras to effectively detect bubbles or uncured areas within the glue; interference from metal reflections, which leads to overexposure and obscures strain gauge edge details; and a low degree of automation, relying on manual visual inspection or single-modal equipment. Through the collaborative detection of polarized light, visible light, and infrared thermal imaging, the present invention enables rapid, fully automatic, and highly accurate assessment of strain gauge patch quality.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: A strain gauge patch microscopic inspection device based on multimodal fusion, the device comprising an imaging device, an illumination and optical component, and an auxiliary module, the imaging device comprising a visible light camera, a polarization camera, and an infrared camera; Visible light camera, used to capture the strain gauge surface topography, such as position deviation and glue overflow; Polarization camera, which suppresses reflections through polarized light and enhances the contrast of defects inside the transparent adhesive layer; Infrared camera (thermal imager) to record temperature changes during the glue curing process and evaluate curing uniformity; The illumination and optical assembly includes a microscope, a ring-shaped LED array, and a polarization assembly; Microscope, used to magnify the observation area and cooperate with visible light camera to achieve micron-level detection; Ring-shaped LED array for brightfield and darkfield illumination, with adjustable brightness to suit different substrate materials; Polarization component, installed in front of the ring-shaped LED array, generates linearly polarized light with an adjustable polarization direction of 0° to 180° to eliminate specular reflections; The auxiliary module includes a heating source and a storage table; A heating source is used to stimulate the curing of the glue to release heat; The stage has XYZ three-axis fine-tuning function to ensure stable focus of the sample being tested.

[0007] Preferably, the heating source is a temperature-controllable heating plate.

[0008] Preferably, the imaging device is connected to a PLC control system, and the PLC hardware triggering ensures synchronization between imaging and mechanical movement.

[0009] The present invention also provides a strain gauge patch microscopic inspection method based on multimodal fusion, which uses the above-mentioned device and includes the following steps: S1. Fix the object to be measured with the strain gauge on a table. After preliminary focusing with a microscope, perform multimodal data acquisition, collecting visible light images, polarized light images, and thermal imaging sequences. S2. Perform multimodal feature fusion on the image obtained in step S1, and perform patch quality assessment through collaborative analysis of visible light images, polarized light images, and thermal imaging sequences; S3, comprehensive judgment logic, generates the final quality judgment result according to the following rules: The position deviation ΔXY < 50 μm, bubble / crack probability < 0.1 and curing score > 0.7 must be met at the same time to be qualified; Exception handling: If any condition is not met, it will be marked as unqualified and the specific defect type and coordinates will be output.

[0010] Preferably, step S1 is: S1-1. Visible light imaging is captured by an industrial camera with a telecentric lens, and the ring-shaped LED array uses bright field illumination mode. The camera is triggered by PLC hardware to ensure that imaging is synchronized with mechanical movement. S1-2. Adjust the linear polarizer in front of the annular LED array so that the linear polarizer and the polarization filter of the polarization camera are cross-polarized to maximize the suppression of specular reflection and capture polarized light images. S1-3. The sequence acquisition of thermal imaging is realized by the infrared thermal imager and the heating source. For the thermal imaging sequence acquisition, the heating source is first started to stimulate the exothermic reaction of the glue, and then the infrared camera takes continuous shots (the thermal imager takes continuous shots at 10fps for 30 seconds) to record the dynamic changes of the temperature field and the ambient temperature at the same time.

[0011] The multimodal feature fusion of the image obtained in step S1 can detect the patch quality of the strain gauge. Through the collaborative analysis of visible light images, polarized light images and thermal imaging sequences, high-precision and automated patch quality assessment can be achieved. As a preferred embodiment, step S2 is: S2-1. Visible light and polarized light feature extraction and fusion: First, the RGB channels of the visible light image are fed into a pre-trained EfficientNet-B3 network to extract high-level semantic features and output feature A. This network uses a compound scaling strategy to balance computational efficiency and feature expression capabilities, capturing macroscopic defects on the strain gauge surface (such as glue overflow and positional offset). Simultaneously, the polarized light image is fed into a lightweight ResNet18 network to extract polarization-specific feature B. After feature A and feature B are concatenated across channels, a cross-attention mechanism is introduced to dynamically assign weights. The visible light feature focuses on glue overflow and positional offset, while the polarized light feature enhances bubble edges and crack details. Finally, the fused feature C is output, achieving efficient integration of complementary information. S2-2. Thermal imaging sequence feature extraction: The thermal imaging sequence is input into the LSTM network for modeling and curing. The LSTM hidden units capture the temporal dependencies of the temperature curve. The fully connected layer outputs feature D, which encodes the curing state and local thermal anomalies (such as the difference in thermal resistance in the debonding area). S2-3, Decision-level fusion and quality assessment: First, align and concatenate the features. Feature C is globally average pooled and then concatenated with feature D. Then, through fusion, multiple defect categories are output. The global feature vector of feature C after global average pooling compression is spliced ​​with the temporal feature of feature D along the dimension to form a joint feature; the joint feature is input into the fully connected network to output the probability distribution of multiple categories of defects.

[0012] Preferably, the polarization degree image is obtained by the following formula: in, The degree of polarization indicates the intensity of the polarization characteristics of light and reflects the scattering ability of the material surface; is the incident light intensity with a polarization direction of 0°, representing the polarization component parallel to the reference direction; is the incident light intensity with a polarization direction of 90°, representing the polarization component perpendicular to the reference direction; Represents a very small constant to prevent the denominator from being zero.

[0013] Preferably, step S1 is specifically as follows: S1-1, visible light imaging, using a ring-shaped LED array for brightfield illumination, with an incident angle of 45° and brightness adjusted to 1200 lux; the industrial camera is triggered by PLC hardware to capture synchronized images with an exposure time of ≤1ms, capturing three images at different focal lengths; S1-2, polarized light imaging, rotate the polarizer in front of the annular LED array to 45° for the metal substrate and 90° for the composite material. The polarization filter of the polarization camera is orthogonal to the polarizer to suppress specular reflection; take polarization images at 0° and 90°, and calculate the polarization degree image , input ResNet18 network; S1-3. Thermal imaging sequence acquisition: Start the heating source at 80°C / 5s to stimulate the exothermic reaction of the glue; the infrared camera continuously shoots at 10fps for 30 seconds, a total of 300 frames, and simultaneously records the ambient temperature; crop the strain gauge area to 128×128 pixels to generate a temperature-time curve.

[0014] Preferably, step S2 is specifically as follows: S2-1. Visible light and polarized light feature extraction and fusion: Feature extraction is performed on both. EfficientNet-B3 is used to extract high-level feature A from visible light images, capturing macro defects such as position offset and glue overflow. ResNet18 is used to extract feature B from polarized light images, focusing on bubble edges and crack details. Feature A and feature B are spliced ​​along the channel, and weights are dynamically assigned using a cross-attention mechanism. Among them, Attention() is the attention function, which realizes the attention weight distribution of query-key-value pairs; () is a normalized exponential function that converts the similarity score into a probability distribution (the sum of the weights is 1); the query matrix , It is feature A, It is a learnable weight matrix in the query direction, which indicates the content features that need to be paid attention to. Each query corresponds to an output position; the key matrix , is feature B, It is a learnable weight matrix in the key direction, representing the reference features of the query, used to calculate the similarity with the query; the value matrix is feature B, It is a learnable weight matrix in the value direction, which stores the feature information that needs to be extracted; key / query dimension After feature A and feature B are spliced ​​along the channel and weighted, fusion feature C is output to enhance the complementary information. S2-2. Thermal imaging time series feature extraction: First, preprocess the temperature curve and perform baseline temperature compensation: in, is the normalized temperature, is the instantaneous measured temperature, is the ambient temperature, and then the temperature-compensated curve is subjected to sliding average filtering: in, The smoothed temperature is the temperature signal after noise is eliminated; is the window width, which controls the number of sampling points for smoothing; is the historical standardized temperature, which represents the relative temperature at time t−k; Then, LSTM modeling is performed, inputting a 300-dimensional temperature time series signal into a single-layer LSTM to capture the solidified dynamic features. The input gate of the LSTM unit selects new information, the forget gate discards old information, the candidate state generates new memories, the memory cell integrates new and old memories, and the output gate controls the output of the hidden state. LSTM dynamically controls the flow of information through a gating mechanism: in, is the hidden state at the previous time step, is the current input feature, is the Sigmoid function, which compresses the output to [0,1], indicating the information retention ratio; is the weight matrix, including the input gate weight , forget gate weight , output gate weight ; is the gate bias vector, which controls the threshold offset of gate activation; Memory cells are the core of LSTM, which updates memory by forgetting old information and adding new information: in, The amount of historical memory retained is controlled by the forget gate. It is to filter candidate memories through the input gate, Is the hidden state of the connection at the previous moment With the current input The weight matrix, is the bias vector corresponding to the candidate memory unit; Finally, the short-term memory of the current time step is output, and the exposure degree of the memory cell is controlled by the output gate: in, is the hidden state of the current time step, which carries short-term memory information and serves as part of the input of the next time step; It is the memory of the current unit Perform nonlinear compression to map long-term memory information to the range of [-1,1], is the activation value of the output gate, which controls which parts of the memory unit can be output as hidden states, thereby determining which information is useful for the next time step; S2-3. Decision-level fusion and quality assessment: First, feature dimensions are aligned. Spatial feature C, derived from the fusion of visible light and polarized light, is compressed into a global feature vector through global average pooling. Temporal feature D, derived from thermal imaging time series analysis, retains its dynamic characteristics. Features C and D are concatenated along the feature dimension to form a joint feature vector, preserving both spatial defect information and temporal state. The joint feature is then fed into a fully connected network for defect classification.

[0015] Preferably, the network structure of the fully connected network is a 128-dimensional hidden layer + ReLU activation function in the first layer and an 8-dimensional output layer in the second layer, corresponding to 8 types of defects; the probability distribution of 8 types of defects is output, including position offset, glue overflow, and warping in macro defects, bubbles, cracks, and wrinkles in micro defects, and insufficient glue and uncured glue in process defects.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention focuses on intelligent detection of strain gauge patch quality, fully considering the challenges of high-precision measurement requirements and complex industrial scenarios, and has the characteristics of multimodal fusion, high-precision recognition, and dynamic self-adaptation. It adopts the visible light-polarized light-thermal imaging multimodal collaborative detection technology based on deep learning as the core method, and uses the position deviation, adhesive layer defects, and curing status of the strain gauge as the core monitoring indicators, which significantly improves the comprehensiveness and reliability of the detection. The invention realizes cross-modal feature fusion through the cross-attention mechanism and LSTM time series modeling, combines adaptive polarized lighting with online feedback control, and can adapt to a variety of materials such as metals, composite materials, and flexible substrates. It is easy to deploy and supports real-time detection, ensuring the "zero defect" standard of the strain gauge patch process and providing a guarantee for high-reliability measurement in the field of precision manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the framework of the device of the present invention; Figure 2 It is a schematic diagram of the visible light-polarized light-thermal imaging multimodal collaborative detection network structure of the present invention. DETAILED DESCRIPTION

[0018] The technical solutions of the present invention are further specifically described below through examples. These examples are provided for the purpose of illustrating the present invention and are not intended to limit the present invention. All other examples obtained by persons of ordinary skill in the art based on the examples in this application without creative work are intended to fall within the scope of protection of this application.

[0019] Reference Figure 1 A strain gauge patch microscopic inspection device based on multimodal fusion includes an imaging device, an illumination and optical component, and an auxiliary module, which collaborate to achieve multimodal data acquisition and processing; the imaging device includes a high-resolution visible light camera, a polarization camera, and an infrared camera; High-resolution visible light camera, using a 50MP industrial camera with a telecentric lens, is used to capture the strain gauge surface topography (position deviation, glue overflow); The polarization camera is equipped with four polarization filters (0°, 45°, 90°, and 135°). It suppresses reflections through polarized light and enhances the contrast of defects inside the transparent adhesive layer. The polarization degree image is obtained using the following formula: in, The degree of polarization indicates the intensity of the polarization characteristics of light and reflects the scattering ability of the material surface; is the incident light intensity with a polarization direction of 0°, representing the polarization component parallel to the reference direction; is the incident light intensity with a polarization direction of 90°, representing the polarization component perpendicular to the reference direction; Represents a very small constant to prevent the denominator from being zero.

[0020] It is used to prevent the divisor from being 0. The closer the polarization degree is to 1, the stronger the scattering characteristics of the area (such as the edge of a bubble); the closer it is to 0, the more specular reflection (such as a metal substrate), which can suppress the reflection of the metal substrate and enhance the contrast of internal defects (bubbles, cracks) in the transparent adhesive layer; Infrared camera (thermal imager), using a microbolometer, to record temperature changes during the glue curing process and evaluate the curing uniformity; The illumination and optical assembly includes a microscope, a ring-shaped LED array, and a polarization assembly; Microscope, equipped with 5X~20X objective lens, works together with high-resolution camera to achieve micron-level defect observation; Ring-shaped LED array, which integrates multiple rings of RGBW LED beads, supports bright field and dark field lighting modes, dynamic brightness adjustment, and is suitable for different substrates such as metal and composite materials; The polarization component is installed in front of the annular LED array and uses a motorized rotating linear polarizer to generate 0° to 180° adjustable polarized light to eliminate specular reflections; The auxiliary module includes a heating source and a storage table; The heating source is a temperature-controllable heating plate, which is used to stimulate the heat release during glue curing; The storage table has XYZ three-axis fine-tuning function, XYZ three-axis electric fine-tuning platform, and integrated vacuum adsorption function to ensure stable focus of the measured sample.

[0021] The method for using the above device for microscopic inspection of strain gauge patches includes the following steps: S1. Multimodal data acquisition requires collecting three types of data: visible light imaging, polarized light imaging, and thermal imaging sequences as input data for the multimodal fusion network; S1-1, visible light imaging, using annular LED brightfield illumination (incident angle 45°) with brightness adjusted to 1200 lux; a high-resolution camera was triggered synchronously via PLC hardware, with an exposure time of ≤1ms, capturing three images at different focal lengths; S1-2, polarized light imaging, rotate the LED front polarizer to the optimal angle (45° for metal substrates, 90° for composite materials), and the camera polarization filter is orthogonal to it to suppress specular reflection; take 0° and 90° polarization images, and calculate the polarization degree image , input ResNet18 network; S1-3, thermal imaging sequence acquisition: start the infrared heating plate (80°C / 5s) to stimulate the exothermic reaction of the glue; the thermal imager continuously shoots at 10fps for 30 seconds (a total of 300 frames), and simultaneously records the ambient temperature (for compensation); crop the strain gauge area (128×128 pixels) to generate the temperature-time curve T(t); S2, multimodal feature fusion, such as Figure 2 As shown, the visible light and polarized light features are fused, the thermal imaging time series features are extracted, and then the three are fused at the decision level; S2-1. Visible light and polarized light features are fused and extracted. EfficientNet-B3 extracts high-level feature A from the visible light image (RGB channels), capturing macro defects such as positional offset and glue overflow. ResNet18 extracts feature B from the polarized light image, focusing on bubble edges and crack details. Features A and B are spliced ​​along the channels, and weights are dynamically assigned using a cross-attention mechanism. The formula is as follows: Among them, Attention() is the attention function, which realizes the attention weight distribution of query-key-value pairs; () is a normalized exponential function that converts the similarity score into a probability distribution (the sum of the weights is 1); the query matrix , It is feature A, It is a learnable weight matrix in the query direction, which indicates the content features that need to be paid attention to. Each query corresponds to an output position; the key matrix , is feature B, It is a learnable weight matrix in the key direction, representing the reference features of the query, used to calculate the similarity with the query; the value matrix is feature B, It is a learnable weight matrix in the value direction, which stores the feature information that needs to be extracted; key / query dimension After feature A and feature B are spliced ​​along the channel and weighted, fusion feature C is output to enhance the complementary information. S2-2. Thermal imaging time series feature extraction: First, preprocess the temperature curve and perform baseline temperature compensation. The formula is as follows: in, is the normalized temperature, is the instantaneous measured temperature, is the ambient temperature, and then the temperature-compensated curve is subjected to sliding average filtering: in, is the ambient temperature. Then perform sliding average filtering on the temperature-compensated curve, and the formula is as follows: in, The smoothed temperature is the temperature signal after noise is eliminated; The window width is 5, which controls the number of sampling points for smoothing. is the historical standardized temperature, which represents the relative temperature at time t−k; Then, LSTM modeling is performed, and a 300-dimensional temperature time series signal is input to a single-layer LSTM to capture the solidified dynamic features. The input gate of the LSTM unit selects new information, the forget gate discards old information, the candidate state generates new memories, the memory cell integrates new and old memories, and the output gate controls the output of the hidden state. LSTM dynamically controls the flow of information through a gating mechanism. The formula is as follows: in, is the hidden state at the previous time step, is the current input feature, is the Sigmoid function, which compresses the output to [0,1], indicating the information retention ratio; is the weight matrix, including the input gate weight , forget gate weight , output gate weight ; is the gate bias vector, which controls the threshold offset of gate activation; Memory cells are the core of LSTM, which updates memory by forgetting old information and adding new information: in, The amount of historical memory retained is controlled by the forget gate. It is to filter candidate memories through the input gate, Is the hidden state of the connection at the previous moment With the current input The weight matrix, is the bias vector corresponding to the candidate memory unit.

[0022] Finally, the short-term memory of the current time step is output, and the exposure degree of the memory cell is controlled by the output gate: in, is the hidden state of the current time step, which carries short-term memory information and serves as part of the input of the next time step; It is the memory of the current unit Perform nonlinear compression to map long-term memory information to the range of [-1,1], is the activation value of the output gate, which controls which parts of the memory unit can be output as hidden states, thereby determining which information is useful for the next time step; S2-3. Decision-level fusion and quality assessment. First, to ensure effective fusion of features from different modalities, the features are dimensionally aligned. The spatial feature C from the fusion of visible light and polarized light is compressed into a global feature vector through global average pooling, while the temporal feature D from thermal imaging time series analysis retains its dynamic characteristics. The two are concatenated along the feature dimension to form a joint feature vector, while retaining both spatial defect information and temporal solidification status. The joint features are input into a fully connected network for defect classification. The network structure consists of a 128-dimensional hidden layer with a ReLU activation function in the first layer and an 8-dimensional output layer in the second layer (corresponding to 8 defect categories). The model outputs the probability distribution of 8 defect categories, including macro defects (position offset, glue overflow, and warping); micro defects (bubbles, cracks, and wrinkles); and process defects (insufficient glue and uncured glue).

[0023] S3. Comprehensive judgment logic generates the final quality judgment result according to the following rules. The result is qualified only if the position deviation ΔXY < 50 μm, bubble / crack probability < 0.1 and curing score > 0.7 are met simultaneously. Exception handling: If any of the conditions are not met, it is marked as "unqualified" and the specific defect type and coordinates are output.

[0024] To verify the performance of the strain gauge patch quality assessment method based on the coordinated detection of polarized light, visible light, and infrared thermal imaging, a comparative experiment was conducted. The experimental design is as follows: Strain gauge samples with different attachment quality states were selected, including high-quality patches, those with tiny air bubble inclusions, edge warping, and local debonding defects. Traditional manual visual inspection methods and the detection system of the present invention were used for inspection and evaluation.

[0025] The test results are as follows: In terms of detection speed, the system of the present invention can detect a single strain gauge in less than 2 seconds, compared with the average time of about 30 seconds for traditional manual visual inspection, and the detection efficiency is improved by about 15 times. This invention can meet the needs of fast and continuous detection scenarios.

[0026] In terms of detection accuracy, the system achieved 98.7% accuracy for 1,000 strain gauge patch quality test samples, a significant improvement over manual visual inspection (approximately 87.2%). When detecting tiny defects with a diameter less than 0.5 mm, the system's missed detection rate was significantly lower than traditional methods and other existing algorithms, demonstrating enhanced defect recognition capabilities.

[0027] In terms of defect location accuracy, this system integrates polarized light, visible light, and infrared thermal imaging information, combined with deep learning image processing algorithms, to achieve submillimeter defect recognition with a spatial resolution of 0.2mm. Compared to conventional thermal imaging systems, this system demonstrates significant advantages in locating tiny structural defects, making it suitable for high-precision manufacturing and structural integrity assessment scenarios.

[0028] In summary, the experimental data fully demonstrate that the present invention can complete the fully automatic assessment of strain gauge patch quality at a speed significantly faster than traditional methods, and has higher detection accuracy and defect location precision, making it suitable for the actual engineering needs of large-scale, high-precision detection.

[0029] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A strain gauge patch microscopic inspection device based on multimodal fusion, characterized in that: The apparatus includes an imaging device, an illumination and optical assembly, and an auxiliary module, wherein the imaging device includes a visible light camera, a polarization camera, and an infrared camera; Visible light camera, used to capture the strain gauge surface topography; Polarization camera, which suppresses reflections through polarized light and enhances the contrast of defects inside the transparent adhesive layer; Infrared camera, which records temperature changes during the glue curing process and evaluates curing uniformity; The illumination and optical assembly includes a microscope, a ring-shaped LED array, and a polarization assembly; Microscope, used to magnify the observation area and cooperate with visible light camera to achieve micron-level detection; Ring-shaped LED array for brightfield and darkfield illumination, with adjustable brightness to suit different substrate materials; Polarization component, installed in front of the ring-shaped LED array, generates linearly polarized light with an adjustable polarization direction of 0° to 180° to eliminate specular reflections; The auxiliary module includes a heating source and a storage table; A heating source is used to stimulate the curing heat of the glue; The stage has XYZ three-axis fine-tuning function to ensure stable focus of the sample being tested.

2. The strain gauge patch microscopic inspection device based on multimodal fusion according to claim 1 is characterized in that: The heating source is a temperature-controllable heating plate.

3. The strain gauge patch microscopic inspection device based on multimodal fusion according to claim 1, characterized in that: The imaging device is connected to a PLC control system.

4. A microscopic inspection method for strain gauge patches based on multimodal fusion, characterized in that: The device according to any one of claims 1 to 3 comprises the following steps: S1. Fix the object to be measured with the strain gauge on a table. After preliminary focusing with a microscope, perform multimodal data acquisition, collecting visible light images, polarized light images, and thermal imaging sequences. S2. Perform multimodal feature fusion on the image obtained in step S1, and perform patch quality assessment through collaborative analysis of visible light images, polarized light images, and thermal imaging sequences; S3, comprehensive judgment logic, generates the final quality judgment result according to the following rules: The position deviation ΔXY < 50 μm, bubble / crack probability < 0.1 and curing score > 0.7 must be met at the same time to be qualified; Exception handling: If any condition is not met, it will be marked as unqualified and the specific defect type and coordinates will be output.

5. The strain gauge patch microscopic inspection method based on multimodal fusion according to claim 4 is characterized in that: Step S1 is: S1-1, visible light imaging is captured by an industrial camera with a telecentric lens, and the ring LED array uses bright field illumination mode; S1-2. Adjust the linear polarizer in front of the annular LED array so that the linear polarizer and the polarization filter of the polarization camera are cross-polarized to maximize the suppression of specular reflection and capture polarized light images. S1-3, thermal imaging sequence acquisition, first start the heating source to stimulate the glue exothermic reaction, then the infrared camera takes continuous shots to record the dynamic changes of the temperature field and the ambient temperature at the same time.

6. The strain gauge patch microscopic inspection method based on multimodal fusion according to claim 5 is characterized in that: Step S2 is: S2-1. Visible light and polarized light feature extraction and fusion: First, the RGB channels of the visible light image are input into the pre-trained EfficientNet-B3 network to extract high-level semantic features and output feature A. Simultaneously, the polarized light image is input into the lightweight ResNet18 network to extract polarization-specific feature B. After feature A and feature B are concatenated across channels, a cross-attention mechanism is introduced to dynamically assign weights. The visible light feature focuses on glue overflow and positional deviation, while the polarized light feature enhances bubble edges and crack details. Finally, the fused feature C is output, achieving efficient integration of complementary information. S2-2. Thermal imaging sequence feature extraction: The thermal imaging sequence is input into the LSTM network for modeling and solidification. The LSTM hidden unit captures the temporal dependency of the temperature curve. The fully connected layer outputs feature D, which encodes the solidification state and local thermal anomalies. S2-3, Decision-level fusion and quality assessment: First, align and concatenate the features. Feature C is globally average pooled and then concatenated with feature D. Then, through fusion, multiple defect categories are output. The global feature vector of feature C after global average pooling compression is spliced ​​with the temporal feature of feature D along the dimension to form a joint feature; the joint feature is input into the fully connected network to output the probability distribution of multiple categories of defects.

7. The strain gauge patch microscopic inspection method based on multimodal fusion according to claim 6, characterized in that: The image polarization degree is obtained by the following formula: in, is the degree of polarization, is the incident light intensity with 0° polarization direction, is the incident light intensity with 90° polarization direction, Represents a very small constant to prevent the denominator from being zero.

8. The strain gauge patch microscopic inspection method based on multimodal fusion according to claim 5, characterized in that: Step S1 is specifically as follows: S1-1, visible light imaging, using a ring-shaped LED array for brightfield illumination, with an incident angle of 45° and brightness adjusted to 1200 lux; the industrial camera is triggered by PLC hardware to capture synchronized images with an exposure time of ≤1ms, capturing three images at different focal lengths; S1-2, polarized light imaging, rotate the polarizer in front of the annular LED array to 45° for the metal substrate and 90° for the composite material. The polarization filter of the polarization camera is orthogonal to the polarizer to suppress specular reflection; take polarization images at 0° and 90°, and calculate the polarization degree image , input ResNet18 network; S1-3. Thermal imaging sequence acquisition: Start the heating source at 80°C / 5s to stimulate the exothermic reaction of the glue; the infrared camera continuously shoots at 10fps for 30 seconds, a total of 300 frames, and simultaneously records the ambient temperature; crop the strain gauge area to 128×128 pixels to generate a temperature-time curve.

9. The strain gauge patch microscopic inspection method based on multimodal fusion according to claim 5, characterized in that: Step S2 is specifically as follows: S2-1. Visible light and polarized light feature extraction and fusion: Feature extraction is performed on both. EfficientNet-B3 is used to extract high-level feature A from visible light images, capturing macro defects such as position offset and glue overflow. ResNet18 is used to extract feature B from polarized light images, focusing on bubble edges and crack details. Feature A and feature B are spliced ​​along the channel, and weights are dynamically assigned using a cross-attention mechanism. Among them, Attention() is the attention function, () is the normalized exponential function, the query matrix , is feature A, is the learnable weight matrix of the query direction; the key matrix , is feature B, is the learnable weight matrix in the key direction; the value matrix is feature B, is a learnable weight matrix in the value direction; key / query dimension After feature A and feature B are spliced ​​along the channel and weighted, fusion feature C is output to enhance the complementary information. S2-2. Thermal imaging time series feature extraction: First, preprocess the temperature curve and perform baseline temperature compensation: in, is the normalized temperature, is the instantaneous measured temperature, is the ambient temperature, and then the temperature-compensated curve is subjected to sliding average filtering: in, is the smoothed temperature; is the window width; is the historical normalized temperature; Then, LSTM modeling is performed, inputting a 300-dimensional temperature time series signal into a single-layer LSTM to capture the solidified dynamic features. The input gate of the LSTM unit selects new information, the forget gate discards old information, the candidate state generates new memories, the memory cell integrates new and old memories, and the output gate controls the output of the hidden state. LSTM dynamically controls the flow of information through a gating mechanism: in, is the hidden state at the previous time step, is the current input feature, is the Sigmoid function, is the weight matrix, including the input gate weight , forget gate weight , output gate weight ; is the gate bias vector; Memory updating is accomplished by forgetting old information and adding new information: in, The amount of historical memory retained is controlled by the forget gate. It is to filter candidate memories through the input gate, Is the hidden state of the connection at the last moment With the current input The weight matrix, is the bias vector corresponding to the candidate memory unit; Finally, the short-term memory of the current time step is output, and the exposure degree of the memory cell is controlled by the output gate: in, is the hidden state at the current time step, It is the memory of the current unit Perform nonlinear compression, is the activation value of the output gate; S2-3. Decision-level fusion and quality assessment: First, the features are dimensionally aligned. The spatial feature C from the fusion of visible light and polarized light is compressed into a global feature vector through global average pooling, while the temporal feature D from thermal imaging time series analysis retains its dynamic characteristics. Feature C and feature D are concatenated along the feature dimension to form a joint feature vector, while retaining the spatial defect information and temporal solidification state. The joint feature is input into a fully connected network for defect classification.

10. The strain gauge patch microscopic inspection method based on multimodal fusion according to claim 9, characterized in that: The network structure of the fully connected network is a 128-dimensional hidden layer + ReLU activation function in the first layer, and an 8-dimensional output layer in the second layer, corresponding to 8 types of defects; the probability distribution of the 8 types of defects is output, including position offset, glue overflow, and warping in macro defects, bubbles, cracks, and wrinkles in micro defects, and insufficient glue and uncured glue in process defects.

Images