A visual detection method based on YOLO multi-modal space-time alignment

Abstract

The application discloses a kind of visual detection methods based on YOLO multimodal space-time alignment, belong to industrial visual detection technical field.The method is by hardware trigger synchronous acquisition first modality image and second modality image, and respectively constructs feature block sequence;The state transition of second modality is dynamically gated modulation using the hidden state of first modality, realize cross-modal space-time alignment and feature fusion;Then, the fusion feature is input into bidirectional visual Mamba backbone network to extract multi-scale global features, and the YOLO detection head is combined to complete target positioning and class recognition.The application can effectively improve the accuracy, stability and real-time performance of multimodal visual detection in high-speed motion scene.

Description

[Technical Field] This invention belongs to the fields of industrial computer vision, non-destructive testing and deep learning technology. Specifically, it relates to a method, system and storage medium that combines the Bi-directional Vision Mamba (Bi-Vim) state space model with the YOLO object detection framework and achieves high-precision real-time detection of defects in objects such as bearings through a multimodal spatiotemporal alignment mechanism. [Background Technology] Bearings, as key basic components in mechanical equipment, are widely used in high-end equipment fields such as aerospace, high-speed rail, automobile manufacturing, and wind power generation. Their quality level directly affects the operational safety and service life of the entire system. During the manufacturing and service process of bearings, defects such as surface scratches, pits, cracks, and internal thermal fatigue can easily occur due to processing defects or long-term fatigue wear. Therefore, efficient and accurate detection of bearing defects is an important link in ensuring the reliable operation of equipment.

[0003] Currently, the industry mainly uses machine vision-based non-destructive testing technology for bearing defect detection. However, after a systematic analysis of existing technologies, it was found that current testing solutions still have the following three main shortcomings under complex working conditions.

[0004] First, single-modal visual inspection has inherent limitations. Most existing solutions rely solely on visible light images for surface defect identification. However, bearing surfaces typically exhibit strong metallic reflectivity and are prone to oil contamination, easily leading to significant artifacts and interference in visible light images. Furthermore, visible light imaging cannot penetrate metal surfaces, making it difficult to identify internal thermal fatigue or subsurface hidden cracks within the bearing. While some studies have attempted to introduce multimodal fusion methods such as visible light and infrared thermal imaging, these often employ simple image stitching or shallow feature weighting fusion strategies, lacking effective deep spatiotemporal alignment mechanisms. Especially in high-speed industrial production line scenarios, differences in physical installation positions and electronic shutter response times between visible light and infrared cameras easily generate millisecond-level temporal deviations and pixel-level spatial parallax, causing fused features to blur and overlap at defect boundaries, leading to false positives and false negatives. Therefore, the multimodal spatiotemporal misalignment problem under high-speed conditions remains a critical technical challenge that urgently needs to be addressed in the field of bearing inspection.

[0005] Second, existing deep learning models suffer from performance bottlenecks. Most current mainstream defect detection algorithms are based on convolutional neural networks (CNNs), especially the YOLO series. Limited by their local receptive field, CNNs struggle to fully model the global contextual information of an image, resulting in limited effectiveness in detecting bearing defects with large morphological spans and discrete spatial distributions (such as through-cracks). In recent years, the Visual Transformer (ViT) has been introduced into the defect detection field to enhance global modeling capabilities; however, the computational complexity of its self-attention mechanism increases quadratically with image resolution, leading to a large parameter scale and slow inference speed, making it difficult to meet the real-time requirements of high-speed online inspection in industrial production lines.

[0006] Third, the application of the emerging Mamba architecture in cross-modal spatiotemporal alignment tasks still has shortcomings. State-space models (SSMs), especially the Mamba architecture, have shown good global modeling potential in the vision field in recent years, achieving a large receptive field while maintaining linear time complexity. However, most existing multimodal Mamba research adopts a two-stream symmetric fusion mechanism, treating different modalities as equal objects for feature exchange. It has not yet designed an asymmetric gating modulation mechanism for spatiotemporal misalignment caused by high-speed motion in industrial production lines, nor has it achieved deep collaboration with the YOLO detection framework, thus making it difficult to effectively cope with multimodal defect detection tasks under complex working conditions. [Summary of the Invention] To address the following core issues in existing technologies: (1) Single-modal detection is susceptible to interference from metal reflections and oil stains, and it is difficult to identify internal defects; (2) Multimodal fusion is prone to dynamic spatiotemporal misalignment in high-speed motion scenarios, resulting in fussing and overlapping of fused features in the defect boundary area; (3) Existing deep learning models are difficult to balance global perception capability and real-time reasoning efficiency. This invention aims to provide a visual detection method based on YOLO multimodal spatiotemporal alignment.

[0008] It should be noted that the technical problem addressed by this invention differs from general cross-modal fusion problems. In high-speed industrial production line inspection scenarios, due to inherent response time differences and installation position deviations between the first modal sensor (such as an infrared thermal imager) and the second modal sensor (such as a visible light camera), the position and shape of the same defect in the two modal images will undergo systematic shifts, resulting in spatiotemporal misalignment. Existing symmetrical fusion schemes assign approximately equal weights to the two modalities, often introducing misalignment information into the fusion result, causing ghosting artifacts and boundary blurring, significantly reducing the detection accuracy of minute defects. This invention, by setting an asymmetric gating mechanism, uses the first modality, which has stronger temporal continuity and more stable physical properties, as the dominant signal to dynamically correct the spatial offset of the second modality, thereby effectively solving the unique spatiotemporal misalignment detection problem in industrial scenarios.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A first aspect of the present invention provides a visual detection method based on YOLO multimodal spatiotemporal alignment, comprising the following steps: Step S1: Synchronously acquire the first modality image and the second modality image of the detected object using a hardware triggering mechanism, and perform preprocessing to generate the first modality feature block sequence and the second modality feature block sequence, respectively; Step S2: Input the first modality feature block sequence and the second modality feature block sequence into the multimodal spatiotemporal alignment module. Use the temporal hidden state of the first modality as a dynamic gating variable to adaptively weight and modulate the state transition matrix of the second modality. This allows the spatial propagation process of the second modality features to be dynamically guided and corrected by the first modality features, thereby achieving dynamic alignment of spatiotemporal features between the two modalities at the hidden state space level. Then, the aligned features are concatenated along the channel dimension and nonlinearly fused through a multilayer perceptron to generate a multimodal fused feature sequence. Step S3: Input the multimodal fusion feature sequence into the bidirectional visual Mamba backbone network, and extract multiple high-level feature maps of different scales containing global context information through bidirectional state space calculation of forward and backward scanning; Step S4: Input the multiple high-level feature maps of different scales into the improved YOLO detection head. After fusion by the feature pyramid network, the decoupling head predicts the coordinates and category of the defect bounding box respectively, and outputs the final defect detection result after non-maximum suppression.

[0010] Furthermore, the cross-modal state transition equation of the multimodal spatiotemporal alignment module is shown in equation (2): Equation (2)

[0011] In equation (2), and Let represent the hidden states of the visible light mode and the infrared mode at time step t, respectively. and These represent the discretized state transition matrix and input matrix, respectively. Indicates the characteristics of visible light input. This represents a nonlinear activation function with a range constraint. Represents the learnable weight matrix. This indicates element-wise multiplication.

[0012] In a second aspect, the present invention provides a training method for a defect detection model based on bidirectional visual Mamba and YOLO multimodal spatiotemporal alignment. The training method includes steps such as constructing a training dataset, performing feature fusion using a multimodal spatiotemporal alignment module, extracting features through bidirectional visual Mamba, calculating a joint loss function, and updating model parameters through backpropagation.

[0013] A third aspect of the present invention provides a defect detection system based on bidirectional visual Mamba and YOLO multimodal spatiotemporal alignment, the system comprising a multimodal perception module, an edge computing host, and an actuator.

[0014] Compared with the prior art, the present invention has the following significant advantages: 1. A multimodal dynamic spatiotemporal alignment mechanism is proposed, which can significantly reduce the false detection rate. Addressing the modal misalignment problem caused by high-speed motion in industrial production lines, this invention proposes an asymmetric cross-modal dynamic spatiotemporal alignment method based on a state-space model. Unlike existing dual-stream symmetric fusion or static alignment schemes, this method utilizes the physical stability of the first mode as an alignment anchor point to dynamically guide and correct the spatial position of the second mode at the feature hidden state level, thereby achieving adaptive compensation for millisecond-level temporal deviations and pixel-level spatial deviations.

[0015] 2. Achieving an effective correspondence between algorithmic features and physical effects. This invention maps the hidden states of the infrared mode to gating weights using an asymmetric state transition equation, and performs element-wise multiplication with the state transition matrix of the visible light mode. When the infrared mode detects an abnormal thermal response in a local region, the corresponding gating weight is increased to enhance the effective propagation of visible light features in that region; in areas without anomalies, the gating weight is decreased to suppress visible light noise that may be introduced by motion blur, thereby improving the stability and interpretability of the cross-modal fusion results.

[0016] 3. Balancing global perception capabilities with high-speed inference performance to meet the requirements of real-time industrial inspection. This invention introduces bidirectional visual Mamba into defect detection tasks to replace traditional CNN or Transformer backbone networks. Through forward and backward state space scanning, bidirectional visual Mamba achieves global context modeling capabilities while keeping computational complexity at a linear level, making it more suitable for real-time inspection scenarios in industrial production lines.

[0017] 4. Achieve deep collaborative optimization of Mamba and YOLO to improve the accuracy of locating minute defects. This invention deeply integrates the multi-scale output of bidirectional visual Mamba with the feature pyramid of YOLO, giving full play to the former's advantages in global representation and the latter's ability in multi-scale target regression, thereby improving the location accuracy and classification accuracy of difficult-to-detect defects such as minute scratches and minor edge damage. [Attached Image Description] Figure 1 The above is an overall flowchart of the method provided in the embodiments of the present invention.

[0019] Figure 2 This is a schematic diagram illustrating the structural principle of the Multimodal Spatiotemporal Alignment Module (MSTA) provided in an embodiment of the present invention.

[0020] Figure 3 A schematic diagram of the bidirectional visual Mamba (Bi-Vim) backbone network structure provided in an embodiment of the present invention.

[0021] Figure 4 This is a hardware architecture block diagram of a bearing defect detection system provided in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram of the YOLO detection head and PANet feature pyramid structure provided in an embodiment of the present invention.

[0023] Figure 6 This is a schematic diagram of the multimodal data preprocessing process provided in an embodiment of the present invention.

[0024] Figure 7 The diagram illustrates the loss function and training process provided in this embodiment of the invention.

Detailed Implementation Methods

[0026] Example 1: A visual detection method based on YOLO multimodal spatiotemporal alignment.

[0027] like Figure 1 As shown, the method provided in this embodiment includes the following specific steps: Step S1: Synchronous acquisition and preprocessing of multimodal data.

[0028] A PLC controller is used to generate synchronous trigger pulse signals to control an industrial-grade visible light camera and an uncooled infrared thermal imager to simultaneously expose and acquire visible light images of the moving bearings on the conveyor belt. and infrared thermal imaging images During the preprocessing of the acquired images, histogram equalization was first performed on the visible light images to suppress the influence of metallic reflections; non-uniformity correction (NUC) was performed on the infrared images to eliminate the inherent noise of the infrared detector. Subsequently, both types of images were uniformly scaled to a resolution of 640×640, and a non-overlapping sliding window was used to divide the image into 16×16 feature patches. These feature patches were then flattened and mapped to a D=256-dimensional feature space using a linear projection layer to obtain the visible light feature sequences. and infrared feature sequences .

[0029] Step S2: Multimodal dynamic spatiotemporal alignment based on state-space model.

[0030] like Figure 2 As shown, the visible light feature sequence and infrared feature sequences Input the Multimodal Spatiotemporal Alignment Module (MSTA). This module utilizes the discretized form of the continuous state-space model to achieve cross-modal alignment. For the infrared mode, its state-space equation is shown in equation (3): Equation (3)

[0031] In equation (3), Indicates a hidden state. and Let represent the state transition matrix and input matrix after discretization using the zero-order hold (ZOH) method, respectively.

[0032] To achieve dynamic spatiotemporal alignment, this invention hides the infrared mode state. As a gating condition, the state transition process of the visible light mode is dynamically modulated, and its dynamic modulation equation is shown in equation (4): Equation (4)

[0033] The execution process and its physical meaning on specific computing devices (such as GPUs) are as follows: (1) Reading and linear transformation: At each time step t, the program reads the infrared modal hidden state vector h_ir(t) from the video memory. This hidden state encodes the temporal change information of the temperature field caused by friction and fatigue during the operation of the bearing. Since infrared radiation is not easily affected by visible light interference factors such as metal reflection and oil stains, it can be used as a physical anchor point for cross-modal alignment.

[0034] (2) Gated signal generation: through a learnable weight matrix The matrix-vector multiplication operation is performed, and the output value is compressed to the [0,1] interval by combining the Sigmoid activation function to generate the gated vector. When the infrared image detects an abnormal temperature distribution in a local area, the gating signal outputs a higher response value close to 1 in the corresponding area; conversely, it outputs a lower response value close to 0.

[0035] (3) Adaptive state transition matrix calculation: The above gate vector and the discretized state transition matrix of the visible light mode are used to calculate the state transition matrix. Element-wise multiplication yields an adaptive state transition matrix guided by infrared modes. In regions with spatiotemporal misalignment, the infrared gating signal can adaptively enhance or suppress the propagation of visible light features in specific directions, thereby achieving dynamic compensation for spatial displacement deviations at the latent state level.

[0036] (4) Hidden state update: The adaptive state transition matrix is ​​used to update the hidden state of the visible light in the previous time step. Perform matrix-vector multiplication and add the contribution of the input at the current time step to update the visible light hidden state. And write it back to the video memory.

[0037] Finally, the aligned features are concatenated along the channel dimension and fused using two multilayer perceptrons to obtain a multimodal fused feature sequence. .

[0038] Step S3: Multi-scale global feature extraction based on bidirectional visual Mamba.

[0039] like Figure 3 As shown, The data is restored to a two-dimensional feature map format and input into the Bi-Vim backbone network, which consists of four stages. In each Vim Block, to overcome the limitation of traditional Mamba, which only supports unidirectional causal sequence modeling, a bidirectional scanning mechanism is employed: first, the forward scan flattens the two-dimensional feature map into a one-dimensional sequence according to the raster order from top left to bottom right of the image and performs forward state space computation; second, the backward scan flattens the sequence according to the reverse raster order from bottom right to top left of the image and performs backward state space computation. Subsequently, the bidirectional features are summed and layer normalization is performed. After multiple stages of processing, the Bi-Vim backbone network outputs three high-level globally perceptive feature maps C3, C4, and C5 at different scales.

[0040] Step S4: YOLO inspection head enhancement and defect prediction.

[0041] like Figure 5 As shown, feature maps C3, C4, and C5 are input into the improved YOLO detection head. This detection head employs a Path Aggregation Network (PANet) structure to fully integrate deep semantic features with shallow spatial detail features. During the prediction phase, a decoupled head design is used, separating the classification and regression tasks into two independent branches. During inference, the network outputs a set of predicted bounding boxes containing defect coordinates, category, and confidence level. Overlapping predicted bounding boxes are then removed using the Non-Maximum Suppression (NMS) algorithm, resulting in the final defect detection result.

[0042] Example 2: Model training method.

[0043] This embodiment provides a training method for the above-mentioned defect detection model, including the following steps: Step T1: Construct a training dataset containing multiple sets of synchronously acquired visible light and infrared image pairs of bearings under high-speed motion conditions, and label and preprocess the training dataset. To enhance the model's robustness to spatiotemporal misalignment, during the data augmentation stage, a random affine transformation is applied synchronously to the visible light and infrared images, and a small random spatiotemporal offset perturbation is intentionally introduced to simulate spatiotemporal misalignment phenomena under extreme industrial scenarios.

[0044] Step T2: Perform feature fusion using the multimodal spatiotemporal alignment module.

[0045] Step T3: Extract features using bidirectional visual Mamba and output prediction results using the YOLO detection head.

[0046] Step T4: Calculate the joint loss function and train the module using the AdamW optimizer with a cosine annealing scheduling strategy for the learning rate. Furthermore, fine-tune the gating weight matrix W_g in the multimodal spatiotemporal alignment module using a small learning rate multiplier to ensure the stability of the cross-modal guidance process.

[0047] Step T5: Update the model parameters end-to-end through backpropagation until the model converges.

[0048] Example 3: Defect detection system based on bidirectional vision Mamba and YOLO multimodal spatiotemporal alignment.

[0049] like Figure 4 As shown, this embodiment provides a hardware system for performing the above method, including: (1) Multimodal sensing module. The multimodal sensing module includes a high-resolution industrial camera and an uncooled infrared thermal imager. The two imaging devices are connected to the same PLC controller via a hard trigger line to ensure that the exposure time difference is kept within a small range.

[0050] (2) Edge computing host. The edge computing host is equipped with an industrial-grade GPU and internally deploys a multimodal Mamba-YOLO model accelerated and optimized by TensorRT. After INT8 quantization, the inference time per frame can be controlled within 15ms, thus meeting the requirements of real-time detection.

[0051] (3) Actuator. The actuator is a high-speed pneumatic rejection device that communicates with the edge computing host via the EtherCAT industrial Ethernet protocol, and is used to reject defective products based on the prediction results.

[0052] The above embodiments are merely preferred embodiments of the present invention and are used for illustration rather than limiting the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A visual detection method based on YOLO multimodal spatiotemporal alignment, characterized in that, Includes the following steps: Step S1: Synchronously acquire the first modality image and the second modality image of the detected object using a hardware triggering mechanism, and preprocess the first modality image and the second modality image to generate the first modality feature block sequence and the second modality feature block sequence, respectively; Step S2: Input the first modality feature block sequence and the second modality feature block sequence into the multimodal spatiotemporal alignment module. Use the temporal hidden state of the first modality as a dynamic gate variable to adaptively weight and modulate the state transition matrix of the second modality to achieve dynamic alignment of spatiotemporal features between the two modalities in the hidden state space. Then, fuse the aligned features to generate a multimodal fused feature sequence. Step S3: Input the multimodal fusion feature sequence into the bidirectional visual Mamba backbone network, and extract multiple high-level feature maps of different scales through bidirectional state space calculation of forward and backward scanning; Step S4: Input the multiple high-level feature maps of different scales into the YOLO detection head, predict the bounding box coordinates and category of the target after feature fusion, and output the final visual detection result through non-maximum suppression.

2. The visual inspection method according to claim 1, characterized in that, The object being detected is a bearing in operation, the first modal image is an infrared thermal imaging image, and the second modal image is a visible light image.

3. The visual inspection method according to claim 1, characterized in that, The preprocessing in step S1 includes: denoising or correcting the first modal image and the second modal image, scaling them uniformly to a preset resolution, then using a non-overlapping sliding window to divide the image into feature blocks of a fixed size, and mapping the feature blocks to a high-dimensional feature space through linear projection.

4. The visual inspection method according to claim 1, characterized in that, The cross-modal state transition equation in step S2 is shown in equation (1): Equation (1) In equation (1), This represents the hidden state of the first mode at time step t. This represents the hidden state of the second mode at time step t. and These represent the discretized state transition matrix and input matrix, respectively. This represents the second modality input features. Represents a non-linear activation function. This represents the learnable weight matrix.

5. The visual inspection method according to claim 4, characterized in that, The nonlinear activation function σ is any one of the Sigmoid function, Tanh function, or Softmax function.

6. The visual inspection method according to claim 1, characterized in that, In step S2, the aligned first modal features and second modal features are spliced ​​along the channel dimension and nonlinearly fused using a multilayer perceptron, wherein the hidden layer activation function of the multilayer perceptron is GELU.

7. The visual inspection method according to claim 1, characterized in that, The bidirectional visual Mamba backbone network in step S3 includes multiple stages, each stage including a Patch Merging downsampling layer and multiple Vim Blocks; In each Vim Block, the forward scan flattens the feature map according to the spatial order of the image from the top left to the bottom right, and the backward scan flattens the feature map according to the spatial order of the image from the bottom right to the top left. After performing state space calculations, the bidirectional features are added together and normalized.

8. The visual inspection method according to claim 7, characterized in that, The backbone network outputs three feature maps at different scales, with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the input image resolution, respectively.

9. The visual inspection method according to claim 1, characterized in that, The YOLO detection head in step S4 adopts a path aggregation network structure, which includes a top-down feature pyramid and a bottom-up path enhancement module, and uses a decoupled head to perform classification prediction and bounding box regression respectively.

10. The visual inspection method according to claim 1, characterized in that, After the trained model is deployed with INT8 quantization optimization by the inference acceleration engine, the inference time per frame does not exceed 15ms.

Images