Real-time horn mouth defect identification method based on multi-modal fusion and dynamic learning

Abstract

The invention discloses a horn mouth real-time defect identification method based on multi-modal fusion and dynamic learning, and the core of the method is that multi-modal data of image-point cloud-vibration signals is collected, the multi-modal data is subjected to enhancement preprocessing and then is input into a cross-modal Transform fusion network, a dynamically optimized composite loss function training model is combined, and finally high-precision real-time detection is realized. The invention discloses a horn mouth real-time defect identification method based on multi-modal fusion and dynamic learning, and aims to solve the problem of low detection rate of micro cracks, three-dimensional curled edges and internal stress defects only depending on images in traditional single-modal detection.

Description

Technical Field

[0001] This invention belongs to the field of industrial defect detection technology, specifically relating to a real-time defect identification method for horn-shaped openings based on multimodal fusion and dynamic adaptive learning. Background Technology

[0002] As a critical connection structure, the performance of steel pipe flared ends is paramount, and they are widely used in the connection parts of core equipment in industries such as petrochemicals, power energy, and intelligent manufacturing. Currently, flared ends are mainly formed through mechanical extrusion. In traditional production processes, each die-casting operation requires manual visual inspection for defects. This intermittent "single die-casting - manual inspection" operation mode severely restricts production efficiency.

[0003] To improve the efficiency of flared die casting, a solution using intelligent image recognition technology to replace manual inspection has been proposed. After implementation, operators can focus on continuous die casting while the system automatically monitors product quality in real time. The system will only trigger an alarm when a defect is detected, at which point operators will need to interrupt the process for intervention. This new "continuous die casting - intelligent monitoring" production model will significantly improve the efficiency and capacity of flared die casting.

[0004] Currently, traditional methods for identifying flared mouth defects (such as penetrant testing and ultrasonic testing) have significant limitations. These methods are typically inefficient and costly; some techniques (such as X-ray testing) require downtime, making them unsuitable for real-time applications; and conventional ultrasonic testing lacks sufficient sensitivity to micron-level early cracks and sealing surface degradation. Research indicates that flared mouth failure often originates from the accumulation of microscopic damage, with the early damage stage accounting for over 70% of the total lifespan. Timely warnings at the microscopic damage stage can significantly reduce the risk of sudden failure.

[0005] In recent years, deep learning-based visual recognition technology has provided new insights for real-time monitoring of flared mouths. By acquiring surface images of the flared mouth using high-resolution industrial cameras and combining them with convolutional neural networks (CNNs), automatic identification of defects such as cracks, notches, curled edges, and burrs can be achieved. However, existing methods still face challenges: variable lighting conditions in industrial environments lead to severe image noise interference; multiple devices operating simultaneously at different shooting angles make it difficult to extract features of minute defects; and it is difficult to balance the efficiency and accuracy of algorithms for real-time identification under dynamic operating conditions. Furthermore, traditional machine learning methods rely on large amounts of labeled data, while the scarcity of defect samples in industrial scenarios limits the generalization ability of the models.

[0006] Existing technologies for detecting horn-shaped defects have limitations. Current mainstream methods, such as ultrasonic testing and penetrant testing, are time-consuming, have low detection rates, and rely on manual operation, making them unsuitable for the real-time demands of continuous production. Traditional rule-based machine vision algorithms, limited by manual feature engineering, suffer from high false detection rates under complex lighting and multi-angle imaging conditions. Furthermore, existing deep learning solutions, due to their large number of model parameters, struggle to achieve millisecond-level real-time inference on edge devices, severely hindering their application in industrial settings. Summary of the Invention

[0007] To address the limitations of existing horn mouth defect detection technologies, such as low detection efficiency, poor real-time performance, insufficient sensitivity to small defects, limited model generalization ability due to the scarcity of defect samples, and weak robustness in complex industrial environments, this invention proposes a real-time horn mouth defect identification method based on multimodal fusion and dynamic learning. This method aims to improve the efficiency, accuracy, and robustness of horn mouth defect detection and meet the needs of real-time detection in industrial settings.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A real-time defect recognition method for horn-shaped mouths based on multimodal fusion and dynamic learning includes the following steps:

[0010] S1) Multimodal data acquisition and preprocessing: The data acquisition method involves using a binocular industrial camera to simultaneously acquire images of the left and right sides of the horn opening, integrating a laser point cloud sensor to obtain three-dimensional contour data, and a piezoelectric sensor to acquire vibration signals; multimodal collaborative preprocessing is performed on the three heterogeneous modal data acquired above. In the multimodal collaborative preprocessing, a multimodal data enhancement method and a multimodal feature extraction method oriented towards defect identification are adopted. The multimodal data enhancement method includes performing ±15° random rotation on the image to simulate attitude shift, performing contrast adaptive adjustment in the color space, and simulating illumination changes by HSV domain perturbation. The multimodal feature extraction method includes extracting surface curvature features from the laser point cloud and extracting frequency domain features from the vibration signal through wavelet transform.

[0011] S2) Dynamic Adaptive Model Construction: A cross-modal Transformer fusion network is adopted. The cross-modal Transformer fusion network includes feature extraction branches for different modalities and a Transformer fusion module. The feature extraction branches include image branches, point cloud branches, and vibration signal branches. The image branch is an improved ConvNeXt-Tiny architecture with 1 / 3 / 5 / 7 convolutional kernels, the point cloud branch is PointNet++, and the vibration signal branch is CNN-LSTM. The features extracted by each branch are used to achieve cross-modal association learning through the Transformer fusion module.

[0012] S3) Model Training: Using a composite loss function

[0013] in Spatial attention weights including the focused sealing surface area This is used to allow the model to actively focus on the key spatial region of the horn opening during training, while reducing the weight of non-key regions. This is an adaptive smoothing binary cross-entropy loss function, used to dynamically adjust the smoothing coefficient according to the sample type; Used to constrain abrupt changes in loss across consecutive frames;

[0014] S4) Real-time detection and alarm: Through a dual-buffered pipeline architecture and ONNX Runtime to accelerate inference, when the defect probability is ≥0.7, an audible and visual alarm with a response latency of <50ms is triggered and uploaded to the cloud for review.

[0015] In step 1, the image is dynamically cropped to 224×224 samples after being standardized to 256×256. The laser point cloud is downsampled to 1024 points after noise reduction and registration. The vibration signal is extracted for peak value and energy spectrum features after wavelet transform.

[0016] The improved ConvNeXt-Tiny architecture loads ImageNet pre-trained weights, and the feature extraction block uses an inverted bottleneck structure, Layer Normalization, and the GELU activation function.

[0017] In the composite loss function, the dynamic Focal Loss expression for spatial attention focusing is:

[0018] in, These are dynamic weighting coefficients; For adaptive focusing parameters; Spatial attention weights, The value should be ≥0.8 in the sealing surface area of ​​the flared mouth and ≤0.3 in non-critical areas;

[0019] The dynamic weighting coefficient ,in The adaptive focusing parameter represents the model's predicted probability for the i-th sample. ,in For the Sigmoid function, .

[0020] The adaptive smoothing cross-entropy expression in the composite loss function is: Among them, the adaptive smoothing coefficient .

[0021] In the composite loss function, the expression for the time consistency constraint term is: ,in, Let be the loss value of the t-th frame. When the difference in loss between consecutive frames exceeds 0.2, The weights are automatically doubled.

[0022] In the real-time detection, a ring-shaped LED light source and an ambient light sensor are integrated. The brightness of the light source is dynamically adjusted through a PID algorithm, with an adjustment range of 500-3000 lux and a camera exposure time of 1-10 ms.

[0023] The model training employs a progressive strategy: the backbone network is frozen for the first 50 rounds, and the loss function is optimized using Label Smoothing (Smoothing Factor=0.1); the network is unfrozen for the next 50 rounds, and the AdamW optimizer is used for end-to-end fine-tuning. The initial learning rate is 0.0003, the weight decay is 0.05, and the proportion of difficult examples in the dynamic batch processing is ≥40%. During training, a dynamic batch processing strategy is used to dynamically adjust the proportion of difficult examples in the batch according to the difficulty of the samples, thereby improving the model's learning performance on difficult examples.

[0024] Real-time defect detection and alarm systems are implemented on-site by deploying four high-frame-rate industrial cameras, laser point cloud sensors, and piezoelectric sensors on four horn-making machines. An integrated ring LED light source ensures stable image quality and monitors ambient light intensity in real time, dynamically adjusting camera exposure parameters and LED brightness accordingly. Multimodal data is transmitted to the main server via gigabit industrial Ethernet in milliseconds. The main server deploys a pre-trained model based on ONNX Runtime for accelerated inference. The system employs a dual-buffered pipeline architecture: when the model detects a defect probability exceeding a threshold (p≥0.7), it triggers an audible and visual alarm (response delay < 50ms) and simultaneously uploads the suspected sample to the cloud for expert review. Operators can view the newly made horn openings for defects in real time and address them promptly.

[0025] The present invention has the following beneficial effects:

[0026] 1. Multimodal data fusion: A unique image data processing and enhancement method, as well as data feature extraction, is designed for horn-mouth products. This allows for the correlation and fusion of multimodal data such as images, laser point clouds, and vibration signals. This enables the comprehensive capture of the surface and internal features of the horn-mouth, improving the ability to identify minute and non-surface defects. Compared with single visual inspection, the detection rate of minute cracks is increased by more than 30%.

[0027] 2. Dynamic Adaptive Model: Employing a Transformer-based cross-modal feature fusion network and a dynamic adaptive loss function, the model parameters and loss weights can be dynamically adjusted according to the difficulty, type, and spatial location of the sample, thereby improving the model's accuracy in identifying difficult samples and defects in key areas. The accuracy of difficult sample identification is improved by more than 25%.

[0028] 3. Strong robustness: Through multimodal data augmentation, dynamic adjustment of camera exposure parameters and LED light source brightness, and temporal consistency constraints, the model can adapt to complex environments such as changes in industrial lighting and noise interference, reducing the false detection rate by more than 40% in strong reflective or shadowy scenes.

[0029] 4. Excellent real-time performance: Adopting a lightweight network architecture, ONNX Runtime inference acceleration, and a double-buffered pipeline architecture, it achieves an alarm response of <50ms, meeting the real-time requirements of continuous die casting and significantly improving production efficiency.

[0030] 5. Efficient sample utilization: Through data augmentation and dynamic batch processing strategies, the problem of scarce defect samples in industrial scenarios is effectively solved, improving the generalization ability of the model. The generalization ability of the model is improved by more than 20% in the case of small samples. Attached Figure Description

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] Figure 1 is a schematic diagram of a standard flared mouth;

[0033] Figure 2 is a schematic diagram of the defective flared mouth;

[0034] Figure 3 is a schematic diagram of the horn-mouth detection system;

[0035] Figure 4 is a schematic diagram of the image detection process;

[0036] Figure 5 is a schematic diagram of the ConvNeXt-Tiny architecture;

[0037] Figure 6 is a schematic diagram of the feature extraction block structure. Detailed Implementation

[0038] To make the objectives, technical solutions, and effects of the present invention clearer, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0039] like Figure 1-6As shown, this invention discloses a real-time defect recognition method for horn-shaped openings based on multimodal fusion and dynamic learning, aiming to solve the problem of low detection rates for traditional single-modal detection, such as relying solely on images for micro-cracks, three-dimensional curling, and internal stress defects. Its core is to collect multimodal data of "image-point cloud-vibration signal," perform enhanced preprocessing, and input the data into a cross-modal Transformer fusion network. This data is then combined with a dynamically optimized composite loss function to train the model, ultimately achieving high-precision real-time detection.

[0040] The specific methods and steps are as follows:

[0041] 1. Multimodal data acquisition:

[0042] In this embodiment, the acquisition system for multimodal data acquisition and enhancement mainly includes the following devices:

[0043] Binocular industrial camera, model: Basler acA2500-14gm: Simultaneously acquires images of the left and right surfaces of the horn opening (resolution 2048×2048) for extracting surface texture features such as scratches and dents;

[0044] Laser point cloud sensor, model: Velodyne VLP-16: acquires 3D contour point cloud of the horn mouth (approximately 20,000 points per frame) for extracting 3D geometric features such as curling and deformation;

[0045] Piezoelectric sensor (model: PCB 352C33): mounted on the surface of the die-casting mold, it collects vibration signals (sampling rate 10kHz) to extract internal stress characteristics (such as vibration anomalies caused by hidden cracks).

[0046] The data acquisition system also includes an LED light source, a main server, an audible and visual alarm device, and a cloud platform. A binocular industrial camera is used to acquire images of the horn-shaped nozzle surface; a laser point cloud sensor is used to obtain 3D contour data of the nozzle; a piezoelectric sensor is used to acquire vibration signals during the die-casting process; a ring-shaped LED light source provides stable illumination; the main server is used for data processing, model inference, and alarm control; the audible and visual alarm device issues an alarm signal when a defect is detected; and the cloud platform is used to store suspicious samples and conduct expert review.

[0047] 2. Multimodal data preprocessing and enhancement

[0048] 2.1 Image preprocessing methods:

[0049] Step 1: Normalize the original image to 256×256 pixels to eliminate size differences;

[0050] Step 2: Perform data augmentation: Rotate the image randomly by ±15° to simulate attitude shift during horn-mouth transmission; use HSV domain perturbation to simulate workshop lighting fluctuations. The image parameters for HSV domain perturbation are set as follows: brightness ±20%, saturation ±15%; set elastic deformation to simulate minute surface deformation, with a scaling factor of 0.1.

[0051] Step 3: Dynamic cropping: Based on the ROI of the sealing surface calibrated by binocular vision, i.e., the coordinates of the region of interest, crop a 224×224 core area of ​​the sealing surface from the enhanced 256×256 image.

[0052] 2.2 Point Cloud Preprocessing:

[0053] Noise reduction: Outliers are removed using statistical filtering with a neighborhood count of 50 and a standard deviation threshold of 1.0.

[0054] Registration: Align the point cloud with the preset horn-shaped 3D model using the ICP algorithm;

[0055] Downsampling: Voxel filtering is used to downsample the point cloud to 1024 points, balancing accuracy and computational cost;

[0056] Feature extraction: Calculate the surface curvature at each point (reflecting variations in surface roughness, formula: ,in These are the eigenvalues ​​of the covariance matrix.

[0057] 2.3 Vibration signal preprocessing:

[0058] Wavelet transform: The signal is decomposed into 5 levels using the db4 wavelet basis to extract the energy spectrum features of the 0-1250Hz frequency band;

[0059] Temporal features: Extracting peak value, kurtosis, and root mean square value (reflecting vibration intensity and abnormal fluctuations).

[0060] 3. Construction of Dynamic Adaptive Model

[0061] 3.1 Image Branch (Improved ConvNeXt-Tiny)

[0062] like Figure 5 As shown in the diagram, the improved ConvNeXt-Tiny architecture mainly consists of the following parts:

[0063] 1) Input layer: Receives pre-processed 224×224×3 image data, which are images of the left and right sides of the horn mouth surface synchronously acquired by a binocular industrial camera.

[0064] 2) Initial convolutional layer (Conv 3×3): Uses 3×3 convolutional kernels for preliminary feature extraction, providing basic feature maps for subsequent feature extraction blocks.

[0065] 3) Feature Extraction Block: As shown in the attached document Figure 6 As shown, these feature extraction blocks are the core components of the ConvNeXt-Tiny architecture, employing an inverted bottleneck structure, multi-scale depthwise separable convolutions, LayerNormalization, and the GELU activation function. Each feature extraction block contains multiple parallel depthwise separable convolutional layers (7×7, 5×5, 3×3, 3×3), fusing multi-scale information through feature concatenation.

[0066] 4) Global Average Pooling: Performs global average pooling on the feature map, converting the feature map into a feature vector.

[0067] 5) Fully connected layer (Linear): The feature vector is processed through two fully connected layers to gradually reduce the feature dimension.

[0068] 6) Prediction layer: Outputs the final defect prediction result, used to determine whether there is a defect in the flared mouth.

[0069] Improvements: One-quarter of the redundant convolutional layers in the original architecture were removed, reducing the number of parameters from 28M to 17M, a reduction of 40%, to meet the real-time requirements of industrial applications.

[0070] Pre-training: Load pre-trained weights from the ImageNet dataset to initialize feature extraction capabilities.

[0071] 3.2 Point Cloud Branch (PointNet++)

[0072] Structure: Includes an input layer (1024 points → 64-dimensional features), a sampling layer (FPS sampling), a grouping layer (ball query), and a feature propagation layer, ultimately outputting a 512-dimensional point cloud feature vector.

[0073] 3.3 Vibration Signal Branch (CNN-LSTM)

[0074] Structure: 1D-CNN (3 convolutional layers, extracting local frequency domain features) + LSTM (2 layers, capturing temporal dependencies), outputting a 512-dimensional signal feature vector.

[0075] 3.4 Transformer Fusion Module

[0076] Encoder: 6 stacked layers, each containing 8 self-attention mechanisms, scaling dot product attention formula: ;

[0077] Fusion logic: The feature vectors of image (512-dimensional), point cloud (512-dimensional), and vibration signal (512-dimensional) are concatenated into a 1536-dimensional input. The encoder learns cross-modal associations, such as the mapping relationship of "image crack - point cloud depression - vibration peak". The output fused features are used for defect classification.

[0078] 4. Model Training

[0079] Using a composite loss function The model is optimized using the following training strategy:

[0080] 4.1 Components of the Loss Function

[0081] Spatial Attention Focal Loss: In the composite loss function, the dynamic Focal Loss expression for spatial attention focusing is:

[0082] In the formula, N is the batch size, which can be 32 in this embodiment; (Real labels: 1 for defect, 0 for normal); The model predicts the probability of defects; Spatial attention weights, The value should be ≥0.8 in the sealing surface area of ​​the flared mouth and ≤0.3 in non-critical areas; dynamic weighting coefficient. ,in The adaptive focusing parameter represents the model's predicted probability for the i-th sample. ,in For the Sigmoid function, .

[0083] Adaptive smooth cross-entropy: The expression is: Among them, the adaptive smoothing coefficient For normal samples (high proportion), the smoothing is stronger, with a value of 0.1; for defective samples (low proportion), the smoothing is weaker, with a value of 0.05, to avoid diluting defective features.

[0084] Timing consistency constraint: The expression is as follows ,in, Let be the loss value of the t-th frame. When the difference in loss between consecutive frames exceeds 0.2, The weights are automatically doubled. Specific function: to constrain loss fluctuations over 5 consecutive frames, when... At that time, the weight doubled from 0.01 to 0.02 to suppress sudden false positives.

[0085] 4.2 Training Strategy

[0086] Equipment: NVIDIA RTX 3090 GPU, batch size=32;

[0087] Optimizer: AdamW (initial learning rate 3e-4, weight decay 0.05);

[0088] Progressive training: The backbone network is frozen for the first 50 rounds, and only the classification head is trained, with a LabelSmoothing factor of 0.1; the entire network is unfrozen for the next 50 rounds, and dynamic batch processing is performed (difficult samples account for 40%); learning rate decay: decays by 10 times every 20 rounds.

[0089] 5. Real-time detection and alarm

[0090] Inference acceleration: The model is exported to ONNX format and accelerated via ONNXRuntime, with a single-frame inference time of <30ms; Dynamic lighting adaptation: Integrated ring LED light source and ambient light sensor, with a range of 0-5000 lux, adjusts the light source brightness in real time via PID algorithm (500-3000 lux) and camera exposure time (1-10ms); Alarm logic: When the defect probability... When the time is right, an audible and visual alarm is triggered with a response time of less than 50ms, and the defect image, point cloud, and vibration signal are uploaded to the cloud for verification.

[0091] 6. Verification of technical effectiveness

[0092] This method achieves a detection rate of 99.2% for microcracks of 0.05-0.1mm on the sealing surface of the flared mouth, with a false detection rate of <0.5% and a frame rate of 30fps, meeting the real-time detection requirements of the die-casting production line.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-modal fusion and dynamic learning based real-time defect recognition method for a trumpet mouth, characterized in that, Comprising the following steps: S1) Multi-modal data acquisition and preprocessing: The data acquisition method is to synchronously collect the surface images of the left and right sides of the horn mouth by using a binocular industrial camera, to obtain three-dimensional contour data by integrating a laser point cloud sensor, and to collect vibration signals by a piezoelectric sensor; the three heterogeneous modal data collected above are subjected to multi-modal collaborative preprocessing, in which a multi-modal data enhancement method and a multi-modal feature extraction method oriented to defect recognition are adopted, the multi-modal data enhancement method includes performing ±15° random rotation on the image to simulate attitude deviation, performing contrast adaptive adjustment in the color space, and simulating illumination change in the HSV domain disturbance, and the multi-modal feature extraction method includes extracting surface curvature features from the laser point cloud and extracting frequency domain features from the vibration signal by wavelet transform; S2) Dynamic adaptive model construction: a cross-modal Transformer fusion network is adopted, which contains feature extraction branches for different modalities and a Transformer fusion module, the feature extraction branches include an image branch, a point cloud branch and a vibration signal branch, wherein the image branch is an improved ConvNeXt-Tiny architecture containing 1 / 3 / 5 / 7 convolution kernels, the point cloud branch is PointNet++, and the vibration signal branch is CNN-LSTM, the features extracted by each branch are subjected to cross-modal correlation learning through the Transformer fusion module; S3) Model training: using a compound loss function ; wherein Spatial attention weight containing a focusing sealing surface area , for making the model actively focus on the key spatial area of the horn mouth during training, and reduce the weight of the non-key area; It is an adaptive smoothing binary cross entropy loss function for dynamically adjusting the smoothing coefficient according to the sample type; For constraining the loss mutation of continuous frames; S4) Real-time detection and alarm: through a double-buffer pipeline architecture and ONNX Runtime acceleration inference, when the defect probability is greater than or equal to 0.7, an audible and visual alarm with a response delay of less than 50ms is triggered, and the cloud is uploaded for review.

2. The method of claim 1, wherein, In step 1, the image is dynamically cropped to 224x224 samples after 256x256 standardization, the laser point cloud is downsampled to 1024 points after denoising and registration, and the vibration signal is extracted after wavelet transform. Peak, energy spectrum features.

3. The method of claim 1, wherein, The improved ConvNeXt-Tiny architecture loads ImageNet pre-training weights, and the feature extraction block adopts an inverted bottleneck structure, Layer Normalization and GELU activation function.

4. The method of claim 1, wherein, In the composite loss function, the dynamic Focal Loss expression of the spatial attention focus is: ; wherein, is a dynamic weight coefficient; is an adaptive focus parameter; is a spatial attention weight, The value of the sealing surface area of the horn mouth is greater than or equal to 0.8, and the value of the non-critical area is less than or equal to 0.

3.

5. The method of claim 4, wherein, The dynamic weight coefficient wherein represents the prediction probability of the model for the i-th sample, the adaptive focus parameter wherein is a Sigmoid function, .

6. The method of claim 1, wherein, In the composite loss function, the adaptive smooth cross-entropy expression is: wherein the adaptive smooth coefficient .

7. The method of claim 1, wherein, In the composite loss function, the expression of the time sequence consistency constraint term is: wherein, is the loss value of the tth frame, when the loss difference of consecutive frames exceeds 0.2, the weight is automatically doubled.

8. The method of claim 1, wherein, In the real-time detection, an annular LED light source and an ambient light sensor are integrated, and the light source brightness is dynamically adjusted by a PID algorithm, with an adjustment range of 500-3000lux and a camera exposure time of 1-10ms.

9. The method of claim 1, wherein, The model training adopts a progressive strategy: the backbone network is frozen for the first 50 rounds, Label Smoothing is used with a factor of 0.1; the network is unfrozen for the last 50 rounds, AdamW optimizer is used with an initial learning rate of 0.0003, weight decay of 0.05, and difficult example sample ratio ≥40% in dynamic batch processing.

Images