Textile fabric defect real-time detection method and system based on multi-modal feature fusion

Abstract

This invention provides a real-time detection method and system for textile fabric defects based on multimodal feature fusion. The method includes: acquiring visual image data, infrared thermal imaging data, and ultrasonic acoustic data of the textile fabric through a multi-sensor array to form a multimodal input; performing temporal alignment and noise filtering preprocessing on the multimodal data to eliminate motion artifacts and environmental interference; and using a parallel feature extraction module to extract texture features from the visual data, temperature distribution features from the thermal imaging data, and acoustic impedance features from the acoustic data. This invention enhances detection capabilities by deeply fusing complementary information from visual, thermal imaging, and ultrasonic modalities. The visual modality captures surface texture details, the thermal imaging modality reveals thermodynamic anomalies related to friction and material properties, and the ultrasonic modality senses subcutaneous structural defects. This allows for the detection of hidden defects that cannot be identified by a single modality, thereby significantly reducing the false negative rate and enabling more accurate differentiation of defect types.

Description

Technical Field

[0001] This invention relates to the field of textile production, and in particular to a method and system for real-time detection of textile defects based on multimodal feature fusion. Background Technology

[0002] In textile production, fabric defect detection is a core step in ensuring product quality. Traditional detection mainly relies on manual visual inspection, which is inefficient, labor-intensive, and susceptible to subjective factors, resulting in poor consistency. In recent years, automatic detection technology based on machine vision has been widely used, but most of them are limited to surface image information collected by visible light cameras. These single-modal methods are insufficient for detecting defects with low contrast similar to the background texture (such as slight oil stains or certain weaving defects), especially for internal defects below the fabric surface (such as interlayer foreign objects), which are completely ineffective. In addition, motion blur caused by high-speed production lines, changes in ambient lighting, and differences between different fabric materials can easily lead to high false alarm and missed detection rates in systems based on a single vision model. Although existing technologies attempt to combine multiple light sources or imaging of different bands, the fusion at the feature level is not deep enough, and the high complementarity between different physical modalities is not fully utilized. Adaptability and real-time performance are often difficult to balance, making it difficult to meet the quality monitoring requirements of modern intelligent textile factories for full-process, high-precision, and zero-latency monitoring.

[0003] Therefore, in order to address the above problems, we are now developing a real-time detection method and system for textile fabric defects based on multimodal feature fusion. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies that attempt to combine multiple light sources or different band imaging, but lack in-depth feature fusion and fail to fully utilize the high complementarity between different physical modes, adaptive capability and real-time performance are often difficult to achieve, making it difficult to meet the quality monitoring needs of modern intelligent textile factories for full-process, high-precision, and zero-latency quality control, this invention provides a real-time detection method and system for textile fabric defects based on multimodal feature fusion.

[0005] The technical solution of this invention is: a real-time detection method for textile fabric defects based on multimodal feature fusion, comprising: Step 1: Simultaneously acquire visual image data, infrared thermal imaging data, and ultrasonic acoustic data of textile fabrics through a multi-sensor array to form a multimodal input; Step 2: Perform time-series alignment and noise filtering preprocessing on the multimodal data to eliminate motion artifacts and environmental interference; Step 3: Use a parallel feature extraction module to extract texture features from visual data, temperature distribution features from thermal imaging data, and acoustic impedance features from acoustic data. The feature extraction uses a combination of lightweight convolutional neural networks and attention mechanisms to improve efficiency. Step 4: Input the extracted multimodal features into the dynamic weighted fusion network, and adaptively adjust the fusion weights according to the feature confidence to generate a robust fusion feature vector; Step 5: Based on the fused feature vector, a real-time classifier is used for defect detection. The classifier integrates a support vector machine and a lightweight recurrent neural network to achieve high accuracy and low latency. Step 6: Dynamically optimize the detection threshold through feedback loops to adapt to changes in fabric material and fluctuations in production line speed.

[0006] As a preferred embodiment of the present invention, in the multimodal data acquisition step, visual image data is captured by a high-resolution linear CCD camera at a rate of 1000 frames per second, infrared thermal imaging data is acquired by a microbolometer sensor in the long-wave infrared band, and ultrasonic acoustic data is acquired by a piezoelectric transducer in the frequency range of 1-10MHz. Furthermore, multiple sensors are synchronized by a hardware synchronization circuit to ensure consistent data timestamps in order to support accurate feature fusion.

[0007] As a preferred embodiment of the present invention, in the feature extraction module, visual feature extraction uses an improved MobileNetV3 network to reduce computational overhead, thermal feature extraction uses Gram matrix analysis to analyze temperature gradients, and acoustic feature extraction is based on short-time Fourier transform to extract frequency domain features, and a channel attention mechanism is introduced to dynamically focus on defect-related feature regions to improve feature discrimination.

[0008] As a preferred embodiment of the present invention, the dynamic weighted fusion network adopts a multi-head attention mechanism to realize feature interaction, wherein the features of each modality are first enhanced by a self-attention layer, and then the correlation between modalities is calculated by a cross-attention layer. The weights are dynamically adjusted according to the feature entropy value, and the feature dimension is compressed after fusion to reduce the computational burden.

[0009] As a preferred embodiment of the present invention, the real-time classifier includes a two-stage detection process: the first stage uses a lightweight model to quickly locate suspicious defect areas, and the second stage uses a bidirectional long short-term memory network to analyze temporal features to distinguish defect types, such as broken yarn, stains or wrinkles, and outputs confidence scores and bounding box coordinates.

[0010] As a preferred embodiment of the present invention, the feedback loop is based on an online learning mechanism, which updates the classifier parameters in real time by collecting false detection and false negative samples, and uses a Kalman filter to predict the movement state of the fabric, thereby adaptively adjusting the detection sensitivity.

[0011] As a preferred embodiment of the present invention, the system further includes the method, the system comprising a multi-sensor integration module, an edge processing unit, and a user interaction module. The multi-sensor integration module includes visual, infrared, and ultrasonic sensor arrays, and data acquisition synchronization is achieved through an FPGA. The edge processing unit adopts a heterogeneous computing architecture, integrating GPU and NPU to accelerate feature extraction and fusion algorithms. The user interaction module provides a web interface to display detection results and alarm information in real time, and supports remote monitoring and data export.

[0012] As a preferred embodiment of the present invention, the edge processing unit also integrates a temperature compensation circuit and a vibration isolation device to reduce the impact of environmental fluctuations on data quality. The system achieves data synchronization with the cloud platform through a 5G module for historical data analysis and model iteration optimization, while employing differential privacy technology to protect the security of production data.

[0013] By adopting the above technical solution, the present invention has the following advantages: 1. This invention significantly improves detection capabilities by deeply integrating complementary information from three modalities: vision, thermal imaging, and ultrasound. The visual modality captures surface texture details, the thermal imaging modality reveals thermodynamic anomalies related to friction and materials, and the ultrasound modality senses subcutaneous structural defects. It can detect hidden flaws that cannot be identified by a single modality, such as slight oil stains or internal fiber clumps on light-colored fabric, thereby greatly reducing the false negative rate and making more accurate distinctions between defect types.

[0014] 2. This invention greatly enhances the robustness and adaptability of the system through a dynamic weighted fusion mechanism and online feedback loop. The system can evaluate the confidence level of each modality data in real time, and automatically adjust the dependency weights when a certain modality is disturbed, ensuring uninterrupted detection process and reliable results. At the same time, it optimizes model parameters by continuously learning from difficult samples on the production line, adapting to different fabric types, environmental changes and production line speed fluctuations, achieving long-term stable operation and reducing manual intervention for parameter tuning.

[0015] 3. This invention successfully achieves the unification of high-precision detection and real-time requirements through an optimized lightweight network architecture, parallel processing pipeline, and edge computing deployment. The feature extraction and classification algorithms are specifically optimized for computational efficiency. Combined with hardware acceleration technology, the entire system can complete the entire process from data acquisition to result output within an extremely short latency, meeting the stringent requirements of high-speed production lines for real-time response and providing immediate decision-making basis for online automatic grading and quality control. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the detection method of the present invention.

[0017] Figure 2 This is a schematic diagram of the system structure of the present invention.

[0018] Figure 3 This is a schematic diagram of the edge processing unit of the present invention. Detailed Implementation

[0019] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] This invention aims to solve the problems of high false negative and false positive rates caused by relying on a single visual modality in the existing textile defect detection, as well as the difficulty in achieving high-precision real-time detection on high-speed production lines. The core lies in the deep integration of complementary information from three modalities: visual, thermodynamic, and acoustic, and the achievement of adaptive and robust real-time detection through innovative algorithm design.

[0021] Real-time detection method for textile fabric defects based on multimodal feature fusion, such as Figure 1 As shown, it includes: Step 1: Simultaneously acquire visual image data, infrared thermal imaging data, and ultrasonic acoustic data of textile fabrics through a multi-sensor array to form a multimodal input; Step 2: Perform time-series alignment and noise filtering preprocessing on the multimodal data to eliminate motion artifacts and environmental interference. Specifically, in hardware, the multi-sensor array integrates hardware synchronization circuitry through an FPGA, uses a high-precision crystal oscillator to generate a synchronization clock signal, and distributes it to the trigger ports of all sensors to achieve synchronous acquisition with microsecond-level precision. For example, when the visual camera captures images at 1000 frames per second, the thermal imager and ultrasonic sensor also work at the same pace. In software, an interpolation resampling algorithm is implemented on the FPGA, performing linear interpolation or spline interpolation on the three data streams based on timestamps. For example, when the fabric moves at a speed of 100 meters per minute, the system calculates the acquisition delay of each sensor and dynamically adjusts the data buffer to ensure that the row data of the visual image, the frame data of the thermal image, and the A-scan signal of the ultrasonic wave are completely corresponding in time, avoiding motion artifacts. Noise filtering preprocessing is tailored to the characteristics of each modality: visual data uses adaptive median filtering, which analyzes the neighborhood of each pixel, dynamically adjusts the size of the filtering window, and removes salt-and-pepper noise and illumination fluctuations; thermal imaging data uses time-domain Gaussian filtering, which applies Gaussian kernel smoothing to consecutive frames to suppress random thermal noise; and ultrasonic data uses wavelet thresholding denoising, which performs discrete wavelet transform on the A-scan signal and suppresses scattering noise through soft thresholding. Step 3: Use a parallel feature extraction module to extract texture features from visual data, temperature distribution features from thermal imaging data, and acoustic impedance features from acoustic data. Feature extraction employs a combination of lightweight convolutional neural networks and an attention mechanism to improve efficiency. Specifically, network pruning reduces computational overhead, redundant layers are removed, and a convolutional block attention module is added after the last convolutional layer. This module includes channel attention and spatial attention: channel attention dynamically weights important feature channels through global average pooling and fully connected layers, focusing on texture-rich frequency bands; spatial attention generates weight maps through convolutional layers to highlight suspicious defects. For regions (such as broken yarns or stains), pre-trained weights are fine-tuned on the fabric dataset during training. The loss function combines cross-entropy and focus loss to handle class imbalance. The final extracted texture features include local binary patterns and gray-level co-occurrence matrix features, representing abnormal warp and weft yarn arrangement on the fabric surface. Thermal feature extraction uses Gram matrix analysis to analyze temperature gradients. The specific calculation algorithm is as follows: for each pixel in the thermal imaging data, calculate its temperature gradient magnitude and direction with its neighborhood, construct a gradient local binary pattern matrix, thereby enhancing the sensitivity to weak thermal anomalies (such as temperature differences caused by oil stains); at the same time, a channel attention mechanism is introduced to dynamically weight abnormal regions in the temperature gradient. Acoustic feature extraction is based on short-time Fourier transform. The ultrasonic A-scan signal is windowed and subjected to STFT to obtain the time spectrum. Then, the Mel frequency cepstral coefficients are extracted as acoustic impedance features to represent subcutaneous structural defects. The attention mechanism focuses on defect-related frequency bands, such as high-frequency bands corresponding to small foreign objects. Step 4: Input the extracted multimodal features into a dynamic weighted fusion network. Adaptively adjust the fusion weights based on feature confidence to generate robust fusion feature vectors. Specifically, firstly, calculate the confidence score for each modality's feature vector in real time. Visual confidence is based on image sharpness and contrast, thermal confidence is based on scene temperature stability, and acoustic confidence is based on the signal-to-noise ratio of the echo signal. Then, use a Softmax layer to normalize the confidence scores and generate fusion weights. For example, when a sudden change in ambient lighting causes a decrease in visual image quality, its weight is automatically reduced, while the weights of thermal and acoustic modalities are increased, thereby maintaining system robustness. Step 5: Based on the fused feature vector, a real-time classifier is used for defect detection. The classifier integrates a support vector machine and a lightweight recurrent neural network to achieve high accuracy and low latency. Specifically, the classifier integrates a support vector machine and a lightweight recurrent neural network to implement a two-stage detection process. In the first stage, a lightweight model is used to quickly locate suspicious defect areas and generate candidate bounding boxes with a high recall rate. In the second stage, the candidate areas are mapped onto thermal and acoustic data, multimodal feature fragments are extracted, and input into Bi-LSTM to analyze temporal features, such as analyzing the evolution pattern of defects in the direction of fabric movement (e.g., the performance of broken yarn in consecutive frames) to distinguish defect types (e.g., broken yarn, stains, or wrinkles), and output confidence scores and bounding box coordinates. Step 6: Dynamically optimize the detection threshold through feedback loops to adapt to changes in fabric material and fluctuations in production line speed. Specifically, the system continuously monitors the detection results, collects false positives and false negatives as hard samples and stores them in the cache. When the number of samples reaches a certain amount, incremental learning is initiated during idle periods (such as when changing rolls) to fine-tune the classifier parameters using these samples. At the same time, a Kalman filter is used to predict the fabric movement state and adaptively adjust the detection sensitivity, for example, dynamically optimizing the threshold based on production line speed fluctuations to reduce false alarms.

[0022] Visual feature extraction utilizes a lightweight convolutional neural network, reducing computational overhead through depthwise separable convolutions and combining an attention mechanism to focus on defective regions. Key formulas include: Depthwise separable convolution formula: Let the input image be I, the depthwise convolution kernel be Wd, the pointwise convolution kernel be Wp, and the output feature map be Fvis. In this context, DepthwiseConv performs independent convolutions on each input channel, while PointwiseConv fuses the channels through 1×1 convolutions. Attention weight formula: Channel attention weights Mc are generated through global average pooling and fully connected layers. Where σ is the Sigmoid function, MLP is a multilayer perceptron that dynamically enhances texture-related feature channels, and the final output is... , where ⊗ represents channel-wise multiplication, thereby enhancing texture-related areas (such as blemishes); Temperature gradient calculation formula: for thermal imaging data (Each pixel represents a temperature value) Calculate the gradient magnitude to highlight areas of temperature variation. Approximate the partial derivatives using the Sobel operator. ,in and It uses a Sobel kernel (e.g., 3×3), and the gradient magnitude G is: ; Gram matrix construction formula: for gradient magnitude plot G local area R(e.g., a 3×3 window), construct a Gram matrix to represent texture features. Let region R contain N pixels, and the feature vector Φ(G) be processed using the ReLU activation function. The Gram matrix is ​​then calculated as follows: ,in The eigenvalues ​​or traces of this matrix can be used as eigenvectors (Fthermal) to distinguish thermal anomalies. Short-time Fourier transform formula: for ultrasonic A-scan signals x ( t (Time series), after windowing and segmentation, Fourier transform is applied, and the discrete formula is: Where w is the window function, H is the frame shift, N is the number of FFT points, m is the frame index, and k is the frequency index. The output spectrum X(m,k) characterizes the change in acoustic impedance. Mel frequency cepstral coefficient extraction formula: Pass the STFT amplitude |X(m,k)| through the Mel scale filter bank {Ml(k)} (l=1,2,…,L) to obtain the Mel spectrum E(m,l): After taking the logarithm, the discrete cosine transform (DCT) is applied to obtain the MFCC coefficients. : ,in c It is the cepstral coefficient index, and usually the first 12-13 coefficients are taken as the feature vector; like Figure 2 As shown, it also includes a real-time textile defect detection system based on multimodal feature fusion, which includes a multi-sensor integration module, an edge processing unit, and a user interaction module. The multi-sensor integration module includes visual, infrared, and ultrasonic sensor arrays, and uses FPGA to achieve synchronous data acquisition. The edge processing unit adopts a heterogeneous computing architecture, integrating GPU and NPU to accelerate feature extraction and fusion algorithms. The user interaction module provides a web interface to display detection results and alarm information in real time, and supports remote monitoring and data export.

[0023] like Figure 3 As shown, the edge processing unit also integrates a temperature compensation circuit and a vibration isolation device to reduce the impact of environmental fluctuations on data quality. The system achieves data synchronization with the cloud platform through a 5G module for historical data analysis and model iteration optimization, while using differential privacy technology to protect the security of production data.

[0024] Specifically, visual image data is captured using a linear CCD camera with a resolution of at least 2048 pixels, mounted along the width of the fabric. The line frequency is strictly matched to the production line speed (e.g., at a speed of 100 meters per minute, the line frequency must be greater than 83 kHz to meet the resolution requirements). The infrared thermal imaging sensor is an uncooled long-wave infrared (8-14 μm) focal plane array with a resolution of at least 640x512 pixels. Its frame rate is synchronized with the CCD camera and it is used to detect minute temperature differences caused by friction, uneven thickness, or material variations. The ultrasonic sensor uses a focused piezoelectric transducer with a center frequency of 5 MHz, operates in pulse-echo mode, and is mounted perpendicular to the fabric surface. It is used to detect subsurface structural defects (such as interlayer foreign objects or fiber clumps). The system uses a high-precision crystal oscillator to generate a synchronous clock signal, which is simultaneously distributed to the trigger ports of all sensors via the FPGA. This ensures that each line of visual data, each frame of thermal image, and each ultrasonic A-scan signal has a timestamp accurate to the microsecond level, laying the hardware foundation for subsequent timing alignment. The timing alignment algorithm is implemented on the FPGA, interpolating and resampling the three data streams based on the timestamps to ensure that the physical position of the same piece of fabric corresponds completely in the data of the three modalities. Noise filtering is tailored to the characteristics of each modality: adaptive median filtering is used to remove salt-and-pepper noise from visual images, time-domain Gaussian filtering is used to smooth random thermal noise from thermal imaging data, wavelet threshold denoising is used to suppress scattering noise from ultrasonic signals, and visual branches are pruned. The subsequent MobileNetV3 removes some redundant layers and adds a convolutional block attention module after the last convolutional layer, allowing the network to focus on texture anomaly regions. Thermal feature extraction does not directly use the original temperature value; instead, it calculates the gradient magnitude and direction of each pixel and its neighborhood, constructing a variant of the local binary mode, namely the gradient local binary mode, to enhance sensitivity to weak thermal anomalies. Acoustic feature extraction first performs STFT on the A-scan signal to obtain the time spectrum, then extracts the Mel-frequency cepstral coefficients as feature vectors characterizing the acoustic impedance of the material. This combination maximizes the discriminative power of the features while ensuring real-time performance. The dynamic weighted fusion network calculates a real-time confidence score for the feature vector of each modality, and visual positioning... The confidence level is based on image sharpness and contrast, the thermal confidence level is based on the temperature stability of the scene, and the acoustic confidence level is based on the signal-to-noise ratio of the echo signal. The fusion weights are proportional to the confidence scores and are normalized through a Softmax layer. For example, when drastic changes in ambient lighting cause a decrease in visual image quality, their weights are automatically reduced, while the weights of the thermal and acoustic modes are correspondingly increased, thus maintaining the robustness of the overall system. The fused feature vectors are then subjected to dimensionality reduction and nonlinear transformation through a fully connected layer to further extract the information most relevant to defects. The two-stage design of the classifier is key to balancing speed and accuracy. In the first stage, the improved lightweight YOLOv5 model (using the Ghost module instead of standard convolution) aims for high recall.The first stage rapidly generates multiple candidate regions that may contain defects across the entire image, allowing for a certain number of false positives. In the second stage, the coordinates of the candidate regions output from the first stage are mapped onto thermal and acoustic data. Multimodal feature fragments of the corresponding regions are extracted and input into a small Bi-LSTM network. The Bi-LSTM network analyzes the temporal evolution pattern of defects along the fabric's movement direction (e.g., the appearance of a broken yarn in consecutive frames), effectively distinguishing between real defects and transient noise, and accurately classifying defect types. A feedback loop continuously monitors the classification results: for multiple consecutive frames classified as defects of the same type with a high confidence level... Areas exceeding the threshold are considered correct detections by the system. For suspected signals with critical confidence levels or those that frequently appear and disappear, the system marks them as difficult samples and stores them, along with their original multimodal data, in a buffer. Once the buffer reaches a certain number of samples, the system initiates an incremental learning process during idle periods (such as when changing rolls), using these samples to fine-tune the classifier, thus achieving adaptive optimization. The Kalman filter is used to predict the real-time position of the fabric, dynamically adjusting the analysis area for the next frame to compensate for image jitter caused by minor fluctuations in production line speed. The edge processing unit utilizes NVIDIA... The Jetson AGX Orin or similar high-performance embedded platform utilizes a built-in GPU and deep learning accelerator for CNN visual feature extraction and LSTM timing analysis, respectively. An FPGA handles the underlying signal preprocessing and data scheduling. The user interface is based on a B / S architecture, using the WebSocket protocol for real-time data push. The interface simultaneously displays the raw data from all three modalities, the fused defect heatmap, classification result charts, and alarm logs. A temperature compensation circuit monitors the ambient temperature in real time using a temperature sensor integrated near the sensor and calibrates the thermal imaging sensor readings using a pre-calibrated lookup table. The vibration isolation device employs an air-floating platform or high-performance damping materials to effectively isolate the interference of mechanical vibrations in the industrial environment on high-precision imaging and ultrasonic detection.

[0025] It should be noted that in continuous textile production processes, such as at the end of the weaving or finishing stages, the fabric to be inspected passes through the inspection area at a constant high speed (e.g., 80-150 meters per minute) under the traction of guide rollers. This system is mounted on a sturdy support on the production line, ensuring that the vision camera, infrared thermal imager, and ultrasonic probe maintain a constant and precisely set distance and angle from the surface of the fabric being inspected. When the fabric passes the inspection point, a synchronous trigger signal causes the three sensors to start working simultaneously. The high-resolution linear CCD camera continuously scans the fabric surface at an extremely high line frequency, generating a one-dimensional image. These image lines are stitched together in real time into a continuous two-dimensional grayscale or color image, clearly representing the warp and weft yarn arrangement, color, and surface texture of the fabric. Simultaneously, an infrared thermal imager captures the temperature distribution map of the fabric surface at the same pace. Because the fabric generates a weak temperature field during production due to friction, residual stress, or environmental factors, and because the heat capacity or thermal conductivity of defective areas (such as oil stains, wet spots, or thicker nodules) differs from normal areas, subtle differences in surface temperature occur. These differences are captured by a highly sensitive infrared sensor. Meanwhile, an ultrasonic probe continuously probes... The fabric emits high-frequency sound pulses and receives echoes. When the sound waves propagate within the fabric, they encounter uniform areas, producing echo signals with specific amplitudes and waveforms. However, when encountering pores, foreign objects, or areas with density variations, the reflection and transmission characteristics of the sound waves change, leading to variations in the amplitude, phase, and spectral characteristics of the echo signals. Three raw data streams are transmitted in real-time to the edge computing unit via a high-speed data interface. During the data preprocessing stage, the system first performs pixel-level alignment of the three data streams based on precise timestamps to ensure that for any physical point on the fabric, the data is accurately timed. Visual appearance, temperature readings, and acoustic echo information are strictly correlated in time. The modal data are then purified using the aforementioned dedicated filtering algorithm. The purified data is fed into a parallel feature extraction pipeline. The visual pipeline quickly analyzes whether the texture's uniformity and periodicity have been disrupted, detecting obvious surface defects such as broken warp or weft threads, and holes. The thermal imaging pipeline sensitively captures areas that are not visually obvious but have abnormal thermal characteristics, such as slight dampness or chemical residues. The acoustic pipeline can detect subcutaneous defects such as foreign objects trapped between fabric layers and fiber clumps. These three features are fed into a dynamic weighted fusion network in real time.

[0026] In the fusion network, the system evaluates the health status of each modality in real time. For example, if the workshop overhead light suddenly flickers, causing a brief overexposure of the visual image, the system immediately reduces the weight of the visual features and relies more on thermal and acoustic features that are unaffected by light for judgment, effectively preventing false alarms. The fused high-level feature vector is fed into the real-time classifier. The first stage of the classifier, the YOLOv5 network, quickly selects all suspicious areas. Then, the second stage, the Bi-LSTM network, analyzes the evolution of these suspicious areas over multiple consecutive frames. A real defect will appear continuously as the fabric moves, while random noise may only appear for one frame and then disappear. Through this temporal context analysis, the system can classify defects extremely accurately as broken yarns, star jumps, thick knots, etc. The system identifies specific types of stains and provides their precise pixel coordinates and confidence levels. Once a defect is confirmed, the system immediately sends a control signal through the interface to trigger a labeling machine or marker to mark the corresponding position on the edge of the fabric, or to issue an audible and visual alarm on the human-machine interface to prompt the operator to handle the issue. Throughout the process, the feedback loop works continuously, and the system self-optimizes when idle, enabling it to gradually adapt to long-term changes such as changes in fabric raw materials and environmental temperature differences caused by seasonal changes, achieving true intelligence. All detection data, alarm records, and defect images are stored and can be uploaded to the cloud-based factory management system via a 5G network for generating quality reports, tracing production batches, and optimizing process parameters, ultimately forming a closed-loop intelligent detection system from perception, analysis, decision-making to optimization.

[0027] The above embodiments are provided for those skilled in the art to implement or use the present invention. Those skilled in the art can make various modifications or changes to the above embodiments without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention is not limited to the above embodiments, but should be the maximum scope that conforms to the innovative features mentioned in the claims.

Claims

1. A real-time detection method for textile fabric defects based on multi-modal feature fusion, characterized in that, Including: Step 1: Simultaneously acquire visual image data, infrared thermal imaging data, and ultrasonic acoustic data of textile fabrics through a multi-sensor array to form multimodal data; wherein, the multi-sensor array ensures that the data timestamps are consistent through a hardware synchronization circuit; Step 2: Perform time-series alignment and noise filtering preprocessing on the multimodal data to eliminate motion artifacts and environmental interference; the time-series alignment preprocessing includes: based on the timestamp, interpolating and resampling the three data streams of visual image data, infrared thermal imaging data, and ultrasonic acoustic data to ensure that the physical position of the same piece of fabric corresponds completely in the three modal data. Step 3: Use a parallel feature extraction module to extract texture features from visual data, temperature distribution features from thermal imaging data, and acoustic impedance features from acoustic data. The feature extraction uses a combination of lightweight convolutional neural networks and attention mechanisms to improve efficiency. Extracting temperature distribution features from thermal imaging data further includes: using Gram matrix analysis to analyze temperature gradients, calculating the magnitude and direction of the temperature gradient between each pixel and its neighborhood, and constructing a gradient local binary pattern matrix to enhance the sensitivity to thermal anomalies. Extracting acoustic impedance features from acoustic data further includes: performing a short-time Fourier transform on the ultrasonic A-scan signal to obtain the time spectrum, and then extracting the Mel frequency cepstral coefficients as acoustic impedance features to characterize subcutaneous structural defects; Step 4: Input the extracted multimodal features into a dynamic weighted fusion network, adaptively adjust the fusion weights based on feature confidence, and generate a robust fusion feature vector; wherein, the dynamic weighted fusion network calculates a confidence score for each modality's feature vector: visual confidence is based on image sharpness and contrast, thermal confidence is based on scene temperature stability, and acoustic confidence is based on the signal-to-noise ratio of the echo signal; the fusion weights are proportional to the confidence scores and are normalized through a Softmax layer; Step 5: Based on the fused feature vector, a real-time classifier is used for defect detection. The classifier integrates a support vector machine and a lightweight recurrent neural network to achieve high accuracy and low latency. Step 6: Dynamically optimize the detection threshold through feedback loops to adapt to changes in fabric material and fluctuations in production line speed.

2. The real-time textile fabric defect detection method based on multi-modal feature fusion according to claim 1, characterized in that, In the multimodal data acquisition step, visual image data is captured by a high-resolution linear CCD camera at a rate of 1000 frames per second, infrared thermal imaging data is acquired by a microbolometer sensor in the long-wave infrared band, and ultrasonic acoustic data is acquired by a piezoelectric transducer in the frequency range of 1-10MHz to support accurate feature fusion.

3. The real-time textile fabric defect detection method based on multi-modal feature fusion according to claim 1, characterized in that, In the feature extraction module, visual feature extraction uses an improved MobileNetV3 network to reduce computational overhead and introduces a channel attention mechanism to dynamically focus on defect-related feature regions. Improve feature discrimination ability.

4. The real-time textile fabric defect detection method based on multi-modal feature fusion according to claim 1, characterized in that, The dynamic weighted fusion network uses a multi-head attention mechanism to achieve feature interaction. The features of each modality are first enhanced by a self-attention layer, and then the correlation between modalities is calculated by a cross-attention layer. The weights are dynamically adjusted according to the feature entropy value. After fusion, the feature dimension is compressed to reduce the computational burden.

5. The real-time textile fabric defect detection method based on multi-modal feature fusion according to claim 1, characterized in that, The real-time classifier includes a two-stage detection process: the first stage uses a lightweight model to quickly locate suspicious defect areas, and the second stage uses a bidirectional long short-term memory network to analyze temporal features to distinguish defect types and output confidence scores and bounding box coordinates.

6. The real-time textile fabric defect detection method based on multi-modal feature fusion according to claim 1, characterized in that, The feedback loop is based on an online learning mechanism. It updates the classifier parameters in real time by collecting false and missed detection samples, and uses a Kalman filter to predict the movement state of the fabric, thereby adaptively adjusting the detection sensitivity.

7. A real-time detection system for textile fabric defects based on multi-modal feature fusion, applied to the real-time detection method of any one of claims 1-6, characterized in that, The system includes a multi-sensor integration module, an edge processing unit, and a user interaction module. The multi-sensor integration module includes visual, infrared, and ultrasonic sensor arrays, and uses an FPGA to achieve synchronous data acquisition. The edge processing unit adopts a heterogeneous computing architecture, integrating GPU and NPU to accelerate feature extraction and fusion algorithms. The user interaction module provides a web interface to display detection results and alarm information in real time, and supports remote monitoring and data export.

8. The real-time textile fabric defect detection system based on multi-modal feature fusion according to claim 7, characterized in that, The edge processing unit also integrates a temperature compensation circuit and a vibration isolation device to reduce the impact of environmental fluctuations on data quality. The system achieves data synchronization with the cloud platform through a 5G module for historical data analysis and model iteration optimization, while employing differential privacy technology to protect the security of production data.

Images