Intelligent sorting method and system based on multi-modal defect feature fusion

Abstract

The invention relates to the technical field of intelligent sorting, and discloses a multi-mode defect feature fusion intelligent sorting method and system, and the method comprises the steps: obtaining and preprocessing a surface image, infrared thermal imaging and voiceprint vibration data of an object; analyzing and generating multi-modal defect feature parameters, fusing to form a fused feature vector set, and constructing a defect detection reference set; performing dynamic matching verification on the multi-modal data based on the reference set, and analyzing the mismatching state of the surface texture and the thermal distribution by using a space alignment technology; and generating a defect form deviation degree index according to a verification result, and judging whether a sorting action is triggered or not. The system comprises a multi-modal acquisition module, a feature fusion modeling module, a form association verification module and a sorting judgment module which are used for respectively realizing data acquisition preprocessing, feature fusion modeling, cross-modal association analysis and sorting decision. Through multi-modal data fusion and cross-modal quantitative analysis, the comprehensiveness of defect detection and the sorting accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent sorting technology, and in particular to an intelligent sorting method and system for multi-modal defect feature fusion. Background Art

[0002] In the fields of industrial production and logistics sorting, the accuracy and efficiency of object defect detection and sorting have always been core industry concerns. Traditional sorting technologies mostly rely on single-modality detection methods, such as analyzing surface defects solely through visual images, detecting temperature anomalies using infrared thermal imaging, or relying on voiceprint vibration data to determine internal structural problems. However, single-modality detection has significant limitations: visual images are easily interfered with by environmental factors such as light and stains, making it difficult to accurately identify internal defects hidden beneath the surface; while infrared thermal imaging can detect temperature distribution, it may miss non-heat-sensitive defects (such as fine cracks and material unevenness); voiceprint vibration analysis is more sensitive to environmental noise, and a single vibration feature cannot fully reflect complex defect morphology. In addition, traditional methods lack the ability to fuse and analyze multi-source data and are unable to integrate complementary information from different modalities, resulting in low defect recognition accuracy and high false detection rates. The limitations of single-modality detection are particularly prominent when dealing with complex objects with multi-dimensional defect characteristics.

[0003] With the development of industrial intelligence, higher requirements are placed on the comprehensiveness, accuracy and real-time performance of object defect detection. Although multimodal data fusion technology has been applied in some fields, existing technologies generally have problems such as insufficient data preprocessing accuracy, single feature fusion methods, and insufficient in-depth spatial alignment and cross-modal correlation analysis when processing multimodal data. For example, in the data preprocessing stage, the noise filtering, spatial registration and normalization processing of different modal data lack specificity, which affects the accuracy of subsequent feature extraction; in the feature fusion process, most of them adopt a simple superposition method with fixed weights, and fail to dynamically adjust the importance of feature parameters according to different defect types; in the defect detection link, there is a lack of quantitative analysis of the spatial correspondence and mismatch state of cross-modal features such as surface texture anomalies and thermal distribution anomalies, making it difficult to accurately locate and evaluate complex defects.

[0004] Furthermore, existing intelligent sorting systems often rely on a single threshold for sorting decision-making, failing to fully consider the dynamic changes in defect characteristics in historical sorting data. This results in a lack of flexibility and adaptability in triggering sorting actions, making it difficult to meet the diverse sorting needs of different production scenarios. Therefore, there is an urgent need for an intelligent sorting method and system that can integrate multimodal defect characteristics, perform cross-modal correlation analysis, and optimize sorting decisions based on dynamic verification and historical data. This approach can improve the comprehensiveness of defect detection and sorting efficiency, and meet the new demands of industrial intelligence development. Summary of the Invention

[0005] The purpose of the present invention is to provide an intelligent sorting method and system for multi-modal defect feature fusion to solve the problems raised in the above background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an intelligent sorting method based on multimodal defect feature fusion, the method comprising: S1: Acquire surface image data, infrared thermal imaging data, and voiceprint vibration data of the object to be sorted, and perform data preprocessing on the surface image data, infrared thermal imaging data, and voiceprint vibration data; S2: Analyze the preprocessed data to generate multimodal defect feature parameters, perform feature fusion based on the multimodal defect feature parameters to form a fused feature vector set, and construct a defect detection benchmark set based on the fused feature vector set; S3: Dynamically match and verify the multimodal data of the object based on the defect detection benchmark set, use multimodal spatial alignment technology to identify the spatial correspondence between the surface image and infrared thermal imaging, and quantitatively analyze the mismatch between surface image texture anomalies and thermal distribution anomalies; S4: Generate a defect morphology deviation index based on the verification result of the defect detection benchmark set, and determine whether to trigger a sorting action based on the defect morphology deviation index.

[0007] Preferably, in S1, the system presets a scanning resolution and obtains regional surface image data, infrared thermal imaging data and voiceprint vibration spectrum data of the object within a preset scanning period; the surface image data includes a grayscale distribution matrix and edge gradient features; the infrared thermal imaging data includes a temperature gradient distribution and hot spot contour data; the voiceprint vibration data includes a spectrum energy distribution and a vibration time domain waveform; data preprocessing is performed on the surface image data, infrared thermal imaging data and voiceprint vibration data, and the data preprocessing includes noise filtering, spatial alignment, data interpolation and normalization; an image feature sequence is generated after the surface image data is preprocessed, and a thermal feature sequence is generated after the infrared thermal imaging data is preprocessed; the image feature sequence and the thermal feature sequence are spatially correlated and matched to form a multimodal data set.

[0008] Preferably, in S2, the following steps are included: S201: extracting morphological feature parameters from the preprocessed image feature sequence, thermal feature sequence, and voiceprint vibration data, respectively. The morphological feature parameters include texture complexity, hot spot area ratio, and spectrum main peak offset. Each morphological feature parameter is fused according to a preset weight to generate a multimodal defect feature parameter. S202: Setting a dynamic screening threshold for multimodal defect feature parameters in the multimodal data set, screening feature parameters that meet the threshold condition to form an initial fusion feature vector set, and discarding feature parameters that do not meet the threshold condition; S203: performing cross-modal comparative analysis on each feature parameter in the initial fused feature vector set, extracting the difference between feature parameters of different modalities in the same spatial region, calculating the relative deviation of the difference, and marking it as an inter-modal difference parameter; S204: Compare the inter-modal difference parameters with the preset difference threshold, filter the feature parameters that exceed the difference threshold and add them to the defect detection benchmark set, and add the spectrum energy mutation feature to the defect detection benchmark set based on the voiceprint vibration data.

[0009] Preferably, in S3, quantitatively analyzing the mismatch between the abnormal surface image texture and the abnormal thermal distribution includes the following steps: S301: Acquire local texture features of the surface image of the object and infrared thermal distribution features of the corresponding area in real time, calculate the ratio of the thermal distribution coverage area to the texture abnormal area, and determine a morphological mismatch event if the ratio exceeds a preset tolerance threshold; S302: Count the number of morphology mismatch events within a preset scanning period and record it as N, and synchronously obtain the rate of change of spectral energy in the voiceprint vibration data and record it as ΔE; S303: Calculate the comprehensive anomaly index based on the number of morphological mismatch events N and the spectrum energy change rate ΔE. The specific calculation method is: Comprehensive anomaly index ; If the comprehensive abnormality index exceeds the preset sorting threshold, the sorting mark will be triggered.

[0010] Preferably, in S4, the verification results of the defect detection benchmark set in the same type of historical sorting data are extracted, and the deviation of each benchmark is dynamically weighted and accumulated to generate a defect morphology deviation index. If the defect morphology deviation index exceeds the preset trigger number within a continuous scanning cycle, it is determined to perform a sorting action.

[0011] Preferably, the present invention also includes an intelligent sorting system for multimodal defect feature fusion, which is used to implement any one of the above-mentioned intelligent sorting methods for multimodal defect feature fusion, and includes a multimodal acquisition module, a feature fusion modeling module, a morphological association verification module, and a sorting judgment module; The multimodal acquisition module is used to acquire surface image data, infrared thermal imaging data, and voiceprint vibration data of an object and perform data preprocessing; The feature fusion modeling module is used to analyze the preprocessed data to generate multimodal defect feature parameters and construct a defect detection benchmark set; The morphological association verification module is used to perform dynamic matching verification on the multimodal data of the object according to the defect detection benchmark set, and analyze the mismatch between the surface texture and the thermal distribution; The sorting determination module is used to generate a defect morphology deviation index and trigger a sorting action.

[0012] Preferably, the multimodal acquisition module includes an image acquisition unit, a thermal imaging acquisition unit and a voiceprint acquisition unit; The image acquisition unit is used to collect regional surface image data and edge gradient features of the object; The thermal imaging acquisition unit is used to obtain the temperature gradient distribution and hot spot contour data of the object; The voiceprint collection unit is used to record the vibration spectrum energy distribution and time domain waveform data of the object.

[0013] Preferably, the feature fusion modeling module includes a parameter extraction unit and a defect construction unit; The parameter extraction unit is used to extract characteristic parameters of texture complexity, hot spot area ratio and spectrum main peak offset from the preprocessed data; The defect construction unit is used to screen characteristic parameters that meet the threshold conditions and add spectrum energy mutation characteristics to the defect detection benchmark set.

[0014] Preferably, the morphological association verification module includes a spatial matching unit and an anomaly calculation unit; The spatial matching unit is used to detect the spatial correspondence between the surface image and the thermal image and identify morphological mismatch events; The anomaly calculation unit is used to calculate a comprehensive anomaly index in combination with the spectrum energy change rate.

[0015] Preferably, the sorting determination module includes an indicator generation unit and an execution unit; The index generating unit is used to dynamically weight and accumulate the deviation of the defect detection benchmark; The execution unit is used to control the action of the sorting mechanism according to the continuous triggering results of the deviation index.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The intelligent sorting method and system for multimodal defect feature fusion provided by the present invention significantly improves the comprehensiveness and accuracy of object defect detection by integrating multimodal data such as surface images, infrared thermal imaging, and voiceprint vibration. Specifically, the multimodal acquisition module can synchronously acquire multi-dimensional data of an object, and ensure the consistency and reliability of data of different modalities through pre-processing operations such as noise filtering and spatial registration, laying a solid foundation for subsequent feature fusion. The feature fusion modeling module can not only extract basic feature parameters such as texture complexity and hot spot area ratio through dynamic weight allocation and cross-modal comparative analysis, but also effectively eliminate redundant information and retain key defect features by setting dynamic screening thresholds and inter-modal difference parameter verification, thereby constructing a more representative defect detection benchmark set, thereby improving the pertinence and effectiveness of feature fusion.

[0017] In the defect detection process, the morphological correlation verification module uses multimodal spatial alignment technology to accurately identify the spatial correspondence between surface images and infrared thermal images. By calculating the ratio of the thermal distribution coverage area to the texture anomaly area, counting the number of morphological mismatch events, and combining the spectral energy change rate of the voiceprint vibration data, it generates a comprehensive anomaly index, achieving a quantitative analysis of the mismatch between surface texture and thermal distribution. This cross-modal correlation analysis method can deeply explore the potential connections between different modal data and effectively identify complex defects that are difficult to detect with a single modality, such as thermal distribution anomalies hidden under surface texture anomalies or internal structural defects accompanied by sudden changes in vibration energy. This significantly improves the accuracy and reliability of defect detection.

[0018] The sorting judgment module extracts historical sorting data of the same type and dynamically weights and accumulates the deviations from the defect detection benchmark to generate a defect morphology deviation index. This index then determines whether to execute the sorting action based on the number of triggers within a continuous scanning cycle. This dynamic decision-making mechanism, based on historical data, fully considers the temporal variation of defect characteristics, avoiding the limitations of a single threshold judgment. This makes the triggering of sorting actions more flexible and intelligent, adapting to the defect detection needs of different production scenarios, effectively reducing the false positive and missed detection rates, and improving sorting efficiency and accuracy.

[0019] Furthermore, the system architecture of the present invention utilizes a modular design. The multimodal acquisition module, feature fusion modeling module, morphological correlation verification module, and sorting and decision module are both independent and collaborative, resulting in excellent scalability and adaptability. The clear division of labor among various functional units (such as the image acquisition unit, thermal imaging acquisition unit, and parameter extraction unit) ensures efficient operation of the system across all aspects of data acquisition, feature processing, defect verification, and sorting decision-making, providing reliable technical support for intelligent and automated sorting in industrial production. By organically combining multimodal data fusion with cross-modal correlation analysis, the present invention not only overcomes the limitations of traditional single-modality detection but also provides a comprehensive, accurate, and intelligent solution for defect detection and sorting of complex objects, with significant industrial application value and market prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a working principle diagram of the intelligent sorting method and system for multi-modal defect feature fusion described in the present invention.

[0021] Figure 2 Design diagrams for data preprocessing and generation of multimodal data sets.

[0022] Figure 3 This is the design diagram of multi-modal defect feature parameter fusion. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] See also Figure 1-Figure 3 The present invention relates to an intelligent sorting method based on multi-modal defect feature fusion, and the specific implementation steps are as follows: S1: Multimodal Data Acquisition and Preprocessing: Surface image data, infrared thermal imaging data, and acoustic vibration data of the objects to be sorted are acquired through a multimodal sensor array. Surface image data is acquired using an industrial camera at a preset scanning resolution, covering the entire surface of the object, generating raw data consisting of a grayscale distribution matrix and edge gradient features. Infrared thermal imaging data is acquired simultaneously by an infrared thermal imager, recording temperature gradient distribution and hot spot contour data. Acoustic vibration data is collected using a microphone array, containing spectral energy distribution and temporal waveform information. Data preprocessing involves median filtering the surface image to remove noise, implementing a spatial registration algorithm to establish a multimodal data coordinate system, filling missing pixels using bilinear interpolation, and normalizing the data range to the [0,1] interval. After preprocessing, the surface image data generates an image feature sequence consisting of pixel grayscale sequences and edge gradient vectors, while the infrared thermal imaging data generates a thermal distribution feature sequence. The two are correlated and matched using spatial coordinate indexing, forming a multimodal data set with a temporal and spatial correspondence.

[0025] S2: Multimodal Feature Fusion and Benchmark Construction: Feature parameter extraction and fusion are performed on the preprocessed image feature sequence, thermal feature sequence, and voiceprint vibration data. First, texture complexity (calculated using the grayscale co-occurrence matrix), hot spot area ratio (the ratio of the number of hot spot pixels to the total area of ​​the region), and spectrum main peak offset (the difference between the measured main peak frequency and the standard frequency) are extracted as morphological feature parameters. Each parameter is weighted and summed according to preset weights (e.g., 40% for texture complexity, 30% for hot spot area ratio, and 30% for spectrum main peak offset) to generate multimodal defect feature parameters. A dynamic screening threshold (e.g., 0.6) is then set to filter out valid features with feature parameter values ​​greater than the threshold to form an initial fused feature vector set, and invalid features are eliminated. The feature parameters in the initial set are compared across modalities, and the difference between different modal parameters within the same spatial region is calculated (e.g., the absolute difference between texture complexity and hot spot area ratio). This difference is then compared with a preset difference threshold (e.g., 0.2). Feature parameters exceeding the threshold are added to the defect detection benchmark set. At the same time, the spectrum energy mutation points in the voiceprint vibration data (such as energy change rate exceeding 15%) are analyzed, and the corresponding mutation features are added to the benchmark set to form a complete defect detection benchmark.

[0026] S3: Dynamic matching verification and mismatch quantification analysis: Use multimodal spatial alignment technology to establish a pixel-level spatial correspondence between the surface image and the infrared thermal imaging. Real-time monitoring of the matching degree between the local texture abnormality area (such as the edge gradient mutation area) of the object surface image and the corresponding infrared thermal distribution area, and calculate the ratio of the thermal distribution coverage area to the texture abnormality area. If the ratio exceeds the preset tolerance threshold (such as 1.2 or 0.8), it is determined to be a morphological mismatch event. Within the preset scanning period (such as 10 seconds), the number of mismatch events N is counted, and the spectral energy change rate ΔE (the proportion of energy mean change per unit time) in the voiceprint vibration data is obtained synchronously. Through the formula "comprehensive anomaly index Calculate the comprehensive abnormality index. If the index exceeds the preset sorting threshold (such as 0.7), a sorting signal is marked for the object.

[0027] S4: Defect Morphology Deviation Calculation and Sorting Triggering: This process extracts the verification results of a defect detection benchmark set from similar historical sorting data. The deviation between the actual measured value and the benchmark value for each benchmark parameter is dynamically weighted and accumulated (weights are set based on the confidence level of the historical data, with more recent data receiving a higher weight). This generates a defect morphology deviation index. If this index exceeds a preset trigger threshold (e.g., 0.8) for three consecutive scan cycles, the object is deemed to have a significant defect, triggering the sorting mechanism to execute the sorting action and remove the object from the production line.

[0028] The present invention will be further described below in conjunction with Examples 1 to 5: Example 1:

[0029] Based on the above overall scheme, this embodiment further defines the specific implementation of data acquisition and preprocessing. The system presets a scanning resolution of 2048×1536 pixels, which ensures accurate capture of surface details of the objects to be sorted. The scanning cycle is set to 5 seconds to balance detection efficiency and data integrity. In each scanning cycle, the system scans the object in different areas. The surface image data is divided into a grid of 20×20 pixels, and the surface of the object is divided into multiple regular sub-areas. Each sub-area generates a corresponding grayscale distribution matrix and edge gradient feature vector, where the edge gradient feature vector contains the gradient amplitude in the x and y directions, which is used to characterize the edge changes in the area.

[0030] Infrared thermal imaging data is processed synchronously using the same grid division method, with each grid area recording the maximum and minimum temperature values ​​and the pixel coordinates of the hot spot outline. The maximum and minimum temperature values ​​reflect the temperature range of the area, while the pixel coordinates of the hot spot outline are used to determine the specific location and shape of the hot spot within the area. The voiceprint vibration data is sampled using a 24-bit ADC with a sampling rate set to 48kHz. This sampling rate can meet the requirements for collecting high-frequency components of vibration signals in most industrial environments, generating a spectral energy distribution matrix and a time-domain waveform sequence with a frequency resolution of 1Hz. The spectral energy distribution matrix can intuitively reflect the energy distribution at different frequencies, while the time-domain waveform sequence records the changing trend of the vibration signal over time.

[0031] The data preprocessing stage is critical to ensuring the accuracy of subsequent analysis. For the surface image, a 5×5 pixel median filter is first used for noise reduction. Median filtering effectively suppresses salt-and-pepper noise in the image while preserving edge detail. Next, a registration algorithm based on the scale-invariant feature transform (SIFT) is used to achieve spatial alignment between the surface image and the infrared thermal image. The SIFT algorithm exhibits excellent scale, rotation, and perspective invariance, accurately identifying and matching feature points in the two images, thereby achieving precise spatial registration. After registration, missing pixels are filled using cubic spline interpolation, which ensures interpolation accuracy while providing a smooth transition effect in the interpolated image.

[0032] Infrared thermal imaging data is smoothed using Gaussian filtering, which effectively reduces temperature gradient noise and makes the temperature distribution smoother and more continuous. Voiceprint vibration data is filtered through a bandpass filter (50Hz-2000Hz) to remove ambient noise. This bandpass filter allows frequency components between 50Hz and 2000Hz to pass, while suppressing low-frequency noise below 50Hz and high-frequency noise above 2000Hz, thereby retaining the effective frequency components related to the object's vibration.

[0033] After preprocessing is completed, the image feature sequence and the thermal feature sequence need to be spatially correlated and matched. Specifically, according to the grid coordinate index, the image features (such as grayscale, gradient, etc.) of each grid area of ​​the surface image are correlated with the thermal features (such as temperature, hot spot contour, etc.) of the corresponding grid area of ​​the infrared thermal imaging to form a multimodal data set containing 4096 spatial points. Each spatial point corresponds to the multidimensional features of a specific position on the surface of the object, including image grayscale, edge gradient amplitude in the x and y directions, maximum temperature, minimum temperature, hot spot contour pixel coordinates, spectral energy distribution value and time domain waveform data. These multidimensional features provide a rich data foundation for subsequent defect feature analysis and fusion, enabling the system to comprehensively detect and evaluate defects in objects from multiple dimensions. Through this regional collection, preprocessing and correlation matching method, the accuracy, consistency and spatial correspondence of multimodal data are ensured, laying a solid foundation for the effectiveness of the entire intelligent sorting method.

[0034] Example 2: This embodiment describes in detail the specific process of feature fusion modeling. In step S2, the morphological feature parameters are extracted using the following method: Texture complexity is calculated using the gray-level co-occurrence matrix (GLCM). The specific process is as follows: In the pre-processed surface image feature sequence, for each 20×20 pixel grid area, gray-level co-occurrence matrices in four directions of 0°, 45°, 90°, and 135° are generated. Each matrix counts the joint distribution probability of the gray levels of pixel pairs within the area. By calculating the four eigenvalues ​​of contrast, entropy, energy, and correlation for each matrix and taking the average of the calculation results in the four directions, the texture complexity index of the area is obtained. This index reflects the roughness and directionality of the image texture.

[0035] Hot spot area ratio is extracted based on infrared thermal imaging data. First, a temperature threshold (e.g., 5°C above ambient temperature) is set and the temperature distribution of each grid area is binarized. Pixels above the threshold are identified as hot spot pixels. The hot spot area ratio is calculated by calculating the ratio of the number of hot spot pixels to the total number of pixels in the area. This parameter quantifies the relative size of abnormal thermal areas in infrared thermal imaging and is used to determine the severity of thermal defects on an object's surface.

[0036] The calculation of the main peak offset of the spectrum is done by converting the time-domain vibration signal into a frequency-domain spectrum using a Fast Fourier Transform (FFT) based on the voiceprint vibration data. The main peak frequency (i.e., the frequency component with the highest energy) in the spectrum is identified and the absolute difference between the frequency and the reference frequency of the standard component (e.g., 500Hz) is calculated to obtain the main peak offset of the spectrum. This parameter is used to measure the degree of abnormality in the vibration frequency of an object and can reflect defects in the internal structure.

[0037] When fusing multimodal defect feature parameters, the hierarchical analysis method is used to determine the weight distribution: the texture complexity weight is set to 0.4, the hot spot area ratio weight is 0.3, and the spectrum main peak offset weight is 0.3. The weight distribution is based on the contribution analysis of multimodal data to defect characterization. Among them, the surface image texture feature is sensitive to visible defects (such as cracks and wear), the infrared thermal feature is sensitive to thermal conduction anomalies (such as internal damage and poor contact), and the voiceprint feature is sensitive to mechanical vibration anomalies (such as looseness and imbalance). The above weights are used to achieve comprehensive consideration of multi-dimensional features. The fusion process is achieved through a linear weighted formula, that is, comprehensive feature parameter = texture complexity × 0.4 + hot spot area ratio × 0.3 + spectrum main peak offset × 0.3, generating multimodal defect feature parameters for each grid area.

[0038] The dynamic screening threshold is set to 0.6. This threshold, determined by the distribution statistics of characteristic parameters of normal and defective parts in historical data, is used to distinguish valid from invalid features. Specifically, the multimodal defect characteristic parameters of each grid area are traversed, and features with parameter values ​​greater than 0.6 are retained, while features below the threshold are eliminated to form an initial fused feature vector set. This step filters out features with noise interference or no significant defect representation, reducing subsequent computational complexity and improving the targeted nature of the feature set.

[0039] Cross-modal comparative analysis is performed for each feature parameter in the initial fused feature vector set. Specifically, within the same grid area, the absolute difference between texture complexity and the hot spot area ratio is calculated, as well as the normalized difference between texture complexity and the offset of the main peak of the spectrum (the difference divided by the mean of the two). If any difference is greater than a preset difference threshold (such as 0.2), it is marked as an inter-modal difference parameter. This process aims to identify inconsistencies between different modal features, such as spatial or intensity mismatches between abnormal surface texture areas and abnormal thermal distribution areas. Such differences often indicate the presence of potential defects.

[0040] The spectral energy mutation feature of voiceprint vibration data is supplemented by real-time monitoring of the dynamic changes in the spectral energy distribution matrix. Specifically, the energy change rate of each frequency point between two adjacent scanning cycles is calculated (the absolute value of the difference between the energy value of the current cycle and the energy value of the previous cycle divided by the energy value of the previous cycle). When the energy change rate of a frequency point exceeds 20%, the center frequency, energy change value, and timestamp of that frequency point are extracted and added as spectral energy mutation features to the defect detection benchmark set. This operation can capture sudden vibration anomalies, such as transient defects such as component breakage and impact, and compensate for the shortcomings of static feature analysis.

[0041] The construction process of the defect detection benchmark set follows a logical chain of "feature validity verification - cross-modal difference screening - dynamic anomaly supplementation": first, significant features are retained through threshold screening, then inconsistency features are discovered through cross-modal difference analysis, and finally, dynamic anomaly information is supplemented through voiceprint mutation features, forming a multimodal, multi-dimensional, and dynamic and static defect characterization system. This benchmark set serves as a reference standard for subsequent dynamic matching verification. Its construction process fully utilizes the complementarity of multimodal data to ensure comprehensive coverage and accurate description of various defects, providing a solid feature foundation for intelligent detection of object defects.

[0042] Example 3: This embodiment further defines the quantitative analysis process for morphological association verification. In step S3, the spatial matching unit achieves precise mapping of pixel coordinates by establishing a perspective transformation matrix between the surface image and the infrared thermal image. Specifically, at least four pairs of feature points with the same name (such as object edge corners and fixed markers) are first selected from the surface image and the infrared thermal image. The perspective transformation matrix H is solved using the direct linear transformation (DLT) algorithm. This matrix contains eight unknown parameters and is obtained by least squares fitting. This allows any pixel coordinate (u, v) in the surface image to be mapped to the corresponding coordinate (u, v) in the infrared thermal image via (u´, v´, 1) = H · (u, v, 1), thereby achieving pixel-level spatial alignment of the two types of images.

[0043] During real-time monitoring, the system performs texture anomaly detection on each 20×20 pixel grid area. The edge gradient amplitude is calculated using the Sobel operator, specifically: for each grid area in the surface image, the horizontal Sobel kernel is applied separately. and the vertical Sobel kernel Perform convolution operation to obtain the horizontal gradient component G x (i, j) and the vertical gradient component G y (i, j), edge gradient amplitude If the edge gradient amplitude in a certain area is greater than a preset threshold (such as 0.3), the area is marked as a texture abnormal area, which is recorded as A tex , whose area is determined by counting the number of abnormal pixels.

[0044] Thermal distribution coverage area A corresponding to the infrared thermal imaging area thr The Ambient temperature is determined by counting the number of pixels whose temperature is 3°C higher than the ambient temperature. The ambient temperature is measured in real time by the temperature sensor in the multimodal acquisition module and used as the reference value. thr With A tex The ratio R=A thr / A texIf R>1.2 or R<0.8, it is determined to be a morphological mismatch event, indicating that there is a significant inconsistency in the spatial coverage between the abnormal surface texture area and the abnormal thermal distribution area, which may indicate the existence of internal defects or composite defects.

[0045] In a preset scanning period (e.g., 10 seconds), the system counts the number of morphological mismatch events as N. Simultaneously, the spectral energy change rate ΔE of the voiceprint vibration data is calculated by dividing the spectral energy distribution matrix of the voiceprint vibration data into continuous frames according to the time series, with each frame lasting 1 second, and calculating the energy mean E of the current frame. cur and the previous frame energy mean E pre The ratio of the absolute value of the difference to the energy mean of the previous frame, that is, , this parameter is used to characterize the dynamic change trend of vibration energy.

[0046] The comprehensive abnormality index is calculated by the formula "comprehensive abnormality index ” calculation. In the formula, the numerator Reflecting the synergistic effect of the number of mismatch events and the energy change rate, the denominator The two are normalized to avoid calculation deviation caused by dimensional differences; The term characterizes the degree of deviation between the two, highlighting abnormal conditions when the number of mismatches is significantly inconsistent with the rate of energy change. When N or ΔE is 0, the denominator is treated as 1 (for example, if N = 0 and ΔE = 0, the denominator is treated as 1 to avoid division by zero errors), ensuring the validity of the formula in extreme cases.

[0047] The preset sorting threshold is 0.7, determined by analyzing the distribution range of the comprehensive anomaly index of normal and defective parts in historical multimodal data. If the calculated comprehensive anomaly index exceeds this threshold, a sorting mark is generated, triggering the subsequent sorting decision process. The entire quantitative analysis process, through a closed loop of spatial alignment, regional matching, parameter calculation, and logical judgment, achieves a quantitative assessment of the mismatch between the surface texture and thermal distribution of an object. Combined with the dynamic characteristics of voiceprint vibration data, a multi-dimensional anomaly detection mechanism is formed, which can effectively identify complex defects that are difficult to detect with a single modality, providing a scientific basis for sorting decisions.

[0048] Example 4:

[0049] This embodiment describes in detail the implementation of the sorting and judgment module. In step S4, the indicator generation unit constructs a calculation model for the defect morphology deviation based on the historical sorting data. Taking a certain type of metal parts as an example, its historical sorting data contains multimodal feature parameter records of normal parts and defective parts, such as the texture complexity of the surface image, the proportion of the hot spot area of ​​infrared thermal imaging, the main peak offset of the spectrum of the soundprint vibration, and other benchmark parameters. For each defect detection benchmark parameter, the system needs to calculate the degree of deviation between the current measurement value and the corresponding benchmark value. For example, the texture complexity benchmark value of a certain grid area is set to 0.5 (based on the statistical mean of similar normal parts). If the texture complexity measurement value of a certain part to be sorted in this area is currently detected to be 0.6, then the absolute deviation between the two is 0.1, and the relative deviation is determined by the ratio of the absolute deviation to the benchmark value, that is, the relative deviation is 0.1 / 0.5=0.2, which represents the relative degree of deviation of the current measurement value from the benchmark.

[0050] To comprehensively evaluate deviations across multiple consecutive scan cycles, the system uses a dynamic weighted accumulation approach to process relative deviations. This weighting strategy prioritizes recent data within the time series. For example, the weights for the data from the last three scan cycles are set to 0.5, 0.3, and 0.2, respectively, while the weights for more recent data gradually decrease. Taking the relative deviation sequence of four consecutive cycles [0.1, 0.3, 0.4, 0.2] as an example (assuming the cycles are ordered from recent to recent), the weighted accumulation result is 0.1 × 0.5 + 0.3 × 0.3 + 0.4 × 0.2 + 0.2 × 0.1 = 0.05 + 0.09 + 0.08 + 0.02 = 0.24, which is an example calculation of the defect morphology deviation indicator. This dynamic weighting approach allows the system to capture recent data trends in real time, avoiding misjudgments caused by occasional interference or single-cycle anomalies.

[0051] The preset trigger condition is that the defect morphology deviation index exceeds the preset threshold value in three consecutive scanning cycles. Taking the threshold value set as 0.8 as an example, if the deviation index of a component in the 1st, 2nd and 3rd cycles are 0.85, 0.82 and 0.83 respectively, all exceeding the threshold value, it is judged as a continuous trigger. The design of the trigger logic is based on the principle of persistence of defect characteristics in industrial scenarios-accidental single-cycle anomalies may be caused by noise or instantaneous interference, while deviations in multiple consecutive cycles are more likely to reflect real defects. For example, when a component has an internal crack, the mismatch between the abnormal surface texture area and the thermal distribution will persist, causing the deviation index of consecutive cycles to remain at a high level, thereby triggering the sorting action.

[0052] The specific implementation of the execution unit involves interoperability with industrial hardware. Once the sorting determination module confirms the triggering condition, the system sends instructions to the sorting actuator via industrial Ethernet (such as Profinet or EtherCAT). For example, on a conveyor belt production line, object positioning sensors are installed at regular intervals along the conveyor belt. When a part triggering sorting moves to a designated sorting station, the execution unit sends an action signal to a robotic arm or pneumatic sorting valve based on the part's real-time position coordinates on the conveyor belt (obtained through visual positioning or encoder data). For example, if a pneumatic sorting valve is used, the execution unit outputs an electrical signal within 0.1 seconds to trigger the solenoid valve, which activates compressed air to push the sorting baffle, removing defective parts from the main conveyor belt and placing them in a defective buffer. If a robotic arm is used for sorting, the system pre-calculates the robotic arm's grasping path and motion parameters to ensure that the robotic arm completes positioning, grasping, and transfer within 0.5 seconds, avoiding disruption to the normal transport of subsequent parts.

[0053] The entire sorting and judgment process must meet industrial real-time requirements, so data processing and logical judgment are completed in edge computing nodes or embedded controllers to reduce network latency. For example, the weighted accumulation calculation of the indicator generation unit can be implemented in parallel through a field programmable gate array (FPGA) or a digital signal processor (DSP), ensuring that the processing delay within each scanning cycle is less than 50 milliseconds. At the same time, the system has a historical data backtracking function and can store at least 100,000 sorting records for later quality tracing and algorithm optimization. For example, when the sorting rate of a batch of parts increases abnormally, the operator can retrieve historical data, analyze the distribution of multimodal characteristic parameters in the corresponding time period, and locate potential problem points in the production process.

[0054] Taking the sorting scenario of automobile engine cylinder blocks as an example, if a cylinder block has a casting sand hole defect, the edge gradient amplitude of its surface image in the sand hole area will increase significantly (texture anomaly). At the same time, due to the low thermal conductivity of the air inside the sand hole, the temperature gradient in the corresponding infrared thermal imaging area will be abnormally distributed, causing the ratio of the thermal coverage area to the area of ​​the texture anomaly area to exceed the preset tolerance threshold, resulting in a morphological mismatch event. In the voiceprint vibration data, sand holes may cause changes in the local vibration frequency, resulting in abnormalities in the offset of the main peak of the spectrum and the energy change rate. The continuous deviation of these multimodal features will cause the defect morphology deviation index to continuously exceed the threshold, triggering the sorting action and removing the unqualified cylinder blocks from the production line, thereby achieving efficient detection and accurate sorting of casting defects. The entire process requires no human intervention and is completely based on the automatic analysis and logical judgment of multimodal data, ensuring the efficiency and reliability of industrial sorting.

[0055] Example 5:

[0056] This example describes the hardware architecture and module-coordinated workflow of an intelligent sorting system. Taking an electronic component production line as an example, the system consists of a multimodal acquisition module, a feature fusion modeling module, a morphological correlation verification module, and a sorting and judgment module. Each module interacts with data via standardized interfaces, meeting the industrial-grade reliability requirements for 24 / 7 continuous operation.

[0057] The multimodal acquisition module, deployed above the production line's conveyor belt, consists of an image acquisition unit, a thermal imaging unit, and a voiceprint acquisition unit. The image acquisition unit uses two 5-megapixel industrial cameras, one mounted on either side of the conveyor belt, to simultaneously capture images of the top and side surfaces of electronic components at a frame rate of 60 fps, with a resolution of 2048 × 1536 pixels. The cameras are equipped with telecentric lenses to eliminate perspective distortion and ensure the accurate proportionality of component dimensions in the image to their actual dimensions. The thermal imaging unit is an uncooled infrared camera with a resolution of 320 × 240 pixels, a temperature detection range of -20°C to 150°C, and an accuracy of ±2%. It is mounted above the industrial camera, parallel to the camera's optical axis, ensuring that the infrared thermal image and the surface image overlap in spatial field of view. The voiceprint acquisition unit features an 8-channel microphone array, evenly distributed on brackets on either side of the conveyor belt. Operating at a sampling rate of 48 kHz, it captures vibration noise generated by electronic components during transport. The microphones are equipped with built-in windshields to reduce interference from ambient airflow. Each acquisition unit is started synchronously through the hardware trigger module to ensure the consistency of the timestamps of the multimodal data. For example, a synchronous acquisition is triggered every 0.1 second, corresponding to a conveyor belt movement of 5 cm.

[0058] The feature fusion modeling module, integrated into the industrial control cabinet, comprises a parameter extraction unit and a defect construction unit. The parameter extraction unit utilizes an FPGA chip to perform real-time feature calculations. For example, for surface image data transmitted by an industrial camera, it extracts the texture complexity of the grayscale co-occurrence matrix for a 20×20 pixel grid frame by frame. Using a parallel computing unit, it completes feature calculations for all 4096 grids in the entire image within 5ms. For infrared thermal image data, the FPGA's pipeline architecture calculates the hotspot area percentage for each grid in real time, with processing latency under 3ms. The defect construction unit utilizes an ARM processor and runs a dynamic threshold filtering algorithm. For example, by setting a texture complexity threshold of 0.6, it filters the feature parameters output by the FPGA, retaining valid features and storing them in a high-speed cache. In cross-modal difference analysis, the ARM processor compares the texture complexity and hotspot area percentage of the same grid. Any discrepancy exceeding 0.2 is flagged as an anomalous feature. The ARM processor also monitors sudden changes in spectral energy in the voiceprint data (e.g., a rate of change exceeding 20% ​​within 500ms) and adds these anomalous features to the defect detection benchmark set. Taking a capacitor component as an example, if the texture complexity of its surface crack area is 0.7 (threshold 0.6), the corresponding infrared hot spot area accounts for 0.3, and the difference between the two is 0.4 (exceeding the threshold 0.2), then this feature will be included in the benchmark set as the basis for defect judgment.

[0059] The morphological correlation verification module consists of a spatial matching unit and an anomaly calculation unit. The spatial matching unit uses an FPGA to perform pixel-level registration between the surface image and the infrared thermal image. Four black cross markers are placed on the component surface. The industrial camera and the infrared thermal imager each identify the marker coordinates. A direct linear transformation algorithm is used to calculate the perspective transformation matrix, achieving image coordinate mapping. During real-time monitoring, the FPGA compares the area ratio of regions with texture anomalies (edge ​​gradient amplitude > 0.3) to regions with thermal anomalies (temperature > 3°C above ambient temperature) on a grid-by-grid basis. For example, if a capacitor component has a cold solder joint, the pin edge gradient amplitude in the surface image is 0.4 (anomaly), while the corresponding temperature in the infrared thermal image is only 1°C above ambient (no hot spots), and the area ratio is 0 (well below the threshold of 0.8), indicating a mismatch event. The anomaly calculation unit, using a DSP chip, counts the number of mismatch events N in real time within 10 seconds and calculates the rate of change ΔE of the spectral energy of the voiceprint data. For example, a component generates vibration noise due to a broken internal pin. Its spectral energy varies by 15% and 20% in two consecutive cycles, respectively. The average ΔE is 17.5%, which is substituted together with the number of mismatches N=3 into the calculation of the comprehensive anomaly index (the specific calculation process is omitted). If the result exceeds the threshold of 0.7, a sorting mark is generated.

[0060] The sorting and determination module consists of an indicator generation unit and an execution unit. The indicator generation unit runs on an edge computing server and builds a defect deviation model based on historical data. For example, if the texture complexity baseline value of a batch of normal capacitor components is 0.5, and the measured values ​​of a component under inspection for three consecutive cycles are 0.65, 0.68, and 0.72, respectively, with relative deviations of 0.3, 0.36, and 0.44, respectively, using exponential weighting (recent weights of 0.6, 0.3, and 0.1), the deviation index is calculated as 0.6 × 0.3 + 0.3 × 0.36 + 0.1 × 0.44 = 0.18 + 0.108 + 0.044 = 0.332 (this is only an example calculation logic and does not represent an actual threshold). If the indicator exceeds a preset threshold (such as 0.8) for three consecutive cycles, the execution unit triggers sorting via the PLC controller. Taking the pneumatic sorting valve as an example, after the PLC receives the sorting signal, it drives the solenoid valve to open within 100ms, and the compressed air pushes the baffle to move the defective component into the buffer box on the right; if a six-axis robotic arm is used for sorting, the system obtains the component coordinates through visual positioning (such as the conveyor belt coordinate X=1000mm, Y=50mm), and the robotic arm moves according to the preset trajectory, completing the grasping and transferring it to the waste bin within 500ms.

[0061] Data is transmitted between modules via the PCIe 3.0 bus. For example, the multimodal acquisition module transmits a set of image, thermal image, and voiceprint data (approximately 15MB) to the feature fusion module every 0.1 seconds. The bus bandwidth (8GB / s) meets real-time transmission requirements. The system software layer uses an embedded Linux operating system, and a real-time scheduling algorithm ensures that the processing delay of each module is less than 200ms, meeting the production line's cycle time requirements. In electronic component sorting scenarios, the system can simultaneously detect multiple defects, including surface cracks (images), welding heat (infrared), and internal looseness (voiceprints). Through multimodal feature fusion and dynamic verification, it can accurately identify complex defects, avoid missed detections and misjudgments caused by single-modal detection, and improve the automation and reliability of industrial sorting.

[0062] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0063] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent sorting method based on multimodal defect feature fusion, characterized in that: The method comprises the following steps: S1: Acquire surface image data, infrared thermal imaging data, and voiceprint vibration data of the object to be sorted, and perform data preprocessing on the surface image data, infrared thermal imaging data, and voiceprint vibration data; S2: Analyze the preprocessed data to generate multimodal defect feature parameters, perform feature fusion based on the multimodal defect feature parameters to form a fused feature vector set, and construct a defect detection benchmark set based on the fused feature vector set; S3: Dynamically match and verify the multimodal data of the object based on the defect detection benchmark set, use multimodal spatial alignment technology to identify the spatial correspondence between the surface image and infrared thermal imaging, and quantitatively analyze the mismatch between surface image texture anomalies and thermal distribution anomalies; S4: Generate a defect morphology deviation index based on the verification result of the defect detection benchmark set, and determine whether to trigger a sorting action based on the defect morphology deviation index.

2. The intelligent sorting method based on multimodal defect feature fusion according to claim 1, characterized in that: In S1, the system presets a scanning resolution and obtains regional surface image data, infrared thermal imaging data and voiceprint vibration spectrum data of the object within a preset scanning period; the surface image data includes a grayscale distribution matrix and edge gradient features; the infrared thermal imaging data includes a temperature gradient distribution and hot spot contour data; the voiceprint vibration data includes a spectrum energy distribution and a vibration time domain waveform; data preprocessing is performed on the surface image data, infrared thermal imaging data and voiceprint vibration data, and the data preprocessing includes noise filtering, spatial alignment, data interpolation and normalization processing; an image feature sequence is generated after the surface image data is preprocessed, and a thermal feature sequence is generated after the infrared thermal imaging data is preprocessed; the image feature sequence and the thermal feature sequence are spatially correlated and matched to form a multimodal data set.

3. The intelligent sorting method based on multimodal defect feature fusion according to claim 2, characterized in that: In S2, the following steps are included: S201: extracting morphological feature parameters from the preprocessed image feature sequence, thermal feature sequence, and voiceprint vibration data, respectively. The morphological feature parameters include texture complexity, hot spot area ratio, and spectrum main peak offset. Each morphological feature parameter is fused according to a preset weight to generate a multimodal defect feature parameter. S202: Setting a dynamic screening threshold for multimodal defect feature parameters in the multimodal data set, screening feature parameters that meet the threshold condition to form an initial fusion feature vector set, and discarding feature parameters that do not meet the threshold condition; S203: performing cross-modal comparative analysis on each feature parameter in the initial fused feature vector set, extracting the difference between feature parameters of different modalities in the same spatial region, calculating the relative deviation of the difference, and marking it as an inter-modal difference parameter; S204: Compare the inter-modal difference parameters with the preset difference threshold, filter the feature parameters that exceed the difference threshold and add them to the defect detection benchmark set, and add the spectrum energy mutation feature to the defect detection benchmark set based on the voiceprint vibration data.

4. The intelligent sorting method based on multimodal defect feature fusion according to claim 1, characterized in that: In S3, quantitative analysis of the mismatch between surface image texture anomaly and thermal distribution anomaly includes the following steps: S301: Acquire local texture features of the surface image of the object and infrared thermal distribution features of the corresponding area in real time, calculate the ratio of the thermal distribution coverage area to the texture abnormal area, and determine a morphological mismatch event if the ratio exceeds a preset tolerance threshold; S302: Count the number of morphology mismatch events within a preset scanning period and record it as N, and synchronously obtain the rate of change of spectral energy in the voiceprint vibration data and record it as ΔE; S303: Calculate the comprehensive anomaly index based on the number of morphological mismatch events N and the spectrum energy change rate ΔE. The specific calculation method is: Comprehensive anomaly index ; If the comprehensive abnormality index exceeds the preset sorting threshold, the sorting mark is triggered.

5. The intelligent sorting method based on multimodal defect feature fusion according to claim 1, characterized in that: In S4, the verification results of the defect detection benchmark set in the same type of historical sorting data are extracted, and the deviation of each benchmark is dynamically weighted and accumulated to generate a defect morphology deviation index. If the defect morphology deviation index exceeds the preset trigger number within a continuous scanning cycle, it is determined to perform a sorting action.

6. An intelligent sorting system for multimodal defect feature fusion, the system being applied to implement the intelligent sorting method for multimodal defect feature fusion according to any one of claims 1 to 5, characterized in that: The system includes a multimodal acquisition module, a feature fusion modeling module, a morphological association verification module and a sorting and judgment module; The multimodal acquisition module is used to acquire surface image data, infrared thermal imaging data, and voiceprint vibration data of an object and perform data preprocessing; The feature fusion modeling module is used to analyze the preprocessed data to generate multimodal defect feature parameters and construct a defect detection benchmark set; The morphological association verification module is used to perform dynamic matching verification on the multimodal data of the object according to the defect detection benchmark set, and analyze the mismatch between the surface texture and the thermal distribution; The sorting determination module is used to generate a defect morphology deviation index and trigger a sorting action.

7. The intelligent sorting system for multimodal defect feature fusion according to claim 6, characterized in that: The multimodal acquisition module includes an image acquisition unit, a thermal imaging acquisition unit and a voiceprint acquisition unit; The image acquisition unit is used to collect regional surface image data and edge gradient features of the object; The thermal imaging acquisition unit is used to obtain the temperature gradient distribution and hot spot contour data of the object; The voiceprint collection unit is used to record the vibration spectrum energy distribution and time domain waveform data of the object.

8. The intelligent sorting system for multimodal defect feature fusion according to claim 6, characterized in that: The feature fusion modeling module includes a parameter extraction unit and a defect construction unit; The parameter extraction unit is used to extract characteristic parameters of texture complexity, hot spot area ratio and spectrum main peak offset from the preprocessed data; The defect construction unit is used to screen characteristic parameters that meet the threshold conditions and add spectrum energy mutation characteristics to the defect detection benchmark set.

9. The intelligent sorting system for multimodal defect feature fusion according to claim 6, characterized in that: The morphological association verification module includes a space matching unit and an anomaly calculation unit; The spatial matching unit is used to detect the spatial correspondence between the surface image and the thermal image and identify morphological mismatch events; The anomaly calculation unit is used to calculate a comprehensive anomaly index in combination with the spectrum energy change rate.

10. The intelligent sorting system for multi-modal defect feature fusion according to claim 6, characterized in that: The sorting determination module includes an indicator generation unit and an execution unit; The index generating unit is used to dynamically weight and accumulate the deviation of the defect detection benchmark; The execution unit is used to control the action of the sorting mechanism according to the continuous triggering results of the deviation index.

Images