Multi-class defect identification system and method applied to intelligent manufacturing

Abstract

The invention discloses a multi-class defect identification system and method applied to intelligent manufacturing, and relates to the technical field of manufacturing defect identification, and the system comprises a data synchronization module which is used for collecting multi-modal data in a production process, and carrying out the data synchronization of the multi-modal data; the preprocessing module is used for preprocessing the synchronized multi-modal data to obtain multi-modal processing data; the independent analysis module is used for carrying out feature extraction on the multi-modal processing data by utilizing a deep learning algorithm so as to carry out defect identification and obtain a single defect result; the fusion analysis module is used for performing fusion processing on the multi-modal processing data to generate a fusion defect result; and the defect verification module is used for performing data comparison based on the single defect result and the fused defect result to obtain a final defect result, and the single defect result is compared with the defect result fused with the multi-modal data, so that the accuracy of single data source detection can be effectively verified.

Description

Technical Field

[0001] The present invention relates to the technical field of manufacturing defect recognition, and in particular to a multi-category defect recognition system and method applied to intelligent manufacturing. Background Art

[0002] With the rapid development of intelligent manufacturing technology, the degree of automation and intelligence in industrial production has been continuously improved. How to efficiently and accurately detect defects in the product production process has become a key issue in the field of intelligent manufacturing.

[0003] At present, traditional defect detection methods mainly rely on a single sensor for detection, such as surface defect detection through visual sensors or internal defect detection of products through ultrasonic sensors. Although these methods can work in specific scenarios, due to the limitations of the sensors themselves, a single data source is susceptible to noise interference and has a limited detection range, resulting in frequent omissions or false alarms, and cannot meet the diverse defect detection needs in complex manufacturing environments.

[0004] To solve this problem, in recent years, multimodal data fusion technology has gradually been applied to the field of intelligent manufacturing defect identification. Multimodal data fusion refers to the comprehensive analysis of data collected by different types of sensors (such as vision, sound waves, vibration, etc.), and the comprehensiveness and accuracy of defect detection are improved through multi-angle and multi-dimensional information complementarity; however, the existing multimodal fusion technology still faces the following challenges in defect identification: first, the data from different sensors differ in time and space; second, the accuracy of the collected multimodal data to ensure the detection results is low. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multi-class defect recognition system and method for intelligent manufacturing.

[0006] A multi-class defect recognition system applied to intelligent manufacturing, the system comprising: Data synchronization module, used to collect multimodal data in the production process and synchronize the multimodal data; A preprocessing module, used for preprocessing the synchronized multimodal data to obtain multimodal processed data; An independent analysis module is used to extract features from multimodal processing data using deep learning algorithms to identify defects and obtain single defect results; A fusion analysis module is used to perform fusion processing on multi-modal processing data to generate fusion defect results; The defect verification module is used to compare data based on single defect results and fused defect results to obtain the final defect results.

[0007] Furthermore, the multimodal data includes visual data, sound wave data and vibration data.

[0008] Furthermore, the steps of synchronizing the multimodal data are as follows: P1, through a high-precision time base, unifies the time of all sensors that collect multimodal data. Multimodal data is accompanied by precise timestamps to ensure that the data has the same time base; P2, through the preset hardware trigger mechanism, when the established trigger event of the production line occurs, the sensor that triggers the multimodal data collection is started, and the trigger event includes but is not limited to the product entering the detection area and the presence of the product passing on the conveyor belt; P3, aligns multimodal data based on timestamps. If the sampling frequencies of multiple sensors are different, time windows are used for processing, and all data are aggregated into the same time period for analysis and comparison.

[0009] Furthermore, the logic of preprocessing the synchronized multimodal data is as follows: The multimodal data containing noise is filtered out, the multimodal data is normalized, and the original data is enhanced.

[0010] Furthermore, the logic of using deep learning algorithms to extract features from multimodal processing data for defect identification is as follows: Q1, analyze visual data through convolutional neural networks to extract visual features related to defects; Q2, analyze the acoustic wave data through a recurrent neural network to identify the acoustic wave features in the abnormal acoustic wave signal; Q3, converts vibration data from the time domain to the frequency domain through fast Fourier transform, extracting spectral features related to equipment status and material properties; Q4, combining convolutional neural network and recurrent neural network, input the above extracted visual features, acoustic wave features and spectral features into the classifier to identify the specific defect type and give a single defect result according to the severity of the defect.

[0011] Furthermore, the logic of fusing the multimodal processing data to generate the fusion defect result is: a1, process the multimodal processing data into feature vectors, where each feature vector represents the defect feature of the corresponding multimodal processing data; a2, adaptively adjust the weights of different modalities based on the attention mechanism, determine the importance of a certain modality based on real-time data analysis, and modify the feature weights based on the established data weights; a3, let the visual feature vector be V, the sound wave feature vector be A, and the vibration feature vector be B. Based on the attention mechanism, the weights of different modalities are adaptively adjusted to αV, αA, and αB, respectively, where αV+αA+αB=0.8694; The eigenvectors of different modes are weighted and summed to obtain the fused eigenvector F. The logical formula of the fused eigenvector F is: F = V × αV + A × αA + B × αB; Set the comprehensive analysis threshold Fy, calculate the defect difference between F and Fy, and classify the defects by comprehensive analysis based on the defect difference.

[0012] Furthermore, the logic of data comparison based on single defect results and fusion defect results is: If the defect detected by the single defect result is the same as the defect detected by the fusion defect result, it is marked as a high-confidence defect and the final defect result is output; if the defect detected by the single defect result is in conflict with the defect detected by the fusion defect result, it is marked as an unreliable defect and the defect detection is performed again.

[0013] A multi-class defect recognition method applied to intelligent manufacturing, applied to the multi-class defect recognition system applied to intelligent manufacturing, characterized in that the method comprises: Collect multimodal data during the production process and synchronize the multimodal data; Preprocessing the synchronized multimodal data to obtain multimodal processed data; Use deep learning algorithms to extract features from multimodal processing data for defect identification and obtain single defect results; Performing fusion processing on multi-modal processing data to generate fusion defect results; Data comparison is performed based on single defect results and fused defect results to obtain the final defect results.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention can effectively verify the accuracy of single data source detection by comparing a single defect result with the defect result of fused multimodal data. When the defect result of a single sensor is consistent with the fusion result, a high-confidence defect result is output to avoid the risk of false alarm or missed alarm of a single sensor, thereby ensuring the accuracy of the detection result. When a single defect result conflicts with a fused defect result, the present invention further screens out more reliable defect information through comparative analysis of multimodal data and investigates the cause of the abnormality. This not only improves the ability to identify complex defects, but also effectively handles conflicts between different sensor data, ensuring that the output defect results are more comprehensive and accurate, and suitable for defect detection needs in various production environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 A module diagram of a multi-class defect recognition system applied to intelligent manufacturing provided in Example 1 of the present invention; Figure 2 A flowchart of a multi-class defect identification method applied to intelligent manufacturing provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example 1 See also Figure 1 As shown, this embodiment discloses a multi-class defect recognition system applied to intelligent manufacturing, and the system includes: The data synchronization module S110 is used to collect multimodal data in the production process and synchronize the multimodal data; Specifically, the multimodal data includes visual data, sound wave data and vibration data; It should be noted that: multimodal data in the production process is collected in real time through corresponding sensors, among which visual data is collected through industrial cameras or visual sensors to obtain product surface images or videos. This process can capture possible defects such as flaws, cracks, deformations, etc. on the product surface; sound wave data is collected using ultrasonic sensors to detect abnormalities in the internal structure of the product and identify possible cracks or voids inside; vibration data is collected through vibration sensors installed on the equipment to detect the vibration of the equipment during the production process and identify defects caused by equipment failure or material abnormalities; Specifically, the steps for synchronizing multimodal data are as follows: P1, through a high-precision time base, unifies the time of all sensors that collect multimodal data. Multimodal data is accompanied by precise timestamps to ensure that the data has the same time base; For example, data from cameras, sound waves, and vibration sensors will record their own timestamps based on a unified clock; It should be noted that: the high-precision time reference includes but is not limited to NTP or IEEE1588PTP; P2, through the preset hardware trigger mechanism, when the established trigger event of the production line occurs, the sensor that triggers the multimodal data collection is started, and the trigger event includes but is not limited to the product entering the detection area and the presence of the product passing on the conveyor belt; It should be noted that: through this synchronous start-up, it is ensured that the acquisition process of the camera, sound wave, and vibration sensor starts at the same time, eliminating timing deviation.

[0019] P3, aligns multimodal data based on timestamps. If the sampling frequencies of multiple sensors are different, time windows are used for processing, and all data are aggregated into the same time period for analysis and comparison.

[0020] A preprocessing module S120 is used to preprocess the synchronized multimodal data to obtain multimodal processed data; Specifically, the logic of preprocessing the synchronized multimodal data is: Filter out the noisy multimodal data, normalize the multimodal data, and perform data enhancement on the original data; For example, operations such as rotation and scaling can be performed on visual data, or noise can be added to vibration data to generate more training samples; An independent analysis module S230 is used to extract features from multimodal processing data using a deep learning algorithm to identify defects and obtain a single defect result; Specifically, the logic of using deep learning algorithms to extract features from multimodal processing data for defect identification is: Q1, analyze visual data through convolutional neural networks to extract visual features related to defects; It is important to understand that the convolutional layer of a convolutional neural network can automatically identify information such as edges, textures, shapes, etc. in an image, thereby identifying surface defects such as scratches, dents, cracks, etc. Q2, analyze the acoustic wave data through a recurrent neural network to identify the acoustic wave features in the abnormal acoustic wave signal; For example, cracks or holes inside a product. In this process, RNN / LSTM captures the temporal characteristics of the signal, which is particularly suitable for processing continuous data. Q3, converts vibration data from the time domain to the frequency domain through fast Fourier transform, extracting spectral features related to equipment status and material properties; Q4, combining convolutional neural network and recurrent neural network, input the above extracted visual features, acoustic wave features and spectral features into the classifier to identify the specific defect type and give a single defect result according to the severity of the defect.

[0021] The fusion analysis module S240 is used to perform fusion processing on the multi-modal processing data to generate a fusion defect result; Specifically, the logic of fusing the multimodal processing data to generate the fusion defect result is: a1, process the multimodal processing data into feature vectors, where each feature vector represents the defect feature of the corresponding multimodal processing data; a2, adaptively adjust the weights of different modalities based on the attention mechanism, determine the importance of a certain modality based on real-time data analysis, and modify the feature weights based on the established data weights; a3, let the visual feature vector be V, the sound wave feature vector be A, and the vibration feature vector be B. Based on the attention mechanism, the weights of different modalities are adaptively adjusted to αV, αA, and αB, respectively, where αV+αA+αB=0.8694; The eigenvectors of different modes are weighted and summed to obtain the fused eigenvector F. The logical formula of the fused eigenvector F is: F = V × αV + A × αA + B × αB; Set the comprehensive analysis threshold Fy, calculate the defect difference between F and Fy, and classify the defects based on the comprehensive analysis; It is important to understand that the correspondence between the comprehensive difference analysis defect and the defect difference is determined by the historical experimental results of the analyst; It should be noted that this step further improves the robustness and accuracy of defect identification by combining random forest with multimodal processing data.

[0022] The defect verification module S250 is used to compare data based on the single defect result and the fusion defect result to obtain the final defect result; Specifically, the logic for data comparison based on single defect results and fusion defect results is: If the defect detected by the single defect result is the same as the defect detected by the fusion defect result, it is marked as a high-confidence defect and the final defect result is output; if the defect detected by the single defect result is in conflict with the defect detected by the fusion defect result, it is marked as an unreliable defect and the defect detection is performed again.

[0023] Example 2 See also Figure 2 As shown, based on the unified inventive concept, this embodiment discloses a multi-class defect recognition method applied to intelligent manufacturing, the method comprising: S210, collecting multimodal data in the production process and synchronizing the multimodal data; Specifically, the multimodal data includes visual data, sound wave data and vibration data; Specifically, the steps for synchronizing multimodal data are as follows: P1, through a high-precision time base, unifies the time of all sensors that collect multimodal data. Multimodal data is accompanied by precise timestamps to ensure that the data has the same time base; P2, through the preset hardware trigger mechanism, when the established trigger event of the production line occurs, the sensor that triggers the multimodal data collection is started, and the trigger event includes but is not limited to the product entering the detection area and the presence of the product passing on the conveyor belt; P3, aligns multimodal data based on timestamps. If the sampling frequencies of multiple sensors are different, time windows are used for processing, and all data are aggregated into the same time period for analysis and comparison.

[0024] S220, preprocessing the synchronized multimodal data to obtain multimodal processed data; Specifically, the logic of preprocessing the synchronized multimodal data is: Filter out the noisy multimodal data, normalize the multimodal data, and perform data enhancement on the original data; For example, operations such as rotation and scaling can be performed on visual data, or noise can be added to vibration data to generate more training samples; S230, using a deep learning algorithm to extract features from the multimodal processing data to perform defect identification and obtain a single defect result; Specifically, the logic of using deep learning algorithms to extract features from multimodal processing data for defect identification is: Q1, analyze visual data through convolutional neural networks to extract visual features related to defects; Q2, analyze the acoustic wave data through a recurrent neural network to identify the acoustic wave features in the abnormal acoustic wave signal; Q3, converts vibration data from the time domain to the frequency domain through fast Fourier transform, extracting spectral features related to equipment status and material properties; Q4, combining convolutional neural network and recurrent neural network, input the above extracted visual features, acoustic wave features and spectrum features into the classifier to identify the specific defect type and give a single defect result according to the severity of the defect; S240, performing fusion processing on the multi-modal processing data to generate a fusion defect result; Specifically, the logic of fusing the multimodal processing data to generate the fusion defect result is: a1, process the multimodal processing data into feature vectors, where each feature vector represents the defect feature of the corresponding multimodal processing data; a2, adaptively adjust the weights of different modalities based on the attention mechanism, determine the importance of a certain modality based on real-time data analysis, and modify the feature weights based on the established data weights; a3, let the visual feature vector be V, the sound wave feature vector be A, and the vibration feature vector be B. Based on the attention mechanism, the weights of different modalities are adaptively adjusted to αV, αA, and αB, respectively, where αV+αA+αB=0.8694; The eigenvectors of different modes are weighted and summed to obtain the fused eigenvector F. The logical formula of the fused eigenvector F is: F = V × αV + A × αA + B × αB; Set the comprehensive analysis threshold Fy, calculate the defect difference between F and Fy, and classify the defects based on the comprehensive analysis; S250, performing data comparison based on the single defect result and the fusion defect result to obtain a final defect result; Specifically, the logic for data comparison based on single defect results and fusion defect results is: If the defect detected by the single defect result is the same as the defect detected by the fusion defect result, it is marked as a high-confidence defect and the final defect result is output; if the defect detected by the single defect result is in conflict with the defect detected by the fusion defect result, it is marked as an unreliable defect and the defect detection is performed again.

[0025] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired network or a wireless network. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD) or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0026] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0027] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0028] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a multi-class defect identification system and method applied to intelligent manufacturing. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0029] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0030] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0031] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0032] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-class defect recognition system for intelligent manufacturing, characterized in that: The system comprises: Data synchronization module, used to collect multimodal data in the production process and synchronize the multimodal data; A preprocessing module, used for preprocessing the synchronized multimodal data to obtain multimodal processed data; An independent analysis module is used to extract features from multimodal processing data using deep learning algorithms to identify defects and obtain single defect results; A fusion analysis module is used to perform fusion processing on multi-modal processing data to generate fusion defect results; The defect verification module is used to compare data based on single defect results and fused defect results to obtain the final defect results.

2. According to claim 1, a multi-class defect recognition system for intelligent manufacturing is characterized in that: The multimodal data includes visual data, sound wave data and vibration data.

3. The multi-class defect recognition system for intelligent manufacturing according to claim 2 is characterized in that: The steps for data synchronization of multimodal data are: P1, through a high-precision time base, unifies the time of all sensors that collect multimodal data. Multimodal data is accompanied by precise timestamps to ensure that the data has the same time base; P2, through the preset hardware trigger mechanism, when the established trigger event of the production line occurs, the sensor that triggers the multimodal data collection is started, and the trigger event includes but is not limited to the product entering the detection area and the presence of the product passing on the conveyor belt; P3, aligns multimodal data based on timestamps. If the sampling frequencies of multiple sensors are different, time windows are used for processing, and all data are aggregated into the same time period for analysis and comparison.

4. The multi-class defect recognition system for intelligent manufacturing according to claim 3 is characterized in that: The logic for preprocessing the synchronized multimodal data is: The multimodal data containing noise is filtered out, the multimodal data is normalized, and the original data is enhanced.

5. The multi-class defect recognition system for intelligent manufacturing according to claim 4 is characterized in that: The logic of using deep learning algorithms to extract features from multimodal processing data for defect identification is: Q1, analyze visual data through convolutional neural networks to extract visual features related to defects; Q2, analyze the acoustic wave data through a recurrent neural network to identify the acoustic wave features in the abnormal acoustic wave signal; Q3, converts vibration data from the time domain to the frequency domain through fast Fourier transform, extracting spectral features related to equipment status and material properties; Q4, combining convolutional neural network and recurrent neural network, input the above extracted visual features, acoustic wave features and spectral features into the classifier to identify the specific defect type and give a single defect result according to the severity of the defect.

6. The multi-class defect recognition system for intelligent manufacturing according to claim 5, characterized in that: The logic of fusing multimodal processing data to generate fusion defect results is: a1, process the multimodal processing data into feature vectors, where each feature vector represents the defect feature of the corresponding multimodal processing data; a2, adaptively adjust the weights of different modalities based on the attention mechanism, determine the importance of a certain modality based on real-time data analysis, and modify the feature weights based on the established data weights; a3, let the visual feature vector be V, the sound wave feature vector be A, and the vibration feature vector be B. Based on the attention mechanism, the weights of different modalities are adaptively adjusted to αV, αA, and αB, respectively, where αV+αA+αB=0.8694; The eigenvectors of different modes are weighted and summed to obtain the fused eigenvector F. The logical formula of the fused eigenvector F is: F = V × αV + A × αA + B × αB; Set the comprehensive analysis threshold Fy, calculate the defect difference between F and Fy, and classify the defects by comprehensive analysis based on the defect difference.

7. The multi-class defect recognition system for intelligent manufacturing according to claim 6 is characterized in that: The logic for data comparison based on single defect results and fusion defect results is: If the single defect result is the same as the defect detected by the fusion defect result, it is marked as a high-confidence defect and the final defect result is output; If there is a conflict between the single defect result and the defect detected by the fusion defect result, it will be marked as an unreliable defect and the defect detection will be performed again.

8. A multi-class defect recognition method for intelligent manufacturing, applied to the multi-class defect recognition system for intelligent manufacturing as claimed in claim 1, characterized in that: The method comprises: Collect multimodal data during the production process and synchronize the multimodal data; Preprocessing the synchronized multimodal data to obtain multimodal processed data; Use deep learning algorithms to extract features from multimodal processing data for defect identification and obtain single defect results; Performing fusion processing on multi-modal processing data to generate fusion defect results; Data comparison is performed based on single defect results and fused defect results to obtain the final defect results.

Images