A cross-scale part multi-modal visual inspection method based on a CAV model
By constructing a multimodal dataset and an improved CAV model, the problems of accuracy, efficiency, and interpretability in cross-scale part inspection are solved, realizing high-precision, high-speed, and low-cost multimodal visual inspection, which is applicable to aerospace, automotive manufacturing, and other fields.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU PUHUI INTELLIGENT TECH CO LTD
- Filing Date
- 2026-03-18
- Publication Date
- 2026-06-23
AI Technical Summary
Existing technologies struggle to efficiently handle multimodal visual inspection of parts across scales, especially the collaborative inspection of minute defects and macroscopic structures, and existing methods cannot meet the requirements for real-time performance and interpretability.
A cross-scale annotation system is constructed, including multimodal datasets of 2D textures, 3D contours, and spectral images. An improved CAV model is used for feature extraction, fusion, and adaptation. Combined with language guidance and incremental training, efficient capture and interpretable output of multimodal features are achieved.
It achieves high-precision and high-speed inspection of cross-scale parts, with an inspection accuracy of 99%, a false negative rate of ≤0.5%, and a false positive rate of ≤0.3%. It adapts to the real-time inspection needs of industrialization, improves generalization ability by 50%, and reduces inspection costs and manual intervention.
Smart Images
Figure CN122265710A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of manufacturing technology, and in particular to a cross-scale multimodal visual inspection method for parts based on the CAV model. Background Technology
[0002] In the process of modern manufacturing transforming towards high precision, automation, and intelligence, the quality inspection of parts is a core link in ensuring product reliability. This is especially true for the inspection of cross-scale parts (i.e., parts that simultaneously possess minute features and macroscopic structures, such as the minute nozzles and overall annular structure of a liquid rocket engine injector, or the blade details and disk outline of an automotive engine turbine disk). This places extremely high demands on inspection accuracy, efficiency, and versatility. These parts are widely used in key fields such as aerospace and high-end equipment manufacturing. They have large size ranges, complex structures, and often exhibit various types of defects, including minute scratches, cracks, deformation, and assembly deviations. Any omissions or misjudgments during inspection can lead to complete machine failure, causing serious safety accidents and economic losses.
[0003] Currently, visual inspection methods for cross-scale parts are mainly divided into two categories: traditional inspection methods and deep learning-based inspection methods. However, both have obvious technical bottlenecks and cannot meet the requirements of high precision, high efficiency, and high versatility in actual production.
[0004] Traditional inspection methods mainly include manual visual inspection, mechanical contact inspection, and single-modal optical inspection. Manual visual inspection relies on the experience of the inspectors, is labor-intensive, inefficient, and easily affected by subjective factors. Its accuracy in identifying millimeter-level defects is extremely low, making it unsuitable for large-scale assembly line production. Mechanical contact inspection (such as coordinate measuring machines) can guarantee high accuracy, but it is slow, expensive, and poses a risk of damaging the part surface, making it unsuitable for inspecting fragile or easily scratched precision parts. It also struggles to detect complex curved surfaces and internal defects. Single-modal optical inspection (such as ordinary visual imaging and laser scanning) can only acquire single-dimensional information about the part (such as two-dimensional image texture or three-dimensional contour), failing to comprehensively characterize the multi-dimensional features of parts across scales. It has poor adaptability to scenarios involving surface reflection, occlusion, and complex textures, easily leading to missed or false defects. Especially in scenarios involving the simultaneous inspection of minute defects and macroscopic structures, it cannot balance inspection accuracy and efficiency.
[0005] Deep learning-based detection methods (such as YOLO and Faster R-CNN) have been widely used in the field of parts inspection in recent years, improving detection efficiency and accuracy to some extent. However, they still have many shortcomings for cross-scale parts inspection. On the one hand, existing deep learning models are mostly trained using single-modal data, lacking the fusion and utilization of multi-dimensional information of parts (such as two-dimensional texture, three-dimensional contour, and spectral features). This makes it difficult to comprehensively capture the feature differences of parts across scales, resulting in insufficient sensitivity for identifying minute defects and insufficient accuracy in locating macroscopic structural anomalies. On the other hand, the feature extraction process of existing models lacks interpretability, and the model decision-making process is difficult to trace. Once a misjudgment occurs, it is impossible to quickly locate the root cause of the problem, which is not conducive to the verification of detection results and the optimization of the model. At the same time, existing deep learning models have poor adaptability to cross-scale features. They usually need to be trained separately for parts or defects of different scales, resulting in weak generalization ability. When the type, size, or defect morphology of the part changes, the model needs to be retrained, increasing the detection cost and cycle.
[0006] As a core technology of interpretable artificial intelligence (XAI), the CAV model (Concept Activation Vector Model) quantifies the model's sensitivity to specific concepts by associating the high-dimensional internal states of a neural network with human-understandable concepts, providing interpretable evidence for model decisions. It has shown promising application potential in fields such as image classification and medical image analysis. However, current CAV model applications are mainly concentrated in image classification and concept interpretation, and have not yet been applied to the field of multimodal visual inspection of cross-scale parts, nor have they been specifically optimized for the characteristics and inspection requirements of cross-scale parts. In existing technologies, CAV model training typically requires a large amount of high-quality labeled data, resulting in high data labeling costs and difficulty in adapting to multimodal data fusion scenarios for cross-scale parts. Furthermore, existing CAV models cannot effectively handle the feature scale differences of cross-scale parts, making it difficult to achieve collaborative detection of minute defects and macroscopic structures, and thus failing to address the multiple pain points in accuracy, efficiency, universality, and interpretability in current cross-scale part inspection.
[0007] Furthermore, industrial production lines have high requirements for real-time inspection, typically needing to complete the inspection of a single part within 10-50ms. Existing inspection methods either struggle to balance real-time performance and accuracy or cannot adapt to the dynamic changes in the production line. Therefore, developing a cross-scale visual inspection method for parts that can integrate multimodal data, adapt to cross-scale features, possess high interpretability and generalization ability, and meet real-time inspection needs has become an urgent requirement for the high-quality development of the manufacturing industry. Summary of the Invention
[0008] In view of the problems mentioned in the background art, the purpose of this invention is to provide a cross-scale multimodal visual inspection method for parts based on the CAV model, so as to solve the problems mentioned in the background art.
[0009] The above-mentioned technical objective of the present invention is achieved through the following technical solution: a cross-scale multimodal visual inspection method for parts based on the CAV model, comprising the following steps: S1, constructing a cross-scale annotation system: constructing a cross-scale multimodal dataset for parts, wherein the multimodal dataset includes two-dimensional texture images, three-dimensional contour images, and spectral images of the parts, and each image sample is labeled with defect type, defect location, defect size, and macroscopic structural parameters of the parts, wherein the defect type includes micro scratches, cracks, dents, and assembly deviations, and the defect size covers millimeter to centimeter levels, forming a cross-scale annotation system.
[0010] S2. Construct an improved CAV model based on language guidance: The improved CAV model includes a feature extraction module, a multimodal fusion module, a CAV concept learning module, a cross-scale feature adaptation module, and a detection decision module.
[0011] S3. Model Training: The improved CAV model is trained using the multimodal dataset constructed in step S1. CAV vectors of part detection-related concepts are generated through language guidance, and model parameters are optimized so that the model can accurately capture multimodal features and defect features of parts across scales.
[0012] S4. Data Acquisition and Preprocessing: Multimodal data acquisition is performed on the multi-scale parts to be tested, acquiring two-dimensional texture images, three-dimensional contour images, and spectral images of the parts to be tested. The acquired multimodal data is preprocessed to eliminate noise, correct image distortion, and unify image size and data format.
[0013] S5. Feature Fusion: Input the preprocessed multimodal data into the trained improved CAV model, extract the features of each modality data through the feature extraction module, and fuse the multimodal features through the multimodal fusion module to obtain the fused feature vector.
[0014] S6. Normalization Processing: The fused feature vector is normalized by the cross-scale feature adaptation module to adapt to part features and defect features of different scales. The similarity between the fused feature vector and each preset defect concept and part structure concept is calculated by the CAV concept learning module.
[0015] S7. Interpretable Output: Based on the similarity results and a preset threshold, the detection decision module determines whether the part under test has defects, the type of defects, the location of defects, and the size of defects. At the same time, it outputs the activation value of the CAV vector to achieve interpretable output of the detection results.
[0016] S8. Incremental Training: Verify and optimize the detection results. If there are misjudgments or missed detections in the detection results, supplement the corresponding multimodal data and correct annotation information into the multimodal dataset and perform incremental training on the improved CAV model.
[0017] Preferably, in step S1, the construction process of the multimodal dataset includes: acquiring two-dimensional texture images, three-dimensional contour images, and spectral images of the parts using an industrial camera, a grating-type three-dimensional contour scanning probe, and a spectral imager, respectively; performing noise reduction, enhancement, and distortion correction preprocessing on the original images; using a combination of manual and automatic annotation for annotation, with magnified annotation for minor defects and overall annotation for macroscopic structures; dividing the dataset into training, validation, and test sets in a 7:2:1 ratio, and expanding the training set using data augmentation techniques.
[0018] Preferably, in step S2, the feature extraction module includes three parallel feature extraction branches: a two-dimensional texture feature extraction branch, a three-dimensional contour feature extraction branch, and a spectral feature extraction branch. The two-dimensional texture feature extraction branch uses a CNN network with an added attention mechanism, the three-dimensional contour feature extraction branch uses a PointNet network, and the spectral feature extraction branch uses a CNN-LSTM network.
[0019] Preferably, in S2, the multimodal fusion module adopts an attention fusion mechanism to calculate the weights of each modal feature and perform weighted fusion of the features extracted from the three branches; the cross-scale feature adaptation module adopts an adaptive scale normalization algorithm to perform multi-scale mapping on the fused feature vector and adapt it to part features and defect features of different scales.
[0020] Preferably, in step S2, the CAV concept learning module adopts a language-guided approach, utilizes a pre-trained visual language model to generate CAV vectors for part detection-related concepts, and quantifies the degree of association between features and concepts by calculating the directional derivatives of feature vectors and CAV vectors.
[0021] Preferably, in S3, the model training process includes: first, pre-training the three branches of the feature extraction module separately, and then connecting them to other modules for joint training; during the training process, the activation sample reweighting technique is used to correct the model prediction results, and the model parameters are adjusted through the validation set. When the detection accuracy of the validation set is ≥99% and the single frame detection time is ≤30ms, the training is stopped.
[0022] Preferably, in step S4, the multimodal data preprocessing includes: using Gaussian filtering and median filtering to eliminate noise; using perspective transformation to correct image distortion; performing point cloud registration on the three-dimensional contour image and normalizing the spectral image; and converting the preprocessed multimodal data into a tensor format recognizable by the model.
[0023] Preferably, in S7, the interpretability output of the detection results includes: outputting the CAV vector activation value corresponding to each defect concept, the similarity heatmap between the feature vector and each CAV vector, the defect location coordinates, the defect size parameters, and the deviation value between the macroscopic structure of the part and the standard model.
[0024] In summary, the present invention has the following main advantages: The invention significantly improves detection accuracy, achieving precise identification of cross-scale defects. By fusing three modal data—two-dimensional texture, three-dimensional contour, and spectral data—the invention can comprehensively capture the multi-dimensional features of cross-scale parts, overcoming the limitation of single-modal detection in fully representing part features. Simultaneously, the improved CAV model generates concept vectors through language guidance, accurately associating part defect features with human-understandable concepts. Combined with the adaptive scale normalization processing of the cross-scale feature adaptation module, it achieves simultaneous and precise detection of millimeter-level micro-defects (such as 0.1mm micro-scratches) and centimeter-level macro-structural anomalies (such as 5cm dents and assembly deviations). The detection accuracy reaches over 99%, with a false negative rate ≤0.5% and a false positive rate ≤0.3%. Compared to existing single-modal detection methods, the detection accuracy is improved by over 30%, and compared to traditional deep learning detection methods, the accuracy of micro-defect identification is improved by over 25%, effectively solving the problem of existing methods "focusing on the large and neglecting the small" or "paying attention to one aspect while neglecting another."
[0025] This invention significantly improves detection efficiency, meeting the real-time detection needs of industrial applications. By optimizing the model structure and employing parallel feature extraction branches and attention fusion mechanisms, the computational load of feature extraction and fusion is reduced. Simultaneously, incremental training avoids repetitive model training, achieving a single-frame detection time of ≤30ms, adaptable to production line speeds of 2-5m / s. Compared to traditional mechanical contact detection (such as coordinate measuring machines), the detection efficiency is improved by more than 100 times, and compared to existing deep learning detection methods, the detection speed is improved by more than 40%. Furthermore, the multimodal data acquisition and preprocessing processes are automated, requiring no manual intervention, significantly reducing the labor intensity of inspection personnel. This enables real-time detection for large-scale assembly line production, such as reducing the full-parameter detection time for liquid rocket engine injector rings from 15 days for 2 people using traditional manual inspection to less than 3.5 hours using automated inspection.
[0026] This invention boasts strong generalization capabilities and is adaptable to the detection of various types of cross-scale parts. The multimodal dataset constructed in this invention covers multiple types of cross-scale parts and defects. Employing data augmentation techniques and a language-guided CAV training method, it reduces the model's dependence on labeled data. Simultaneously, the cross-scale feature adaptation module can adaptively adapt to cross-scale parts of different sizes and structures, eliminating the need for separate model training for different part types. The incremental training mechanism continuously optimizes model performance; when part type, size, or defect morphology changes, only a small amount of supplementary data is needed for incremental training to achieve model adaptation. Compared to existing detection methods, the generalization capability is improved by more than 50%, enabling its widespread application in the detection of various cross-scale parts in aerospace, automotive manufacturing, and precision instrument fields, such as key components like liquid rocket engine injectors, impellers, and automotive engine turbine disks. Attached Figure Description
[0027] Figure 1 This is a flowchart of the present invention. Detailed Implementation
[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0029] refer to Figure 1 A cross-scale multimodal visual inspection method for parts based on the CAV model includes the following steps: S1, constructing a cross-scale annotation system: constructing a cross-scale multimodal dataset for parts, wherein the multimodal dataset includes two-dimensional texture images, three-dimensional contour images, and spectral images of the parts, and each image sample is labeled with defect type, defect location, defect size, and macroscopic structural parameters of the parts. The defect types include micro scratches, cracks, dents, and assembly deviations, and the defect sizes cover millimeter to centimeter levels, forming a cross-scale annotation system.
[0030] S2. Construct an improved CAV model based on language guidance: The improved CAV model includes a feature extraction module, a multimodal fusion module, a CAV concept learning module, a cross-scale feature adaptation module, and a detection decision module.
[0031] S3. Model Training: The improved CAV model is trained using the multimodal dataset constructed in step S1. CAV vectors of part detection-related concepts are generated through language guidance, and model parameters are optimized so that the model can accurately capture multimodal features and defect features of parts across scales.
[0032] S4. Data Acquisition and Preprocessing: Multimodal data acquisition is performed on the multi-scale parts to be tested, acquiring two-dimensional texture images, three-dimensional contour images, and spectral images of the parts to be tested. The acquired multimodal data is preprocessed to eliminate noise, correct image distortion, and unify image size and data format.
[0033] S5. Feature Fusion: Input the preprocessed multimodal data into the trained improved CAV model, extract the features of each modality data through the feature extraction module, and fuse the multimodal features through the multimodal fusion module to obtain the fused feature vector.
[0034] S6. Normalization Processing: The fused feature vector is normalized by the cross-scale feature adaptation module to adapt to part features and defect features of different scales. The similarity between the fused feature vector and each preset defect concept and part structure concept is calculated by the CAV concept learning module.
[0035] S7. Interpretable Output: Based on the similarity results and a preset threshold, the detection decision module determines whether the part under test has defects, the type of defects, the location of defects, and the size of defects. At the same time, it outputs the activation value of the CAV vector to achieve interpretable output of the detection results.
[0036] S8. Incremental Training: Verify and optimize the detection results. If there are misjudgments or missed detections in the detection results, supplement the corresponding multimodal data and correct annotation information into the multimodal dataset and perform incremental training on the improved CAV model.
[0037] In S1, the construction process of the multimodal dataset includes: acquiring two-dimensional texture images, three-dimensional contour images, and spectral images of the parts using an industrial camera, a grating-type three-dimensional contour scanning probe, and a spectral imager, respectively; preprocessing the original images by denoising, enhancing, and correcting distortion; labeling using a combination of manual and automatic annotation, with magnified annotation for minor defects and overall annotation for macroscopic structures; dividing the dataset into training, validation, and test sets in a 7:2:1 ratio, and expanding the training set using data augmentation techniques.
[0038] In S2, the feature extraction module includes three parallel feature extraction branches: a two-dimensional texture feature extraction branch, a three-dimensional contour feature extraction branch, and a spectral feature extraction branch. The two-dimensional texture feature extraction branch uses a CNN network with an added attention mechanism, the three-dimensional contour feature extraction branch uses a PointNet network, and the spectral feature extraction branch uses a CNN-LSTM network.
[0039] In S2, the multimodal fusion module uses an attention fusion mechanism to calculate the weights of each modal feature and perform weighted fusion of the features extracted from the three branches; the cross-scale feature adaptation module uses an adaptive scale normalization algorithm to perform multi-scale mapping on the fused feature vector to adapt to part features and defect features of different scales.
[0040] In S2, the CAV concept learning module adopts a language-guided approach, uses a pre-trained visual language model to generate CAV vectors for part detection-related concepts, and quantifies the degree of association between features and concepts by calculating the directional derivatives of feature vectors and CAV vectors.
[0041] In S3, the model training process includes: first, pre-training the three branches of the feature extraction module separately, and then connecting them to other modules for joint training; during the training process, the activation sample reweighting technique is used to correct the model prediction results, and the model parameters are adjusted through the validation set. When the validation set detection accuracy is ≥99% and the single frame detection time is ≤30ms, the training is stopped.
[0042] In S4, the multimodal data preprocessing includes: using Gaussian filtering and median filtering to eliminate noise; using perspective transformation to correct image distortion; performing point cloud registration on the three-dimensional contour image and normalizing the spectral image; and converting the preprocessed multimodal data into a tensor format that the model can recognize.
[0043] In S7, the interpretability output of the detection results includes: outputting the CAV vector activation value corresponding to each defect concept, the similarity heatmap between the feature vector and each CAV vector, the defect location coordinates, the defect size parameters, and the deviation value between the macroscopic structure of the part and the standard model.
[0044] This invention significantly improves detection accuracy, enabling precise identification of defects across scales. By fusing three modal data—two-dimensional texture, three-dimensional contour, and spectral data—it can comprehensively capture the multi-dimensional features of parts across scales, overcoming the limitation of single-modal detection in fully representing part features. Simultaneously, the improved CAV model generates concept vectors through language guidance, accurately associating part defect features with human-understandable concepts. Combined with the adaptive scale normalization processing of the cross-scale feature adaptation module, it achieves simultaneous and precise detection of millimeter-level micro-defects (such as 0.1mm scratches) and centimeter-level macro-structural anomalies (such as 5cm dents or assembly deviations). The detection accuracy reaches over 99%, with a false negative rate of ≤0.5% and a false positive rate of ≤0.3%. Compared to existing single-modal detection methods, the detection accuracy is improved by over 30%, and compared to traditional deep learning detection methods, the accuracy of micro-defect identification is improved by over 25%, effectively solving the problem of existing methods "focusing on the large and neglecting the small" or "paying attention to one aspect while neglecting another."
[0045] This invention significantly improves detection efficiency, meeting the real-time detection needs of industrial applications. By optimizing the model structure and employing parallel feature extraction branches and attention fusion mechanisms, the computational load of feature extraction and fusion is reduced. Simultaneously, incremental training avoids repetitive model training, achieving a single-frame detection time of ≤30ms, adaptable to production line speeds of 2-5m / s. Compared to traditional mechanical contact detection (such as coordinate measuring machines), the detection efficiency is improved by more than 100 times, and compared to existing deep learning detection methods, the detection speed is improved by more than 40%. Furthermore, the multimodal data acquisition and preprocessing processes are automated, requiring no manual intervention, significantly reducing the labor intensity of inspection personnel. This enables real-time detection for large-scale assembly line production, such as reducing the full-parameter detection time for liquid rocket engine injector rings from 15 days for 2 people using traditional manual inspection to less than 3.5 hours using automated inspection.
[0046] This invention boasts strong generalization capabilities and is adaptable to the detection of various types of cross-scale parts. The multimodal dataset constructed in this invention covers multiple types of cross-scale parts and defects. Employing data augmentation techniques and a language-guided CAV training method, it reduces the model's dependence on labeled data. Simultaneously, the cross-scale feature adaptation module can adaptively adapt to cross-scale parts of different sizes and structures, eliminating the need for separate model training for different part types. The incremental training mechanism continuously optimizes model performance; when part type, size, or defect morphology changes, only a small amount of supplementary data is needed for incremental training to achieve model adaptation. Compared to existing detection methods, the generalization capability is improved by more than 50%, enabling its widespread application in the detection of various cross-scale parts in aerospace, automotive manufacturing, and precision instrument fields, such as key components like liquid rocket engine injectors, impellers, and automotive engine turbine disks.
[0047] The present invention offers several advantages. Firstly, its detection results are highly interpretable, facilitating defect tracing and quality improvement. Existing deep learning detection models are "black box models," making the decision-making process difficult to trace. The improved CAV model of this invention, by outputting the activation values of CAV vectors and similarity heatmaps, can intuitively demonstrate the model's decision-making basis, clearly defining the correlation between the characteristics of the tested part and the defect concept, thus making the detection results interpretable and verifiable. Secondly, the output defect location coordinates, dimensional parameters, and macroscopic structural deviation values provide precise data support for subsequent defect repair and production process optimization, helping companies quickly locate problems in the production process, reduce product defect rates, and improve product quality.
[0048] This invention reduces detection costs and improves the practicality and economy of detection: It employs a language-guided CAV training method, eliminating the need for large amounts of high-quality labeled data and reducing data labeling costs; the model training and detection processes are automated, reducing manual intervention and labor costs; simultaneously, the multimodal data acquisition equipment uses commonly used industrial equipment, making it easy to deploy and maintain, and reducing equipment costs by more than 60% compared to traditional high-precision detection equipment (such as coordinate measuring machines); the incremental training mechanism reduces model update time and costs, further improving the economy of the detection method and facilitating its application in small and medium-sized enterprises.
[0049] Among them, the present invention has strong adaptability and can cope with complex detection scenarios: through multimodal data fusion, the present invention can effectively cope with complex detection scenarios such as surface reflection, occlusion, and complex textures of parts, overcoming the problem of poor adaptability of single-modal detection in complex scenarios; at the same time, noise elimination, distortion correction and other operations in the preprocessing process further improve the model's adaptability to complex environments, and can operate stably in industrial production environments with different lighting and temperatures, improving detection stability by more than 45%.
[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A multi-modal visual inspection method for cross-scale parts based on the CAV model, characterized in that: Includes the following steps: S1. Construct a cross-scale annotation system: Construct a cross-scale multimodal dataset of parts. The multimodal dataset includes two-dimensional texture images, three-dimensional contour images, and spectral images of the parts. Each image sample is labeled with defect type, defect location, defect size, and macroscopic structural parameters of the parts. The defect types include micro scratches, cracks, dents, and assembly deviations. The defect sizes cover millimeter to centimeter levels, forming a cross-scale annotation system. S2. Construct an improved CAV model based on language guidance: The improved CAV model includes a feature extraction module, a multimodal fusion module, a CAV concept learning module, a cross-scale feature adaptation module, and a detection decision module; S3. Model Training: The improved CAV model is trained using the multimodal dataset constructed in step S1. CAV vectors of part detection-related concepts are generated through language guidance, and model parameters are optimized so that the model can accurately capture multimodal features and defect features of parts across scales. S4. Data Acquisition and Preprocessing: Multimodal data acquisition is performed on the multi-scale parts to be tested, acquiring two-dimensional texture images, three-dimensional contour images, and spectral images of the parts to be tested. The acquired multimodal data is preprocessed to eliminate noise, correct image distortion, and unify image size and data format. S5. Feature Fusion: Input the preprocessed multimodal data into the trained improved CAV model, extract the features of each modality data through the feature extraction module, and fuse the multimodal features through the multimodal fusion module to obtain the fused feature vector; S6. Normalization Processing: The fused feature vector is normalized by the cross-scale feature adaptation module to adapt to the part features and defect features of different scales. The similarity between the fused feature vector and each preset defect concept and part structure concept is calculated by the CAV concept learning module. S7. Interpretable Output: Based on the similarity results and a preset threshold, the detection decision module determines whether the part under test has defects, the type of defects, the location of defects, and the size of defects. At the same time, it outputs the activation value of the CAV vector to achieve interpretable output of the detection results. S8. Incremental Training: Verify and optimize the detection results. If there are misjudgments or missed detections in the detection results, supplement the corresponding multimodal data and correct annotation information into the multimodal dataset and perform incremental training on the improved CAV model.
2. The method for multimodal visual inspection of cross-scale parts based on the CAV model according to claim 1, characterized in that: In step S1, the construction process of the multimodal dataset includes: acquiring two-dimensional texture images, three-dimensional contour images, and spectral images of the parts using an industrial camera, a grating-type three-dimensional contour scanning probe, and a spectral imager, respectively; preprocessing the original images by denoising, enhancing, and correcting distortion; labeling using a combination of manual and automatic annotation, with magnified annotation for minor defects and overall annotation for macroscopic structures; dividing the dataset into training, validation, and test sets in a 7:2:1 ratio, and expanding the training set using data augmentation techniques.
3. The method for multimodal visual inspection of cross-scale parts based on the CAV model according to claim 1, characterized in that: In S2, the feature extraction module includes three parallel feature extraction branches: a two-dimensional texture feature extraction branch, a three-dimensional contour feature extraction branch, and a spectral feature extraction branch. The 2D texture feature extraction branch uses a CNN network with an added attention mechanism, the 3D contour feature extraction branch uses a PointNet network, and the spectral feature extraction branch uses a CNN-LSTM network.
4. The method for multimodal visual inspection of cross-scale parts based on the CAV model according to claim 1, characterized in that: In S2, the multimodal fusion module adopts an attention fusion mechanism to calculate the weight of each modal feature and perform weighted fusion of the features extracted from the three branches; the cross-scale feature adaptation module adopts an adaptive scale normalization algorithm to perform multi-scale mapping on the fused feature vector and adapt to part features and defect features of different scales.
5. The method for multimodal visual inspection of cross-scale parts based on the CAV model according to claim 1, characterized in that: In S2, the CAV concept learning module adopts a language-guided approach, uses a pre-trained visual language model to generate CAV vectors for part detection-related concepts, and quantifies the degree of association between features and concepts by calculating the directional derivatives of feature vectors and CAV vectors.
6. The method for multimodal visual inspection of cross-scale parts based on the CAV model according to claim 1, characterized in that: In S3, the model training process includes: first, pre-training the three branches of the feature extraction module separately, and then connecting them to other modules for joint training; during the training process, activation sample reweighting technology is used to correct the model prediction results, and the model parameters are adjusted through the validation set. When the validation set detection accuracy is ≥99% and the single frame detection time is ≤30ms, the training is stopped.
7. The method for multimodal visual inspection of cross-scale parts based on the CAV model according to claim 1, characterized in that: In step S4, the multimodal data preprocessing includes: eliminating noise by using Gaussian filtering and median filtering; correcting image distortion by using perspective transformation; performing point cloud registration on the three-dimensional contour image and normalizing the spectral image; and converting the preprocessed multimodal data into a tensor format that the model can recognize.
8. The method for multimodal visual inspection of cross-scale parts based on the CAV model according to claim 1, characterized in that: In S7, the interpretability output of the detection results includes: outputting the CAV vector activation value corresponding to each defect concept, the similarity heatmap between the feature vector and each CAV vector, the defect location coordinates, the defect size parameters, and the deviation value between the macroscopic structure of the part and the standard model.