Multi-modal and visual model collaborative magnetic particle flaw detection method and system
Patent Information
- Application Number
- CN202610894981.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-22
- Publication Date
- 2026-10-30
AI Technical Summary
[0005]为了解决上述问题，本申请提供了一种多模态与视觉模型协同的磁粉探伤缺陷检测方法及系统，以解决现有技术中难以进行有效的磁粉探伤的问题
[0052]采用基于封闭集精确识别与开放集因果推理协同的分级检测策略，能够有效整合图像视觉特征与缺陷工程知识，实现从已知缺陷精确分类到未知异常智能判别的全流程覆盖。该系统通过置信度驱动的在线协同决策机制，显著提升了检测过程的自动化与智能化水平，降低了人工判读的主观性与劳动强度。同时，其内置的反馈优化闭环能够持续利用生产现场数据对模型进行迭代增强，从而在复杂的工业环境下实现高效、稳定、可进化的无损检测，满足了航空制造业对关键结构件高质量、高可靠性与智能化检测的迫切需求。
Smart Images
Figure CN123015364A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of aviation quality and reliability, and specifically relates to a method and system for detecting defects in magnetic particle testing using a multimodal and visual model collaboration. Background Technology
[0002] As the aerospace manufacturing industry moves towards digitalization and intelligence, higher demands are being placed on the non-destructive testing of key structural components. Magnetic particle testing, as a widely used technology, currently relies mainly on manual visual interpretation, which suffers from problems such as strong subjectivity, low efficiency, and poor stability, making it difficult to meet the demands of modern intelligent manufacturing for automated quality inspection.
[0003] To address the shortcomings of manual interpretation, the industry has introduced computer vision technology. Traditional visual inspection methods typically identify defects through image processing and feature engineering, but they face numerous challenges in practical applications. Magnetic particle inspection environments are complex, with diverse workpiece surface conditions and uneven lighting, resulting in complex and variable image backgrounds. Simultaneously, defect morphologies vary widely, from minute cracks to obvious magnetic traces, differing in contrast, size, and shape, making it difficult for features extracted using traditional methods to effectively distinguish them. Furthermore, traditional methods rely solely on image information, lacking the integration of knowledge about defect causal analysis. While commonly used visual neural network technologies have improved feature extraction capabilities to some extent, their application in this field remains limited by insufficient high-quality labeled data and the constraints of closed-set scenarios, presenting technical limitations in handling unknown defects.
[0004] Therefore, existing technologies lack an efficient and reliable magnetic particle inspection method that can detect open set defects and perform engineering analysis based on the causes of defects. Summary of the Invention
[0005] To address the aforementioned issues, this application provides a method and system for detecting magnetic particle defects using a multimodal and visual model collaboration approach, thereby resolving the problem of difficulty in performing effective magnetic particle testing in the prior art.
[0006] The technical solution of this application is: a magnetic particle inspection defect detection method that combines multimodal and visual models, comprising:
[0007] The defect data and image data in the original magnetic particle inspection dataset were preprocessed to construct a defect target formatted dataset for visual model training and a part surface image-text aligned dataset for fine-tuning of multimodal large model instructions.
[0008] We design a neural architecture search method to construct a visual feature recognition model for defect classification; we extract defect data from the defect target formatted dataset, train the visual features, and obtain the trained visual feature recognition model after convergence.
[0009] A modular parameter efficient fine-tuning strategy is adopted to construct a large-scale magnetic particle inspection health diagnosis model; image data is extracted from the part surface image-text alignment dataset for health diagnosis training, and the trained large-scale magnetic particle inspection health diagnosis model is obtained after convergence.
[0010] The magnetic particle inspection surface image of the part to be inspected is acquired. Through the trained visual feature recognition model and the magnetic particle inspection health diagnosis model, online collaborative detection is performed to identify and mark the suspected defect areas in the magnetic particle inspection surface image.
[0011] Preferably, the preprocessing of defect data and image data in the original magnetic particle inspection dataset includes:
[0012] The defect data includes defect regions of abnormal samples. The defect regions of abnormal samples in the original magnetic particle inspection dataset are labeled. The labeling information includes the bounding box coordinates and category labels of the defects, and a formatted dataset for defect target recognition is constructed.
[0013] For each image in the original magnetic particle inspection dataset, a standardized text description of the overall state and detailed features is performed; for abnormal images, the text description includes the type, shape, size and location information of the defect, and a part surface image-text aligned dataset is constructed.
[0014] Preferably, the original magnetic particle inspection dataset includes normal samples and abnormal samples of various typical defect types.
[0015] Preferably, the neural architecture search method is designed as follows:
[0016] Obtain the dynamic scaling weight factor of the visual feature recognition model A training loss function is established using a structured sparse constraint term R. Cross-layer connections are adapted through a cross-layer connection network structure, and then a resource constraint optimization objective is set for neural network training. After the training loss function converges, a neural network model under the resource constraint optimization objective is obtained.
[0017] Preferably, the weighting factor is dynamically scaled. During training, the optimal scaling factor for each network layer of the visual feature recognition model is adaptively learned in a continuously differentiable manner. The calculation formula is:
[0018]
[0019] in Let be the probability mapping function, and the scaling factor. Mapped in ( , The range of intervals; Minimum scaling factor Maximum scaling ratio;
[0020] Structured sparse constraint term R: based on the scaling factor For low-value characteristic channels Ln-norm sparse regularization is implemented, and redundant channels are gradually sparsified through iterative training. The specific formula for the sparse regularization term is as follows:
[0021] ;
[0022] In the formula, The total number of network layers. For layer index, For the first Total number of channels in the layer feature map; For the first Layer channel number, Representing the Layer, First The complete feature map tensor corresponding to each channel Let be the order of the regularization norm.
[0023] Preferably, cross-layer connection adaptation: the fused output feature norm is used as the compression basis for input-output dimension matching;
[0024] Resource-constrained optimization objective: Using the number of parameters or FLOPs as the resource constraint metric, the model complexity is controlled by the regularization term in the training loss function to generate a lightweight visual feature recognition model. The specific formula for the resource constraint term is as follows:
[0025] ;
[0026] in To balance hyperparameters, By dynamically scaling weight factor Determines the actual resource consumption of the model. To preset the target resource cost;
[0027] The training loss function is:
[0028] ;
[0029] in For network weights, The main loss function for the task. Control the strength of sparsity regularization.
[0030] Preferably, a large-scale model for health diagnosis of magnetic particle inspection is constructed, specifically as follows:
[0031] The visual encoder, visual-language adapter, and language model are trained and configured. Based on the actual task requirements and computing resources, specific network layer modules are frozen or fine-tuned, model weights are fine-tuned, and a multimodal large model is built.
[0032] A certain amount of knowledge in the field of magnetic particle inspection is acquired and input into a multimodal large model. The LoRA fine-tuning method is integrated, and the magnetic particle inspection knowledge is adapted and trained by setting hyperparameters, including adjusting the rank r and scaling factor. ;
[0033] Within the multimodal large model, a phased fine-tuning mechanism is adopted. In the first phase, the language model weights are frozen, and only the visual encoder and multimodal fusion module are fine-tuned to train on normal sample features. In the second phase, based on the accumulated anomaly detection data, some layer parameters of the language model are selectively unfrozen to train on the anomaly detection data.
[0034] After training on normal sample features and anomaly detection data, the multimodal large model is fine-tuned using the cross-entropy loss function based on the autoregressive language modeling task to obtain the magnetic particle inspection health diagnosis large model.
[0035] Preferably, online collaborative detection is performed, specifically as follows:
[0036] The input is defined as the magnetic particle inspection image to be inspected. The system's collaborative detection process is determined by the following decision function. describe:
[0037] ;
[0038] in The macroscopic state discriminant function of the large-scale magnetic particle inspection health diagnosis model satisfies:
[0039] ;
[0040] when When the system outputs "no defects", it directly outputs the conclusion; when... At that time, the system triggers the precise classification and localization process of the visual feature recognition model, which is handled by a function. express:
[0041] ;
[0042] Where U represents the set and N is the number of potential defective regions detected; Let i be the defect category predicted by the visual model for the i-th region. The set of known defect categories; This refers to the location information for this area; The final status of this area is determined by the following formula:
[0043] ;
[0044] in, The preset confidence threshold, is the prediction confidence level for the i-th defect region detected.
[0045] Preferably, the confirmation results of the "suspected" defect areas are used as newly labeled data and fed back to the defect target recognition formatted dataset and the part surface image-text alignment dataset, and used for periodic incremental optimization training of the magnetic particle inspection health diagnosis model and the visual feature recognition model.
[0046] Another technical solution of this application is: a multimodal and visual model collaborative magnetic particle inspection defect detection method, comprising:
[0047] The image acquisition and preprocessing module preprocesses the defect data and image data in the original magnetic particle inspection dataset, and constructs a defect target formatted dataset for visual model training, and a part surface image-text aligned dataset for fine-tuning of multimodal large model instructions.
[0048] The visual feature recognition module is used to design neural architecture search methods and build a visual feature recognition model for defect classification; it extracts defect data from the defect target formatted dataset, trains visual features, and obtains the trained visual feature recognition model after convergence.
[0049] The health diagnosis module is used to construct a large-scale health diagnosis model for magnetic particle inspection by adopting a modular parameter efficient fine-tuning strategy; it extracts image data from the part surface image-text alignment dataset, performs health diagnosis training, and obtains the trained large-scale health diagnosis model for magnetic particle inspection after convergence;
[0050] The online collaborative detection and decision-making module is used to acquire the magnetic particle inspection surface image of the part to be inspected, and to perform online collaborative detection through the trained visual feature recognition model and the magnetic particle inspection health diagnosis model to identify and mark the suspected defect areas in the magnetic particle inspection surface image.
[0051] The multimodal and visual model-based magnetic particle inspection defect detection method and system of this application have the following advantages:
[0052] Employing a hierarchical detection strategy based on precise closed-set identification and causal reasoning in open sets, this system effectively integrates image visual features with defect engineering knowledge, achieving full-process coverage from precise classification of known defects to intelligent identification of unknown anomalies. Through a confidence-driven online collaborative decision-making mechanism, the system significantly improves the automation and intelligence of the detection process, reducing the subjectivity and labor intensity of manual interpretation. Simultaneously, its built-in feedback optimization loop continuously utilizes production site data to iteratively enhance the model, thereby achieving efficient, stable, and evolvable non-destructive testing in complex industrial environments. This meets the urgent needs of the aerospace manufacturing industry for high-quality, high-reliability, and intelligent testing of critical structural components. Attached Figure Description
[0053] Figure 1 This is a flowchart of the overall system decision-making process for this application. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are only some, not all, of the embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0055] The first aspect of this application provides a multimodal and visual model-based magnetic particle inspection method for defect detection, such as... Figure 1 It includes the following steps:
[0056] Step S1, Image Acquisition and Preprocessing: Preprocess the defect data and image data in the original magnetic particle inspection dataset to construct a defect target formatted dataset for visual model training, and a part surface image-text aligned dataset for fine-tuning of multimodal large model instructions.
[0057] The original magnetic particle inspection dataset is a collection of part surface images acquired by a high-resolution industrial camera under standard industrial inspection conditions. It covers different workpiece batches, lighting conditions, and shooting angles, and includes normal (defect-free) samples and abnormal samples with various typical defect types.
[0058] Preferably, the preprocessing of defect data and image data in the original magnetic particle inspection dataset includes:
[0059] The defect data includes defect regions of abnormal samples. The defect regions of abnormal samples in the original magnetic particle inspection dataset are finely annotated. The annotation information includes the bounding box coordinates and category labels of the defects, and a formatted dataset for defect target recognition is constructed.
[0060] For each image in the original magnetic particle inspection dataset, a standardized text description of the overall state and detailed features is performed. For abnormal images, the text description includes information on the type, shape, size, and location of defects, as well as a qualitative analysis of possible causes. This process is used to construct a part surface image-text aligned dataset.
[0061] Step S2: Design a neural architecture search method to construct a visual feature recognition model for defect classification; extract defect data from the defect target formatted dataset, train visual features, and obtain the trained visual feature recognition model after convergence.
[0062] The visual feature recognition model is the ability to recognize features of a closed set of labeled defect types. The specific recognition content includes the type and coordinate information of the defect data image.
[0063] Preferably, the neural architecture search method is designed as follows:
[0064] Obtain the dynamic scaling weight factor of the visual feature recognition model A training loss function is established using a structured sparse constraint term R. Cross-layer connections are adapted through a cross-layer connection network structure, and then a resource constraint optimization objective is set for neural network training. After the training loss function converges, a neural network model under the resource constraint optimization objective is obtained.
[0065] Dynamic scaling weight factor During training, the optimal scaling factor for each network layer of the visual feature recognition model is adaptively learned in a continuously differentiable manner. The calculation formula is:
[0066]
[0067] in Let be the probability mapping function, and the scaling factor. Mapped in ( , The range of intervals; Minimum scaling factor Maximum scaling ratio;
[0068] Structured sparse constraint term R: based on scaling rate For low-value characteristic channels Ln-norm sparse regularization is implemented, and redundant channels are gradually sparsified through iterative training to enhance the effectiveness of preserved channels and achieve efficient single-pass compression. The specific formula for the sparse regularization term is as follows:
[0069] ;
[0070] In the formula, The total number of network layers. For layer index, For the first Total number of channels in the layer feature map; For the first Layer channel number, Representing the Layer, First The complete feature map tensor corresponding to each channel Let be the order of the regularization norm.
[0071] Cross-layer connection adaptation: The fusion output feature norm is used as the basis for compression to match the input and output dimensions, ensuring that the input and output dimensions of each module match after compression and avoiding feature fusion conflicts.
[0072] Resource constraint optimization objective: Using the number of parameters or FLOPs as the resource constraint index, the model complexity is controlled by the regularization term in the training loss function to balance classification accuracy and computational efficiency, generating a lightweight visual feature recognition model. The specific formula for the resource constraint term is as follows:
[0073] ;
[0074] in To balance hyperparameters, By dynamically scaling weight factor Determines the actual resource consumption of the model. To preset the target resource cost;
[0075] The training loss function is:
[0076] ;
[0077] in For network weights, The main loss function for the task. Control the strength of sparsity regularization.
[0078] Step S3: Using a modular parameter efficient fine-tuning strategy, construct a large-scale magnetic particle inspection health diagnosis model; extract image data from the part surface image-text alignment dataset, perform health diagnosis training, and obtain the trained large-scale magnetic particle inspection health diagnosis model after convergence.
[0079] Preferably, a large-scale model for health diagnosis of magnetic particle inspection is constructed, specifically as follows:
[0080] Modular and configurable training: The visual encoder, visual-language adapter and language model are trained and configured. Based on the actual task requirements and computing resources, specific network layer modules can be frozen or fine-tuned, model weights can be fine-tuned, and a multimodal large model can be built.
[0081] Low-rank adaptive technology integration: A certain amount of knowledge in the magnetic particle inspection domain, including open-set real-time images, is acquired and input into a multimodal large model. The LoRA fine-tuning method is integrated for macroscopic discrimination within the multimodal large model. Hyperparameters are set to adapt the model to the magnetic particle inspection domain knowledge, ensuring that the model efficiently learns the professional descriptive logic of industrial inspection scenarios while maintaining its general capabilities. Hyperparameters include adjusting the rank r and scaling factor. Hyperparameters, etc.
[0082] Progressive unfreezing strategy: Within the multimodal large model, a phased fine-tuning mechanism is adopted. In the first phase, the language model weights are frozen, and only the visual encoder and multimodal fusion module are fine-tuned to train on normal sample features. In the second phase, based on the accumulated anomaly detection data, some layer parameters of the language model are selectively unfrozen, and the model is trained on the anomaly detection data to achieve end-to-end optimization of anomaly description capabilities.
[0083] Loss function optimization: After training with normal sample features and anomaly detection data, based on the autoregressive language modeling task, the cross-entropy loss function is used to fine-tune the multimodal large model to obtain a large model for magnetic particle inspection health diagnosis.
[0084] Step S4: Obtain the magnetic particle inspection surface image of the part to be inspected, and perform online collaborative detection using the trained visual feature recognition model and the magnetic particle inspection health diagnosis model to identify and mark suspected defect areas in the magnetic particle inspection surface image.
[0085] Preferably, online collaborative detection is performed, specifically as follows:
[0086] The input is defined as the magnetic particle inspection image to be inspected. The system's collaborative detection process is determined by the following decision function. describe:
[0087] ;
[0088] in The macroscopic state discriminant function of the large-scale magnetic particle inspection health diagnosis model satisfies:
[0089] ;
[0090] when When the system outputs "no defects", it directly outputs the conclusion; when... At that time, the system triggers the precise classification and localization process of the visual feature recognition model, which is handled by a function. express:
[0091] ;
[0092] Where U represents the set and N is the number of potential defective regions detected; Let i be the defect category predicted by the visual model for the i-th region. The set of known defect categories; This refers to the location information for this area; The final status of this area is determined by the following formula:
[0093] ;
[0094] in, The preset confidence threshold, is the prediction confidence level for the i-th defect region detected.
[0095] The "pending suspected" defect areas are transferred to the manual judgment queue for judgment. The confirmed results are used as new labeled data and fed back to the defect target recognition formatted dataset and the part surface image-text alignment dataset. They are then used for periodic incremental optimization training of the magnetic particle inspection health diagnosis model and the visual feature recognition model to achieve closed-loop iteration and continuous performance improvement.
[0096] In summary, this application has the following advantages:
[0097] By annotating the raw magnetic particle inspection data, a closed-set defect identification dataset and an open-set image-text alignment dataset are established. Based on this, a visual feature recognition model for accurate defect classification is constructed using a neural architecture search method with resource constraints as the optimization objective. Simultaneously, instruction fine-tuning technology is used to optimize the multimodal large model, forming a large-scale magnetic particle inspection health diagnosis model. Finally, by establishing an online collaborative decision-making mechanism between the large-scale magnetic particle inspection health diagnosis model and the visual feature recognition model, automated intelligent diagnosis of magnetic particle inspection images is achieved, thereby significantly improving the reliability and stability of the inspection process.
[0098] This method effectively combines multimodal semantic understanding and visual feature extraction capabilities to achieve end-to-end intelligent processing from raw images to defect cause analysis. Through joint training with an open-set image-text alignment dataset and a closed-set defect recognition data, it significantly enhances the detection and generalization capabilities for unknown defect types while ensuring accurate identification of known defect types. This approach is suitable for complex and varied magnetic particle inspection scenarios, possesses strong engineering practicality and reliability, and can effectively improve the automation level of inspection and the stability of judgment.
[0099] The second aspect of this application provides a multimodal and visual model-based magnetic particle inspection defect detection method, including:
[0100] The image acquisition and preprocessing module preprocesses the defect data and image data in the original magnetic particle inspection dataset, and constructs a defect target formatted dataset for visual model training, and a part surface image-text aligned dataset for fine-tuning of multimodal large model instructions.
[0101] The visual feature recognition module is used to design neural architecture search methods and build a visual feature recognition model for defect classification; it extracts defect data from the defect target formatted dataset, trains visual features, and obtains the trained visual feature recognition model after convergence.
[0102] The health diagnosis module is used to construct a large-scale health diagnosis model for magnetic particle inspection by adopting a modular parameter efficient fine-tuning strategy; it extracts image data from the part surface image-text alignment dataset, performs health diagnosis training, and obtains the trained large-scale health diagnosis model for magnetic particle inspection after convergence;
[0103] The online collaborative detection and decision-making module is used to acquire the surface image of the part to be inspected by magnetic particle inspection. Through the trained visual feature recognition model and the large-scale magnetic particle inspection health diagnosis model, online collaborative detection is performed to identify and mark the suspected defect areas in the surface image of magnetic particle inspection.
[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for detecting defects in magnetic particle testing using a multimodal and visual model collaboration, characterized in that, include: The defect data and image data in the original magnetic particle inspection dataset were preprocessed to construct a defect target formatted dataset for visual model training and a part surface image-text aligned dataset for fine-tuning of multimodal large model instructions. We designed a neural architecture search method and constructed a visual feature recognition model for defect classification. Defect data is extracted from the defect target formatted dataset, visual feature training is performed, and the trained visual feature recognition model is obtained after convergence. A modular parameter efficient fine-tuning strategy is adopted to construct a large-scale magnetic particle inspection health diagnosis model; image data is extracted from the part surface image-text alignment dataset for health diagnosis training, and the trained large-scale magnetic particle inspection health diagnosis model is obtained after convergence. The magnetic particle inspection surface image of the part to be inspected is acquired. Through the trained visual feature recognition model and the magnetic particle inspection health diagnosis model, online collaborative detection is performed to identify and mark the suspected defect areas in the magnetic particle inspection surface image.
2. The magnetic particle inspection defect detection method based on multimodal and visual model collaboration as described in claim 1, characterized in that, The preprocessing of defect data and image data in the original magnetic particle inspection dataset is as follows: The defect data includes defect regions of abnormal samples. The defect regions of abnormal samples in the original magnetic particle inspection dataset are labeled. The labeling information includes the bounding box coordinates and category labels of the defects, and a formatted dataset for defect target recognition is constructed. For each image in the original magnetic particle inspection dataset, a standardized text description of the overall state and detailed features is performed; for abnormal images, the text description includes the type, shape, size and location information of the defect, and a part surface image-text aligned dataset is constructed.
3. The magnetic particle inspection defect detection method based on multimodal and visual model collaboration as described in claim 2, characterized in that, The original magnetic particle inspection dataset includes normal samples and abnormal samples of various typical defect types.
4. The magnetic particle inspection defect detection method based on multimodal and visual model collaboration as described in claim 1, characterized in that, The neural architecture search method is designed as follows: Obtain the dynamic scaling weight factor of the visual feature recognition model A training loss function is established using a structured sparse constraint term R. Cross-layer connections are adapted through a cross-layer connection network structure, and then a resource constraint optimization objective is set for neural network training. After the training loss function converges, a neural network model under the resource constraint optimization objective is obtained.
5. The magnetic particle inspection defect detection method based on multimodal and visual model collaboration as described in claim 4, characterized in that, Dynamic scaling weight factor During training, the optimal scaling factor for each network layer of the visual feature recognition model is adaptively learned in a continuously differentiable manner. The calculation formula is: ； in Let be the probability mapping function, and the scaling factor. Mapped in ( , The range of intervals; Minimum scaling factor Maximum scaling ratio; Structured sparse constraint term R: based on the scaling factor For low-value characteristic channels Ln-norm sparse regularization is implemented, and redundant channels are gradually sparsified through iterative training. The specific formula for the sparse regularization term is as follows: ； In the formula, The total number of network layers. For layer index, For the first Total number of channels in the layer feature map; For the first Layer channel number, Representing the Layer, First The complete feature map tensor corresponding to each channel Let be the order of the regularization norm.
6. The magnetic particle inspection defect detection method based on multimodal and visual model collaboration as described in claim 5, characterized in that, Cross-layer connectivity adaptation: The fusion output feature norm is used as the compression basis to match the input and output dimensions; Resource-constrained optimization objective: Using the number of parameters or FLOPs as the resource constraint metric, the model complexity is controlled by the regularization term in the training loss function to generate a lightweight visual feature recognition model. The specific formula for the resource constraint term is as follows: ； in To balance hyperparameters, By dynamically scaling weight factor Determines the actual resource consumption of the model. To preset the target resource cost; The training loss function is: ； in For network weights, The main loss function for the task. Control the strength of sparsity regularization.
7. The magnetic particle inspection defect detection method based on multimodal and visual model collaboration as described in claim 1, characterized in that, Construct a large-scale model for health diagnosis of magnetic particle inspection, specifically as follows: The visual encoder, visual-language adapter, and language model are trained and configured. Based on the actual task requirements and computing resources, specific network layer modules are frozen or fine-tuned, model weights are fine-tuned, and a multimodal large model is built. A certain amount of knowledge in the field of magnetic particle inspection is acquired and input into a multimodal large model. The LoRA fine-tuning method is integrated, and the magnetic particle inspection knowledge is adapted and trained by setting hyperparameters, including adjusting the rank r and scaling factor. ; Within the multimodal large model, a phased fine-tuning mechanism is adopted. In the first phase, the language model weights are frozen, and only the visual encoder and multimodal fusion module are fine-tuned, while training is performed on normal sample features. The second stage, based on the accumulated anomaly detection data, selectively unfreezes some layer parameters of the language model and trains the model on the anomaly detection data. After training on normal sample features and anomaly detection data, the multimodal large model is fine-tuned using the cross-entropy loss function based on the autoregressive language modeling task to obtain the magnetic particle inspection health diagnosis large model.
8. The magnetic particle inspection defect detection method based on multimodal and visual model collaboration as described in claim 1, characterized in that, Online collaborative detection is performed, specifically as follows: The input is defined as the magnetic particle inspection image to be inspected. The system's collaborative detection process is determined by the following decision function. describe: ； in The macroscopic state discriminant function of the large-scale magnetic particle inspection health diagnosis model satisfies: ； when When the system outputs "no defects", it directly outputs the conclusion; when... At that time, the system triggers the precise classification and localization process of the visual feature recognition model, which is handled by a function. express: ； Where U represents the set and N is the number of potential defective regions detected; Let i be the defect category predicted by the visual model for the i-th region. The set of known defect categories; This refers to the location information for this area; The final status of this area is determined by the following formula: ； in, The preset confidence threshold, is the prediction confidence level for the i-th defect region detected.
9. The magnetic particle inspection defect detection method based on multimodal and visual model collaboration as described in claim 8, characterized in that, The confirmation results of the "pending suspected" defect areas are used as new labeled data and fed back to the defect target recognition formatted dataset and the part surface image-text alignment dataset, and used for periodic incremental optimization training of the magnetic particle inspection health diagnosis model and the visual feature recognition model.
10. A method for detecting defects in magnetic particle inspection using a multimodal and visual model synergy, employing the method described in any one of claims 1-9, characterized in that, include: The image acquisition and preprocessing module preprocesses the defect data and image data in the original magnetic particle inspection dataset, and constructs a defect target formatted dataset for visual model training, and a part surface image-text aligned dataset for fine-tuning of multimodal large model instructions. The visual feature recognition module is used to design neural architecture search methods and build a visual feature recognition model for defect classification. Defect data is extracted from the defect target formatted dataset, visual feature training is performed, and the trained visual feature recognition model is obtained after convergence. The health diagnosis module is used to construct a large-scale health diagnosis model for magnetic particle inspection by adopting a modular parameter efficient fine-tuning strategy; it extracts image data from the part surface image-text alignment dataset, performs health diagnosis training, and obtains the trained large-scale health diagnosis model for magnetic particle inspection after convergence; The online collaborative detection and decision-making module is used to acquire the magnetic particle inspection surface image of the part to be inspected, and to perform online collaborative detection through the trained visual feature recognition model and the magnetic particle inspection health diagnosis model to identify and mark the suspected defect areas in the magnetic particle inspection surface image.