Multimodal industrial defect detection method, apparatus, device, and storage medium

Abstract

The application provides a multi-modal industrial defect detection method, device, equipment and storage medium, and relates to the technical field of industrial anomaly detection. The method comprises the following steps: performing two-dimensional contour registration on an RGB image and a CAD model of a part to be detected to generate an aligned sample information file; filling a five-section thinking chain template including positioning, evidence, process, root cause and suggestion to obtain a thinking chain example set file; dividing the thinking chain example set file into multiple domain subsets according to part types, each domain subset comprising at least two examples, and respectively extracting a text input identification sequence and an image pixel tensor to form a data set; using the data set to supervise fine-tuning of a large model to obtain a fine-tuned large model; using a reference-query comparison sample pair to reinforce fine-tuning of the fine-tuned large model to obtain a multi-modal industrial defect detection model, and realizing detection of the part to be detected. The method can significantly improve the positioning accuracy of small defects and the cross-line generalization ability.

Description

Technical Field

[0001] This application relates to the field of industrial anomaly detection technology, and in particular to a multimodal industrial defect detection method, apparatus, equipment and storage medium. Background Technology

[0002] Industrial defect detection plays a crucial role in today's 3C (computer, communication, and consumer electronics) manufacturing industry, and is widely used in various stages such as mid-frame stamping line screening, camera module lens inspection, PCB solder paste printing monitoring, and wafer defect screening. These applications rely on in-depth analysis of massive amounts of multimodal process data (appearance images, equipment parameters, operating logs, etc.) to accurately locate minute defects and identify their potential root causes, thereby ensuring consistent product yield and production cycle time. 3C component defects are complex in shape, span a wide range of dimensions, and often closely resemble normal textures. This requires inspection systems not only to see anomalies but also to understand the physical mechanisms underlying them, thus supporting engineers in making traceable quality decisions.

[0003] However, traditional industrial defect detection solutions still face significant challenges in terms of both understanding and generalization. The existing public benchmark, MVTec-AD, primarily contains only artificially created single defects, with a scale of just over 5,000 images. This leads to the saturation of mainstream deep learning models in terms of image-level metrics, making it difficult to evaluate the true performance of the algorithms. In contrast, the latest 3CAD dataset, directly derived from 3C production lines, contains 27,039 high-resolution images, 24 defect types, and allows for multiple defects to occur concurrently, significantly improving data realism and detection difficulty. On this dataset, many methods achieve a pixel-level average precision (AP) of less than 10%, highlighting their insufficient adaptability to small defects and scenarios with multiple defect types coexisting.

[0004] From a technological evolution perspective, industrial anomaly detection methods have generally gone through three generations:

[0005] The first generation, based on rules or traditional machine vision, relies on manual features and threshold judgments, is easily affected by changes in lighting and pose, and is difficult to extend to new defects.

[0006] The second generation, based on classic machine learning, uses algorithms such as SVM and random forest to classify handmade texture features. It requires a lot of parameter tuning for specific scenarios and is sensitive to data quality.

[0007] The third generation comprises unsupervised / weakly supervised methods based on deep learning, such as PaDiM (Embedded Distribution Modeling), FastFlow, and Teacher-Student Distillation (RD). These methods automatically extract features using convolutional or Transformer structures, improving detection accuracy, but still suffer from three shortcomings: semantic isolation—only outputting heatmaps, unable to explain the root causes of defects and their impact on subsequent processes; data saturation—achieving near-perfect scores on small-scale, single-defect datasets, but struggling to generalize to the complex conditions of real production lines; and high maintenance costs—extremely sensitive to drift in lighting, material, or equipment parameters, requiring frequent retraining.

[0008] Therefore, current industrial defect detection technologies have the following shortcomings:

[0009] 1) The lack of structured priors and traceable reasoning mechanisms leads to false positives, false negatives, and inconvenient diagnosis.

[0010] Current industrial defect detection methods primarily rely on image texture difference or simple heatmap output, failing to deeply integrate the CAD geometric alignment parameters, tolerance information, and complete process flow of parts. When the same part exhibits normal machining marks (such as tool marks or embossing) in different processes, or when surface texture changes due to batch material variations, the model is prone to misclassifying these legitimate features as defects; furthermore, it may miss defects when they are extremely small and diverse in shape. Moreover, these methods typically only output probabilities or masks, lacking the ability to reason along a chain of ideas, and cannot explain the specific process steps and physical causes corresponding to defects. This makes it difficult for engineers to conduct traceable and verifiable root cause diagnosis and process optimization based on the model results.

[0011] 2) Insufficient model generalization ability and high maintenance costs in multi-model and multi-process environments.

[0012] During product iterations, part design updates, or process parameter fine-tuning on 3C production lines, imaging conditions, geometric characteristics, and defect distribution often undergo significant changes. Existing supervised and weakly supervised inspection methods are sensitive to these domain shifts, typically requiring the re-collection of a large number of normal or labeled samples and complete network training. Methods relying on only a small number of samples or without labels struggle to fully utilize CAD and process priors, leading to significant performance degradation. Frequent retraining processes are not only time-consuming but also significantly increase labeling and computational resource costs, making it difficult to meet the demands of rapid production line changeovers and small-batch, multi-variety production. Summary of the Invention

[0013] This application provides a multimodal industrial defect detection method, apparatus, equipment, and storage medium that combines visual information with the process flow and introduces Chain-of-Thought (CoT) reasoning to gradually generate an interpretable reasoning trajectory from anomaly occurrence to root cause analysis and parameter tuning suggestions. Simultaneously, supervised fine-tuning rapidly aligns large model knowledge to the real 3CAD scene, and a reinforcement fine-tuning (GRPO) strategy combining IoU reward and inference chain formatting reward is used for continuous optimization, which can significantly improve the accuracy of small defect location and cross-production line generalization ability.

[0014] In a first aspect, this application provides a multimodal industrial defect detection method, including:

[0015] The RGB image and CAD model of the part to be inspected are acquired, and two-dimensional contour registration is performed on the RGB image and CAD model of the part to be inspected to generate an alignment sample information file; wherein, the RGB image includes the appearance image of the part, and the CAD model contains the geometric contour and tolerance information of the part;

[0016] A five-segment thinking chain template, including location, evidence, process, root cause, and suggestion, is constructed. The five-segment thinking chain template is then filled in based on the alignment sample information file to obtain a thinking chain example set file.

[0017] The thought chain example set file is divided into multiple domain subsets according to the part type. Each domain subset includes at least two examples, resulting in a domain-specific thought chain example file. The prompt word string of each example in the domain-specific thought chain example file is extracted, the prompt word string is encoded, and a domain label is inserted at the beginning of the encoded sequence to generate a text input identifier sequence. The aligned RGB image corresponding to the domain subset is preprocessed to obtain an image pixel tensor. The text input identifier sequence and the image pixel tensor are combined to form a dataset. The dataset is used to perform supervised fine-tuning on the large model to obtain the supervised fine-tuned large model.

[0018] A reference-query comparison sample pair is constructed, which includes a reference image and a query image. The reference image is an aligned RGB image of a normal part, and the query image is an aligned RGB image of a defective part from the same batch of parts. The reference-query comparison sample pair is used to enhance and fine-tune the supervised fine-tuning of the large model to obtain a multimodal industrial defect detection model. The detection of the part to be inspected is realized based on the multimodal industrial defect detection model.

[0019] In one possible design, the RGB image and CAD model of the part to be inspected are registered in two dimensions to generate an alignment sample information file, including:

[0020] The CAD model is projected into a binary contour map, and Canny edge detection is performed on the RGB image to generate a binary edge map. The pixel plane is determined based on the binary edge map. In the binary contour map, different pixel values ​​represent the background area and the target position, and the target position includes the edge or contour position of the part model.

[0021] Based on a set combination of parameters, a grid search is performed on the binarized contour map within the pixel plane. Using the pixel ratio and intersection-to-union (IoU) as the registration metric, the parameters that maximize the IoU from the set combination of parameters are selected as the optimal alignment parameters. The parameter combination includes translation parameters. and rotation parameters , The translation amount along the X and Y axes is represented by px, which stands for pixel and is used as the unit of measurement for the translation parameter.

[0022] Based on the optimal alignment parameters, the RGB image and CAD model of the part to be inspected are aligned to obtain a CAD contour overlay. If the IoU of the CAD contour overlay is greater than or equal to a set cross-comparison threshold, the RGB image, CAD contour overlay, optimal alignment parameters, and process information are encapsulated into an alignment sample information file. If the IoU of the CAD contour overlay is less than the set cross-comparison threshold, the current part to be inspected is marked as having failed alignment and a log is recorded. The process information includes process identifier, machine number, tool number, rotation speed, and / or temperature.

[0023] In one possible design, the formula for calculating the pixel ratio and intersection ratio is:

[0024] ;

[0025] In the formula, This represents the pixel-level intersection ratio between the CAD contour map and the edge detection map. This represents an overlay of CAD outlines. This represents the set of edge pixels obtained from the RGB image to be detected using an edge detection algorithm.

[0026] In one possible design, the five-segment thought chain template is populated based on the alignment sample information file to obtain a thought chain example set file, including:

[0027] The candidate box coordinates, optimal alignment parameters and process information are extracted from the alignment sample information file and filled into the positioning, evidence and process in the five-segment thinking chain template to obtain the filled template. The filled template is then encoded into a text input identifier sequence using BPE. At the same time, the RGB image is adjusted to the model preset size and obtained as a pixel tensor after center cropping and resampling.

[0028] The text input identifier sequence and pixel tensor are input into the large model. Based on the set generation hyperparameters, the large model's generation interface is called to complete the root causes and suggestions in the filled template, generating the analysis results. If generation fails, it is repeated up to n times. If all n generation attempts fail, a simplified thought chain containing localization and anomaly detection is output, and an inference error log file is recorded. The generation hyperparameters include temperature. , top_p, top_k and repetition_penalty, where n is a positive integer greater than 1.

[0029] In one possible design, the text input identifier sequence and image pixel tensor are combined to form a dataset, and the dataset is used to perform supervised fine-tuning on a large model to obtain a supervised fine-tuned large model, including:

[0030] A batch size is set, and the text input identifier sequence and image pixel tensor are input into the large model in batches. The total loss is calculated according to the set loss function. The model is trained by gradient descent to adapt it to the geometric and process characteristics of different parts, resulting in a supervised fine-tuned large model. The loss function is expressed as follows:

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] In the formula, For the first Example of the length of the thought chain. and These are the tokens of the thought chain and the label of the judgment, respectively; B is the batch size; i is the batch number; and t is the total number of tokens in the thought chain area of ​​the i-th sample. For the cross-entropy loss of the thought chain region, Let θ represent the conditional probability distribution of the model. It is the natural logarithm. For the sequence of generated tokens preceding the t-th token in the i-th sample, For the first Image pixel tensor For the cross-entropy loss of the answer region, For the complete thought chain text sequence of the i-th training sample, For intra-segment alignment loss, The segment numbers in the thought process template correspond sequentially to location, evidence, process, root cause, and suggestion. For the word set predicted by the model, For keyword set, To monitor the total loss during fine-tuning, This is the intra-segment alignment loss weight function.

[0036] In one possible design, the large model after supervised fine-tuning is enhanced and fine-tuned using the reference-query comparison samples to obtain a multimodal industrial defect detection model, including:

[0037] Set a composite reward function, expressed as follows:

[0038] ;

[0039] In the formula, Indicates a compound reward. , and These represent the weights of the IOU (Interval to Unit) location reward and the KL (Kidney to Unit) penalty, respectively.

[0040] For formatted rewards, a formatted reward is only awarded when the large model after supervised fine-tuning outputs all five segments completely and uses the preset correct judgment format.

[0041] The location reward for IOU is calculated using the following formula:

[0042] ;

[0043] In the formula, express, express, express;

[0044] The penalty for KL is calculated using the following formula:

[0045] ;

[0046] In the formula, For reference strategies on freezing, As the current strategy, The output action generated by the model at time step t. The time step index indicates which generation action or token it is. It is the natural logarithm;

[0047] For the same input, G candidate outputs are sampled from the current policy, and the composite reward for each input is calculated according to the composite reward function, and then normalized using the following formula:

[0048] ;

[0049] In the formula, The normalized composite reward for the i-th candidate output is... The composite reward for the i-th candidate output. Let be the average composite reward of the i-th candidate output, and j be the index of the candidate output. The composite reward for the j-th candidate output;

[0050] According to normalized composite reward The advantage is calculated using the following formula. :

[0051] ;

[0052] in , This indicates that the i-th candidate performs better than the group average, and the probability of generating this output by the current strategy needs to be increased. This indicates that the performance of the i-th candidate is below the group average, and it is necessary to suppress the tendency of the current strategy to generate such output;

[0053] The probability ratio is calculated using the following formula. :

[0054] ;

[0055] In the formula, For the i-th candidate output, Provide conditional inputs or prompt text when the model generates output. Generate candidate outputs for the current policy model when given input p. The probability, Generate candidate outputs for the old policy model with the same input p. The probability of;

[0056] Based on probability ratios and advantage Design a clipping function, which is expressed as follows:

[0057] ;

[0058] In the formula, The magnitude of probability change. This represents the probability ratio between the current strategy and the old strategy. This is the clipping function;

[0059] Based on the pruning function, the objective function for constructing a single candidate output is:

[0060] ;

[0061] In the formula, This is the tailored objective function value constructed for the i-th candidate output. It is a minimum value function;

[0062] Based on the objective function, the loss of G candidate outputs sampled from the same input is calculated using the following formula:

[0063] ;

[0064] In the formula, The total loss function is obtained by averaging the objective function values ​​of G candidate outputs for the same input.

[0065] Based on the calculated loss, the model parameters are adjusted. Update the policy and repeat the training at least once more until the policy performance converges or the predetermined number of iterations is reached.

[0066] In one possible design, based on the calculated loss, the model parameters are adjusted using the following formula. Update:

[0067] ;

[0068] In the formula, For learning rate, For weight decay, This is the descent gradient.

[0069] Secondly, this application provides a multimodal industrial defect detection device, the device comprising:

[0070] The data alignment module is configured to acquire the RGB image and CAD model of the part to be inspected, and perform two-dimensional contour registration on the RGB image and CAD model of the part to be inspected to generate an alignment sample information file; wherein, the RGB image includes the appearance image of the part, and the CAD model contains the geometric contour and tolerance information of the part;

[0071] The thought chain generation module is configured to construct a five-segment thought chain template including location, evidence, process, root cause, and suggestion. The five-segment thought chain template is populated based on the alignment sample information file to obtain a thought chain example set file.

[0072] The supervised fine-tuning module is configured to divide the thought chain example set file into multiple domain subsets according to part type, with each domain subset including at least two examples, to obtain a domain-specific thought chain example file. It extracts the prompt word string from each example in the domain-specific thought chain example file, encodes the prompt word string, and inserts a domain label at the beginning of the encoded sequence to generate a text input identifier sequence. It preprocesses the aligned RGB image corresponding to the domain subset to obtain an image pixel tensor. It combines the text input identifier sequence and the image pixel tensor to form a dataset, and uses this dataset to perform supervised fine-tuning on the large model, resulting in a supervised fine-tuned large model.

[0073] The enhancement and fine-tuning module is configured to construct a reference-query comparison sample pair, which includes a reference image and a query image. The reference image is an aligned RGB image of a normal part, and the query image is an aligned RGB image of a defective part from the same batch of parts. The reference-query comparison sample pair is used to enhance and fine-tune the large model after supervised fine-tuning to obtain a multimodal industrial defect detection model. The detection of the part to be inspected is realized based on the multimodal industrial defect detection model.

[0074] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the multimodal industrial defect detection method as described in the first aspect and various possible designs of the first aspect.

[0075] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the multimodal industrial defect detection method described in the first aspect and various possible designs of the first aspect.

[0076] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the multimodal industrial defect detection method described in the first aspect and various possible designs of the first aspect.

[0077] The multimodal industrial defect detection method, apparatus, equipment, and storage medium provided in this application have at least the following beneficial effects:

[0078] 1) This application structures CAD-image alignment parameters and process metadata into a five-segment thought chain, enabling the output of a complete chain-like diagnostic process in a single model inference, providing engineers with intuitive and readable root cause analysis and optimization suggestions. Compared to traditional industrial defect detection methods that only output confidence levels or heatmaps, this solution significantly reduces subsequent manual diagnostic costs and improves decision-making efficiency and quality traceability on the production floor.

[0079] 2) This application employs a two-stage fine-tuning strategy combining domain-specific SFT and GRPO with formatting and IOU rewards, enabling rapid model migration and online optimization for different part types and process scenarios. The first stage uses domain label injection and intra-segment alignment loss to ensure the model masteres the geometric and process characteristics of each model. The second stage utilizes a trimmed GRPO under a comparative prompt to continuously adjust the decoding strategy, simultaneously improving the format compliance of the thought chain output and the accuracy of defect localization. Compared to existing technologies that lack fine-grained domain adaptation and precise quantitative localization, this application maintains high accuracy and low maintenance costs during multi-model switching and on-site process changes. Attached Figure Description

[0080] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0081] Figure 1 A flowchart of a multimodal industrial defect detection method provided in this application embodiment;

[0082] Figure 2 A schematic diagram illustrating a fixed thought chain example input provided for an embodiment of this application;

[0083] Figure 3 When inputting a fixed thought chain example as an embodiment of this application, fill in the framework diagram;

[0084] Figure 4 A schematic diagram of a JSON object provided in an embodiment of this application;

[0085] Figure 5 A schematic diagram of the text input identifier sequence input_ids for inputting the large model Qwen2.5-VL, provided in an embodiment of this application;

[0086] Figure 6 Example diagram of Reference-Query pairs provided in embodiments of this application;

[0087] Figure 7 This is a structural diagram of the multimodal industrial defect detection device provided in the embodiments of this application.

[0088] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0089] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0090] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0091] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0092] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0093] This application provides a multimodal industrial defect detection method. Addressing the issues of pure visual models lacking CAD, tolerance, and process priors, and being prone to misjudgment and missed detection, this method proposes a geometry-process ternary semantic lattice-driven thought chain anomaly detection scheme. This scheme first maps the part's CAD depth / normal projection, tolerance grid, and process parameters into formatted vectors, then fuses them with visual ROI features in a dual-branch collaborative network to output BoundingBox, pixel mask, and geometric deviation. Subsequently, a traceable inference chain is generated through the CoT inference head according to location → evidence → root cause → process suggestion, achieving accurate and interpretable diagnosis of defect type, severity, and cause. To address the problems of frequent process iterations leading to high model retraining costs and insufficient cross-production line generalization, this method proposes a supervised-reinforcement dual-stage few-sample adaptive fine-tuning scheme. The scheme first uses 3CAD pixel annotations and CAD / process information to perform multi-task supervised fine-tuning to complete the domain alignment; then, under conditions of few or even no labeled samples, it adopts GRPO strategy gradient combined with IoU reward and inference chain logical consistency reward for reinforcement fine-tuning, continuously optimizing the model's robustness to unknown defect morphology and process drift online, and achieving rapid transfer without full retraining.

[0094] In its implementation, this method first uses visual-CAD contour alignment to precisely match the real-world images with the CAD model at the pixel level, and attaches complete process information to provide a unified geometric and semantic benchmark for subsequent inference. Then, in the Prompt-driven inference stage, structured templates are used to load location, evidence, and process information into the model, outputting readable results containing location and inference chains in real time. Next, supervised fine-tuning (SFT) is used to train the large model on specific tasks with domain-customized example data, enabling it to master the five-segment thought chain format and adapt to different part categories. Finally, in the reinforcement fine-tuning (GRPO) stage, a composite strategy of formatting rewards and IOU location rewards is used to optimize the decoding strategy online under a comparative Prompt, achieving high compliance and accurate location of the thought chain output.

[0095] Specifically, such as Figure 1 As shown, the multimodal industrial defect detection method includes the following steps S10 to S40.

[0096] S10: Acquire the RGB image and CAD model of the part to be inspected, and perform two-dimensional contour registration on the RGB image and CAD model of the part to be inspected to generate an alignment sample information file; wherein, the RGB image includes the appearance image of the part, and the CAD model contains the geometric contour and tolerance information of the part.

[0097] Step S10 is the step to achieve visual-CAD contour alignment. To ensure that subsequent reasoning is based on a consistent geometric benchmark, step S10 performs two-dimensional contour registration between the part's RGB photograph and its CAD model. Specifically, this is done by generating a black and white image from the CAD projection, where a pixel value of 1 (white) represents the edge or contour position of the part model, and a pixel value of 0 (black) represents the background area. This accurately depicts the CAD contour at the pixel level for registration with the image edges. After two-dimensional contour registration of the CAD model, the alignment parameters and process information are formatted and stored in the sample entry. After completing the CAD contour rendering, the workpiece's STP / STL format CAD model and the camera intrinsic parameter matrix K are read, and OpenCascade is used to project the CAD model into a binary contour image. Then, Canny edge detection is applied to the original RGB image to generate a binarized edge map. Subsequently, within the pixel plane... Translate and rotation The grid search uses pixel intersection-over-union ratio (IoU) as the registration metric to select the grid that maximizes IoU. This serves as the final alignment parameter. The formula for calculating the pixel ratio and intersection ratio is:

[0098] ;

[0099] In the formula, This represents the pixel-level intersection ratio between the CAD contour map and the edge detection map. This represents an overlay of CAD outlines. This represents the set of edge pixels obtained from the RGB image to be detected using an edge detection algorithm.

[0100] If the maximum IoU is less than 0.7, the sample is marked as an alignment failure and logged; otherwise, the original RGB image is saved. and Write the `cad_alignment` field to the `aligned_sample.json` file (alignment sample information file) and save it as the part's... Parameters are provided for direct use in subsequent processes. After obtaining the parameters, the process identifier (process_id), machine number (machine_id), tool number (tool_id), and key process parameters (including speed_rpm and temperature_temp_C) are read from aligned_sample.json and compared with... The alignment results are encapsulated together into a unified field called `alignment`. This field, along with the original RGB image, the overlaid CAD contour image, and the process information, is written to `aligned_sample.json`. If any required element is missing or fails format validation, it is moved to the `invalid / ` directory, indicating that the entry is invalid. After execution, each sample carries the original photo, the precisely aligned CAD contour and parameters, and complete process metadata, which can be directly used for subsequent multimodal detection and thought chain inference module calls.

[0101] S20: Construct a five-segment thinking chain template including location, evidence, process, root cause, and suggestion. Fill in the five-segment thinking chain template based on the alignment sample information file to obtain a thinking chain example set file.

[0102] The purpose of step S20 is to achieve prompt-driven localization and thought chain generation. After completing the vision-CAD alignment, step S20 will solidify the five-segment thought chain of localization → evidence → process → root cause → suggestion from the training example into the model capability, and generate the complete structure in real time during reasoning. In specific implementation, it can be implemented through the following steps S201-S202.

[0103] S201: Fixed thought chain example input.

[0104] Based on a template containing a five-part thought chain specification (location, evidence, process, root cause, recommendation) and JSON output format requirements, the tokens are encoded in one go using the Qwen2.5-VL BPE vocabulary (length ≤ 300 tokens). If the file is missing or the placeholders are insufficient, it is written to prompt_error.log as an exception handling.

[0105] Based on the aligned_samples.json output in step S10, read the candidate bounding box coordinates (x, y, w, h) for each sample and align the offsets. The process fields are process_id, machine_id, tool_id, speed_rpm, and temp_C. The normal / abnormal image identifiers in the aligned_sample.json file are based on the prompt template. <think> …< / think> <answer> …< / answer> Complete the input, and a diagram illustrating a fixed thought process chain example is shown below. Figure 2 As shown. This example set is called sft_cot_examples.json, and it contains approximately 3000 entries.

[0106] S202: Generate real-time thought chains.

[0107] Extract from the aligned_sample.json file of the new sample. Candidate bounding boxes (x, y, w, h) and process parameters: process_id, machine_id, tool_id, speed_rpm, temp_C, filled in as follows: Figure 3 The frame shown.

[0108] When filling in the framework, the values ​​and text from steps 1, 2, and 3 need to be filled into the first two paragraphs and the process paragraph, leaving the root cause and suggestion paragraphs empty. After obtaining the prompt, the complete prompt is encoded into input_ids (int32 sequence, length ≤ 1024) using the BPE vocabulary of Qwen2.5-VL, and [EOS] is appended to the end. When processing images, the RGB images are adjusted to the model's expected size (e.g., 1024×4 or 512×512), and necessary center cropping or resampling is performed to obtain pixel_values ​​(float32 tensor, [3,H,W]). To control the model output, maintain the fluency and logical consistency of the generated content, and avoid the model from getting stuck in repetition or excessive divergence, a retry strategy (increasing temperature, resampling) is adopted, retrying a maximum of two times; if it still fails, infer_error.log is recorded and a simplified version of the thought chain (only localization and judgment) is output. This is set accordingly. ,in =Number of generated tokens, where higher temperature T results in more diverse outputs; lower temperature results in more deterministic outputs, concentrated on high-probability words. Regarding hyperparameters, top_p=0.9 means random selection is only made from words that best cover 90% of the probability quality, ensuring no output repetition; top_k=50 restricts sampling to the top K words with the highest probabilities; repetition_penalty=1.1 raises the sampling threshold for already generated tokens, suppressing model repetition of the same content and making subsequent selections more inclined towards new words. Through these hyperparameters, we can find a balance between "diversity vs. stability": maintaining the fluency and logical consistency of generated content while avoiding model repetition or excessive divergence. During inference, input_ids and pixel_values ​​are input to processor.apply_chat_template, and .generate() is called to generate the tokens. The model gradually completes the root cause and suggestion segments until it encounters or exceeds max_new_tokens. After the model inference is complete, the parsed result is described as follows: Figure 4 The JSON object shown.

[0109] S30: Divide the thought chain example set file into multiple domain subsets according to the part type. Each domain subset includes at least two examples to obtain a domain-specific thought chain example file. Extract the prompt word string of each example in the domain-specific thought chain example file, encode the prompt word string, and insert a domain label at the beginning of the encoded sequence to generate a text input identifier sequence. Preprocess the aligned RGB image corresponding to the domain subset to obtain the image pixel tensor. Combine the text input identifier sequence and the image pixel tensor to form a dataset. Use the dataset to perform supervised fine-tuning on the large model to obtain the supervised fine-tuned large model.

[0110] Building upon steps S10 and S20, step S30 involves domain-specific fine-tuning to ensure that the large model not only grasps the thought chain structure but also adapts to the geometric and technological characteristics of different part types. It's important to note that the large model discussed here refers to an existing large model. Steps S30 and S40 in this embodiment aim to adjust the parameters of the large model through improvements in the training method. For example, the large model could be selected as Qwen2.5-VL.

[0111] In practice, step S30 includes the following steps S301-S304.

[0112] S301: Based on the part types in the 3CAD dataset (e.g., aluminum alloy housings and camera brackets), the samples are divided into several domain subsets. For each subset, using the alignment parameters from step S10 and the script template (five-segment thought chain template) from step S20, a domain-specific [database name] is generated. <think> …< / think> <answer> …< / answer> For example, ensure that the numerical values ​​in the location and evidence match the process descriptions of the part. Then, build approximately 500–800 examples for each domain subset, totaling about 3,000, and save them in the domain-specific thought chain example file sft_cot_{category}.json. This fine-tuning by part type allows the model to learn the distribution of common defects and process terminology for different parts, improving the adaptability of the reasoning.

[0113] S302: Extract the prompt string prompt_str from the example (including the mind chain template identifier). <think>and <answer>The text input identifier sequence `input_ids` is converted into a text input identifier sequence using BPE, with a domain tag token (e.g., `[DOMAIN_ALUMINUM]`) added at the beginning to inject domain information into the model context. The length is limited to 512 tokens; any exceeding this limit is truncated. Subsequently, the obtained prompt text is split into sub-tokens using the BPE tokenizer of Qwen2.5-VL, with a domain tag token (e.g., `[DOMAIN_ALUMINUM]`) inserted at the beginning of the sequence, followed by the ID of each word. This yields the text input identifier sequence `input_ids` used to input the large model Qwen2.5-VL, as detailed below. Figure 5 As shown. The total length L ≤ 512; if it exceeds this, it is truncated from back to front to ensure that the leading domain label, location, and evidence segments are preserved.

[0114] S303: After the text encoding is completed, the image is preprocessed.

[0115] The aligned RGB image is scaled down to 256 pixels on its shorter side while maintaining the aspect ratio. This ensures consistent relative scale across all areas of the image, preventing over-stretching. Then, a 1024×1024 pixel crop is taken from the center of the scaled image. Since defects often appear near the alignment box, the central area of ​​the 1024×1024 crop usually includes this box. If the alignment box deviates from the center, an additional ±20 pixel translation compensation is applied to the central cropped area based on the box coordinates (x, y, w, h) to ensure the defect is not cropped out. Finally, pixel normalization is performed according to the formula... Where I is the cropped pixel value, and μ and σ are the mean and standard deviation during model pre-training, respectively (e.g., μ=[0.485,0.456,0.406], σ=[0.229,0.224,0.225]). The formula outputs the result. The values ​​fall within the range of [-2.0, 2.0], which helps the model converge stably.

[0116] S304: Perform supervised fine-tuning on the large model to obtain the supervised fine-tuned large model.

[0117] Through the S302 and S303 processes, the neighborhood + Prompt text and the aligned image can be converted into the input_ids required by the large model Qwen2.5-VL while retaining the key information of the defects.

[0118] After constructing the input required for Qwen2.5-VL, set B=16, meaning each batch loads 16 examples simultaneously. Each example contains a text sequence. With image pixel tensor ,in .

[0119] Each forward pass will batch text Input the model to obtain the predicted text sequence For thought chain paragraphs ( <think>(area) and judgment paragraph ( <answer>Calculate the cross-entropy for each region:

[0120] ;

[0121] ;

[0122] in For the first Example of the length of the thought chain. and These are the tokens of the thought chain and the label of the judgment, respectively; B is the batch size; i is the batch number; and t is the total number of tokens in the thought chain area of ​​the i-th sample. For the cross-entropy loss of the thought chain region, Let θ represent the conditional probability distribution of the model. It is the natural logarithm. For the sequence of generated tokens preceding the t-th token in the i-th sample, For the first Image pixel tensor For the cross-entropy loss of the answer region, For the complete thought chain text sequence of the i-th training sample, For intra-segment alignment loss, The segment numbers in the thought process template correspond sequentially to location, evidence, process, root cause, and suggestion. For the word set predicted by the model, For keyword set, To monitor the total loss during fine-tuning, This is the intra-segment alignment loss weight function.

[0123] Next, consider the intra-segment alignment loss: for each example and each segment... Let the set of keywords that should be included in reality be... The model predicts the word set as follows: Then the intra-segment alignment loss for:

[0124] ;

[0125] The total loss can be obtained by weighting and combining the three losses. :

[0126] ;

[0127] The above loss calculation method not only... The two core segments are supervised using cross-entropy, and intra-segment alignment loss is also applied. Enforcing that each paragraph must include key figures and process terms ensures both the formatting and consistency of the thought process and improves the accuracy of information citation. This design retains the advantages of both formatting and correctness while meeting the stringent requirements of industrial defect detection scenarios for numerical values ​​and keywords, making the model output more interpretable and practical for engineering applications.

[0128] S40: Construct a reference-query comparison sample pair, which includes a reference image and a query image. The reference image is an aligned RGB image of a normal part, and the query image is an aligned RGB image of a defective part in the same batch. The reference-query comparison sample pair is used to enhance and fine-tune the large model after supervised fine-tuning to obtain a multimodal industrial defect detection model. The detection of the part to be inspected is realized based on the multimodal industrial defect detection model.

[0129] After completing the supervised fine-tuning, in order to further improve the logical coherence and accuracy of the thought chain generated by the model in real-time inference, step S40 uses Group Relative Policy Optimization (GRPO) for enhanced fine-tuning, which can be implemented through the following steps S401-S403.

[0130] S401: Construct data.

[0131] To ensure the model focuses on difference reasoning from normal to abnormal during the reinforcement phase, contrastive examples are used. First, Reference-Query pairs need to be constructed. The reference image is an aligned RGB image of a normal part; the query image is an aligned image of a part with defects from the same batch. A total of approximately 1000 sample pairs are generated, covering typical defect types such as scratches, dents, and cracks. An example of a Reference-Query pair is shown below. Figure 6 As shown.

[0132] S402: Set the reward function.

[0133] To balance the formatted compliance and defect localization accuracy of the thought chain output, and to maintain policy stability, the reward function is defined as consisting of three parts: formatting reward, IOU localization reward, and KL penalty. The reward function is defined as follows:

[0134] ;

[0135] in,

[0136] 1) Formatted rewards

[0137] Formatting ensures that the model output conforms to the thought chain format defined in step two, improving interpretability and consistency.

[0138] definition:

[0139]

[0140] This means that the model can only receive this reward if it outputs all five segments completely and uses the correct judgment format.

[0141] 2) IOU Location Rewards

[0142] To address the stringent positioning accuracy requirements in industrial scenarios, IOU is used to measure the overlap between the bounding box coordinates and the ground truth box in the "positioning" segment, enabling the model to provide a more accurate location description.

[0143] If the model location segment generation box is known With truth box ,but:

[0144] ;

[0145] The larger the IOU, the more accurate the positioning and the higher the reward.

[0146] 3) KL Penalty

[0147] To maintain consistency between the fine-tuned model output and the pre-training distribution, a KL constraint is introduced to prevent excessive deviation. The following definitions are possible:

[0148] ;

[0149] in For reference strategies on freezing, This is the current strategy.

[0150] The greater the difference between the generated model and the reference model, the stronger the penalty, ensuring the stability of the generated quality.

[0151] Through the design of the above reward function, this application not only adopts the formatting and stability reward approach in the enhanced fine-tuning, but also introduces IOU reward specifically for the positioning requirements of industrial defect detection, thereby comprehensively improving the format accuracy, positioning precision and strategy robustness of the thought chain output.

[0152] S403: Enhanced fine-tuning.

[0153] In the enhancement and fine-tuning phase, this embodiment uses Group Relative Policy Optimization (GRPO) to adjust the decoding strategy parameters. Update.

[0154] To eliminate differences in reward scales between different prompts (inputs) and to guide policy updates based on relative performance, we first normalize the reward and calculate the advantage.

[0155] For the same Prompt We can start with the current strategy. Sample G candidate outputs And calculate their compound rewards respectively:

[0156] ;

[0157] Each of them Both are real numbers, representing the first... The candidate is evaluated based on a comprehensive score relative to formatting, positioning accuracy, and strategy stability.

[0158] To ensure that each candidate is compared to other candidates in the same group, reflecting their relative merits rather than absolute scores, the group rewards are normalized:

[0159] ;

[0160] If a candidate's composite reward Higher than the group average After normalization If below average, then In this way, the reward comparison under different prompts is based on the group average, making the subsequent calculation of relative advantage more equitable.

[0161] To ensure stable updates and eliminate the advantages This directly guides the strategy to increase or decrease the probability of a certain output, avoiding directional deviations in parameter updates due to the inherent difficulty of certain prompts. Therefore, based on the normalized composite reward... The advantage is defined as:

[0162] ;

[0163] in;

[0164] : This indicates that the i-th candidate performs better than the group average, and the probability of generating this output by the current strategy needs to be increased.

[0165] : This indicates that the performance of the i-th candidate is lower than the group average, and the tendency of the current strategy to generate such output needs to be suppressed.

[0166] Numerical range: .

[0167] Through the above-mentioned reward evaluation, intra-group normalization, and advantage calculation, GRPO can select the truly fitting output from a variety of candidates, and then use the pruning strategy objective to update the model accurately and stably.

[0168] After obtaining the output, in order to find a balance between encouraging high-dominance outputs and suppressing over-updates, a pruning GRPO objective function is designed.

[0169] In GRPO, this embodiment aims to increase the probability ratio of the policy output. Its advantages Pruning is performed to confine each update to a safe range, preventing excessive policy jumps. Specifically, for the first update under the same Prompt... candidate outputs :

[0170] First, calculate the probability ratio between the current policy and the old policy, which is used to measure the probability of the same policy under the same conditions. Below, the current strategy Generate candidate output The probability, relative to the old strategy Ratio:

[0171] ;

[0172] like This indicates that the new strategy is more inclined to generate [something] than the old strategy. ;like If so, the tendency weakens.

[0173] Leveraging the advantages gained from the previous normalization step Constructing an unpruned target .

[0174] when When the value exceeds this interval, it is replaced with the interval boundary value to limit the maximum step size of the policy update and prevent... Excessive deviation The clipping function can then be designed as follows:

[0175] ;

[0176] Ultimately for each Taking the smaller of the two values, and based on the pruning loss, the objective function for a single candidate output is:

[0177] ;

[0178] First item If no pruning is performed, optimization is carried out by multiplying the original probability ratio by the advantage, which can encourage or suppress the strategy.

[0179] Second item The cropped version, used when the original... Replace the settings when the value exceeds the safe range, in order to limit the extent of optimization.

[0180] Minimum value: To prevent situations where the advantage is positive. Too large; to prevent situations where the advantage is negative. Too small.

[0181] Here (Values ​​of 0.1–0.2) define the acceptable range of probability variation; if the policy probability ratio jumps out... For intervals, boundary values ​​are used to limit the amount of updates, thereby ensuring training stability.

[0182] Subsequently, batch merging and parameter updates were performed.

[0183] Sampled from the same Prompt The candidate outputs are combined and their losses are merged as follows:

[0184] ;

[0185] The purpose of this loss is to apply to the same Prompt. The average loss of each sample is taken to ensure that the update has a consistent impact on each Prompt.

[0186] Then the AdamW optimizer was used to optimize the model parameters. An update will be made:

[0187] ;

[0188] Learning rate Weight decay With learning rate and weight decay Gradient descent is applied to the model parameters to complete one policy update. This update step is repeated for 1–2 epochs until policy performance convergence is detected or the predetermined number of iterations is reached.

[0189] By using the aforementioned pruning objective function, we can ensure that the policy strengthens high-quality candidates while preventing unstable jumps during training, thus achieving accurate and stable online optimization in the industrial defect thought chain generation task.

[0190] The method proposed in this application is applicable to online quality inspection systems in smart manufacturing workshops, particularly in the automated assembly and acceptance stages of 3C products. First, industrial cameras and a digital twin platform are deployed on the production line to synchronously acquire and align the RGB images of each workpiece with its corresponding CAD model and process parameters (tool number, rotation speed, temperature, etc.), constructing a geometric-texture-process ternary semantic lattice. When the assembled mobile phone camera module enters the inspection station, the system's multimodal detection network automatically identifies micron-level cracks or hidden stains on the camera housing and generates a complete reasoning chain through CoT-Head: anomaly location → CAD deviation evidence → machining process comparison → root cause inference → process adjustment suggestions. For example, if the system determines that a batch of camera housings has localized indentations accompanied by deviations exceeding tolerances, it not only marks the crack location but also indicates that the defect is likely caused by wear on the CNC rough milling tool and suggests replacing the tool or reducing the feed rate.

[0191] In subsequent production iterations, workshop quality control engineers no longer need to re-annotate massive amounts of samples. They only need to upload a small number of examples or normal parts after a new model goes live, and the system quickly aligns the new parts' features through supervised fine-tuning. Furthermore, GRPO is used to enhance fine-tuning during daily operation. A simple review of 50 randomly selected samples in the online buffer is all that's needed for the system to continuously optimize detection thresholds and inference parameters, addressing domain drift caused by material batches or process optimizations. This not only reduces equipment downtime but also lowers rework rates, achieving the dual benefits of high-precision detection and low maintenance costs.

[0192] This application also provides a multimodal industrial defect detection device, such as... Figure 7 As shown, the multimodal industrial defect detection device includes:

[0193] The data alignment module 701 is configured to acquire an RGB image and a CAD model of the part to be inspected, and to perform two-dimensional contour registration on the RGB image and the CAD model of the part to be inspected to generate an alignment sample information file; wherein, the RGB image includes an image of the part's appearance, and the CAD model contains the part's geometric contour and tolerance information;

[0194] The thought chain generation module 702 is configured to construct a five-segment thought chain template including location, evidence, process, root cause and suggestion, and to fill the five-segment thought chain template based on the alignment sample information file to obtain a thought chain example set file.

[0195] The supervised fine-tuning module 703 is configured to divide the thought chain example set file into multiple domain subsets according to part type, with each domain subset including at least two examples, to obtain a domain-specific thought chain example file; extract the prompt word string for each example in the domain-specific thought chain example file; encode the prompt word string and insert a domain label at the beginning of the encoded sequence to generate a text input identifier sequence; preprocess the aligned RGB image corresponding to the domain subset to obtain an image pixel tensor; combine the text input identifier sequence and the image pixel tensor to form a dataset; and use the dataset to perform supervised fine-tuning on the large model to obtain the supervised fine-tuned large model.

[0196] The enhancement and fine-tuning module 704 is configured to construct a reference-query comparison sample pair, which includes a reference image and a query image. The reference image is an aligned RGB image of a normal part, and the query image is an aligned RGB image of a defective part from the same batch of parts. The reference-query comparison sample pair is used to enhance and fine-tune the large model after supervised fine-tuning to obtain a multimodal industrial defect detection model. The detection of the part to be inspected is realized based on the multimodal industrial defect detection model.

[0197] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0198] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0199] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0200] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0201] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the multimodal industrial defect detection method described in the above embodiments.

[0202] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the multimodal industrial defect detection method in the above embodiments.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0204] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0205] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0206] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0207] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0208] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0209] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0210] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0211] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0212] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.< / answer> < / think> < / answer> < / think>

Claims

1. A multimodal industrial defect detection method, characterized in that, The method includes: The RGB image and CAD model of the part to be inspected are acquired, and two-dimensional contour registration is performed on the RGB image and CAD model of the part to be inspected to generate an alignment sample information file; wherein, the RGB image includes the appearance image of the part, and the CAD model contains the geometric contour and tolerance information of the part; A five-segment thinking chain template, including location, evidence, process, root cause, and suggestion, is constructed. The five-segment thinking chain template is then filled in based on the alignment sample information file to obtain a thinking chain example set file. The thought chain example set file is divided into multiple domain subsets according to the part type. Each domain subset includes at least two examples, resulting in a domain-specific thought chain example file. The prompt word string of each example in the domain-specific thought chain example file is extracted, the prompt word string is encoded, and a domain label is inserted at the beginning of the encoded sequence to generate a text input identifier sequence. The aligned RGB image corresponding to the domain subset is preprocessed to obtain an image pixel tensor. The text input identifier sequence and the image pixel tensor are combined to form a dataset. The dataset is used to perform supervised fine-tuning on the large model to obtain the supervised fine-tuned large model. A reference-query comparison sample pair is constructed, which includes a reference image and a query image. The reference image is an aligned RGB image of a normal part, and the query image is an aligned RGB image of a defective part in the same batch of parts. The reference-query comparison sample pair is used to enhance and fine-tune the supervised fine-tuning of the large model to obtain a multimodal industrial defect detection model. The detection of the part to be inspected is realized based on the multimodal industrial defect detection model. The five-segment thinking chain template is populated based on the alignment sample information file to obtain a thinking chain example set file, including: The candidate box coordinates, optimal alignment parameters and process information are extracted from the alignment sample information file and filled into the positioning, evidence and process in the five-segment thinking chain template to obtain the filled template. The filled template is then encoded into a text input identifier sequence using BPE. At the same time, the RGB image is adjusted to the model preset size and obtained as a pixel tensor after center cropping and resampling. The text input identifier sequence and pixel tensor are input into the large model. Based on the set generation hyperparameters, the large model's generation interface is called to complete the root causes and suggestions in the filled template, generating the analysis results. If generation fails, it is repeated up to n times. If all n generation attempts fail, a simplified thought chain containing localization and anomaly detection is output, and an inference error log file is recorded. The generation hyperparameters include temperature. , top_p, top_k and repetition_penalty, where n is a positive integer greater than 1.

2. The multimodal industrial defect detection method according to claim 1, characterized in that, Two-dimensional contour registration is performed on the RGB image and CAD model of the part to be inspected to generate an alignment sample information file, including: The CAD model is projected into a binary contour map, and Canny edge detection is performed on the RGB image to generate a binary edge map. The pixel plane is determined based on the binary edge map. In the binary contour map, different pixel values ​​represent the background area and the target position, and the target position includes the edge or contour position of the part model. Based on a set combination of parameters, a grid search is performed on the binarized contour map within the pixel plane. Using the pixel ratio and intersection-to-union (IoU) as the registration metric, the parameters that maximize the IoU from the set combination of parameters are selected as the optimal alignment parameters. The parameter combination includes translation parameters. and rotation parameters , It represents the translation amount along the X and Y axes, with px representing the unit of pixels; Based on the optimal alignment parameters, the RGB image and CAD model of the part to be inspected are aligned to obtain a CAD contour overlay image. If the IoU of the CAD contour overlay image is greater than or equal to a set cross-comparison threshold, the RGB image, CAD contour overlay image, optimal alignment parameters, and process information are encapsulated into an alignment sample information file. If the IoU of the CAD contour overlay image is less than the set cross-comparison threshold, the current part to be inspected is marked as having failed alignment and a log is recorded. The process information includes process identifier, machine number, tool number, rotation speed, and / or temperature.

3. The multimodal industrial defect detection method according to claim 2, characterized in that, The formula for calculating the pixel ratio and crossover ratio is: In the formula, This represents the pixel-level intersection ratio between the CAD contour map and the edge detection map. This represents the overlay of CAD binary contours after translation and rotation parameter transformations. This represents the set of edge pixels obtained from the RGB image to be detected using an edge detection algorithm.

4. The multimodal industrial defect detection method according to claim 1, characterized in that, The text input identifier sequence and image pixel tensor are combined to form a dataset. This dataset is then used to perform supervised fine-tuning of a large model, resulting in a supervised fine-tuned large model, including: A batch size is set, and the text input identifier sequence and image pixel tensor are input into the large model in batches. The total loss is calculated according to the set loss function. The model is trained by gradient descent to adapt it to the geometric and process characteristics of different parts, resulting in a supervised fine-tuned large model. The loss function is expressed as follows: In the formula, For the first Example of the length of the thought chain. and These are the tokens of the thought chain and the label of the judgment, respectively; B is the batch size; i is the batch number; and t is the total number of tokens in the thought chain area of ​​the i-th sample. For the cross-entropy loss of the thought chain region, Let θ represent the conditional probability distribution of the model. It is the natural logarithm. For the i-th sample, the sequence of generated tokens preceding the t-th token is... For the first Image pixel tensor For the cross-entropy loss of the answer region, For the complete thought chain text sequence of the i-th training sample, For intra-segment alignment loss, The segment numbers in the thought process template correspond sequentially to location, evidence, process, root cause, and suggestion. For the word set predicted by the model, For keyword set, To monitor the total loss during fine-tuning, This is the intra-segment alignment loss weight function.

5. The multimodal industrial defect detection method according to claim 1, characterized in that, The large model after supervised fine-tuning is enhanced and fine-tuned using the reference-query comparison samples to obtain a multimodal industrial defect detection model, including: Set a composite reward function, expressed as follows: In the formula, Indicates a compound reward. , and These represent the weights of the formatting reward, IOU positioning reward, and KL penalty, respectively. For formatted rewards, a formatted reward is only awarded when the large model after supervised fine-tuning outputs all five segments completely and uses the preset correct judgment format. The location reward for IOU is calculated using the following formula: In the formula, Generate a box. Represents a truth box; The penalty for KL is calculated using the following formula: In the formula, For reference strategies on freezing, As the current strategy, The output action generated by the model at time step t. The time step index indicates which generation action or token it is. It is the natural logarithm; For the same input, G candidate outputs are sampled from the current policy, and the composite reward for each input is calculated according to the composite reward function, and then normalized using the following formula: In the formula, The normalized composite reward for the i-th candidate output is... The composite reward for the i-th candidate output. Let be the average composite reward of the i-th candidate output, and j be the index of the candidate output. The composite reward for the j-th candidate output; According to normalized composite reward The advantage is calculated using the following formula. : in , This indicates that the i-th candidate performs better than the group average, and the probability of generating this output by the current strategy needs to be increased. This indicates that the performance of the i-th candidate is below the group average, and it is necessary to suppress the tendency of the current strategy to generate such output; The probability ratio is calculated using the following formula. : In the formula, For the i-th candidate output, Provide conditional inputs or prompt text when the model generates output. Generate candidate outputs for the current policy model when given input p. The probability, Generate candidate outputs for the old policy model with the same input p. The probability of; Based on probability ratios and advantage Design a clipping function, which is expressed as follows: In the formula, The magnitude of probability change. This represents the probability ratio between the current strategy and the old strategy. This is the clipping function; Based on the pruning function, the objective function for constructing a single candidate output is: In the formula, This is the tailored objective function value constructed for the i-th candidate output. It is a minimum value function; Based on the objective function, the loss of G candidate outputs sampled from the same input is calculated using the following formula: In the formula, The total loss function is obtained by averaging the objective function values ​​of G candidate outputs for the same input. Based on the calculated loss, the model parameters are adjusted. Update the policy and repeat the training at least once more until the policy performance converges or the predetermined number of iterations is reached.

6. The multimodal industrial defect detection method according to claim 5, characterized in that, Based on the calculated loss, the model parameters are adjusted using the following formula. Update: In the formula, For learning rate, For weight decay, This is the descent gradient.

7. A multimodal industrial defect detection device for implementing the method as described in any one of claims 1-6, characterized in that, The device includes: The data alignment module is configured to acquire the RGB image and CAD model of the part to be inspected, and perform two-dimensional contour registration on the RGB image and CAD model of the part to be inspected to generate an alignment sample information file; wherein, the RGB image includes the appearance image of the part, and the CAD model contains the geometric contour and tolerance information of the part; The thought chain generation module is configured to construct a five-segment thought chain template including location, evidence, process, root cause, and suggestion. The five-segment thought chain template is populated based on the alignment sample information file to obtain a thought chain example set file. The supervised fine-tuning module is configured to divide the thought chain example set file into multiple domain subsets according to part type, with each domain subset including at least two examples, to obtain a domain-specific thought chain example file. It extracts the prompt word string from each example in the domain-specific thought chain example file, encodes the prompt word string, and inserts a domain label at the beginning of the encoded sequence to generate a text input identifier sequence. It preprocesses the aligned RGB image corresponding to the domain subset to obtain an image pixel tensor. It combines the text input identifier sequence and the image pixel tensor to form a dataset, and uses this dataset to perform supervised fine-tuning on the large model, resulting in a supervised fine-tuned large model. The enhancement and fine-tuning module is configured to construct a reference-query comparison sample pair, which includes a reference image and a query image. The reference image is an aligned RGB image of a normal part, and the query image is an aligned RGB image of a defective part from the same batch of parts. The reference-query comparison sample pair is used to enhance and fine-tune the large model after supervised fine-tuning to obtain a multimodal industrial defect detection model. The detection of the part to be inspected is realized based on the multimodal industrial defect detection model.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the multimodal industrial defect detection method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the multimodal industrial defect detection method as described in any one of claims 1-6.

Images