Industrial metal multi-defect detection method, medium, and device

Abstract

The present application relates to the field of artificial intelligence and industrial machine vision technology, and more particularly to a kind of industrial metal multi-defect detection method, medium and equipment, method includes constructing industrial metal defect multimodal instruction dataset;Build pre-training visual large model base;Improved low-rank adapter module is inserted in the model architecture block of base model;Introduce defect perception prompt learning;Cross-scale feature fusion;Model deployment and reasoning;Real-time detection, adaptive threshold and continuous optimization.The method of the present application is efficient, low consumption, multi-scale, high precision, strong generalization and few sample learning, strong anti-interference ability, through model light weight compression and adaptive threshold decision, the best balance of detection accuracy and real-time is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and industrial machine vision technology, and in particular to a method, medium and equipment for detecting multiple defects in industrial metals. Background Technology

[0002] In the manufacturing sector, the quality of metal surfaces directly determines product performance and lifespan. Currently, mainstream industrial visual defect detection relies primarily on manual inspection or methods based on traditional machine learning (such as SVM and manual feature extraction) and early deep learning (such as CNN and YOLO series small models). However, these methods suffer from the following significant drawbacks: 1. Weak generalization ability: When faced with different materials, different lighting conditions and unknown defect types generated by new processes, traditional models need to collect a large amount of data again and train from scratch, resulting in high costs.

[0003] 2. Difficulty in detecting multi-scale defects: Due to the large differences in the morphology of metal defects in practical applications, ranging from minute cracks to macroscopic pits, a single-scale convolution kernel is difficult to take into account.

[0004] 3. Difficulty in deploying large-scale models: In recent years, large-scale vision models have demonstrated powerful feature extraction and generalization capabilities, but their parameter count is enormous. If full fine-tuning is performed directly, it requires extremely high GPU memory resources and computing power, and is very likely to lead to "catastrophic forgetting", that is, the model forgets the general background features while learning new defects, thus leading to an increase in the false detection rate.

[0005] While existing fine-tuning techniques (such as LoRA) reduce memory requirements, they often struggle to converge or achieve sufficient accuracy when dealing with extremely unbalanced defect samples (many normal samples and very few defect samples) and subtle texture features in industrial scenarios.

[0006] Therefore, it is necessary to provide a fine-tuning method that can both retain the generalization ability of large models and efficiently adapt to the multi-defect detection scenarios of industrial metals. Summary of the Invention

[0007] The main objective of this invention is to provide a method, medium, and equipment for detecting multiple defects in industrial metals, aiming to solve the technical problem of insufficient accuracy in existing industrial visual defect detection.

[0008] To achieve the above objectives, the present invention proposes a method for detecting multiple defects in industrial metals, comprising the following steps: S1. Collect industrial metal data; S2. Construct a multimodal instruction dataset for industrial metal defects; S3. Build a pre-trained large visual model base. Specifically, based on the industrial metal defect multimodal instruction dataset, select a pre-trained large visual model as the base model for fine-tuning, freeze the parameters of its bottom and middle layers, and retain the top layer parameters as adjustable; connect a lightweight detection head at the decoding end. S4. Insert the improved low-rank adapter module into the model architecture block of the base model to obtain the adapted base model. S5. Introduce defect-aware prompt learning, specifically: add the learned prompt vector to the input end of the adapted base model; S6. Cross-scale feature fusion, specifically: using the cue vectors introduced in S5 and the multi-scale perceptual low-rank adapter module in S4, by freezing all parameters of the base model, only on the trainable set... Optimize the model to adapt to industrial data distribution; unfreeze the last two layers of the base model architecture, jointly train the multi-scale perception low-rank adapter module and cue vector, and optimize multi-scale feature fusion; introduce hard example mining, optimize the weighted focus loss function for false detection samples, fine-tune the detection head parameters, and obtain the fine-tuned base model. S7. Model Deployment and Inference: Specifically, the fine-tuned base model is compressed using operator fusion and quantization techniques and deployed to industrial edge computing devices; the optimized model is then subjected to structured pruning, knowledge distillation, and INT8 quantization. S8 features real-time detection, adaptive thresholding, and continuous optimization. Specifically, it utilizes the S7 industrial edge computing device for real-time detection. The adaptive thresholding module dynamically calculates thresholds based on the confidence distribution of historical normal samples, identifies defects, and displays the results in real time. During the detection process, false positives and false negatives are automatically collected, reviewed, and added to the training set. Periodic incremental fine-tuning is performed to adapt the model to changes in production line conditions, forming a data loop and enabling continuous model evolution. S9. Output the detection results.

[0009] A further improvement of the present invention for the detection method of multiple defects in industrial metals is that S4 further includes: In the multi-scale perceptual low-rank adapter module, a lightweight dynamic gating network automatically learns the fusion weights of high-frequency and low-frequency branches in the multi-scale perceptual low-rank adapter module based on the local or global statistical characteristics of the current input feature map. The steps are as follows: S401. Input Preparation: Let the input feature map of the current model architecture layer be... ,in, For batch size, For spatial height dimension, For the spatial width dimension, The number of channels; the feature map is fed into two parallel branches simultaneously: The output of the high-frequency branch is: ; The output of the low-frequency branch is: ; S402. Design a lightweight multilayer perceptron global pooling structure to generate fusion weights for each sample. S403, For each sample index The value range is from 1 to Its final output through the adaptive fusion formula is: ; in: To indicate the first The final fused output feature map of each sample The scalar weights output by the gating network, The output feature map of the corresponding branch is then used; the fusion weights are then integrated into the overall process of the multi-scale perception low-rank adapter module.

[0010] A further improvement of the present invention for the multi-defect detection method for industrial metals is that the cue vector includes visual cue and text cue; S5 includes the following steps: S501, Visual Cue Construction: Define a set of learnable visual cue vectors : ; in: To indicate the length, For the hidden layer dimension of the model, It is the set of real numbers; Image block embedding: embedding the input image Divided into Each patch is used to obtain the image embedding through linear projection. ; Cue splicing: combining visual cue vectors spliced ​​before the image embedding sequence: ; in: For the final visual input sequence, For splicing operations, The image embedding sequence defined above; Location encoding: a visual cue vector Assign independent location codes; S502, Text Prompt Construction: Template Design: Define a collection of text templates ; Learnable category embeddings: Learn independent vectors for each defect category : ; in: For the number of defect categories, For the embedding dimension of the model; Text encoding process: Convert the fixed text portion in the template into an integer number; simultaneously perform category vector replacement, changing the text in the template... Replace the placeholder position with the corresponding ,in For category index; final input is: ; in: The text segment before the placeholder. The text fragment following the placeholder. For the final text input sequence, The total length of the sequence. For text encoding / embedding functions.

[0011] A further improvement of the present invention for the multi-defect detection method for industrial metals lies in the following steps when using the cue vector in training S5 and the multi-scale perceptual low-rank adapter module in S4 to adapt the model to the industrial data distribution by freezing all parameters of the base model: S601, Parameter Freezing and Grouping Strategy: Freezing and grouping model parameter sets Divided into frozen sets With trainable set : Among them, frozen collection It includes all self-attention layers and feedforward neural network layer parameters of the pedestal model, as well as the main parameters of the visual encoder; ; Trainable set Includes visual cue vectors introduced in S5 Independent vectors for defect category learning And the parameters of the multi-scale sensing low-rank adapter module in S4 ; ; S602, Differentiated Learning Rate Configuration: Assign differentiated learning rates to the trainable parameter set. The configuration is as follows: Hint vector group Assign learning rate ; Adapter group Assign learning rate ; S603. Construction of Joint Loss Function: Construct a joint loss function that includes both classification and feature alignment tasks. : ; in: Cross-entropy loss is used for constraint defect classification and prediction. With real labels Consistent; To compare the loss, bring the image features closer together. Corresponding text features Distance in the embedded space; This is the balance coefficient.

[0012] A further improvement of the present invention for the multi-defect detection method for industrial metals lies in the following steps when jointly training the multi-scale perceptual low-rank adapter module and the cue vector in the last two layers of the unfrozen base model architecture block, and optimizing the multi-scale feature fusion: S604, Parameter Unfreezing Strategy: While keeping most base parameters frozen, selectively unfreeze the final parameters of the base model. Layered model architecture blocks: ; The total trainable parameter set is now updated to: ; S605, Cross-scale feature fusion mechanism: Assuming the model extracts... Feature maps at different scales By fusing features through a scale attention mechanism, features at different scales are first... Unify to the same dimension through linear mapping Then, the attention weights for each scale feature are calculated. : ; Finally, generate fusion features. : ; S606, Multi-scale Consistency Loss: Introducing consistency regularization loss : ; in: For the set of thawing parameters, For the total set of trainable parameters, The second to last layer of the model. This is the last layer of the model. To convert the model's raw output into a probability distribution, It is a multilayer perceptron. For average pooling, This is a scale index. The separator, in the KL divergence formula, indicates the calculation of the difference between the previous distribution and the next distribution. For the first The original prediction output at each scale, For the first +1 scale of original prediction output, Let KL divergence be denoted as KL divergence.

[0013] A further improvement of the present invention for the multi-defect detection method for industrial metals is that, when introducing hard example mining and optimizing the weighted loss focus function for false detection samples, the following steps are included: S607, Difficult Example Discovery Strategy: Before the start of each round, use the current model to perform inference on the training set once to define a set of difficult examples. ; where false positives are the predicted categories Compared to the real category Inconsistent samples; while low-confidence samples are correctly predicted and have the highest class probability. The sample, For threshold; ; S608, Sample weighting mechanism: For each sample Assign weights Difficult cases are given higher weight: ; in, For the first One sample and its label, For the set of difficult examples, For logical OR, For the maximum class probability, For difficult cases, the magnification factor is used. S609, Weighted Focus Loss Function: A focus loss mechanism is introduced to further reduce the weight of easily separable samples and focus on difficult examples: ; in: This represents the loss value for the third stage. The total number of samples, To predict the probability of the true class for the model; For focusing parameters; Balance the weights for each category.

[0014] A further improvement of the present invention for the multi-defect detection method for industrial metals lies in that, during real-time detection using an S7 industrial edge computing device, dynamic patching, multi-expert collaborative reasoning, cross-scale feature fusion, and temporal context fusion are performed on the real-time images of the production line to output the confidence score of the defect candidate box, including the following steps: S801, Input to the Dynamic Patch Partitioning Algorithm: Input image ; The base patch size is ; sliding step : Output: Patch collection ; The implementation steps are as follows: S80101, Image Complexity Evaluation: Calculate the image gradient map: ; Calculate the local complexity score: ; Generate a complexity heatmap: ; S80102, Adaptive Patch Size Determination: For each candidate region Computational average complexity : ; Determine the patch size: ; in: This is a complexity scaling factor. For image gradient, The neighborhood radius is used to calculate local complexity. For image gradient maps, coordinates Local complexity score at the location, These are the coordinates of the image pixels. This is the relative offset within the local window. The offset index within the neighborhood window. coordinates The complexity heatmap values ​​at the location, This refers to the patch size after regional adjustment. The average complexity of the region; S80103, constraint patch size is Within the range, among which, Minimum height, For maximum height, Minimum width, Maximum width; S80104, Patch Generation and Overlap Control: Generate patch set according to adaptive size ; Controlling the overlap rate of adjacent patches: Filter out patches whose area is smaller than the preset minimum defect size; S80105, Return to Patch Collection ; S803, Cross-scale feature fusion, includes the following steps: S80301, Multi-scale Feature Extraction: The system extracts multi-scale feature maps from the backbone network, acquiring a series of feature maps across different layers, spanning multiple scales from shallow to deep. The deepest feature maps typically contain the richest semantic information, while other layers correspond to different resolutions and channel counts, collectively forming a multi-scale feature representation. Each scale feature map has specific height, width, and channel count; these parameters determine the spatial resolution and feature dimension of the feature map.

[0015] S80302, Feature Pyramid Fusion: A top-down feature pyramid structure is used for fusion: ; in: For the first Features after layer fusion For 1x1 convolution, For upsampling operation, This is the current processing level; S80303, Scale Attention Weighting: Assign attention weights to each scale feature: ; ; in: For the first Attention weights of layers For the first to be weighted Layer features, For global average pooling; S804, Timing Context Fusion: S80401, Time-Sequence Feature Cache: Maintenance length is Feature cache queue: ; in: The fusion features of the current frame, For feature cache queue; S80402, Temporal Attention Fusion: Calculate the attention weights for the current frame and historical frames: ; in: For attention weights, It is an exponential function. The cosine similarity function; S80403, Timing Enhancement Features: ; in: For temporal enhancement features, Here, represents the fusion features of historical frames, and represents the attention weights. This is the time offset. This represents the length of the time window.

[0016] A further improvement of the present invention for the multi-defect detection method for industrial metals is that, when the adaptive threshold module dynamically calculates the threshold based on the confidence distribution of historical normal samples, it includes the following steps: S805, Historical Normal Sample Confidence Distribution Modeling: S80501, Collection of Confidence Levels for Normal Samples: Collecting history The highest background confidence score for a normal sample: ; in: For the first The highest class confidence score output by the model among all normal samples; S80502, Distribution Fitting: Fit the confidence distribution of normal samples using a Gaussian mixture model: ; in: The number of Gaussian components. For the first The mixing coefficient of each component, For the first The mean of the Gaussian components, For the first The standard deviation of each Gaussian component For background confidence, The function is a Gaussian distribution (normal distribution). Let be the probability density function of the confidence level; S80503, Online Update of Distributed Parameters: The distribution parameters are updated using an exponential moving average: ; ; in: For smoothing coefficients, This is an estimate for the current batch. This is the updated mean. This is the average value from the previous time step. The updated standard deviation, The standard deviation of the previous time step. This represents the estimated standard deviation for the current batch. S806, Dynamic Threshold Calculation Steps: S80601. Calculate the cumulative distribution function: ; in: The cumulative distribution function is... For confidence level variables; S80602. Determine the basic threshold: Solution: ; Right now: ; in: This represents the cumulative probability at the threshold. The preset extremely low probability (false alarm rate). Based on the threshold, The formula for the tail probability; S80603, Confidence Calibration: Considering defect size factors: ; Considering defect category factors: ; in: For category-specific offsets, The threshold after size adjustment. This is the size adjustment factor. The area of ​​the current defect. For reference area; Considering timing consistency factors: ; in: The threshold after time-series adjustment. The threshold after category adjustment. This is the consistency weighting coefficient. The temporal consistency score is an indicator that measures whether the defect behavior in the current frame is consistent with that in the preceding and following frames. Threshold boundary constraints: ; in: The lower limit of the threshold. For the final dynamic threshold, The upper limit of the threshold; Prevent threshold fluctuations: ; Finally return ; in: The final threshold after smoothing. The currently calculated threshold, This is the threshold value from the previous time step.

[0017] The present invention also provides a readable storage medium storing a computer program adapted to be loaded by a processor and executed as described above for the method for detecting multiple defects in industrial metals.

[0018] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, runs the multi-defect detection method for industrial metals as described above.

[0019] The technical solution of the present invention has the following beneficial effects: 1) High efficiency and low power consumption: Through the multi-scale perception low-rank adapter module (MSP-LoRA) and cue learning, this invention only needs to update less than 1% of the model parameters, reducing memory usage by more than 70%, so that large models with hundreds of billions of parameters can be fine-tuned on a single card or even an edge device.

[0020] 2) Multi-scale high precision: The parallel multi-scale adaptation mechanism of this invention effectively solves the detection problem when fine cracks and macro-scale pits coexist on the metal surface, and significantly improves the recall rate of small target defects.

[0021] 3) Strong generalization and few-shot learning: By combining the general knowledge and defect perception prompts of the pre-trained large model, high accuracy can still be maintained even when only a small number of defect samples are used, which greatly reduces the cost of data collection.

[0022] 4) Strong anti-interference capability: The dynamic gating mechanism of this invention enables the model to adapt to different lighting, reflection and complex background interference, reducing the false alarm rate in industrial fields.

[0023] 5) This invention achieves the best balance between detection accuracy and real-time performance through lightweight model compression and adaptive threshold decision-making, meeting the stringent requirements of industrial production lines.

[0024] 6) This invention constructs a data closed loop and continuous learning mechanism to enable the model to continuously evolve as production line conditions change, significantly reducing implementation costs and maintenance cycles. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the multi-defect detection method for industrial metals according to the present invention; Figure 2 A comparison of PR curves for different methods in metal defect detection; Figure 3 This is a graph showing the comprehensive trade-off analysis of speed, accuracy, and volume for different methods in metal defect detection. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0028] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0029] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0030] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0031] like Figure 1 As shown, this invention proposes a method for multi-defect detection in industrial metals. By constructing a multi-scale sensing adapter, introducing a defect-sensing prompting learning mechanism, and designing a hierarchical freezing strategy, it achieves high-precision, multi-category metal defect detection on large models with low computational power consumption. Specifically, it includes the following steps: S1. Collect industrial metal data; S2. Construct a multimodal instruction dataset for industrial metal defects; collect images of metal surfaces on industrial production lines, label defect categories (including dents, inclusions, patches, scratches, and foreign objects) and their locations; transform the detection task into a vision-language instruction format, and construct a dataset containing "image-instruction-response" triples, where the instruction contains a natural language description of the defect type, location, and severity, and the response is the corresponding detection box coordinates or segmentation mask and text conclusion.

[0032] S3. Build a pre-trained large-scale visual model base. Specifically, based on the industrial metal defect multimodal instruction dataset, select a pre-trained large-scale visual model as the base model for fine-tuning, freeze the parameters of its bottom and middle layers, and retain the top layer parameters as adjustable; connect a lightweight detection head to the decoding end. S301. Select an open-source multimodal or pure vision large model as the base, such as Qwen3.5-35B-A3B or a dedicated vision model such as ViT-Large / 14, Swin-L, InternVL, LLaVA-NeXT, etc. Use HuggingFaceTransformers (an open-source deep learning framework) or ModelScope to load pre-trained weights.

[0033] S302, Parameter Freezing Strategy: Freeze the bottom and middle layer parameters, leaving only the top layer adjustable, to balance transfer capability and computational efficiency; freeze the first 80% to 90% of the model framework (Transformer) layers (e.g., in Qwen3.5, which has 60 layers, freeze the first 48 to 54 layers); unfreeze the last 2 to 4 layers, all multi-scale perceptual low-rank adapter (MSP-LoRA) modules, and Prompt Tokens (the complete input content sent to the model).

[0034] S303. A lightweight detection head is connected to the end of the decoder for regressing defect coordinates and classification; its structural design is as follows: Input: Tokens from the last Transformer layer or global average pooling features.

[0035] Structure: 2-3 layers of MLP (Multilayer Perceptron) with Sigmoid / RReLU activation output Similar to class logits (class logits).

[0036] Optional: Add FPN (Feature Pyramid Network) or multi-scale fusion to improve small target detection capabilities.

[0037] S304, Loss function configuration for joint optimization of localization and classification, the function is as follows: ;in This represents the total loss, used for backpropagation optimization of the model. This is the localization loss, used to measure the deviation between the predicted bounding box and the ground truth bounding box. Classification loss is used to measure the accuracy of class prediction. , This represents the weighting coefficient, used to balance the contributions of the two losses. Represents the predicted bounding box. Represents the actual bounding box; Represents the predicted class probability. This represents the actual category label.

[0038] S4. Embedded multi-scale sensing low-rank adapter: An improved low-rank adapter module is inserted into the model architecture block of the base model to obtain the adapted base model. Unlike the traditional LoRA (low-rank adapter), the MSP-LoRA (multi-scale sensing low-rank adapter) in this embodiment includes parallel high- and low-frequency branches: The high-frequency branch uses a smaller rank to focus on capturing high-frequency fine texture features such as cracks and micro-scratches. The low-frequency branch uses a larger rank and focuses on capturing low-frequency macroscopic morphological features such as pits and patches. A dynamic gating mechanism is introduced to adaptively fuse the output weights of high- and low-frequency branches based on the feature map of the input image.

[0039] The dynamic gating mechanism adaptive fusion method includes the following steps: In the multi-scale sensing low-rank adapter module, a lightweight dynamic gating network automatically learns the fusion weights of high-frequency and low-frequency branches based on the local or global statistical characteristics of the current input feature map, thereby achieving adaptive response to different defect types such as microcracks or large pits. The steps are as follows: S401. Input Preparation: Let the input feature map of the current model architecture layer be... ,in, For batch size, For spatial height dimension, For the spatial width dimension, The number of channels; the feature map is fed into two parallel branches simultaneously: The output of the high-frequency branch is: ; The output of the low-frequency branch is: ; S402. Construct a dynamic gating network: Design a lightweight multilayer perceptron (MLP) global pooling structure to generate fusion weights for each sample; Global average pooling (GAP) is used to extract global contextual information from the entire feature map. Two fully connected layers are used for compression and expansion to learn the mapping from global features to fused weights. Finally, a softmax function is used to ensure that the sum of the two weights is 1, guaranteeing numerical stability.

[0040] S403, Adaptive Fusion Formula: For each sample index The value range is from 1 to Its final output through the adaptive fusion formula is: ; in: To indicate the first The final fused output feature map of each sample is the feature map after weighted summation. The scalar weights output by the gating network, This is the output feature map for the corresponding branch.

[0041] S404. Then, integrate the fusion weights into the overall process of the multi-scale perception low-rank adapter module.

[0042] S405, Training Strategy Recommendations: Initialization: The weights of the gated network are initialized to be evenly distributed (e.g., fc2.weight.data.fill_(0.5)) to avoid bias towards a certain branch in the early stages.

[0043] Regularization: An entropy regularization term can be added to the loss function to encourage the gating network to make explicit choices on certain samples (non-uniform distribution). ; in: This represents the gated regularization loss value. This is the regularization coefficient (weight hyperparameter). No. Each sample is assigned a weight (probability) to either the "high branch" or the "main path". No. Each sample is assigned a weight (probability) to either the "low branch" or the "secondary path". Visualization analysis: After training, graphs of each sample can be plotted. Distribution, to verify whether it has the ability to distinguish different types of defects (e.g., crack samples tend to be assigned high weights to high-frequency branches), among which For the first Each sample (or location) is assigned a weight (probability value, ranging from 0 to 1) to the high-frequency branch. For the first Each sample (or location) is assigned a weight (probability value, ranging from 0 to 1) to the low-frequency branch.

[0044] S5. Introduce defect-aware prompt learning, specifically: add the learned prompt vector to the input end of the adapted base model; The cue vectors include visual cues and text cues. Visual cues append a set of optimizable continuous vectors to the image patch sequence, guiding the model to focus on potential defect regions. Text cues construct specific templates for metal defects (e.g., "a metal surface with [category] defect") and fine-tune the category embedding vectors in the templates, enhancing the model's semantic understanding of rare defect categories. Specifically, the steps include: S501, Visual Cue Construction: Before the image enters the visual encoder, learnable continuous vectors are inserted to guide the model to focus on potential defect areas. Define a set of learnable visual cue vectors : ; in: To indicate the length (e.g.) =10), This is the hidden layer dimension of the model (consistent with the hidden layer dimension hidden_size in Qwen3.5, such as 2048). Let be the set of real numbers, indicating that the vector is a matrix composed of real numbers; Image block embedding: embedding the input image Divided into Each patch is used to obtain the image embedding through linear projection. ; Cue splicing: combining visual cue vectors spliced ​​before the image embedding sequence: ; in: For the final visual input sequence, For splicing operations, The image embedding sequence defined above; Location encoding: a visual cue vector Assign independent location codes (or before multiplexing) (Each location is encoded) to ensure that the model can distinguish between prompts and real image content; S502, Text Prompt Construction: Construct specific templates for metal defects and replace category words with fine-tunable embedding vectors to enhance semantic understanding.

[0045] Template Design: Define a collection of text templates ;For example: "A metal surface with {cls} defects" "Defect {cls} was found during industrial testing" Damage to metal materials Learnable category embeddings: Instead of using fixed word vectors for "scratches" and "cracks" in the tokenizer, independent vectors are learned for each defect category. : ; in: For the number of defect categories, For the embedding dimension of the model; Text encoding process: Template tokenization, converting fixed text portions in the template (such as "a sheet with") into integer token IDs; simultaneously, category vector replacement, replacing the text in the template... Replace the placeholder position with the corresponding ,in For category index; final input is: ; in: The text segment before the placeholder. The text fragment following the placeholder. For the final text input sequence, The total length of the sequence. For text encoding / embedding functions.

[0046] S6. Cross-scale feature fusion: This step involves fine-tuning the large model in three processes: First, using the cue vectors introduced in S5 and the multi-scale perceptual low-rank adapter module in S4, all parameters of the base model are frozen, and only the trainable set is fused. The first stage involves optimizing the model to adapt to the distribution of industrial data. The second stage involves unfreezing the last two layers of the base model architecture, jointly training the multi-scale perception low-rank adapter module and cue vector, and optimizing multi-scale feature fusion. The third stage involves introducing hard example mining, optimizing the weighted focus loss function for false detection samples, fine-tuning the detection head parameters, and obtaining the fine-tuned base model. The main task of the first stage is to enable the model to quickly adapt to the distribution characteristics of industrial defect data without destroying the pre-trained knowledge. This includes the following steps: S601, Parameter Freezing and Grouping Strategy: Freezing and grouping model parameter sets Divided into frozen sets With trainable set : Among them, frozen collection Includes all self-attention layers and feedforward neural network layer parameters of the pedestal model (LLM), as well as the main parameters of the visual encoder; ; Trainable set Includes visual cue vectors introduced in S5 Independent vectors for defect category learning And the parameters of the multi-scale sensing low-rank adapter (MSP-LoRA) module in S4. ; ; S602, Differentiated Learning Rate Configuration: Assign differentiated learning rates to the trainable parameter set. To balance convergence speed and stability, the main configuration is as follows: Hint vector group Assign a higher learning rate (e.g., 1×10) -3 ); Adapter group Distribute a moderate learning rate (e.g., 1×10) -4 ); S603. Construction of Joint Loss Function: Construct a joint loss function that includes both classification and feature alignment tasks. : ; in: Cross-entropy loss is used for constraint defect classification and prediction. With real labels Consistent; To compare the loss, bring the image features closer together. Corresponding text features Distance in the embedded space; This is the balance coefficient (usually set to 0.3).

[0047] The main task of the second stage is to enhance the model's ability to fuse and express multi-scale defect features by unfreezing part of the base layer, including the following steps: S604, Parameter Unfreezing Strategy: While keeping most base parameters frozen, selectively unfreeze the final parameters of the base model. Layered model architecture blocks (usually) =2): ; The total trainable parameter set is now updated to: ; S605, Cross-scale feature fusion mechanism: Assuming the model extracts... Feature maps at different scales By fusing features through a scale attention mechanism, features at different scales are first... The features are unified to the same dimension through linear mapping; then the attention weights for each scale feature are calculated. : ; Finally, generate fusion features. : ; S606, Multi-scale Consistency Loss: To prevent excessive differences in multi-scale feature distributions, a consistency regularization loss is introduced. : ; in: The unfrozen parameter set refers to the set of model parameters that transition from a frozen state to a trainable state. For the total set of trainable parameters, The second to last layer of the model. This is the last layer of the model. To convert the model's raw output into a probability distribution, It is a multilayer perceptron used to learn the weights of features. This is average pooling, used to compress the spatial dimension of the feature map. For is the index of the count, The separator, in the KL divergence formula, indicates the calculation of the difference between the previous distribution and the next distribution. For the first The original prediction output at each scale, For the first +1 scale of original prediction output, Let KL divergence be denoted as KL divergence.

[0048] The third phase involves weighted optimization of samples prone to false positives to further reduce the false positive rate and improve the recall rate for rare categories. This includes the following steps: S607, Difficult Example Discovery Strategy: Before the start of each round, use the current model to perform inference on the training set once to define a set of difficult examples. ; where false positives are the predicted categories Compared to the real category Inconsistent samples; while low-confidence samples are correctly predicted and have the highest class probability. The sample, This is the threshold (usually set to 0.7); ; S608, Sample Weighting Mechanism: This mechanism primarily encourages the model to focus more on samples it deems "difficult" during training. The process involves first identifying "difficult examples"—those with low prediction confidence or incorrect predictions—and categorizing them into a difficult example set. Then, when calculating the loss, difficult examples are assigned higher weights, while ordinary samples retain their standard weights. This strategy forces the model to prioritize challenging problems during optimization, thereby improving its ability to fit complex data and avoiding wasting time on simple samples.

[0049] For each sample Assign weights Difficult cases are given higher weight: ; in, For the first One sample and its label, For the set of difficult examples, For logical OR, For the maximum class probability, For difficult cases, the magnification factor is used. S609, Weighted Focus Loss Function: A focus loss mechanism is introduced to further reduce the weight of easily separable samples and focus on difficult examples: ; in: This is the loss value for the third stage, which is the final error value calculated for backpropagation. This represents the total number of samples (or the batch size). To predict the probability of the true class for the model; This is the focusing parameter (usually set to 2.0), used to adjust the weight balance between easy and difficult samples; To balance the weights across categories, higher weights are assigned to rare defect categories.

[0050] S7. Model Deployment and Inference: Specifically, the fine-tuned base model is compressed using operator fusion and quantization techniques (such as INT8 quantization) and deployed to industrial edge computing devices (such as GPU industrial control computers or embedded AI modules) for real-time online detection. To meet the real-time requirements of industrial production lines (>30fps), the optimized model undergoes structured pruning, knowledge distillation (using the large model as a teacher to guide student models), and INT8 quantization. While ensuring a mAP reduction of <1%, the inference speed is improved by more than 3 times, and the number of parameters is compressed to a deployable version. The lightweight model is then deployed to edge computing devices and integrated into the production line detection system. S701, Structured Pruning: The task of structured pruning is to remove redundant structured components (such as channels, attention heads, and Transformer layers) from the model, reducing the number of parameters and computational cost while maintaining the regularity of the model architecture.

[0051] 1) Definition of pruning objects: The pruning structure in the model is formally defined as shown in Table 1: Table 1. Formal Definition of Prunable Structures in the Model

[0052] In the table, Let be the weight tensor of the convolution, representing all learnable parameters of that layer. Let the set be the set of real numbers, indicating that the weight values ​​are real numbers. The number of output channels, i.e., the number of convolutional kernels, determines the depth of the output feature map. The number of input channels represents the depth of the input feature map. Let be the kernel size, indicating that the kernel is a K×K square matrix. This refers to the set of all attention heads in a multi-head attention mechanism. For the total number of attention heads, The set of all layers in the Transformer model. For the number of Transformer layers, The rank of the low-rank matrix in LoRA (Low-Rank Adaptation) controls the number of parameters in the adapter. This is the maximum rank value allowed by LoRA, and is usually set by the user.

[0053] 2) Importance Scoring Criteria To determine which parts of the model to keep and which to remove, we need to calculate an "importance score" for each pruning unit. A lower score indicates that the unit is less important and more likely to be pruned. The scoring criteria are as follows for different model structures: 2.1 Channel Importance Scoring (L1 Norm Criterion): For convolutional layers, the importance of channels is evaluated by calculating the sum of the absolute values ​​of the weights. If the weight values ​​corresponding to an output channel are generally very small (close to 0), it means that the features extracted by this channel contribute little to the final result. The absolute values ​​of all weights of all channels, including the input channel dimension, the height of the convolution kernel, and the width dimension, are added together. The smaller the sum, the less important the channel is.

[0054] 2.2 Attention Importance Score: For the multi-head attention mechanism in Transformers, we evaluate its importance by calculating the L2 norm of the gradient of the loss function with respect to the output of each attention head. If a small change in the output of an attention head causes a drastic fluctuation in the final task loss, it indicates that this head is crucial (sensitive). Conversely, if its loss remains unchanged regardless of changes, it is not important. On the evaluation dataset, we calculate the L2 norm of the gradient of the task loss with respect to the output of a specific attention head and take the average value. The larger this value, the more important the attention head is.

[0055] 2.3 Importance scoring of the Transformer layer: For the entire Transformer layer, its importance is evaluated by comparing the "feature differences before and after skipping the layer". For example, if removing the 3rd layer (i.e., directly connecting the 2nd and 4th layers) results in very little change in the feature map, it indicates that this layer does not perform any substantial work and is redundant. 3) Post-pruning structural constraints To ensure that the pruned model not only theoretically becomes smaller but also runs efficiently on real-world edge devices, the following hard constraints must be imposed: The pruned architecture constraints aim to ensure that the model remains efficient and maintains performance after compression. Specifically, the number of channels after pruning must be adjusted to a multiple of 8 to meet hardware requirements for memory alignment and avoid computational inefficiency. Simultaneously, to maintain the model's expressive power, at least half the original Transformer layers must be retained to prevent accuracy collapse due to insufficient layers. Furthermore, the LoRA rank cannot be lower than 4 to ensure sufficient fitting ability to learn task knowledge. These hard constraints collectively guarantee the practicality and stability of the pruned model on edge devices.

[0056] S702, Knowledge Distillation: The task of knowledge distillation is to use the large model before pruning as the teacher to guide the learning of the lightweight student model after pruning, so as to maintain detection accuracy while reducing the number of parameters.

[0057] 1) The teacher-student model is defined as shown in Table 2: Table 2 Teacher-Student Model Definition Table

[0058] 2) Distillation loss function design: The total loss function consists of two parts: hard-label loss and soft-label loss. ; 2.1 Hard Label Loss: ; in: Labels for actual defect categories. For student model prediction, For the total loss of knowledge distillation, This is the balance coefficient (weighting factor). Hard label loss measures the difference between the student model's predictions and the true labels. The soft label loss measures the difference between the predicted distribution of the student model and the predicted distribution of the teacher model (usually calculated using KL divergence). Cross-entropy loss function is the most commonly used loss function in classification tasks, used to calculate the difference between two probability distributions; 2.2 Soft Tag Loss (Teacher Knowledge Transfer): ; in: The output is the logits (raw scores) of the teacher model. The output is the logits (raw scores) for the student model. For temperature (usually) ∈[3,10]), used to soften the probability distribution, Divergence is used to measure the difference between two probability distributions. 2.3 Intermediate layer characteristic distillation (optional enhancement): In addition to output layer distillation, it can also align intermediate layer feature representations: ; in: The set of layer indices that need to be aligned; For feature projection function (mapping teacher feature dimensions to student dimensions); For the first Feature map of the layer The feature distillation loss is an indicator that measures the difference between the student model and the teacher model in the intermediate layer feature representation; the smaller the loss, the closer the features learned by the student model are to those of the teacher model.

[0059] 3) Temperature parameter scheduling strategy: Temperature parameters To improve the smoothness of soft labels, a gradual cooling strategy is employed. ; in: In the early stages of training, soft tags are smoother and easier for students to learn; To improve the decision boundary in the later stages of training, the labels are closer to hard labels. This represents the total number of epochs (rounds) in the distillation training.

[0060] 4) Recommendations for distillation weighting are shown in Table 3: Table 3. Recommended Distillation Weight Configuration

[0061] S703, INT8 Quantization: 1) Quantification object: For model weights and activation value Perform 8-bit integer quantization: ; ; in: The quantized weights (8-bit integers). This is a rounding function. The original weights (32-bit floating-point numbers). This is the quantization scaling factor for the weights. This is the quantized activation value (an 8-bit integer). The quantization scaling factor for the activation value; 2) Quantitative perception training: Quantization simulation is introduced in the later stages of distillation training to reduce the loss of quantization accuracy. 1. Insert pseudo-quantization nodes during forward propagation to simulate INT8 quantization error; 2. Gradients bypass quantization nodes during backpropagation (pass-through estimator); 3. Fine-tune the model over 5 epochs to adapt it to quantization noise; 3) Calibration after quantization; The core purpose of post-quantization calibration is to determine the optimal scaling factor for each layer using a small calibration set (such as 500 images) to minimize quantization error. Specifically, it involves finding a scaling factor that ensures the output of the quantized model is as consistent as possible with the output of the original floating-point model when given the same calibration data, thus minimizing the accuracy loss of the quantized model.

[0062] S704, model compression and deployment effect expectations are shown in Table 4: Table 4 Expected Effects of Model Compression and Deployment

[0063] S8. Real-time detection, adaptive threshold and continuous optimization: During the detection process, false detection and missed detection images are automatically collected, and after manual review, they are added to the training set. Incremental fine-tuning is performed regularly to make the model adapt to changes in production line conditions, forming a data closed loop and continuous model evolution.

[0064] After loading a lightweight model, the edge detection device performs real-time detection using the S7 industrial edge computing device. It performs dynamic patching, multi-expert collaborative reasoning, cross-scale feature fusion, and temporal context fusion (if enabled) on real-time production line images, outputting defect candidate box confidence scores. This includes the following steps: S801, Dynamic Patch Division: Adaptively divides the detection region based on image content, balancing detection accuracy and inference speed, including the following steps: Input to the dynamic patch partitioning algorithm: Input image ; The base patch size is ; sliding step : Output: Patch collection ; The implementation steps are as follows: S80101, Image Complexity Evaluation: Calculate the image gradient map: ; Calculate the local complexity score: ; Generate a complexity heatmap: ; S80102, Adaptive Patch Size Determination: The size of the processing patch is dynamically adjusted based on the complexity of the image region. The process first calculates the average complexity of each candidate region, which is the average of the local complexity scores of all pixels within that region. Then, based on this average complexity, the base patch size is scaled up by a certain factor (typically between 0.2 and 0.5). This means that for highly complex regions with rich textures and intricate details, the system automatically allocates a larger patch size to capture more contextual information; while for simpler regions, a smaller size is maintained, thus optimizing the allocation of computational resources while ensuring processing accuracy.

[0065] For each candidate region Computational average complexity : ; Determine the patch size: ; in: This is a complexity scaling factor. For image gradient, The neighborhood radius is used to calculate local complexity. For image gradient maps, coordinates Local complexity score at the location, These are the coordinates of the image pixels. This is the relative offset within the local window. The offset index within the neighborhood window. coordinates The complexity heatmap values ​​at the location, This refers to the patch size after regional adjustment. The average complexity of the region; S80103, constraint patch size is within Within the range, Minimum height, For maximum height, Minimum width, The maximum width is set; specifically, regardless of the height and width calculated in step S80102, they must ultimately fall within the preset minimum and maximum values. If the calculated value exceeds this range, it will be forcibly truncated to the boundary value. This mechanism ensures that the patch size is always within a reasonable and controllable range, thereby guaranteeing the stability and standardization of subsequent processing procedures.

[0066] S80104, Patch Generation and Overlap Control: Generate a collection of patches according to adaptive size. ; Controlling the overlap of adjacent patches: Filter out patches whose area is too small to the preset minimum defect size; S80105, Return to patch collection ; S803, Cross-scale Feature Fusion: This method fuses feature maps from multiple scales to improve the detection capability of defects of different sizes. It includes the following steps: S80301, Multi-scale Feature Extraction: Extract from backbone network Feature maps at each scale: ; in: This typically refers to the deepest feature map (lowest resolution, largest receptive field, and richest semantic information). For the first Feature maps at various scales For the first The height of the feature map at each scale. The width of the feature map. The number of channels in the feature map; scale. The larger the feature map, the lower its resolution and the richer its semantic information.

[0067] S80302, Feature Pyramid Fusion: A top-down feature pyramid structure is used for fusion: ; in: For the first Features after layer fusion For 1x1 convolution, For upsampling operation, This is the current processing level; S80303, Scale Attention Weighting: Assign attention weights to each scale feature: ; ; in: For the first Attention weights of layers For the first to be weighted Layer features, For global average pooling; S804, Temporal Context Fusion (if enabled): Utilizes historical information from video sequences to improve detection stability in dynamic scenes. First, a short-term memory (SSM) is established to cache features from the current frame and past video frames (typically 5-10 frames). Next, attention weights are assigned by calculating the similarity between the current frame and historical frames, thereby identifying which historical moments are most valuable for the current judgment. Finally, these weights are used to perform weighted fusion of historical features to generate "temporally enhanced features" that include temporal context information. This process allows the model to make judgments that no longer rely on isolated single frames, but rather combine coherent historical information, effectively reducing jitter and making the detection results smoother and more stable.

[0068] S80401, Time-Sequence Feature Cache: Maintenance length is Feature cache queue: ; in: The fusion features of the current frame, For feature cache queue; S80402, Temporal Attention Fusion: Calculate the attention weights for the current frame and historical frames: ; in: For attention weights, It is an exponential function. The cosine similarity function; S80403, Timing Enhancement Features: ; in: For temporal enhancement features, Here, represents the fusion features of historical frames, and represents the attention weights. This is the time offset. This represents the length of the time window.

[0069] S805, Adaptive Threshold Module: The adaptive threshold module dynamically calculates the threshold based on the confidence distribution of historical normal samples, identifies defects, and displays the results in real time; it includes the following steps: Modeling the confidence distribution of historical normal samples: S80501, Collection of Confidence Levels for Normal Samples: Collecting history The highest background confidence score for a normal sample (without defects): ; in: For the first The highest class confidence score output by the model among all normal samples; S80502, Distribution Fitting: Fit the confidence distribution of normal samples using a Gaussian mixture model (GMM): ; in: This represents the number of Gaussian components (usually set to 2~3). For the first The mixing coefficient of each component, ; For the first The mean of the Gaussian components, For the first The standard deviation of each Gaussian component For background confidence, The function is a Gaussian distribution (normal distribution). Let be the probability density function of the confidence level; S80503, Online Update of Distributed Parameters: The distribution parameters are updated using an exponential moving average (EMA). A weighted fusion method combines historical statistical parameters (mean and standard deviation) from the previous time step with real-time estimates from the current batch to calculate the updated distribution parameters. During this process, a relatively large smoothing coefficient (typically between 0.9 and 0.95) is introduced, meaning the updated parameters primarily rely on long-term historical accumulation and are only slightly affected by the current batch data. This strategy not only makes parameter updates smoother and more stable but also effectively filters out potential noise interference from the current batch data, ensuring a more accurate and reliable tracking of the data distribution.

[0070] The distribution parameters are updated using an exponential moving average: ; ; in: For smoothing coefficients, This is an estimate for the current batch. This is the updated mean. This is the average value from the previous time step. The updated standard deviation, The standard deviation of the previous time step. This represents the estimated standard deviation for the current batch. S806, Dynamic Threshold Calculation Steps: S80601. Calculate the cumulative distribution: ; in: The cumulative distribution function is... For confidence level variables; The cumulative distribution function accumulates the confidence probability distributions of background samples through integration. Simply put, for any given confidence value, this function calculates the total probability that the background confidence is less than or equal to that value. By constructing this cumulative distribution function, the system can clearly understand the overall distribution of background confidence and determine how many background samples fall within different confidence intervals.

[0071] S80602. Determine the basic threshold: Solution: ; Right now: ; in: This represents the cumulative probability at the threshold. The preset extremely low probability (false alarm rate). Based on the threshold, The formula for the tail probability; The system sets an extremely small probability value to find a specific threshold point, such that the probability of a background sample's confidence exceeding this threshold is exactly equal to the false alarm rate. This process utilizes the inverse operation of the cumulative distribution function to truncate a very small region at the tail of the probability distribution, thereby ensuring that only a very small number of background noise samples are incorrectly identified as high-confidence targets.

[0072] S80603, Confidence Calibration: Considering defect size factors: ; Considering defect category factors: ; in: For category-specific offsets, The threshold after size adjustment. This is the size adjustment factor. The area of ​​the current defect. For reference area; Consider timing consistency factors (if enabled): ; in: The threshold after time-series adjustment. The threshold after category adjustment. This is the consistency weighting coefficient. The temporal consistency score is an indicator that measures whether the defect behavior in the current frame is consistent with that in the preceding and following frames. Threshold boundary constraints: ;generally ; in: The lower limit of the threshold. For the final dynamic threshold, The upper limit of the threshold; Prevent threshold fluctuations: ; Finally return ; in: (The left side of the equals sign) is the final threshold after smoothing. (The right side of the equals sign) represents the currently calculated threshold. This is the threshold value from the previous time step.

[0073] S9. Output the detection results.

[0074] The present invention also includes a readable storage medium storing a computer program adapted to be loaded by a processor and executed as described above for the method of detecting multiple defects in industrial metals.

[0075] The present invention also includes a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, performs the multi-defect detection method for industrial metals as described above.

[0076] This invention aims to address the problems of weak generalization ability, easy omission of small targets, high cost of cross-production line migration, and insufficient real-time performance of traditional models in industrial visual inspection. The invention's workflow is designed for common defect types on industrial metal surfaces (dents, inclusions, patches, scratches, foreign objects), encompassing a complete technical solution from data preparation, model selection, model fine-tuning, optimization to deployment and continuous learning. It is suitable for the development of high-precision visual defect detection systems in intelligent manufacturing scenarios. Figure 2 and Figure 3 As shown, the following is an embodiment of this defect detection method: 1) Data preparation: Five thousand images of steel surfaces from the factory's production line were collected, covering three types of defects: scratches, pinholes, and cracks. Due to the scarcity of defect samples, a generative adversarial network (GAN) was used to synthesize some rare crack samples, and the training and validation sets were divided in a 4:1 ratio. The labeled data was converted into an instruction format, such as input images. ,instruction =“Surface defects detected in the image”, expected output =“A scratch was found, coordinates [x1,y1,x2,y2]”.

[0077] 2) Model building: The open-source Qwen3.5-35B-A3B was chosen as the base model. An MSP-LoRA module was connected in parallel after the Self-Attention Layer in each of its Transformers.

[0078] Set the rank of high-frequency branches Low-frequency branch rank .

[0079] 3) Prompt for initialization: Initialize 10 learnable Visual Prompt tokens and design a text template “This is an image of an industrial metal with a {cls} defect”, where {cls} is a learnable vector.

[0080] 4) Fine-tuning training: Training was performed on an NVIDIA RTX A6000 graphics card (Note: If video memory is insufficient, DeepSpeed ​​or quantization techniques may be required).

[0081] In the first phase, training is conducted for 20 epochs, with a learning rate set at [value missing]. Only the Prompt and MS-LoRA parameters are updated.

[0082] In the second phase, train for 10 epochs and set the learning rate. Unfreeze the last two Transformer Blocks.

[0083] The loss function uses Focal Loss to address the imbalance between positive and negative samples, and IoU Loss is added to optimize localization accuracy.

[0084] 5) Result verification: To more intuitively demonstrate the superiority of this invention (hereinafter referred to as "this solution") over the prior art, we compared three mainstream methods under the same hardware environment (NVIDIA RTX A6000) and the same dataset (5000 images of steel surfaces): 1. Baseline Method 1: Traditional industrial inspection model YOLOv8.

[0085] 2. Baseline Method 2: Full Fine-tuning for large models.

[0086] 3. The method of this invention: a fine-tuning method based on Qwen3.5-35B-A3B+MSP-LoRA.

[0087] 5.1.1 Comparison Table of Key Performance Indicators: Table 5 shows the comparison results of the three methods in terms of detection accuracy, resource consumption, and inference speed: Table 5 Comparison of detection accuracy, resource consumption, and inference speed

[0088] Comparative analysis: Accuracy Advantage: This solution outperforms the traditional YOLOv8 model (+4.2%) in mAP, mainly due to the powerful contextual understanding capabilities of the large model. In microcrack detection, the false negative rate is only 3.2%, far lower than the 18.5% of the traditional model, demonstrating the effective capture of high-frequency detail features (such as crack edges) by the MSP-LoRA module.

[0089] Efficiency advantage: Compared with full fine-tuning, this invention reduces the video memory usage from 300GB to 64GB, significantly lowering the hardware threshold for industrial application.

[0090] 6) Lightweight Model and Industrial Deployment: To meet the real-time requirements of industrial production lines (typically >30fps), model compression and deployment are performed: a) Structured pruning: Based on Taylor expansion or BN layer gamma values, redundant channels are removed, and the model volume is compressed by 30% to 50%.

[0091] b) Knowledge distillation: A small student model (such as ResNet-50+ lightweight detector head) is trained using the finely tuned large model as the teacher, and the accuracy is maintained through feature distillation and output distillation.

[0092] c) INT8 quantization: Using TensorRT or OpenVINO to quantize the model with 8 bits can improve inference speed by 2 to 3 times while keeping the accuracy loss within 1%.

[0093] d) Deployment: The quantized model is deployed to edge computing devices and integrated into the detection system.

[0094] 7) Continuous learning and model evolution: a) New sample refluxing: During production line operation, false positive and false negative images are automatically collected, and after rapid manual review, they are added to the training set.

[0095] b) Incremental fine-tuning: At regular intervals (e.g., weekly), incremental learning is performed on the deployed model using new samples to adapt the model to changes in production line conditions (e.g., light source attenuation, new batches of materials).

[0096] c) Version Management: Record the model version, training data, and performance metrics for each update to create a traceable model lifecycle archive.

[0097] By following the seven steps above, a dedicated large-scale model for multi-defect detection in industrial metals can be built, which takes into account high precision, high generalization ability and real-time performance, and can continuously evolve through continuous learning to meet the high requirements of quality control in intelligent manufacturing.

[0098] The above description is only a preferred embodiment of the present invention and does not limit the scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the description and drawings of the present invention, or direct / indirect applications in other related technical fields, are included within the protection scope of the present invention.

Claims

1. A method for detecting multiple defects in industrial metals, characterized in that, Includes the following steps: S1. Collect industrial metal data; S2. Construct a multimodal instruction dataset for industrial metal defects; S3. Build a pre-trained large visual model base. Specifically, based on the industrial metal defect multimodal instruction dataset, select a pre-trained large visual model as the base model for fine-tuning, freeze the parameters of its bottom and middle layers, and retain the top layer parameters as adjustable; connect a lightweight detection head at the decoding end. S4. Insert the improved low-rank adapter module into the model architecture block of the base model to obtain the adapted base model. In the multi-scale perceptual low-rank adapter module, a lightweight dynamic gating network automatically learns the fusion weights of high-frequency and low-frequency branches in the multi-scale perceptual low-rank adapter module based on the local or global statistical characteristics of the current input feature map. The steps are as follows: S401. Input Preparation: Let the input feature map of the current model architecture layer be... ,in, For batch size, For spatial height dimension, For the spatial width dimension, The number of channels; the feature map is fed into two parallel branches simultaneously: The output of the high-frequency branch is: ; The output of the low-frequency branch is: ; S402. Design a lightweight multilayer perceptron global pooling structure to generate fusion weights for each sample. S403, For each sample index The value range is from 1 to Its final output through the adaptive fusion formula is: ； in: To indicate the first The final fused output feature map of each sample The scalar weights output by the gating network, The output feature map of the corresponding branch is then used; the fusion weights are then integrated into the overall process of the multi-scale perception low-rank adapter module. S5. Introduce defect-aware prompt learning, specifically: add the learned prompt vector to the input end of the adapted base model; S6. Cross-scale feature fusion, specifically: using the cue vectors introduced in S5 and the multi-scale perceptual low-rank adapter module in S4, by freezing all parameters of the base model, only on the trainable set... Optimize the training set Includes visual cue vectors introduced in S5 Independent vectors for defect category learning And the parameters of the multi-scale sensing low-rank adapter module in S4 Adapt the model to the distribution of industrial data; Unfreeze the last two layers of the base model architecture, jointly train the multi-scale perception low-rank adapter module and cue vector, and optimize multi-scale feature fusion; Introduce hard example mining, optimize the weighted focus loss function for false detection samples, fine-tune the detection head parameters, and obtain the fine-tuned base model; S7. Model Deployment and Inference: Specifically, the fine-tuned base model is compressed using operator fusion and quantization techniques and deployed to industrial edge computing devices; the optimized model is then subjected to structured pruning, knowledge distillation, and INT8 quantization. S8 features real-time detection, adaptive thresholding, and continuous optimization. Specifically, it utilizes the S7 industrial edge computing device for real-time detection. The adaptive thresholding module dynamically calculates thresholds based on the confidence distribution of historical normal samples, identifies defects, and displays the results in real time. During the detection process, false positives and false negatives are automatically collected, reviewed, and added to the training set. Periodic incremental fine-tuning is performed to adapt the model to changes in production line conditions, forming a data loop and enabling continuous model evolution. S9. Output the detection results.

2. The method for detecting multiple defects in industrial metals according to claim 1, characterized in that, Cue vectors include visual cues and textual cues; S5 includes the following steps: S501, Visual Cue Construction: Define a set of learnable visual cue vectors : ； in: To indicate the length, For the hidden layer dimension of the model, It is the set of real numbers; Image block embedding: embedding the input image Divided into Each patch is used to obtain the image embedding through linear projection. ; Cue splicing: combining visual cue vectors spliced ​​before the image embedding sequence: ； in: For the final visual input sequence, For splicing operations, The image embedding sequence defined above; Location encoding: a visual cue vector Assign independent location codes; S502, Text Prompt Construction: Template Design: Define a collection of text templates ; Learnable category embeddings: Learn independent vectors for each defect category : ； in: For the number of defect categories, For the embedding dimension of the model; Text encoding process: Convert the fixed text portion in the template into an integer number; simultaneously perform category vector replacement, changing the text in the template... Replace the placeholder position with the corresponding ,in For category index; final input is: ； in: The text segment before the placeholder. The text fragment following the placeholder. For the final text input sequence, The total length of the sequence. For text encoding / embedding functions.

3. The method for detecting multiple defects in industrial metals according to claim 2, characterized in that, When using the cue vectors from training S5 and the multi-scale perceptual low-rank adapter module from S4 to adapt the model to the industrial data distribution by freezing all parameters of the pedestal model, the following steps are included: S601, Parameter Freezing and Grouping Strategy: Freezing and grouping model parameter sets Divided into frozen sets With trainable set : Among them, frozen collection It includes all self-attention layers and feedforward neural network layer parameters of the pedestal model, as well as the main parameters of the visual encoder; ; ； S602, Differentiated Learning Rate Configuration: Assign differentiated learning rates to the trainable parameter set. The configuration is as follows: Hint vector group Assign learning rate ; Adapter group Assign learning rate ; S603. Construction of Joint Loss Function: Construct a joint loss function that includes both classification and feature alignment tasks. : ； in: Cross-entropy loss is used for constraint defect classification and prediction. With real labels Consistent; To compare the loss, bring the image features closer together. Corresponding text features Distance in the embedded space; This is the balance coefficient.

4. The method for detecting multiple defects in industrial metals according to claim 3, characterized in that, In the final two layers of the unfrozen pedestal model architecture, when jointly training the multi-scale perceptual low-rank adapter module and cue vectors, and optimizing multi-scale feature fusion, the following steps are included: S604, Parameter Unfreezing Strategy: While keeping most base parameters frozen, selectively unfreeze the final parameters of the base model. Layered model architecture blocks: ； The total trainable parameter set is now updated to: ; S605, Cross-scale feature fusion mechanism: Assuming the model extracts... Feature maps at different scales By fusing features through a scale attention mechanism, features at different scales are first... Unify to the same dimension through linear mapping Then, the attention weights for each scale feature are calculated. : ； Finally, generate fusion features. : ； S606, Multi-scale Consistency Loss: Introducing consistency regularization loss : ； in: For the set of thawing parameters, For the total set of trainable parameters, The second to last layer of the model. This is the last layer of the model. To convert the model's raw output into a probability distribution, It is a multilayer perceptron. For average pooling, This is a scale index. The separator, in the KL divergence formula, represents the calculation of the difference between the previous distribution and the next distribution. For the first The original prediction output at each scale, For the first +1 scale of original prediction output, Let KL divergence be denoted as KL divergence.

5. The method for detecting multiple defects in industrial metals according to claim 4, characterized in that, When introducing hard example mining and optimizing the weighted loss focus function for false positive samples, the following steps are included: S607, Difficult Example Discovery Strategy: Before the start of each round, use the current model to perform inference on the training set once to define a set of difficult examples. ; where false positives are the predicted categories Compared to the real category Inconsistent samples; while low-confidence samples are correctly predicted and have the highest class probability. The sample, For threshold; ； S608, Sample weighting mechanism: For each sample Assign weights Difficult cases are given higher weight: ； in, For the first One sample and its label, For the set of difficult examples, For logical OR, For the maximum class probability, For difficult examples, the magnification factor is... S609, Weighted Focus Loss Function: A focus loss mechanism is introduced to further reduce the weight of easily separable samples and focus on difficult examples: ； in: This represents the loss value for the third stage. The total number of samples, To predict the probability of the true class for the model; For focusing parameters; To balance the weights for each category.

6. The method for detecting multiple defects in industrial metals according to claim 1, characterized in that, When performing real-time inspection using the S7 industrial edge computing device, dynamic patching, multi-expert collaborative reasoning, cross-scale feature fusion, and temporal context fusion are performed on the real-time images of the production line to output the confidence score of the defect candidate box. The steps include: S801, Input to the Dynamic Patch Partitioning Algorithm: Input image ; The base patch size is ; sliding step : Output: Patch collection ; The implementation steps are as follows: S80101, Image Complexity Evaluation: Calculate the image gradient map: ; Calculate the local complexity score: ; Generate a complexity heatmap: ; S80102, Adaptive Patch Size Determination: For each candidate region Computational average complexity : ； Determine the patch size: ; in: This is a complexity scaling factor. For image gradient, The neighborhood radius is used to calculate local complexity. For image gradient maps, coordinates Local complexity score at the location, These are the coordinates of the image pixels. This is the relative offset within the local window. The offset index within the neighborhood window. coordinates The complexity heatmap values ​​at the location, This refers to the patch size after regional adjustment. The average complexity of the region; S80103, constraint patch size is Within the range, among which, Minimum height, For maximum height, Minimum width, Maximum width; S80104, Patch Generation and Overlap Control: Generate patch set according to adaptive size ; Controlling the overlap rate of adjacent patches: Filter out patches whose area is smaller than the preset minimum defect size; S80105, Return to Patch Collection ; S803, Cross-scale feature fusion, includes the following steps: S80301, Multi-scale Feature Extraction: The system extracts multi-scale feature maps from the backbone network. It obtains a series of feature maps from different layers of the backbone network, which cover multiple scales from shallow to deep layers. Among them, the feature maps of the deepest layer usually contain the richest semantic information, while the feature maps of other layers correspond to different resolutions and number of channels. Together, they constitute a multi-scale feature representation. Each scale feature map has its specific height, width, and number of channels. These parameters determine the spatial resolution and feature dimension of the feature map. S80302, Feature Pyramid Fusion: A top-down feature pyramid structure is used for fusion: ； in: For the first Features after layer fusion For 1x1 convolution, For upsampling operation, This is the current processing level; S80303, Scale Attention Weighting: Assign attention weights to each scale feature: ； ； in: For the first Attention weights of layers For the first to be weighted Layer features, For global average pooling; S804, Timing Context Fusion: S80401, Time-Sequence Feature Cache: Maintenance length is Feature cache queue: ； in: The fusion features of the current frame, For feature cache queue; S80402, Temporal Attention Fusion: Calculate the attention weights for the current frame and historical frames: ； in: For attention weights, It is an exponential function. The cosine similarity function; S80403, Timing Enhancement Features: ； in: For temporal enhancement features, Here, represents the fusion features of historical frames, and represents the attention weights. This is the time offset. This represents the length of the time window.

7. The method for detecting multiple defects in industrial metals according to claim 6, characterized in that, When the adaptive threshold module dynamically calculates the threshold based on the confidence distribution of historical normal samples, the following steps are included: S805, Historical Normal Sample Confidence Distribution Modeling: S80501, Collection of Confidence Levels for Normal Samples: Collecting history The highest background confidence score for a normal sample: ； in: For the first The highest class confidence score output by the model among all normal samples; S80502, Distribution Fitting: Fit the confidence distribution of normal samples using a Gaussian mixture model: ； in: The number of Gaussian components. For the first The mixing coefficient of each component, For the first The mean of the Gaussian components, For the first The standard deviation of each Gaussian component For background confidence, The Gaussian distribution function is... Let be the probability density function of the confidence level; S80503, Online Update of Distributed Parameters: The distribution parameters are updated using an exponential moving average: ； ； in: For smoothing coefficients, This is an estimate for the current batch. This is the updated mean. This is the average value from the previous time step. The updated standard deviation, The standard deviation of the previous time step. This represents the estimated standard deviation for the current batch. S806, Dynamic Threshold Calculation Steps: S80601. Calculate the cumulative distribution function: ； in: The cumulative distribution function is... For confidence level variables; S80602. Determine the basic threshold: Solution: ; Right now: ; in: This represents the cumulative probability at the threshold. For a preset extremely small probability, Based on the threshold, The formula for the tail probability; S80603, Confidence Calibration: Considering defect size factors: ； Considering defect category factors: ； in: For category-specific offsets, The threshold after size adjustment. This is the size adjustment factor. The area of ​​the current defect. For reference area; Considering timing consistency factors: ； in: The threshold after time-series adjustment. The threshold after category adjustment. This is the consistency weighting coefficient. The temporal consistency score is an indicator that measures whether the defect behavior in the current frame is consistent with that in the preceding and following frames. Threshold boundary constraints: ； in: The lower limit of the threshold. For the final dynamic threshold, The upper limit of the threshold; Prevent threshold fluctuations: ； Finally return ; in: The final threshold after smoothing. The currently calculated threshold, This is the threshold value from the previous time step.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-7 for the detection of multiple defects in industrial metals.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, runs the method for detecting multiple defects in industrial metals according to any one of claims 1-7.

Images