A multi-modal industrial intelligent detection method and system based on efficient reinforcement fine-tuning and rule checking

Abstract

The application discloses a kind of based on efficient reinforcement fine tuning and rule check multi-modal industrial intelligent detection method and system, method includes: S1.to industrial detection data is divided, and reference sample library and industrial acceptance rule base are constructed;S2.in frozen multi-modal base model, introduce interlayer collaborative adjustment module, and build feature space;S3.in feature space, extract locatable abnormal evidence, and output structured detection result that can be directly accessed in industrial acceptance rule base and check;S4.structurized detection result is input to industrial acceptance rule base and is checked, obtains industrial rule check result;S5.based on industrial rule check result, the related parameter of interlayer collaborative adjustment module is reinforced fine tuning, obtains multi-modal detection model;S6.based on multi-modal detection model, the abnormal detection and industrial acceptance determination of sample to be detected on production line are carried out, obtains final detection result.

Description

Technical Field

[0001] This invention belongs to the field of intelligent detection technology in industrial manufacturing, specifically relating to a multimodal industrial intelligent detection method and system based on efficient enhancement fine-tuning and rule verification. Background Technology

[0002] With the evolution of intelligent manufacturing technology, intelligent detection technology based on deep learning and multimodal large models has become a core means of surface quality control for industrial products. Existing technologies mainly utilize visual or multimodal pre-trained basic models, achieving the identification and localization of abnormal regions in industrial images through pre-training on large-scale general datasets and fine-tuning under specific scenarios. However, when facing manufacturing scenarios with complex and variable process standards and extremely scarce defect samples, existing mainstream technical solutions still have the following significant limitations:

[0003] 1. Existing models lack a mechanism to transform discrete process rules into reward and penalty signals in the optimization closed loop, making it difficult to accurately determine the practical significance of defects based on physical properties.

[0004] Existing technologies, whether unsupervised methods based on reconstruction errors or multimodal methods based on semantic alignment, primarily rely on statistical distribution differences in sample data or visual-semantic similarity for optimization objectives. However, in practical industrial applications, defect judgment logic does not depend solely on visual differences but is strictly constrained by specific process attributes. Existing technologies generally lack an effective mechanism to directly map industrial acceptance rules into reward / penalty signals during model training. This results in models passively fitting data labels rather than actively utilizing rule feedback to calibrate their understanding of the physical meaning of features. When faced with the same physical feature having drastically different judgment criteria under different products or operating conditions (i.e., physical meaning changing with context), existing models struggle to adaptively adjust judgment boundaries, leading to detection results that do not strictly conform to actual process acceptance specifications.

[0005] 2. Existing fine-tuning methods lack direct communication and coordination mechanisms between different network layers, which limits their ability to quickly adapt features to small batches of data.

[0006] In industrial production line changeover scenarios, models are typically required to achieve rapid adaptation with minimal training data. However, existing Parameter Efficient Fine-Tuning (PEFT) methods usually involve inserting adapters or low-rank matrices into specific layers of the pre-trained model and relying on the network's inherent sequential connections for backpropagation updates. This hierarchical update approach is limited by the properties of Hidden Markov Chains (HMCs), meaning that the state update of each layer is mainly influenced by its adjacent layers, lacking direct information interaction and collaborative adjustment capabilities across layers. This limitation makes it difficult for supervisory signals from the output to directly and effectively affect the early layers (shallow layers) of the network, resulting in a lack of inter-layer communication. Under conditions of small-batch training and fewer iterations, early layers struggle to break the sequential propagation constraint to quickly correct the response patterns of basic features, thus hindering the rapid convergence and improvement of the overall model performance.

[0007] Therefore, existing technologies lack a detection method that can both integrate industrial rule verification into a reinforcement learning reward and punishment loop to solve the problem of physical meaning determination, and achieve rapid feature adaptation under small sample conditions by establishing an efficient inter-layer communication mechanism. This restricts the application of intelligent detection systems in complex production lines. Summary of the Invention

[0008] This invention aims to address the shortcomings of existing technologies and provides the following solutions:

[0009] A multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification includes the following steps:

[0010] S1. Divide the industrial testing data and construct a reference sample library and an industrial acceptance rule library;

[0011] S2. Under the premise of freezing the parameters of the multimodal basic model, an inter-layer collaborative adjustment module is introduced into the frozen multimodal basic model to perform consistent modeling of the multi-layer features output by the visual encoder and construct a feature space suitable for anomaly comparison and rule verification.

[0012] S3. Extract locatable abnormal evidence from the feature space and output structured detection results that can be directly accessed and verified in the industrial acceptance rule base;

[0013] S4. Input the structured test results into the industrial acceptance rule base for verification to obtain the industrial rule verification results;

[0014] S5. Based on the industrial rule verification results, and on the premise of freezing the multimodal basic model parameters, the relevant parameters of the inter-layer collaborative adjustment module are strengthened and fine-tuned to obtain the multimodal detection model;

[0015] S6. Based on the multimodal detection model, perform anomaly detection and industrial acceptance judgment on the samples to be tested on the production line to obtain the final test results.

[0016] Preferably, S1 includes:

[0017] Define the industrial testing data, and divide the industrial testing data into training set, validation set and test set;

[0018] Defect bounding boxes are automatically generated from defect masks;

[0019] Based on the training set, the reference sample library is constructed from the normal sample subset;

[0020] Based on the defect boundary box, the industrial acceptance rule base is constructed.

[0021] Preferably, S2 includes:

[0022] A single image is obtained from the training set as an input image sample and input into a pre-trained visual encoder for feature extraction.

[0023] Based on the extracted features, an inter-layer collaborative adjustment module is introduced into the frozen multimodal basic model, and the normalized collaborative weight and collaborative adjustment vector of the inter-layer collaborative adjustment module are calculated.

[0024] To control the influence of the collaborative adjustment vector on the original features, a gating mechanism is introduced, and a low-rank projection method is used to inject the adjustment information into the original features to obtain the adjusted features;

[0025] The modified feature terms of the adjusted feature are added to the original feature to obtain a consistent feature representation, and the feature space suitable for anomaly comparison and rule verification is constructed.

[0026] Preferably, S3 includes:

[0027] The input image samples are input into the frozen visual encoder and passed through the inter-layer collaborative adjustment module to obtain consistent patch-level features of the input samples.

[0028] Reference image samples are retrieved from the reference sample library and input into the frozen visual encoder and passed through the inter-layer collaborative adjustment module to obtain the uniformized patch-level features of the reference image.

[0029] Calculate the similarity between the uniformized patch-level features of the input sample and the uniformized patch-level features of the reference image, and perform reference feature alignment based on the similarity.

[0030] After feature alignment is completed, a difference is constructed and noise is suppressed by gating to obtain the abnormal evidence;

[0031] The anomalous evidence is mapped to interpretable industrial defect semantics to obtain anomalous semantic features;

[0032] The structured detection result is obtained based on the aforementioned abnormal semantic features.

[0033] Preferably, S4 includes:

[0034] The structured inspection results are input into the industrial acceptance rule base to calculate the structured consistency rule, defect type legality rule, context attribute constraint rule, appearance consistency rule, and evidence consistency rule.

[0035] Each of the obtained rules is verified to obtain the verification results of the industrial rules.

[0036] Preferably, S5 includes:

[0037] Based on the industrial rule verification results, and under the premise of freezing the multimodal basic model parameters, the reward function and the advantage function are calculated.

[0038] Based on the reward function and the advantage function, the gradient of the objective function of the inter-layer collaborative adjustment module is calculated and the LCN parameters are updated. The value network parameters are updated synchronously to obtain the multimodal detection model.

[0039] Preferably, S6 includes:

[0040] The sample to be tested is obtained from the production line and input into the multimodal detection model to obtain the structured detection result of the sample to be tested;

[0041] The structured test results of the sample to be tested are input into the industrial acceptance rule base, and each rule is verified one by one to obtain the rule verification results of the sample to be tested.

[0042] Based on the rule verification results of the sample to be tested, the rule satisfaction score is calculated, and the final detection result is generated.

[0043] This invention also provides a multimodal industrial intelligent inspection system based on efficient enhancement fine-tuning and rule verification. The system applies the above-mentioned methods and includes: a database construction module, a model improvement module, a structured result generation module, a verification result generation module, a model adjustment module, and a final inspection module.

[0044] The database construction module is used to divide industrial testing data and build a reference sample library and an industrial acceptance rule library.

[0045] The model improvement module is used to introduce an inter-layer collaborative adjustment module into the frozen multimodal base model, under the premise of freezing the parameters of the multimodal base model, to perform consistent modeling of the multi-layer features output by the visual encoder and construct a feature space suitable for anomaly comparison and rule verification.

[0046] The structured result generation module is used to extract locatable abnormal evidence in the feature space and output structured detection results that can be directly accessed and verified in the industrial acceptance rule base.

[0047] The verification result generation module is used to input the structured test results into the industrial acceptance rule base for verification, and obtain the industrial rule verification result;

[0048] The model adjustment module, based on the industrial rule verification results and under the premise of freezing the multimodal basic model parameters, strengthens and fine-tunes the relevant parameters of the inter-layer collaborative adjustment module to obtain a multimodal detection model.

[0049] The final detection module performs anomaly detection and industrial acceptance judgment on the samples to be tested on the production line based on the multimodal detection model, and obtains the final detection result.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] (1) This invention solves the problem of inconsistent physical feature judgment standards in cross-category inspection and improves the process adaptability of inspection: The invention innovatively integrates industrial acceptance rule verification into the reward signal closed loop of reinforcement learning, breaking through the limitation of traditional models that rely solely on semantic differences for judgment. By introducing contextual attribute constraints, the model can evaluate the compliance of features based on the process logic of specific products and automatically adapt to the differentiated standards of "the same feature is defective in product A and normal in product B". This mechanism fundamentally eliminates false alarms and false negatives caused by the ambiguity of physical feature definitions, ensuring that the inspection results are not only visually accurate but also conform to the actual production process acceptance specifications.

[0052] (2) Significantly improves model convergence speed and training efficiency under small sample conditions: This invention effectively overcomes the gradient decay problem in the backpropagation process of traditional deep neural networks by constructing a gradient feedback mechanism based on inter-layer collaboration. This mechanism allows the supervision signal from the end layer to bypass the deep backbone network and directly act on the shallow feature extractor, enabling the model to quickly correct its response patterns to basic textures, edges, and other features under the constraints of small batches of training data and very few iterations. This makes this invention particularly suitable for scenarios where industrial production lines frequently change models and defect samples are scarce, significantly reducing the time cost and computing power consumption of model deployment. Attached Figure Description

[0053] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Example 1:

[0058] In this embodiment, as Figure 1 As shown, a multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification includes the following steps:

[0059] S1. Divide the industrial testing data and construct a reference sample library and an industrial acceptance rule library.

[0060] S1 includes: defining industrial inspection data, dividing the industrial inspection data into training set, validation set and test set; automatically generating defect bounding boxes from defect masks; constructing a reference sample library from a normal sample subset based on the training set; and constructing an industrial acceptance rule library based on the defect bounding boxes.

[0061] In this embodiment, in order to achieve aligned modeling, rule-based verification, and automatic acceptance of subsequent anomaly detection results, the industrial detection data is first clearly divided, and a reference sample library and an industrial acceptance rule library are built on this basis, thereby providing a unified data foundation and acceptance standard for subsequent feature modeling, anomaly evidence extraction, and rule-driven optimization.

[0062] Industrial testing data is defined as:

[0063] ,

[0064] in, For industrial testing data, N is the total number of samples in the dataset; Ii Let be the i-th industrial image sample, denoted as . H and W represent the image height and width, respectively; c i The sample is identified by its product category, production line condition, or workstation, used to distinguish different product structures or shooting conditions; M i In this embodiment, a pixel-level defect mask is used to label the defects. ,in This indicates that pixel p belongs to the defect area. This indicates the normal area.

[0065] Dataset Divided into three non-overlapping parts:

[0066] ,

[0067] in, This is the training set, used for offline training and parameter updates of the model; This serves as a validation set, used for performance monitoring and model selection during the training process. This is a test set used for evaluation during the inference phase and for online applications; it does not participate in any parameter updates.

[0068] Furthermore, based on the presence or absence of defects, the training set is divided into:

[0069] ,

[0070] ,

[0071] in, This is a subset of normal samples in the training set that do not contain defects. This is a subset of defective samples in the training set. The reference samples are derived solely from... During the training phase, the samples input to the model in each round (hereinafter referred to as "training input samples") are from... The extraction process therefore includes both normal and abnormal samples; the "test samples" in the inference phase come from... And newly collected data from the production line is strictly separated from the training data.

[0072] To unify subsequent structured output, rule verification, and location evaluation, this implementation method uses defect annotation information M. i Automatically generate defect bounding boxes, denoted as:

[0073] ,

[0074] Among them, B i For the defect bounding box, (x i ,y i) represents the coordinates of the top-left corner of the bounding box; w i and h i Bounding box width and height; BBox() is the minimum bounding rectangle operator, for all... Calculate the minimum bounding rectangle of the set of pixels; when M i When all values ​​are zero (normal sample), B i Record it as an empty frame.

[0075] For subsequent rule calculations, define the bounding box area function:

[0076] ,

[0077] Here, Area() represents the bounding box area function.

[0078] To support subsequent extraction of anomalous evidence based on reference alignment, a reference sample library is constructed from a subset of normal samples:

[0079] ,

[0080] Among them, I r For reference image, c r Meta is used to identify the product category or operating condition of the reference sample. r Additional information for the reference sample (such as shooting location, lighting conditions, etc.) is used as a constraint for subsequent reference retrieval.

[0081] To improve retrieval efficiency, the index features of each reference sample are pre-calculated and cached:

[0082] ,

[0083] Where Ψ() is the index feature extraction function, used to map the image into a d-dimensional global representation; F r These are index features used for fast retrieval of reference samples. These index features are used only for reference retrieval and are independent of the patch-level features used in subsequent difference modeling.

[0084] To enable the test results to directly support production line re-inspection, traceability and acceptance decisions, this implementation method formalizes industrial acceptance requirements into automatically executable rule verification items and performs consistency checks on the model output.

[0085] Let the final structured detection result output by the model be:

[0086] ,

[0087] in, For structured detection results, defect_exist∈{0,1} indicates whether a defect was detected; defect_type is the defect type label; B is the defect bounding box; mask is the defect region mask; s∈[0,1] is the detection confidence level; and severity is the defect severity level or impact level.

[0088] Define the industrial acceptance rule base:

[0089] ,

[0090] in, For the industrial acceptance rule base, each rule r K A corresponding executable verification function:

[0091] ,

[0092] in, This is an executable validation function. Rule types include, but are not limited to, the following:

[0093] (1) Defect type rules

[0094] Used to verify whether the defect type in the detection results is valid:

[0095] ,

[0096] in, For defect type rules, A predefined set of defect types for the production line;

[0097] (2) Rules for consistency of defect appearance

[0098] Used to verify whether the appearance of a defect conforms to its type characteristics. Let it be... The extracted morphological features are (e.g., connectivity, directionality, boundary complexity, etc.), then:

[0099] ,

[0100] in, Let Φ() be the rule for consistency of defect appearance, and let Φ() be the set of morphological feature constraints corresponding to different defect types.

[0101] (3) Constraint rules based on context attributes

[0102] For the same physical feature in different product contexts (c prod To address the issue of inconsistent judgment criteria (e.g., surface roughness is considered a defect in the "Tile" category but normal texture in the "Wood" category), a context-sensitive attribute validation function should be constructed:

[0103] ,

[0104] in, For context-based constraint rules, This is an indicator function that evaluates to 1 if the condition within the curly braces {} is true, and 0 otherwise. For all attribute dimensions k (i.e., universal quantifiers), this represents the requirement for each extracted quantized physical attribute. All of them must fall within their respective valid value ranges. Only then will the rule be considered to have passed the overall verification; c prod This serves as the product category context identifier for the current input sample. The k-th dimension of the quantized physical properties (such as texture roughness, color variance, grayscale mean, etc.) is extracted from the detection region. This is a dynamic threshold mapping function, based on product category c. prod Returns the range of valid values ​​for this attribute. This rule ensures that the model can determine defects based on the specific product's manufacturing standards (rather than a single visual anomaly).

[0105] (4) Dynamic severity and handling rules based on comprehensive judgment (revised):

[0106] This is used to map the validation results of the above rules to the defect's own attributes, resulting in a final action decision, and to verify the executability of the decision. First, the severity is calculated based on the defect type and the degree of attribute violation:

[0107] ,

[0108] Where severity refers to the severity level. This is the severity assessment function. For example, if the defect type is a critical defect (such as Crack), or although it is a general defect, its attribute value significantly exceeds f. context If the defined upper limit of the threshold is reached, the severity level is determined to be high. Then, it is verified whether this severity level corresponds to a valid production line action.

[0109] ,

[0110] in, The validity (or legality) verification result of the handling action is used to determine whether the handling action triggered by the current defect severity falls within the scope of actions that can be executed on the current production line (if yes, the value is 1; otherwise, the value is 0); Action is the mapping function from severity to handling action (or a specific handling instruction generated based on severity), used to convert the pre-calculated defect severity into the actual production line action that needs to be executed; A line A set of processing actions that support the production line (such as automatic rejection, manual re-inspection, shutdown alarm, and release).

[0111] The validation results of each rule are summarized into a rule satisfaction score:

[0112] ,

[0113] Among them, S rule ∈[0,1] represents the satisfaction score; a larger value indicates that the test result better meets the production line acceptance rules. k Let be the importance weight of the k-th rule. This score will be used for feedback optimization during the training phase and as the output for automatic acceptance during the inference phase. Through step S1, the standardized partitioning of the dataset and the definition of context identifiers were completed, and an industrial acceptance rule base containing morphological consistency, context attribute constraints, and dynamic handling logic was constructed, laying the foundation for subsequent rule-based feedback optimization processes.

[0114] S2. Under the premise of freezing the parameters of the multimodal base model, an inter-layer collaborative adjustment module is introduced into the frozen multimodal base model to perform consistent modeling of the multi-layer features output by the visual encoder, and to construct a feature space suitable for anomaly comparison and rule verification.

[0115] S2 includes: obtaining a single image from the training set as an input image sample and inputting it into a pre-trained visual encoder for feature extraction; based on the extracted features, introducing an inter-layer collaborative adjustment module into the frozen multimodal base model, and calculating the normalized collaborative weights and collaborative adjustment vectors of the inter-layer collaborative adjustment module; to control the influence of the collaborative adjustment vectors on the original features, a gating mechanism is introduced, and a low-rank projection method is used to inject the adjustment information into the original features to obtain the adjusted features; the feature modification terms of the adjusted features are added to the original features to obtain a consistent feature representation, and a feature space suitable for anomaly comparison and rule verification is constructed.

[0116] In this embodiment, in order to reduce the inconsistency in scale and semantic expression of features at different levels and to avoid the influence of cross-layer distribution shift on subsequent reference sample alignment and differential modeling, this step introduces an inter-layer collaborative adjustment module on the premise of freezing the multi-modal basic model parameters. This module performs consistent modeling of the multi-layer features output by the visual encoder, thereby constructing a stable feature space suitable for anomaly comparison and rule verification.

[0117] Visual encoder input and basic feature extraction (frozen parameters):

[0118] In this embodiment, the input image sample I obtained in step one is fed into a pre-trained visual encoder for feature extraction, wherein:

[0119] ;

[0120] The visual encoder can be an image coding network based on a convolutional structure or a Transformer, and its parameters are frozen in this embodiment to provide general visual representation capabilities. Let the visual encoder contain L feature extraction layers, then the intermediate feature representation output by the l-th feature layer is denoted as:

[0121] ,

[0122] in, This represents the mapping function of the l-th feature extraction layer in the visual encoder; N represents the feature matrix output of the l-th layer; l C represents the number of spatial units (e.g., the number of patches or feature points) in the l-th layer features; l Let N be the channel dimension of the features at layer l. In a Transformer-based visual encoder, N... l The corresponding number of patch tokens will be processed in subsequent processing steps according to the token dimension.

[0123] To adjust the representation of features at different layers without significantly modifying the basic model structure, a trainable hierarchical adjustment vector is introduced for each feature layer:

[0124] ,

[0125] Among them, u l This represents a trainable vector used to adjust the intensity and direction of the feature response in layer l. The number of adjustment vectors and the channel dimension are much smaller than the size of the base model parameters, thus ensuring efficient parameter updates. To avoid independent effects of adjustment parameters in each layer leading to new inter-layer inconsistencies, an inter-layer collaborative weight matrix is ​​introduced:

[0126] ,

[0127] in, R represents the weight of the collaborative influence of the k-th layer adjustment parameters on the l-th layer features. In forward propagation, R is used to fuse multi-layer contextual information; in backpropagation, R acts as a short-circuit propagation channel for gradients. When the gradient is non-zero, the rule reward / penalty gradient from the last layer L can be directly propagated back to the first layer 1 without going through... The layer-by-layer chain-like decay principle allows shallow features to quickly perceive the semantic constraints of higher layers, thus enabling coordinated adjustment between layers.

[0128] For each feature layer l, calculate its corresponding normalized collaborative weights based on the collaborative weight matrix R:

[0129] ,

[0130] in, This represents the contribution ratio of the regulation information from layer k to layer l; normalization is used to ensure that the regulation information from different layers is numerically comparable. Based on the above weights, the regulation vectors of all layers are merged into the collaborative regulation vector of layer l:

[0131] ,

[0132] in, This represents the collaborative adjustment vector of the l-th layer. This is expressed as a dimension-aligned projection function (used when the number of channels between layers is different). Using this formula, the adjustment vector of the l-th layer... It aggregates feature adjustment information from all network layers.

[0133] To control the influence of the collaborative adjustment vector on the original features, a gating mechanism is introduced to adaptively scale the adjustment information:

[0134] ,

[0135] in, W represents the gating vector of the l-th layer, used to limit the adjustment range; g and b g Let represent the trainable linear mapping parameters; σ() is the Sigmoid activation function. Then, a low-rank projection method is used to inject modulation information into the original features to avoid significantly increasing computational complexity.

[0136] ,

[0137] in, Represents the dimension-reduced projection matrix; Represents the up-dimensional projection matrix; Used to ensure that the parameter scale of the adjustment module is controllable; ⊙ indicates channel-by-channel multiplication operation.

[0138] Adding the adjusted feature correction term to the original feature yields the uniform feature representation of the l-th layer:

[0139] ,

[0140] in, This represents the feature representation after inter-layer collaborative adjustment, which is used for subsequent reference sample alignment and differential modeling. Through inter-layer collaboration and gated injection, the features at different levels are more consistent in semantic expression and numerical distribution, reducing the interference of cross-layer offset on the extraction of abnormal evidence and rule verification.

[0141] Through step S2, this embodiment constructs a parameter-efficient inter-layer collaborative mechanism. Unlike traditional fine-tuning that relies solely on layer-by-layer backpropagation, this mechanism utilizes matrix R to shorten the propagation path of the feedback signal. When reinforcement learning in subsequent steps generates reward / penalty signals based on contextual attribute rules, these signals can be quickly transmitted to the shallow feature extraction module via the direct path formed by the R matrix. This forces the shallow network to rapidly suppress non-compliant feature responses (such as texture noise in specific frequency bands) in very few training iterations, thereby significantly improving the model's rule adaptation efficiency and convergence speed under small sample conditions.

[0142] S3. Extract locatable anomaly evidence in the feature space and output structured test results that can be directly accessed and verified in the industrial acceptance rule base.

[0143] S3 includes: inputting input image samples into a frozen visual encoder and passing them through an inter-layer collaborative adjustment module to obtain uniform patch-level features of the input samples; retrieving reference image samples from a reference sample library, inputting the reference image samples into the frozen visual encoder and passing them through an inter-layer collaborative adjustment module to obtain uniform patch-level features of the reference images; calculating the similarity between the uniform patch-level features of the input samples and the uniform patch-level features of the reference images, and aligning the reference features based on the similarity; after completing the feature alignment, constructing a difference and suppressing noise through gating to obtain anomaly evidence; mapping the anomaly evidence to interpretable industrial defect semantics to obtain anomaly semantic features; and obtaining structured detection results based on the anomaly semantic features.

[0144] In this embodiment, in order to extract stable and locatable abnormal evidence from the consistent feature space constructed in step S2 and output structured detection results that can be directly entered into the industrial acceptance rule base for verification, this step unifies reference sample retrieval, feature alignment, gated differential modeling, and text conditional multimodal fusion into a continuous forward process, thereby realizing the closed-loop input preparation of "abnormal evidence → structured output".

[0145] Input image sample I is fed into the frozen visual encoder and passed through the inter-layer collaborative adjustment module in step two to obtain the uniformized patch-level features of the input sample:

[0146] ,

[0147] Where I is the input image, X is the consistent patch-level feature matrix of the input samples; N is the number of patches / tokens; C is the feature dimension of each patch / token.

[0148] Reference sample library constructed from step S1 Reference sample image I was retrieved from the middle r And by using the same feature extraction and adjustment path as the input samples, consistent patch-level features of the reference samples are obtained:

[0149] ,

[0150] Among them, I r For reference, from Y is the uniformized patch-level feature matrix of the reference sample, which has the same dimensional definition as X.

[0151] To obtain a normal template that matches the input sample in terms of category / condition, the global index features used for retrieval are first calculated for the input sample:

[0152] ,

[0153] Where F represents the global index feature; the index feature pre-cached for each reference sample in the reference library. Calculate the similarity and select the most similar reference sample:

[0154] ,

[0155] Where sim(⋅,⋅) is the similarity metric function (e.g., cosine similarity); I r For the reference image obtained from the retrieval; F r This step is used to select reference samples; the difference modeling still uses patch-level features X and Y.

[0156] Since the patch response order and local semantic correspondence formed during feature extraction between the input sample and the reference sample may not be completely consistent, directly subtracting position by position according to the token index would introduce a large amount of difference noise unrelated to defects. Therefore, this implementation constructs the correspondence between the input token and the reference token before difference.

[0157] (1) Construct a similarity matrix:

[0158] ,

[0159] Where S is the similarity matrix; This involves matrix multiplication to obtain a similarity score between each input token and each reference token. S is a scaling factor used to stabilize the numerical range; S corresponds to S0. i,j This represents the similarity score between the i-th input feature token and the j-th reference feature token, where: .

[0160] (2) Determine the alignment index (matching the input token to the reference token):

[0161] ,

[0162] Where π(i) is the reference token index corresponding to the input token i.

[0163] (3) Rearrange and align the reference features:

[0164] ,

[0165] in, For alignment operators; For the aligned reference feature; This refers to the i-th token vector of the aligned reference features. The token vector with index π(i) in the reference features.

[0166] After feature alignment is completed, differentials are constructed and noise is suppressed by gating to obtain anomalous evidence features.

[0167] (1) Fundamental difference:

[0168] ,

[0169] Where Δ is the alignment difference feature matrix.

[0170] (2) Calculation of differential gating weights:

[0171] ,

[0172] in, , These are trainable parameters; G is a column vector of all 1s, used to broadcast the bias term to N tokens; G is a gating weight matrix with values ​​ranging from (0,1).

[0173] (3) Gated difference and normalization yield the characteristics of anomalous evidence:

[0174] ,

[0175] Where LN() is LayerNorm normalization; D is the anomaly evidence feature, which has spatial orientation (corresponding to the token position) and is numerically stable.

[0176] To map anomalous evidence into interpretable industrial defect semantics, a textual prompt or defect description text T is constructed and its text features are obtained through a text encoder:

[0177] ,

[0178] Where T represents text input (defect type description, rule hints, etc.); T f The text feature is denoted by T; M represents the number of text tokens. The anomaly evidence feature D is used as the query, and the text feature T... f Compute cross-modal attention for keys and values:

[0179] ,

[0180] ,

[0181] in, Z is a trainable projection matrix; Z represents the semantically enhanced features after text conditionalization (consistent with the dimension of D). Gated fusion is introduced to ensure that textual information primarily affects regions of significant anomalous evidence.

[0182] ,

[0183] ,

[0184] in, , For trainable parameters; G m To integrate gating weights; F fuse These are the fused abnormal semantic features.

[0185] To ensure that the structured detection results reflect the reliability of the detection, this implementation combines the classification probability output by the detection head with the strength of abnormal evidence to generate a confidence score s.

[0186] (1) Probability of defect existence: The logit of the detection head outputting the existence of defects is z. exist ,but:

[0187] ,

[0188] Where, p exist This indicates the probability of a defect existing.

[0189] (2) Defect type probability: The logits of the defect type output by the detection head is a vector z. typeThe probabilities of each type are obtained through Softmax, and the probability corresponding to the predicted type is taken.

[0190] ,

[0191] in, p represents the probability distribution vector of all predefined defect categories (obtained by mapping the logits output by the network through the Softmax function). type This represents the confidence probability value (i.e., the extracted probability vector p) corresponding to a specific defect category predicted by the model. type (the maximum value in the range).

[0192] (3) Locating quality items (given by anomalous evidence): identifying the characteristics of anomalous evidence. Convert to token-level evidence strength graph:

[0193] ,

[0194] Among them, e i D represents the scalar anomaly evidence strength at the i-th token position (obtained by calculating the L2 norm of the corresponding eigenvector; a larger value indicates a more significant anomaly at that position). i Let represent the feature vector corresponding to the i-th token in the anomalous evidence feature matrix D. Map the predicted bounding box B to the token set Ω(B), calculate the average evidence strength within the box, and normalize it to obtain the localization quality term:

[0195] ,

[0196] Here, qloc represents the location quality term, and Norm represents the normalization function that maps the values ​​to [0,1] (e.g., min-max normalization or Sigmoid mapping).

[0197] (4) The final confidence score is defined as:

[0198] ,

[0199] Where s∈[0,1] is the confidence field in the structured output; this definition makes the confidence level reflect "whether there is a defect", "the reliability of the defect type judgment" and "the consistency between the abnormal evidence and the location".

[0200] Based on the fused feature F, the detection head outputs structured anomaly evidence:

[0201] ,

[0202] Where, defect_exist∈{0,1} is the defect existence identifier; defect_type∈C is the defect type (corresponding to the defect type set in step S1); B=[x,y,w,h] is the defect location bounding box; mask is the defect region mask; s∈[0,1] is the confidence score. The fields are consistent with the input fields for rule base verification in step one, so we can directly proceed to the subsequent "automatic verification and acceptance judgment of industrial rules".

[0203] By retrieving normal templates from reference samples and performing patch-level alignment and gated differential modeling in a uniform feature space, stable abnormal evidence features are extracted in this step. Furthermore, through text-conditional multimodal fusion and structured abnormal evidence output, the abnormal evidence is transformed into rule-verifiable detection results, providing standardized input for subsequent rule verification, traceability, and acceptance.

[0204] S4. Input the structured test results into the industrial acceptance rule base for verification to obtain the industrial rule verification results.

[0205] S4 includes: inputting the structured inspection results into the industrial acceptance rule base, calculating the structured consistency rules, defect type legality rules, context attribute constraint rules, appearance consistency rules, and evidence consistency rules; verifying each of the obtained rules to obtain the industrial rule verification results.

[0206] In this embodiment, this step is located in the "evaluation phase" of the training loop. Its core task is to input the structured detection result y output in step three into the industrial acceptance rule base constructed in step one for verification. This step does not generate the final decision for production, but rather calculates a quantitative indicator of "whether the current model output conforms to the rules" (i.e., rule satisfaction score and attribute reward / penalty), providing a directional supervision signal (Reward / Penalty) for parameter enhancement and fine-tuning in step five.

[0207] During the forward propagation phase of training, the structured output of the training samples is obtained. Simultaneously, the context identifier (ContextID) of the sample is read from the training data annotations.

[0208] ,

[0209] Among them, Input train This indicates the context identifier. To make rule validation executable, this step further extracts an "evidence strength index" from the anomalous evidence generated in steps S3 / S4, serving as a unified basis for rule and severity calculation. Let the anomalous evidence characteristics obtained in steps three / four be... The token-level evidence strength is defined as follows: And map the bounding box B to a set of token indices. Thus, the mean value of evidence within the frame is obtained:

[0210] ,

[0211] Where, μ in Ω(B) represents the mean value of evidence within the bounding box, N is the number of tokens (determined by the visual encoder patch division); C is the feature dimension of each patch / token; Ω(B) represents the set of token indices falling within the bounding box region.

[0212] To ensure that the rules cover common industrial acceptance points, this embodiment divides the rules into five categories: structural consistency, type legality, location occurrence, appearance consistency, impact and handling consistency, and additionally introduces "evidence consistency" to bind abnormal evidence in steps S3 / S4, so that rule verification not only "looks at the output fields" but also "looks at whether the evidence supports it".

[0213] (1) Structured Consistency Rules (output is parsable and verifiable): Industrial systems first require that the results be logically self-consistent, avoiding contradictions between fields. Definition:

[0214] ,

[0215] in, This represents the structured consistency check result (used to characterize whether the detection results output by the model are consistent in logical fields; for example, there must be no bounding box when there are no defects, and there must be a bounding box and a clear defect type when there are defects. Consistency is 1, otherwise it is 0). This is an indicator function; it takes the value 1 if the condition is true, and 0 otherwise. This indicates an empty set (in this case, it means that no bounding box was generated or no defect type was specified).

[0216] (2) Defect type legality rules (type must be resolvable): The production line usually defines a set of resolvable defect types:

[0217] ,

[0218] It also requires that the output type belongs to this set, defined as follows:

[0219] ,

[0220] in, This indicates the result of the defect type validity check (used to characterize whether the defect type output by the model belongs to the predefined set of manageable defect types in the production line; 1 indicates belonging and 0 indicates not belonging). c represents the set of defect types. KThis represents the name of the Kth defect type (e.g., a production line terminology such as "scratch / crack / contamination").

[0221] (3) Contextual attribute constraint rules: For the same physical feature in different product contexts (c prod To address the inconsistency in judgment criteria within the detection results, dynamic verification rules based on attributes are defined. The physical attribute features extracted from the detection results are... (such as texture roughness, grayscale mean, etc.), and the rule base defines the legal threshold mappings for different products:

[0222] ,

[0223] in, Represents a valid threshold mapping. This represents the result of the context rule validation, with values ​​of {0, 1}, where 1 indicates that the validation passed (valid) and 0 indicates that the validation failed (violation). It is the basic component for calculating rule satisfaction. This indicates an indicator function; the function value is 1 when the condition within the curly braces is true, and 0 otherwise. It also serves as a key, used to look up the corresponding threshold standard in the rule base. This represents the quantified physical property feature value, used to analyze the detection results. Specific physical indicators extracted from the defective area, such as texture roughness, grayscale mean, and aspect ratio, are the objects being verified. and These represent the dynamic lower threshold and the dynamic upper threshold, respectively. These two values ​​are not fixed, but rather vary according to a function. Based on the input product c prod These are dynamically returned. They constitute the "legal range" within which the current product allows this attribute to exist. Example: If ,but (Strict); if ,but (Relaxed).

[0224] Example: If the current product c prod For "Wood", the allowable roughness threshold is... The threshold is relatively high; if the region detected by the model is coarse but does not exceed this threshold, then f is relatively high. context =1 (Validation passed, meaning the feature is considered valid for the current product and should not be judged as a defect). This rule directly constrains the model to learn "what features are allowed for what products".

[0225] (4) Appearance consistency rule (appearance must match type):

[0226] To avoid situations where the type appears correct but the appearance is inconsistent, a function for extracting appearance features is defined:

[0227] ,

[0228] And for each type of defect, a set of permissible morphological constraints Ω(c) is set, thus:

[0229] ,

[0230] in, This indicates the result of the appearance consistency verification (used to determine whether the extracted appearance features conform to the prior morphological constraints of this type of defect; 1 is set if they conform, and 0 otherwise). The function represents the extraction function of appearance and morphology features. Morph() represents the morphological indicators that can be extracted, such as the number of connected components, aspect ratio, thinness, and boundary roughness (the implementation is defined by the production line and fixed as rule parameters); Ω(c) represents the set of morphological ranges allowed for type c.

[0231] (5) Evidence consistency rule (the result must be supported by anomalous evidence)

[0232] Using the obtained μ in Establish an "evidence-outcome consistency" check with the confidence level s:

[0233] ,

[0234] in, The result of the evidence consistency check (used to determine whether the final detection result is effectively supported by the underlying anomalous evidence strength; 1 is set if the threshold condition is met, and 0 otherwise) represents the result. e δ is the threshold for the strength of evidence. s The confidence thresholds are all set based on production line acceptance experience. This rule ensures that the output location is consistent with the differential evidence, thereby enhancing traceability.

[0235] Each rule in the rule base is validated to obtain a pass vector:

[0236] ,

[0237] in, Represented by vectors; defining rule satisfaction scores:

[0238] ,

[0239] in, Represents the rule satisfaction score; generates a set of failure reason indexes:

[0240] ,

[0241] in, This represents the set of failure reason indices, used to output "which rules failed," supporting re-examination and traceability. Furthermore, to feed "serious errors" back to the training process, this step derives a dynamic severity penalty coefficient based on rule validation. Severity is no longer just an output label, but serves as a weighting factor in training:

[0242] ,

[0243] Where, γ severity The weighting term is represented by λ, which is the severity scaling factor, used to control the amplification of severity on the overall loss function. Its value is set according to the production line quality control strategy. g() is the severity mapping function, which outputs discrete levels (e.g., {1,2,3} represent slight / moderate / severe, respectively). Its mapping threshold is determined by the production line process standards. μ is introduced. in To ensure that the severity aligns with the strength of anomalous evidence, and is interpretable and verifiable. severity The weighting coefficients used as the loss function in step S5 force the model to prioritize correcting errors of high severity. The final output feedback signal set is as follows:

[0244] ,

[0245] Among them, z train This represents the set of output feedback signals. This set of signals will be directly input into step S5 to drive the parameter update of the interlayer coordinated adjustment module (LCM).

[0246] S5. Based on the industrial rule verification results, and under the premise of freezing the multimodal basic model parameters, the relevant parameters of the inter-layer collaborative adjustment module are strengthened and fine-tuned to obtain the multimodal detection model.

[0247] S5 includes: calculating the reward function and advantage function based on the industrial rule verification results and under the premise of freezing the multimodal basic model parameters; calculating the gradient of the objective function of the inter-layer collaborative adjustment module and updating the LCN parameters based on the reward function and advantage function, and synchronously updating the value network parameters to obtain the multimodal detection model.

[0248] In this embodiment, this step aims to use the industrial rule verification results obtained in step four to transform "whether the production line acceptance constraint is met" into a quantifiable training feedback signal. Under the premise of freezing the multimodal basic model parameters, only the relevant parameters of the inter-layer collaborative adjustment module (LCM) are strengthened and fine-tuned, thereby forming a closed-loop optimization process of "detection-verification-feedback-update", so that the model output gradually approaches the production line acceptance target and maintains inference stability.

[0249] During the training phase, sample images I (containing both normal and abnormal samples) are extracted from the training dataset, and steps three / four are performed to obtain the structured detection output. Then, perform step four to obtain the rule satisfaction score. During this process: the visual encoder parameters remain frozen; the text encoder parameters remain frozen; only the parameter set of the Inter-Layer Co-tuning Module (LCM) is allowed to participate in the update. The trainable parameter set of the LCM is:

[0250] ,

[0251] in, This represents the set of trainable parameters to be optimized, which accounts for only a very small percentage (usually less than 1%) of the total number of parameters in the model. These parameters all fall under the categories of lightweight adjustment / gating / projection, and are significantly smaller than the number of parameters in the base model, thus representing efficient parameter fine-tuning.

[0252] Perform rule validation on each training sample to obtain a rule satisfaction score:

[0253] ,

[0254] If the training samples have location annotations (boundary boxes generated by the defect mask M), Then calculate the predicted bounding box B and the label box. Crossover ratio:

[0255] ,

[0256] Among them, Q IoU The Intersection over Union (IoU) represents the ratio of the predicted bounding box B to the ground truth bounding box B. The ratio of the intersection area to the union area between the samples (i.e., the intersection-union ratio calculation function); based on the feedback signal from step S4, a comprehensive reward function r for a single sample is constructed. In this implementation, LCM is regarded as a policy network, the detection result is regarded as an action, and the rule feedback is regarded as a reward.

[0257] ,

[0258] Where r is the rule feedback reward during the training phase; α, β, η≥0 are weight coefficients; : Indicates a context rule violation. This item is 1 only when the model violates context attribute constraints (such as falsely reporting a normal texture), triggering the penalty mechanism. Indicates the severity of the attribute violation. The calculation method is as follows: This is used to implement "gradient penalty," where the more severe the violation (e.g., roughness far exceeding a threshold), the greater the penalty, forcing the model to adjust its parameters more aggressively. This reward simultaneously constrains "production line rule pass rate" and "detection and localization quality," and explicitly avoids "output field inconsistency."

[0259] To leverage rule feedback for enhanced fine-tuning, this implementation treats the LCM as a parameterized module of a "learnable strategy," which generates a set of modulating variables given an input image, thereby influencing subsequent anomaly evidence modeling and structured output. Definition: State s t This indicates that the data is composed of multi-layered consistent feature statistics of the current input sample (e.g., the multi-layered features output from step S2). (obtained after pooling), denoted as:

[0260] ,

[0261] Here, Pool represents the pooling operation. Action a t The set of modulation intensity / gating coefficients generated by LCM (e.g., a low-dimensional representation of the gating vectors of each layer) is denoted as:

[0262] ,

[0263] Among them, strategy This represents the policy network / regulation module parameterized by parameter Θ; reward r t This represents the reward r obtained from the rules above.

[0264] To stabilize training and avoid performance fluctuations caused by excessive parameter updates, this implementation adopts a policy gradient with a baseline and introduces pruning constraints to construct a reinforcement fine-tuning objective function. First, a value baseline is defined, where... The value network parameters are denoted as:

[0265] ,

[0266] in, Represents network parameters, These are the parameters for the value network (also a lightweight module, which can share some features with LCM or be implemented independently). Define the advantage function:

[0267] ,

[0268] Among them, A t Describe the advantage function; define the probability ratio between the old and new strategies:

[0269] ,

[0270] in, This represents the ratio of the probabilities of the new and old strategies. This indicates the old strategy parameters saved before the update. Indicates that in state s t Take action a t The probability of [something]. The objective of constructing the pruning strategy:

[0271] ,

[0272] in, Indicates the goal of the trimming strategy. This represents the empirical mathematical expectation calculated based on the batch of data sampled using the current strategy (i.e., the average of the sampled data over the batch). `clip()` represents the truncation function. This represents the pruning factor, used to limit the magnitude of a single update. To ensure that the structured results output by the strategy do not deviate from the basic detection capability (especially when there is a ground truth), a supervised consistency term is introduced (which can be set to zero if there is no ground truth):

[0273] ,

[0274] in, The term represents the supervised consistency term, and CE() represents the cross-entropy loss. Indicates the weight of the supervised item; defect_exist gt defect_type gt B gt The labels are then assigned to the training samples. Simultaneously, the regression loss of the value network is updated.

[0275] ,

[0276] in, This represents the regression loss of the value network. The final reinforcement fine-tuning objective function is defined as:

[0277] ,

[0278] Its equivalent minimization form is:

[0279] ,

[0280] in, This represents the objective function.

[0281] Parameter update and closed-loop training process: (1) Forward evaluation: obtain the detection results With characteristic state s t (2) Rule verification: Execute step four to calculate And derive the severity penalty γ severity (3) Reward calculation: Construct reward r t And calculate advantage A t(4) Parameter update: Calculate the objective function Calculate the gradient and update the LCM parameters: Synchronously update value network parameters: Where η represents the LCM learning rate, η v This represents the learning rate of the value network.

[0282] When the rule satisfaction score S on the validation set rule Once the system reaches a stable state or the maximum number of training rounds, stop updating and solidify the parameters, and proceed to the inference and deployment phase.

[0283] S6. Based on the multimodal detection model, perform anomaly detection and industrial acceptance judgment on the samples to be tested on the production line to obtain the final test results.

[0284] S6 includes: acquiring the sample to be tested on the production line and inputting the sample to be tested into the multimodal detection model to obtain the structured detection result of the sample to be tested; inputting the structured detection result of the sample to be tested into the industrial acceptance rule base, performing verification on each rule to obtain the rule verification result of the sample to be tested; calculating the rule satisfaction score based on the rule verification result of the sample to be tested, and generating the final detection result.

[0285] In this embodiment, after completing the training phase in step five and fixing the model parameters, this step performs anomaly detection and industrial acceptance judgment on the test dataset or newly arrived unknown samples on the production line, and outputs the detection results, rule verification information, and acceptance conclusions in a structured and traceable JSON format. This step only performs forward model computation and rule verification, without updating any parameters, thereby ensuring the stability and consistency of the system during actual deployment.

[0286] During the inference and testing phases, the system receives the image to be detected as an input sample, denoted as:

[0287] ,

[0288] Among them, I test This represents the industrial image sample to be detected. During this stage, the model is in an inference state, specifically including: freezing the visual encoder parameters; freezing the text encoder parameters; freezing the interlayer collaborative adjustment module (LCM) parameters; not calculating gradients and not performing any parameter updates.

[0289] Input sample I test The input is fed into the trained model, and the forward computation process, consistent with steps S3 / S4, is executed, including feature unification, reference alignment, gated differencing, and multimodal inference, thereby generating a structured detection result, denoted as:

[0290] ,

[0291] in, This represents the structured test result of the sample to be tested, which serves as the unified input for subsequent industrial rule verification and acceptance decisions.

[0292] Structure the detection results of the sample to be tested Input the industrial acceptance rule base and verify each rule one by one. Unlike simply outputting "pass / fail", this implementation further records key judgment information related to the rule when it fails, to support subsequent re-inspection and quality traceability. For any failed rule r... k Construct a failure rule logging object:

[0293] ,

[0294] Among them, Fail k The object representing the failure rule record, ID k Indicates a unique rule number. (Reason) k The description of why the rule failed (e.g., "Context attribute constraint validation failed"). Val represents the valid threshold corresponding to the current context. obs This represents the actual observed value of the current sample. Through this method, the rule validation results are transformed from "Boolean judgments" into engineering records that are interpretable and traceable.

[0295] Calculate the rule satisfaction score S based on the rule validation results. rule And generate the final acceptance conclusion (Decision) based on the preset decision logic:

[0296] ,

[0297] Here, Decision represents the final acceptance conclusion, and REJECT(NG) indicates that the test result failed the critical rule validation, meaning that at least one critical rule exists. Not satisfied ( REVIEW (Unsure) indicates that the current sample has serious defects or violates critical process constraints, and should be judged as a non-conforming product, requiring rejection, rework, or shutdown; REVIEW (Unsure) indicates that although the test results meet all critical rules ( However, the overall rule satisfaction score S rule Below the preset release threshold This indicates that the test results are uncertain or that some non-critical rules are biased, requiring manual re-inspection or further confirmation. ACCEPT (OK) indicates that the test results meet all critical rules and the overall rule satisfaction score is S. rule Not lower than the threshold This indicates that the current sample meets the industrial acceptance standards and can be directly released into the subsequent production or shipment process. This represents a set of key rules (such as contextual attribute constraints and type validity). This indicates the threshold score for approval. If a key rule is passed but the overall score is low (e.g., slight deviation in morphology), the output will be "Pending Re-inspection".

[0298] Finally, the detection results, rule verification details, and acceptance conclusions are organized into a single JSON output object. By adopting the JSON output format, this embodiment achieves re-inspection, traceability, and direct acceptance of abnormal detection results during the inference and testing phases, significantly improving the engineering usability of the industrial intelligent detection system in actual production lines.

[0299] Example 2:

[0300] In this embodiment, a multimodal industrial intelligent inspection system based on efficient enhancement fine-tuning and rule verification includes: a database construction module, a model improvement module, a structured result generation module, a verification result generation module, a model adjustment module, and a final inspection module.

[0301] The database construction module is used to divide industrial testing data and build a reference sample library and an industrial acceptance rule library.

[0302] The model improvement module is used to introduce an inter-layer collaborative adjustment module into the frozen multimodal base model, under the premise of freezing the parameters of the multimodal base model, to perform consistent modeling of the multi-layer features output by the visual encoder, and to construct a feature space suitable for anomaly comparison and rule verification.

[0303] The structured results generation module is used to extract localizable anomaly evidence in the feature space and output structured test results that can be directly accessed and verified in the industrial acceptance rule base.

[0304] The verification result generation module is used to input the structured test results into the industrial acceptance rule base for verification, and obtain the industrial rule verification results.

[0305] The model adjustment module is based on the industrial rule verification results, and under the premise of freezing the multimodal basic model parameters, it strengthens and fine-tunes the relevant parameters of the inter-layer collaborative adjustment module to obtain the multimodal detection model.

[0306] The final testing module uses a multimodal testing model to perform anomaly detection and industrial acceptance judgment on the samples to be tested on the production line, and obtains the final testing results.

[0307] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification, characterized in that, Includes the following steps: S1. Divide the industrial testing data and construct a reference sample library and an industrial acceptance rule library; S2. Under the premise of freezing the parameters of the multimodal basic model, an inter-layer collaborative adjustment module is introduced into the frozen multimodal basic model to perform consistent modeling of the multi-layer features output by the visual encoder and construct a feature space suitable for anomaly comparison and rule verification. S3. Extract locatable abnormal evidence from the feature space and output structured detection results that can be directly accessed and verified in the industrial acceptance rule base; S4. Input the structured test results into the industrial acceptance rule base for verification to obtain the industrial rule verification results; S5. Based on the industrial rule verification results, and on the premise of freezing the multimodal basic model parameters, the relevant parameters of the inter-layer collaborative adjustment module are strengthened and fine-tuned to obtain the multimodal detection model; S6. Based on the multimodal detection model, perform anomaly detection and industrial acceptance judgment on the samples to be tested on the production line to obtain the final test results.

2. The multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification according to claim 1, characterized in that, S1 includes: Define the industrial testing data, and divide the industrial testing data into training set, validation set and test set; Defect bounding boxes are automatically generated from defect masks; Based on the training set, the reference sample library is constructed from the normal sample subset; Based on the defect boundary box, the industrial acceptance rule base is constructed.

3. The multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification according to claim 2, characterized in that, S2 includes: A single image is obtained from the training set as an input image sample and input into a pre-trained visual encoder for feature extraction. Based on the extracted features, an inter-layer collaborative adjustment module is introduced into the frozen multimodal basic model, and the normalized collaborative weight and collaborative adjustment vector of the inter-layer collaborative adjustment module are calculated. To control the influence of the collaborative adjustment vector on the original features, a gating mechanism is introduced, and a low-rank projection method is used to inject the adjustment information into the original features to obtain the adjusted features; The modified feature terms of the adjusted feature are added to the original feature to obtain a consistent feature representation, and the feature space suitable for anomaly comparison and rule verification is constructed.

4. The multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification according to claim 3, characterized in that, S3 includes: The input image samples are input into the frozen visual encoder and passed through the inter-layer collaborative adjustment module to obtain consistent patch-level features of the input samples. Reference image samples are retrieved from the reference sample library and input into the frozen visual encoder and passed through the inter-layer collaborative adjustment module to obtain the uniformized patch-level features of the reference image. Calculate the similarity between the uniformized patch-level features of the input sample and the uniformized patch-level features of the reference image, and perform reference feature alignment based on the similarity. After feature alignment is completed, a difference is constructed and noise is suppressed by gating to obtain the abnormal evidence; The anomalous evidence is mapped to interpretable industrial defect semantics to obtain anomalous semantic features; The structured detection result is obtained based on the aforementioned abnormal semantic features.

5. The multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification according to claim 1, characterized in that, S4 includes: The structured inspection results are input into the industrial acceptance rule base to calculate the structured consistency rule, defect type legality rule, context attribute constraint rule, appearance consistency rule, and evidence consistency rule. Each of the obtained rules is verified to obtain the verification results of the industrial rules.

6. The multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification according to claim 1, characterized in that, S5 includes: Based on the industrial rule verification results, and under the premise of freezing the multimodal basic model parameters, the reward function and the advantage function are calculated. Based on the reward function and the advantage function, the gradient of the objective function of the inter-layer collaborative adjustment module is calculated and the LCN parameters are updated. The value network parameters are updated synchronously to obtain the multimodal detection model.

7. The multimodal industrial intelligent detection method based on efficient enhancement fine-tuning and rule verification according to claim 1, characterized in that, S6 includes: The sample to be tested is obtained from the production line and input into the multimodal detection model to obtain the structured detection result of the sample to be tested; The structured test results of the sample to be tested are input into the industrial acceptance rule base, and each rule is verified one by one to obtain the rule verification results of the sample to be tested. Based on the rule verification results of the sample to be tested, the rule satisfaction score is calculated, and the final detection result is generated.

8. A multimodal industrial intelligent detection system based on efficient enhancement fine-tuning and rule verification, wherein the system applies the method described in any one of claims 1-7, characterized in that, include: The system includes a database construction module, a model improvement module, a structured result generation module, a validation result generation module, a model adjustment module, and a final detection module. The database construction module is used to divide industrial testing data and build a reference sample library and an industrial acceptance rule library. The model improvement module is used to introduce an inter-layer collaborative adjustment module into the frozen multimodal base model, under the premise of freezing the parameters of the multimodal base model, to perform consistent modeling of the multi-layer features output by the visual encoder and construct a feature space suitable for anomaly comparison and rule verification. The structured result generation module is used to extract locatable abnormal evidence in the feature space and output structured detection results that can be directly accessed and verified in the industrial acceptance rule base. The verification result generation module is used to input the structured test results into the industrial acceptance rule base for verification, and obtain the industrial rule verification result; The model adjustment module, based on the industrial rule verification results and under the premise of freezing the multimodal basic model parameters, strengthens and fine-tunes the relevant parameters of the inter-layer collaborative adjustment module to obtain a multimodal detection model. The final detection module performs anomaly detection and industrial acceptance judgment on the samples to be tested on the production line based on the multimodal detection model, and obtains the final detection result.

Images