An industrial defect detection method and system based on bidirectional semantic flow and boundary perception refinement

Abstract

The application discloses an industrial defect detection method and system based on bidirectional semantic flow and boundary perception refinement. The method comprises the following steps: collecting industrial product images and natural language defect description instructions, obtaining image high-level semantic features and fine-grained pixel-level features; extracting conditional embedding vectors and segmentation embedding vectors and initializing decoder queries; realizing bidirectional interaction and collaborative updating of visual features and text semantics in the whole decoding process through semantic cross attention, adaptive fusion gate and conditional refinement mechanism; introducing a boundary perception mask refinement module to improve the defect segmentation boundary precision; and finally deploying pixel-level defect segmentation positioning and classification output by adopting end-to-end multi-task joint training to complete model optimization. The application can support open vocabulary natural language instruction guidance, has the advantages of high detection precision, fine boundary, strong universality and expansibility, and can effectively meet the requirements of high-precision and high-robustness intelligent quality inspection of industrial production lines.

Description

Technical Field

[0001] This invention relates to the field of industrial machine vision and intelligent quality inspection technology, specifically to an industrial defect detection method and system based on bidirectional semantic flow and boundary awareness refinement. Background Technology

[0002] Surface defect detection of industrial products is a key link in ensuring product quality and reducing defect rates in the manufacturing industry. Traditional manual visual inspection methods are inefficient and greatly affected by the subjective state of the inspectors, making it difficult to meet the inspection needs of modern high-speed production lines. Therefore, automatic defect detection technology based on machine vision has been widely researched and applied.

[0003] Early industrial defect detection methods mainly relied on manually designed image processing algorithms, such as threshold segmentation, edge detection, and texture feature analysis. These methods were effective for specific types of defects, but lacked versatility and robustness. With the development of deep learning technology, classifiers and segmentation models based on convolutional neural networks were introduced into the field of defect detection, significantly improving detection accuracy and efficiency.

[0004] Existing deep learning-based industrial defect detection methods face the following main challenges: First, traditional methods mostly perform closed-set detection for defects of fixed categories, lacking the ability to respond to open text queries (i.e., operators describing defect types in natural language), resulting in insufficient flexibility; Second, for pixel-level defect segmentation tasks, the visual feature extractor and semantic understanding module in existing frameworks are independent of each other, lacking sufficient deep interaction between the two, leading to semantic misalignment between the generated mask candidates and the text description; Third, existing methods lack the ability to refine the mask boundary region, resulting in coarse defect segmentation boundaries that are difficult to meet the requirements of high-precision quantitative measurement.

[0005] In language-guided image segmentation paradigms, current mainstream methods typically employ a "propose first, then select" approach. The segmentation decoder generates a set of candidate masks, and then matches these candidate masks with textual conditions using final embedding similarity to determine the optimal mask output. However, in this design, textual conditional embedding only participates in the final classification matching stage, lacking semantic constraints during the iterative refinement process of mask generation. This results in the evolution of the decoding query being entirely driven by visual features, wasting a significant amount of the rich semantic guidance provided by the textual information. Through systematic analysis of typical failure cases, it can be found that in a considerable proportion of failed predictions, the candidate mask set actually already contains masks that highly match the ground truth (Intersection over Union (IoU) exceeding 80%). The root cause of the failure is not insufficient mask generation quality, but rather semantic misalignment in the selection stage, i.e., the model "segments accurately but matches incorrectly."

[0006] For example, Chinese invention patent application CN201910559978.4 discloses an edge defect detection method, an edge defect detection device, and a quality inspection equipment. Based on this invention, a pre-defined ideal edge curve can be used as a detection reference for the target image, thus reducing the interference of image noise on the accuracy of the edge curve. Furthermore, regionalized defects can be delineated based on the distribution of fine candidate defect points obtained from fine detection, thereby reducing the probability of missed defects and improving the accuracy of defect detection. In addition, this invention supports manual setting or automatic detection of ideal edge curves based on sample images, which helps in the precision control of ideal edge curves; this invention supports manual setting or automatic generation of detection areas, which helps improve the accuracy of targeted positioning for defect detection; this invention can introduce a mask to shield predictable areas of strong interference; the defect types that this invention can identify may include concave / convex defects and fracture defects.

[0007] For example, Chinese invention patent application number CN202510480124.2 discloses a semantically guided and texture-prior-based dual-branch surface defect segmentation method and medium, relating to the field of surface defect segmentation. The method includes acquiring an image of the object to be detected; inputting the image into a defect segmentation model to obtain the surface defect segmentation result of the object; the defect segmentation model includes a semantic and texture dual-branch feature extraction network, a semantically guided and texture-prior-based feature fusion network, and a decoder connected sequentially; the dual-branch feature extraction network is used to extract semantic information, defect texture, and edge features from the image of the object to be detected, obtaining a semantic feature map and a texture feature map; the feature fusion network is used to fuse the semantic feature map and the texture feature map; and the decoder is used to output the surface defect segmentation result of the object to be detected based on the fused feature map.

[0008] The core problem with existing technologies is that the interaction between visual features and semantic conditions during segmentation and decoding is unidirectional and superficial, lacking a bidirectional collaborative mechanism that runs through the entire decoding process. To address this, this invention provides an industrial defect detection method and system based on bidirectional semantic flow and boundary-aware refinement. Summary of the Invention

[0009] To address the aforementioned technical problems in existing technologies, this invention provides an industrial defect detection method and system based on bidirectional semantic flow and boundary-aware refinement.

[0010] The present invention adopts the following technical solution:

[0011] This invention provides an industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement, comprising:

[0012] Step 1: Collect images of industrial products and corresponding natural language defect description instructions to construct image-text instruction pairs;

[0013] Step 2: Construct a dual encoder feature extraction module to perform high-level semantic feature extraction and fine-grained pixel-level feature extraction on the input image respectively;

[0014] Step 3: Construct a language model processing module to perform semantic parsing and tag extraction on text instructions, obtain conditional embedding vectors and segmentation embedding vectors, and initialize decoder queries with segmentation embedding vectors;

[0015] Step 4: Construct a segmentation decoder based on bidirectional semantic flow. In the multi-layer decoding iteration, a semantic cross-attention mechanism, an adaptive fusion gate mechanism, and a conditional refinement mechanism are used to achieve bidirectional interaction and collaborative updating of visual features and text semantics.

[0016] Step 5: Construct a boundary-aware mask refinement module, locate the defect boundary region based on morphological gradient, perform selective residual refinement on the boundary region, and obtain a refined defect mask;

[0017] Step 6: Use a multi-task loss function that includes language modeling loss and segmentation loss to perform end-to-end joint training on the feature extraction module, language model processing module, bidirectional semantic stream segmentation decoder, and boundary-aware mask refinement module.

[0018] Step 7: Deploy the trained model in an industrial quality inspection scenario, input the image to be inspected and the defect detection command, and output the pixel-level defect segmentation results and category confidence.

[0019] Furthermore, in step 1, images of industrial products and corresponding natural language defect description instructions are collected to construct image-text instruction pairs, including:

[0020] Industrial products are captured on the production line using industrial cameras or vision sensors. The objects captured include metal castings, electronic components, textiles, leather, and printed circuit boards. The defect description instructions captured include, but are not limited to: cracks, pores, scratches, contamination, oxide spots, missing materials, and open circuits. Natural language image-text instruction pairs are constructed to support operators in specifying the type of defect to be inspected through flexible query descriptions.

[0021] Furthermore, in step 2, a dual-encoder feature extraction module is constructed, including a semantic encoder and a pixel-level encoder:

[0022] Semantic encoders for input industrial images High-level semantic features are extracted, and the extracted features are then processed by a lightweight projection network. Mapping to the embedding space of the language model enables cross-modal alignment between vision and language;

[0023] Pixel-level encoder for input image Extracting fine-grained pixel-level features This provides accurate local positioning information for subsequent segmentation decoders;

[0024] The dual encoder design decouples semantic understanding from pixel-level localization, enabling features at both levels to interact collaboratively on the decoder side through bidirectional semantic flow.

[0025] Furthermore, in step 3, a language model processing module is constructed, including:

[0026] Introduce special markers 、 , <seg>Annotate the text instructions, where and Used to define the range of phrases for the target defect category. <seg>Used to indicate the location of the split output;

[0027] The large language model processes the concatenated text and visual tokens, and outputs the hidden state. Extract conditional embeddings from them Corresponding to phrase tag position and segment embedding ,correspond <seg>Mark the location;

[0028] Through projection network Conditional embeddings and segmentation embeddings are mapped to the decoder space to obtain... and ;

[0029] Initial decoder query By segmentation embedding Perform additive initialization, that is This allows for the injection of global multimodal priors at the beginning of the decoding phase.

[0030] Furthermore, in step 4, a segmentation decoder based on bidirectional semantic stream is constructed. The segmentation decoder is a... It consists of cascaded bidirectional semantic stream decoding units, which include Each layer, including the previous one, performs the following operation:

[0031] A semantic cross-attention step is introduced after standard visual cross-attention, in the decoder's... Layer, Decoding Query Conditional embedding Attention operations are performed on keys and values, first layer hour For the initial conditional embedding, the semantic cross-attention step is as shown in Equation (1):

[0032] (1),

[0033] in, As a standard multi-head attention mechanism, the three parameter positions represent the query, key, and value, respectively; For conditional embedding, For visual queries, This is the output of the semantic cross-attention step, i.e., the semantic query;

[0034] An adaptive fusion gate mechanism calculates the dynamic fusion ratio between visual and semantic queries, with the fusion weight for each query determined by a gating signal. The decision is as shown in equations (2) to (3):

[0035] (2),

[0036] (3),

[0037] in, For learnable weights in the adaptive fusion gate mechanism, For the Sigmoid function, For feature splicing operations, Element-wise product; gate signal Calculated at the query granularity, allowing different queries to adaptively adjust the fusion ratio based on the importance of current visual and semantic information;

[0038] Merged query By capturing the global context between queries through self-attention operations, a refined query is obtained. As shown in equation (4):

[0039] (4),

[0040] To establish a true bidirectional semantic flow, the conditional embedding also incorporates visual evidence by updating the decoded query through a multi-head attention mechanism, as shown in equation (5):

[0041] (5),

[0042] Among them, conditional embedding is used as a query, and the refined decoded query is used. As keys and values, multi-head attention mechanisms and semantic cross-attention together constitute the forward flow. and reverse flow An explicit two-way closed loop; refined conditional embedding in the final stage. With decoding query output Cosine similarity between them is used to classify and select masks.

[0043] Furthermore, in step 5, a boundary-aware mask refinement module is constructed, including:

[0044] Based on the original mask probability map The boundary pixel region is located using morphological gradient operations, as shown in equation (6):

[0045] (6),

[0046] in, For binary boundary mask, To control the threshold of boundary sensitivity, and These are expansion and erosion operators, respectively. The indicator function is defined as follows: if the position in the probability map meets the above conditions, the corresponding value is 1; otherwise, it is 0. Thus, the boundary position is located using morphological gradients.

[0047] Refined Network Original mask With compressed network Pixel-level features after dimensionality reduction The process involves stitching and merging, calculating bounded residual updates, and updating only in the boundary regions. The internal refinement is applied as shown in equations (7) to (8):

[0048] (7),

[0049] (8),

[0050] in, For learnable scaling factor, For feature splicing operations, It is an element-wise product; the refinement operation is strictly limited to the boundary uncertainty region to avoid affecting the prediction of the already high-confidence internal region.

[0051] Furthermore, in step 6, the feature extraction module, language model processing module, bidirectional semantic stream segmentation decoder, and boundary-aware mask refinement module are jointly trained end-to-end. The total loss of the end-to-end multi-task training is... The sum of the language modeling loss and the segmentation loss is shown in Equation (9):

[0052] (9),

[0053] Among them, language modeling loss Using the standard next-word prediction cross-entropy loss, For text and images, The sequence of text terms generated before time t. Generate text words for the current moment. The conditional probability is shown in equation (10):

[0054] (10)

[0055] Segmentation loss The prediction mask is aligned with the ground truth label using Hungarian matching, as shown in Equation (11):

[0056] (11),

[0057] in For classification cross-entropy loss, and These are Dice loss and mask binary cross-entropy loss, respectively. and To balance the contributions of each loss to the hyperparameters, deep supervision is applied to all decoder layers to stabilize the optimization process of multi-layer semantic propagation.

[0058] Furthermore, in step 7, the image to be inspected and the defect detection instructions are input, including:

[0059] Images of industrial products and natural language defect description instructions As input, the instructions may contain defect_name The target phrase is labeled in a specific format to precisely constrain the detected targets; the model outputs a refined pixel-level defect segmentation mask. The positive region in the mask is the location of the detected defect, and the confidence score of the corresponding defect category is output at the same time; it supports parallel output of independent detection masks for multiple defect categories in a single image.

[0060] Furthermore, in step 2, both the semantic encoder and the pixel-level encoder employ pre-trained visual backbone networks; the features extracted by the semantic encoder are processed by a lightweight multilayer perceptron. Mapped to the embedding space of a large language model to support vision-language alignment; pixel-level features extracted by the pixel-level encoder. It is directly fed into the segmentation decoder and boundary-aware refinement module to provide fine-grained structural information for accurate positioning.

[0061] This invention also provides an industrial defect detection system based on bidirectional semantic flow and boundary-aware refinement, comprising:

[0062] The image acquisition unit is used to acquire image frames of products to be inspected on the industrial production line.

[0063] The text instruction unit is used to generate natural language detection instructions containing descriptions of defect categories based on a preset list of defect categories or operator input.

[0064] The dual encoder feature extraction module is used to extract high-level semantic features and fine-grained pixel-level features of images;

[0065] The language model processing module is used to parse text instructions and output conditional embeddings and segmentation embeddings;

[0066] The bidirectional semantic stream decoding module contains multi-layered cascaded decoder units, each layer containing semantic cross-attention, adaptive fusion gates, and conditional refinement sub-modules, which perform visual-semantic bidirectional interaction and generate candidate masks;

[0067] The boundary-aware mask refinement module uses morphological operations to locate the boundary region and then applies selective residual refinement to output the refined defect segmentation mask.

[0068] The decision output unit determines whether a product is qualified or not based on the defect mask and confidence score, and outputs the defect category and location information to trigger production line alarms or rejection actions.

[0069] Compared with the prior art, the superior effects of the present invention are as follows:

[0070] 1. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement described in this invention fundamentally solves the problem of decoupling between visual features and semantic conditions during segmentation and decoding. By introducing semantic cross-attention at each decoding layer, textual condition constraints are integrated throughout the entire mask generation process, effectively eliminating semantic mismatch and significantly improving the semantic consistency between the detection mask and the text description.

[0071] 2. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement described in this invention has an adaptive fusion gate mechanism that allows each decoding query to independently and dynamically adjust the fusion ratio of visual and semantic information, adapting to the diverse detection needs of different defect types and different image regions, and improving the robustness of the system in complex industrial scenarios.

[0072] 3. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement described in this invention establishes a reverse information flow from decoding visual queries to embedding linguistic conditions through a condition refinement mechanism. This allows linguistic conditions to continuously absorb visual evidence and dynamically evolve during the decoding process, forming a bidirectional interactive mechanism for the co-evolution of vision and semantics, which effectively improves the accuracy of the final classification and matching.

[0073] 4. The industrial defect detection method and system based on bidirectional semantic flow and boundary-aware refinement described in this invention, the boundary-aware mask refinement module significantly improves the fineness of defect segmentation boundaries without interfering with internal high-confidence prediction through morphological boundary detection and selective residual update, which is beneficial for subsequent refined detection tasks such as defect area calculation and severity quantification.

[0074] 5. The industrial defect detection method based on bidirectional semantic flow and boundary awareness refinement described in this invention uses natural language as the detection command interface, supports defect description with open vocabulary, does not require retraining a fixed classification head for each new type of defect, and has good generalization and scalability for mixed production lines with multiple varieties, processes, and scenarios. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the overall system architecture in an embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of the bidirectional semantic stream decoding layer in an embodiment of the present invention. Detailed Implementation

[0077] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0078] Example

[0079] like Figure 1 As shown, the industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement includes:

[0080] Step 1: Collect images of industrial products and corresponding natural language defect description instructions to construct image-text instruction pairs;

[0081] Industrial products are captured on the production line using industrial cameras or vision sensors. The captured products include various types such as metal castings, electronic components, textiles, leather, and printed circuit boards. Defect types captured include, but are not limited to, typical defects in industrial scenarios such as cracks, porosity, scratches, contamination, oxide spots, missing materials, and open circuits. Natural language text-based inspection command pairs are constructed, allowing operators to flexibly specify the defect category using query descriptions. The image resolution is no less than 224×224 pixels. For each defect image captured, natural language text command pairs are constructed according to the defect category, such as: "Please label the defects in the image..." crack "Area", "Please locate" pores "Defects", among which and Used to define the range of phrases for the target defect category;

[0082] Step 2: Construct a dual encoder feature extraction module to perform high-level semantic feature extraction and fine-grained pixel-level feature extraction on the input image respectively;

[0083] Based on a pre-trained backbone network with good visual representation capabilities, the dual-encoder feature extraction module comprises two sub-networks: a semantic encoder and a pixel-level encoder. The semantic encoder (such as a ViT-based encoder based on a visual Transformer architecture) extracts high-level semantic features from the input image. These features are then projected onto the embedding space of a large language model via a lightweight MLP (Multi-Layer Perceptron), supporting cross-modal alignment between vision and language. The size is denoted as [size not specified in the original text]. Pixel-level encoders, based on convolutional architectures or SAM (Segment Anything Model), extract pixel-level feature maps with higher spatial resolution. The dimensions are H'×W'× The structured information is directly fed into the segmentation decoder and boundary refinement module to provide structural information for accurate local localization. The dual encoder design separates macroscopic semantic understanding from microscopic pixel localization functions, while organically integrating the two on the decoder side through bidirectional semantic flow.

[0084] Step 3: Construct a language model processing module to perform semantic parsing and tag extraction on text instructions, obtain conditional embedding vectors and segmentation embedding vectors, and initialize decoder queries with segmentation embedding vectors;

[0085] Introduce special markers 、 The boundaries of target defect category phrases in the annotation text instructions are marked to achieve phrase-level semantic extraction; <seg>The markers indicate the segmentation output positions. Pre-trained large language models, such as the LLaMA (Large Language Model Meta AI) series and the InternLM (Intern Language Model) series, are selected to jointly process the text instructions and visual token sequences. The language model outputs the hidden state of the sequence. , shape , For the embedding dimension of the large language model, extract: (1) the conditional embedding corresponding to the phrase tag position. , shape Conditional embedding of the decoder space is obtained through linear projection. , shape , It is the decoder embedding dimension; (2) <seg>Segmentation embedding corresponding to the marked position , <seg>Used to indicate the segmentation output position, the segmentation embedding is obtained by projection. , shape Initial decoder query set ( (To determine the number of queries) the embedding is split using addition. Initialization complete;

[0086] Through projection network Conditional embeddings and segmentation embeddings are mapped to the decoder space to obtain... and ;

[0087] Initial decoder query By segmentation embedding Perform additive initialization, that is This allows for the injection of global multimodal priors at the beginning of the decoding phase;

[0088] Step 4: Construct a segmentation decoder based on bidirectional semantic flow. In the multi-layer decoding iteration, a semantic cross-attention mechanism, an adaptive fusion gate mechanism, and a conditional refinement mechanism are used to achieve bidirectional interaction and collaborative updating of visual features and text semantics.

[0089] Following standard visual cross-attention, a semantic cross-attention step is introduced, in the first... The decoding layer decodes the conditional embeddings output from the previous layer. Attention operations are performed on keys and values, and semantic constraints are directly injected into the query update of each layer, so that text conditions participate in the entire mask generation process, rather than just being used for similarity matching at the end.

[0090] Different queries and different decoding stages have different degrees of dependence on visual and semantic information. The adaptive fusion gate calculates the fusion weights separately for each query. The system dynamically adjusts the mixing ratio of visual and semantic features to maintain a visual-semantic balance in the early stages of decoding, while shifting towards semantic constraints in the later stages to refine mask-condition alignment, ultimately resulting in a fused query. It models global dependencies between queries through self-attention operations and outputs refined queries. ;

[0091] In addition to the forward guidance of conditional embedding for decoding queries, the visual context emerging in decoding queries should also feed back into conditional embedding. The conditional refinement mechanism enables conditional embedding to use itself as the query and the decoded query refined by the current layer's self-attention as the key value for cross-attention updates, thereby absorbing visual evidence layer by layer throughout the decoding process and forming an explicit bidirectional information flow closed loop together with semantic cross-attention.

[0092] After the decoding process is complete, the query features are calculated and output. Embedded with final refinement conditions The cosine similarity between the two is used to perform semantic matching and classification on each candidate mask, and output the defect segmentation result corresponding to the text description.

[0093] Step 5: Construct a boundary-aware mask refinement module, locate the defect boundary region based on morphological gradient, perform selective residual refinement on the boundary region, and obtain a refined defect mask;

[0094] To address the issue of accurate predictions for high-confidence internal regions but uncertainties in boundary regions, the boundary-aware mask refinement module first uses morphological gradient operations (performing dilation and erosion differences on the original mask probability map) to automatically locate boundary pixels in the transition region, generating a binary boundary mask B. The refinement network then fuses the original mask predictions with compressed pixel-level features to generate bounded residual updates. Residual stacking is applied only to the boundary region B, while high-confidence internal regions are not included in the update. This achieves a conservative refinement strategy of "enhancement rather than replacement," effectively improving the accuracy of defect segmentation boundaries.

[0095] Step 6: Use a multi-task loss function that includes language modeling loss and segmentation loss to perform end-to-end joint training on the feature extraction module, language model processing module, bidirectional semantic stream segmentation decoder, and boundary-aware mask refinement module.

[0096] A joint training objective is adopted to simultaneously optimize language modeling capability and segmentation accuracy. The segmentation loss is a linear combination of semantic classification cross-entropy loss, Dice loss and mask binary cross-entropy loss. Supervision is performed after aligning the predicted candidates with the ground truth through Hungarian matching. Deep supervision is applied to all decoding layers to promote the stable propagation of bidirectional semantic flow in multi-layer training.

[0097] Step 7: Deploy the trained model in an industrial quality inspection scenario, input the image to be inspected and the defect detection command, and output the pixel-level defect segmentation results and category confidence.

[0098] In industrial quality inspection reasoning, after receiving industrial images and natural language defect description instructions, the system performs feature extraction, language model processing, bidirectional semantic stream decoding, and boundary-aware refinement, and finally outputs a refined pixel-level defect segmentation mask and corresponding category confidence. Positive regions of the mask are marked as defect regions, which are used to trigger production line alarms or rejection actions. The system supports parallel detection of multiple types of defects in the same image, and the detection instructions can be dynamically configured at runtime to adapt to multi-product mixed production line scenarios.

[0099] After the system is deployed on the industrial production line, the image acquisition unit transmits product images to the host computer in real time. These images are then concatenated with pre-configured text detection instructions and sent to the system. After completing forward inference, the system outputs a refined mask. The system can input multiple text commands in parallel for scenarios that require detecting multiple types of defects, including the semantic matching confidence of each candidate. The system outputs independent mask predictions for each defect type, which are then superimposed and fused to obtain a complete defect distribution map. The final result is sent to the decision unit, which determines whether the product is qualified based on the preset confidence threshold and defect area threshold. The system also records information such as defect category, pixel-level coordinates, and area for use in production quality database archiving and production line feedback control.

[0100] In some specific embodiments of the present invention, in step 4, a segmentation decoder based on bidirectional semantic stream is constructed, which is a decoder composed of... It consists of cascaded bidirectional semantic stream decoding units, which include Each layer, including the previous one, performs the following operation:

[0101] Step 4.1, Visual Cross-Attention: Decoding the Query pixel-level features Using key-value pairs, visual information is extracted through multi-head attention to obtain visually enhanced queries. As shown in equation (12):

[0102] (12),

[0103] Step 4.2, Semantic Cross-Attention: Visually Enhanced Queries Conditional embedding of the output of the previous layer Using key-value pairs, semantic constraints are injected through multi-head attention to obtain semantically enhanced queries. As shown in equation (1):

[0104] (1),

[0105] in, As a standard multi-head attention mechanism, the three parameter positions represent the query, key, and value, respectively. For conditional embedding, For visual queries, The output of the semantic cross-attention step, i.e., the semantic query, is that this operation enables textual conditions to participate in constraints at every layer of mask generation, rather than just as a static matching benchmark in the final stage.

[0106] Step 4.3, Adaptive Fusion: Gated Signal The result is obtained by concatenating visually enhanced queries and semantically enhanced queries and then calculating using the Sigmoid function. The mixing ratio of visual and semantic information is dynamically adjusted according to the query granularity, as shown in equations (2) to (3):

[0107] (2),

[0108] (3),

[0109] in, For learnable weights in the adaptive fusion gate mechanism, For the Sigmoid function, For feature splicing operations, Element-wise product; gate signal Calculated at the query granularity, allowing different queries to adaptively adjust the fusion ratio based on the importance of current visual and semantic information. For learnable projection matrix;

[0110] Step 4.4, Self-Attention: Fusion Query By performing self-attention operations between queries, the global dependencies between different queries are modeled, resulting in refined queries. As shown in equation (4):

[0111] (4),

[0112] Step 4.5, Condition Refinement: Condition Embedding Use itself as the query, and refine the query based on the current layer. As the key value, it absorbs visual context through a multi-head attention mechanism and is updated to... As shown in equation (5):

[0113] (5),

[0114] Among them, the conditional embedding gradually incorporates visual evidence as the number of decoding layers increases, resulting in refined conditional embeddings. Used in the final layer for output queries Calculate cosine similarity to complete mask selection and defect category determination, and then perform refined decoding query. As keys and values, multi-head attention mechanisms and semantic cross-attention together constitute the forward flow. and reverse flow An explicit two-way closed loop; refined conditional embedding in the final stage. With decoding query output Cosine similarity between them is used to classify and select masks;

[0115] Step 4.6, Feedforward Network: Refined Decoding Query feedforward network The final query update is output as shown in equation (13):

[0116] (13)

[0117] through After layer iteration, the decoding query is updated simultaneously at each layer. and conditional embedding This allows semantic guidance to continue throughout the mask generation process, and the decoder output contains... Candidate mask logic , shape and query output , shape ;

[0118] In some specific embodiments of the present invention, in step 5, based on the morphological gradient defect boundary region: the original mask probability map is processed. (After Sigmoid activation) Perform morphological gradient operations to generate a boundary binary mask. As shown in equation (6):

[0119] (6),

[0120] in, As a binary boundary mask, the size of the expansion nucleus and the erosion nucleus are... The value is a hyperparameter that controls the boundary width and detection sensitivity. To control the threshold of boundary sensitivity, and These are expansion and erosion operators, respectively. The indicator function is defined as follows: if the position in the probability map meets the above conditions, the corresponding value is 1; otherwise, it is 0. Thus, the boundary position is located using morphological gradients.

[0121] In step 5, selective residual refinement is performed on the boundary region: pixel-level features. Lightweight compressed convolution After dimensionality reduction, compared with the original mask In channel-dimensional splicing, input refined network. Predicted residual update and the boundary area Apply bounded residual superposition, as shown in equations (7) to (8):

[0122] (7),

[0123] (8),

[0124] in, For feature splicing operations, Limit the residual to Within the range, A learnable scaling factor ensures that the refinement level is controlled; For element-wise multiplication, Guarantee only the boundary area The predictions within the range were revised, while the high-confidence regions within the range remained unchanged;

[0125] In some specific embodiments of the present invention, in step 6, end-to-end joint training is performed using an industrial defect dataset labeled with defect categories and pixel-level mask ground truth values ​​to train the system end-to-end, with an overall training loss. As shown in equation (9):

[0126] (9),

[0127] Language modeling loss The standard next-word prediction cross-entropy loss, where For text and images, The sequence of text terms generated before time t. Generate text words for the current moment. The conditional probability is shown in equation (10):

[0128] (10)

[0129] Segmentation loss Cross-entropy loss based on semantic classification Dice loss and mask binary cross-entropy loss The combination is shown in equation (11):

[0130] (11),

[0131] in, For classification cross-entropy loss, and These are Dice loss and mask binary cross-entropy loss, respectively. and To balance the contribution of each loss parameter, typical values ​​are 0.5 and 2.0. During training, the Hungarian matching algorithm is used to... After assigning each predicted candidate to a ground value in a one-to-one manner, the segmentation loss is calculated, and then... Apply deep supervision to each decoding layer;

[0132] Furthermore, in step 2, both the semantic encoder and the pixel-level encoder can employ pre-trained visual backbone networks. The semantic encoder processes the input industrial image... I High-level semantic features are extracted, and the extracted features are then processed by a lightweight projection network. Mapping to the embedding space of the language model enables cross-modal alignment between vision and language;

[0133] Pixel-level encoder for input image Extracting fine-grained pixel-level features It is directly fed into the segmentation decoder and boundary-aware refinement module to provide accurate local positioning information for the subsequent segmentation decoder;

[0134] The dual encoder design decouples semantic understanding from pixel-level localization, enabling features from both levels to interact collaboratively on the decoder side through bidirectional semantic flow.

[0135] The semantic encoder uses a pre-trained visual encoder based on the ViT-Large architecture to extract a 1024-dimensional semantic feature vector sequence from the image, which is then projected onto the 4096-dimensional embedding space of an LLM (such as LLaMA-7B) through two layers of MLP. The pixel-level encoder uses an image encoder based on SAM to extract pixel-level feature maps with a spatial resolution of 1 / 16 of the input image and a channel dimension of 256.

[0136] Furthermore, in step 4, the number of decoder layers... It can be set to 3 levels, with a query count of... It can be set to 100, the decoder embedding dimension. The number of attention heads in the multi-head attention mechanism can be set to 8, which can be set to 256. The projection matrix of the adaptive fusion gate... It has a shape of 256×512 and is implemented using standard linear layers.

[0137] Furthermore, in step 5, the size of both the expansion nucleus and the erosion nucleus in the morphological operation can be set to 3×3, with a boundary sensitivity threshold. It can be set to 0.1; refine the network. It can consist of 3 convolutional layers with 256, 64, and 1 channels respectively, and is a compressed convolution. A single-layer 1×1 convolution compresses pixel-level features from 256 dimensions to 64 dimensions; a learnable scaling factor is available. It is initialized to 0.5 and optimized in an end-to-end manner with other parameters during training.

[0138] Furthermore, step 7 involves inputting the image to be inspected and the defect detection instructions, including:

[0139] Images of industrial products and natural language defect description instructions As input, the instructions may contain defect_name The target phrase is labeled in a specific format to precisely constrain the detected targets; the model outputs a refined pixel-level defect segmentation mask. The positive region in the mask is the detected defect location, and the confidence score of the corresponding defect category is output at the same time; it supports parallel output of independent detection masks for multiple defect categories in a single image;

[0140] When the production line needs to detect both "cracks" and "porosity" defects simultaneously, two separate instructions are generated: "Please label the defects in the image..." crack "Area" and "Please label the area in the image" pores The industrial defect detection system outputs two independent mask images after parallel inference, which are superimposed on the acquired image. The decision unit then comprehensively judges whether the product is qualified, and the relevant defect information is uploaded to the industrial defect detection system in real time.

[0141] This invention also provides an industrial defect detection system based on bidirectional semantic flow and boundary-aware refinement, comprising:

[0142] The image acquisition unit is used to acquire image frames of products to be inspected on the industrial production line.

[0143] The text instruction unit is used to generate natural language detection instructions containing descriptions of defect categories based on a preset list of defect categories or operator input.

[0144] The dual encoder feature extraction module is used to extract high-level semantic features and fine-grained pixel-level features of images;

[0145] The language model processing module is used to parse text instructions and output conditional embeddings and segmentation embeddings;

[0146] The bidirectional semantic stream decoding module contains multi-layered cascaded decoder units, each layer containing semantic cross-attention, adaptive fusion gates, and conditional refinement sub-modules, which perform visual-semantic bidirectional interaction and generate candidate masks;

[0147] The boundary-aware mask refinement module uses morphological operations to locate the boundary region and then applies selective residual refinement to output the refined defect segmentation mask.

[0148] The decision output unit determines whether a product is qualified or not based on the defect mask and confidence score, and outputs the defect category and location information to trigger production line alarms or rejection actions.

[0149] This invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims.< / seg> < / seg> < / seg> < / seg> < / seg> < / seg>

Claims

1. An industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement, characterized in that, include: Step 1: Collect images of industrial products and corresponding natural language defect description instructions to construct image-text instruction pairs; Step 2: Construct a dual encoder feature extraction module to perform high-level semantic feature extraction and fine-grained pixel-level feature extraction on the input image respectively; Step 3: Construct a language model processing module to perform semantic parsing and tag extraction on text instructions, obtain conditional embedding vectors and segmentation embedding vectors, and initialize decoder queries with segmentation embedding vectors; Step 4: Construct a segmentation decoder based on bidirectional semantic flow. In the multi-layer decoding iteration, a semantic cross-attention mechanism, an adaptive fusion gate mechanism, and a conditional refinement mechanism are used to achieve bidirectional interaction and collaborative updating of visual features and text semantics. Step 5: Construct a boundary-aware mask refinement module, locate the defect boundary region based on morphological gradient, perform selective residual refinement on the boundary region, and obtain a refined defect mask; Step 6: Use a multi-task loss function that includes language modeling loss and segmentation loss to perform end-to-end joint training on the feature extraction module, language model processing module, bidirectional semantic stream segmentation decoder, and boundary-aware mask refinement module. Step 7: Deploy the trained model in an industrial quality inspection scenario, input the image to be inspected and the defect detection command, and output the pixel-level defect segmentation results and category confidence.

2. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement according to claim 1, characterized in that, In step 1, images of industrial products and corresponding natural language defect description instructions are acquired to construct image-text instruction pairs, including: Industrial products are captured on the production line using industrial cameras or vision sensors. The objects captured include metal castings, electronic components, textiles, leather, and printed circuit boards. The defect description instructions captured include, but are not limited to: cracks, pores, scratches, contamination, oxide spots, missing materials, and open circuits. Natural language image-text instruction pairs are constructed to support operators in specifying the type of defect to be inspected through flexible query descriptions.

3. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement according to claim 1, characterized in that, In step 2, a dual-encoder feature extraction module is constructed, including a semantic encoder and a pixel-level encoder: Semantic encoders for input industrial images I High-level semantic features are extracted, and the extracted features are then processed by a lightweight projection network. Mapping to the embedding space of the language model enables cross-modal alignment between vision and language; Pixel-level encoder for input image Extracting fine-grained pixel-level features This provides accurate local positioning information for subsequent segmentation decoders; The dual encoder design decouples semantic understanding from pixel-level localization, enabling features at both levels to interact collaboratively on the decoder side through bidirectional semantic flow.

4. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement according to claim 1, characterized in that, Step 3 involves constructing a language model processing module, including: Introduce special markers 、 , <seg>Annotate the text instructions, where and Used to define the range of phrases for the target defect category. <seg> Used to indicate the location of the split output;< / seg> < / seg> The large language model processes the concatenated text and visual tokens, and outputs the hidden state. Extract conditional embeddings from them Corresponding to phrase tag position and segmentation embedding ,correspond <seg> Mark the location;< / seg> Through projection network Conditional embeddings and segmentation embeddings are mapped to the decoder space to obtain... and ; Initial decoder query By segmentation embedding Perform additive initialization, that is This allows for the injection of global multimodal priors at the beginning of the decoding phase.

5. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement according to claim 1, characterized in that, Step 4 involves constructing a segmentation decoder based on bidirectional semantic streams, including: A semantic cross-attention step is introduced after standard visual cross-attention, in the decoder's... Layer, Decoding Query Conditional embedding Attention operations are performed on keys and values, first layer hour For the initial conditional embedding, the semantic cross-attention step is as shown in Equation (1): （1）， in, As a standard multi-head attention mechanism, the three parameter positions represent the query, key, and value, respectively; For conditional embedding, For visual queries, This is the output of the semantic cross-attention step, i.e., the semantic query; An adaptive fusion gate mechanism calculates the dynamic fusion ratio between visual and semantic queries, with the fusion weight for each query determined by a gating signal. The decision is as shown in equations (2) to (3): （2）， （3）， in, For learnable weights in the adaptive fusion gate mechanism, For the Sigmoid function, For feature splicing operations, Element-wise multiplication; gate signal Calculated at the query granularity, allowing different queries to adaptively adjust the fusion ratio based on the importance of current visual and semantic information; Merged query By capturing the global context between queries through self-attention operations, a refined query is obtained. As shown in equation (4): （4）， To establish a true bidirectional semantic flow, the conditional embedding also incorporates visual evidence by updating the decoded query through a multi-head attention mechanism, as shown in equation (5): （5）， The conditional embedding is used as a query, and the refined decoded query is used as such. As keys and values, this mechanism, together with semantic cross-attention, constitutes the forward flow. and reverse flow An explicit two-way closed loop; refined conditional embedding in the final stage. With decoding query output Cosine similarity between them is used to classify and select masks.

6. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement according to claim 1, characterized in that, In step 5, a boundary-aware mask refinement module is constructed, including: Based on the original mask probability map The boundary pixel region is located using morphological gradient operations, as shown in equation (6): （6）， in, For binary boundary mask, To control the threshold of boundary sensitivity, and These are expansion and erosion operators, respectively. The indicator function is defined as follows: if the position in the probability map meets the above conditions, the corresponding value is 1; otherwise, it is 0. Thus, the boundary position is located using morphological gradients. Refined Network Original mask With compressed network Pixel-level features after dimensionality reduction The process involves stitching and merging, calculating bounded residual updates, and updating only in the boundary regions. The internal refinement is applied as shown in equations (7) to (8): （7）， （8）， in, For learnable scaling factor, For feature splicing operations, It is an element-wise product; the refinement operation is strictly limited to the boundary uncertainty region to avoid affecting the prediction of the already high-confidence internal region.

7. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement according to claim 1, characterized in that, In step 6, the feature extraction module, language model processing module, bidirectional semantic stream segmentation decoder, and boundary-aware mask refinement module are jointly trained end-to-end. The total loss of the end-to-end multi-task training is... The sum of the language modeling loss and the segmentation loss is shown in Equation (9): （9）， Among them, language modeling loss Using the standard next-word prediction cross-entropy loss, Text and images respectively. The sequence of text terms generated before time t. Generate text words for the current moment. The conditional probability is shown in equation (10): （10）， Segmentation loss The prediction mask is aligned with the ground truth label using Hungarian matching, as shown in Equation (11): （11）， in For classification cross-entropy loss, and These are Dice loss and mask binary cross-entropy loss, respectively. and To balance the contributions of each loss to the hyperparameters, deep supervision is applied to all decoder layers to stabilize the optimization process of multi-layer semantic propagation.

8. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement according to claim 1, characterized in that, In step 7, input the image to be inspected and the defect detection command, including: Images of industrial products and natural language defect description instructions As input, the instructions may contain defect_name The target phrase is labeled in a specific format to precisely constrain the detected targets; the model outputs a refined pixel-level defect segmentation mask. The positive region in the mask is the location of the detected defect, and the confidence score of the corresponding defect category is output at the same time; it supports parallel output of independent detection masks for multiple defect categories in a single image.

9. The industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement according to claim 3, characterized in that, In step 2, both the semantic encoder and the pixel-level encoder can employ pre-trained visual backbone networks; the features extracted by the semantic encoder are processed by a lightweight multilayer perceptron. Mapped to a large language model embedding space to support vision-language alignment; Pixel-level features extracted by a pixel-level encoder It is directly fed into the segmentation decoder and boundary-aware refinement module to provide fine-grained structural information for accurate positioning.

10. An industrial defect detection system based on bidirectional semantic flow and boundary-aware refinement, applied to the industrial defect detection method based on bidirectional semantic flow and boundary-aware refinement as described in any one of claims 1 to 9, characterized in that, include: The image acquisition unit is used to acquire image frames of products to be inspected on the industrial production line. The text instruction unit is used to generate natural language detection instructions containing descriptions of defect categories based on a preset list of defect categories or operator input. The dual encoder feature extraction module is used to extract high-level semantic features and fine-grained pixel-level features of images; The language model processing module is used to parse text instructions and output conditional embeddings and segmentation embeddings; The bidirectional semantic stream decoding module contains multi-layered cascaded decoder units, each layer containing semantic cross-attention, adaptive fusion gates, and conditional refinement sub-modules, which perform visual-semantic bidirectional interaction and generate candidate masks; The boundary-aware mask refinement module uses morphological operations to locate the boundary region and then applies selective residual refinement to output the refined defect segmentation mask. The decision output unit determines whether a product is qualified or not based on the defect mask and confidence score, and outputs the defect category and location information to trigger production line alarms or rejection actions.

Images