Capsule material visual defect detection method and system based on hybrid neural network

Abstract

This application provides a method and system for visual defect detection of capsule materials based on a hybrid neural network. The method includes simultaneously inputting the acquired visual image of the capsule material into parallel local feature extraction branches and global feature extraction branches; inputting the local feature map and global feature vector into a gated cross-attention interactive fusion module for bidirectional interaction and fusion to generate a fused feature vector; and inputting the fused feature vector into a classification head for classification and recognition, outputting the defect category of the capsule material. This application addresses the fundamental differences in feature levels between "damage" and "wrinkles" on the surface of capsule materials, designing a parallel hybrid model that can consider both local and global features and perform feature complementarity and intelligent fusion. This solves the core problems of low accuracy, poor generalization ability, and difficulty in real-time deployment faced by surface defect detection in the automated production process of capsule materials.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to a method and system for visual defect detection of capsule materials based on hybrid neural networks. Background Technology

[0002] The capsule material is a crucial foundational material for airships, and its surface must be free of any defects that could affect airtightness and structural strength, such as minor breaks, cracks, or large-area wrinkles. Currently, the industry generally relies on manual visual inspection for quality control. This method is not only inefficient, but more seriously, the inspection results are highly susceptible to fluctuations due to the subjective fatigue and experience differences of quality inspectors. For defects like wrinkles, which are difficult to distinguish from normal material deformation, the rate of missed detection is particularly high, becoming a bottleneck restricting production automation and quality consistency.

[0003] Early automated production lines attempted to replace manual labor with automated optical inspection (AOI) technology based on traditional image processing. However, it relied on manually set thresholds, filters, and feature extraction, and had poor generalization ability for defects in capsule materials with varying shapes, sizes, and contrasts (e.g., wrinkles of varying depths and irregularly distributed damage), making it unable to adapt to complex production line environments.

[0004] In recent years, with the rise of deep learning, convolutional neural networks (CNNs) have begun to show their potential in this field. CNNs, after learning, possess extremely high generalization capabilities, demonstrating detection rates far exceeding those based on traditional image processing for local detail features (such as scratches and pinholes). However, for defects that require judgment based on the macroscopic structure of the entire image, such as slight and widespread tension unevenness or shallow wrinkles blending into the background texture, the limitations of their local receptive field make it difficult to establish long-distance dependencies, leading to misjudgments or missed detections.

[0005] The Visual Transformer (ViT) model effectively captures global contextual information through its self-attention mechanism, which precisely compensates for the aforementioned shortcomings of CNNs. However, the standard ViT model loses fine-grained local textures that are crucial for defect detection when processing high-resolution images, and its high computational complexity makes it difficult to deploy on industrial edge devices that require real-time performance.

[0006] Furthermore, what's even more challenging is the inherent contradiction in the defect morphology of the capsule material: "Folds" are a macroscopic structural problem, requiring a model to "see the whole picture" in order to perceive their overall shape and trend.

[0007] "Damage" is a microscopic, local problem that requires the model to be "observant" in order to capture abrupt changes in its edges and textures.

[0008] Existing technologies, whether single CNNs, ViTs, or the serial stacked hybrid architecture used in D1 (CN117993442B), struggle to effectively address this contradiction. The serial structure leads to "path dependence" in feature extraction; earlier modules (whether CNNs or ViTs) pass their inherent inductive biases to subsequent modules, limiting the model's ability to adaptively balance local and global feature weights based on specific flaws. Simple feature concatenation or addition methods cannot achieve this intelligent, dynamic trade-off.

[0009] Therefore, in the specific task of defect detection of capsule materials, how to deeply integrate the local detail perception capability of CNN with the global structure modeling capability of ViT, and meet the requirements of lightweight and real-time industrial deployment while ensuring high accuracy, is a current unresolved technical problem.

[0010] The foregoing statements are for informational purposes only and are not intended to provide background information in connection with this application. Unless otherwise stated herein, the content described in this section is not prior art to the rest of this application. Summary of the Invention

[0011] To overcome the shortcomings of existing technologies, this invention provides a method and system for visual defect detection of capsule materials based on hybrid neural networks. Its core objective is to design a parallel hybrid model that can take into account both local and global features and perform feature complementarity and intelligent fusion, addressing the fundamental differences in feature levels between "damage" and "wrinkles" on the surface of capsule materials.

[0012] Specifically, this invention uses parallel lightweight CNN branches and ViT branches to accurately capture the microscopic texture and macroscopic structure of images, respectively. Furthermore, through an innovative gated cross-attention interaction fusion module (GCA-IFM), deep bidirectional interaction between the two branches is achieved: global features guide local features to focus on key regions related to structural defects, while the enhanced local information, in turn, feeds back and refines the global features.

[0013] This architecture ensures that the model relies more heavily on global contextual information when dealing with suspected "wrinkles," while amplifying the contribution of local detailed features when judging "damage." Ultimately, without significantly increasing computational overhead, it significantly improves the accuracy and robustness of identifying various complex defects such as "wrinkles" and "damage," thereby meeting the stringent requirements of industrial automated production for detection accuracy and efficiency.

[0014] According to a first aspect of the embodiments of this application, a method for visual defect detection of capsule materials based on hybrid neural networks is provided, comprising the following steps: Acquire visual imaging images of the capsule material; The image is simultaneously input into both the local feature extraction branch and the global feature extraction branch in parallel. Local features of the image are extracted using a lightweight convolutional neural network in the local feature extraction branch, generating a local feature map. Global features of the image are extracted using the visual Transformer model in the global feature extraction branch, generating a global feature vector. Local feature maps and global feature vectors are input into a gated cross-attention interaction fusion module for bidirectional interaction and fusion to generate a fused feature vector. The fused feature vector is input into the classification head for classification and identification, and the defect category of the capsule material is output.

[0015] In some embodiments of this application, local feature maps and global feature vectors are input into a gated cross-attention interaction fusion module for bidirectional interaction and fusion to generate a fused feature vector, including: Using global features as queries and local features as keys and values, a context vector weighted by global information is calculated through a cross-attention mechanism. The context vector and local features are added and aggregated to generate a local information summary vector; and a modulation gate is dynamically generated based on the local information summary vector and global features through a gating unit to modulate the global features. The modulated global features are concatenated and linearly transformed with the local information summary vector to generate a fused feature vector.

[0016] In some embodiments of this application, a context vector weighted by global information is calculated through a cross-attention mechanism, including: Flatten the local feature map in the spatial dimension into a sequence of local feature tokens; Local feature token sequences and global feature vectors are mapped to a unified interaction dimension through linear projection; Using the projected global features as the query and the projected local feature sequence as the key and value, cross-attention computation is performed to obtain the context vector.

[0017] In some embodiments of this application, the gating unit employs the Sigmoid activation function to dynamically generate a modulation gate with a range of (0,1) based on the local information digest vector and global features, and modulates the global features through element-wise multiplication.

[0018] In some embodiments of this application, the local feature extraction branch uses MobileNetV3 as a lightweight convolutional neural network to output a two-dimensional local feature map that preserves spatial structure information.

[0019] In some embodiments of this application, the classification head is a multilayer perceptron, including one or more fully connected layers, and finally outputs the probability distribution corresponding to each defect category through a linear layer with K output units and a Softmax function, where K is the number of defect categories.

[0020] In some embodiments of this application, defect categories include conformity, breakage, and wrinkles.

[0021] According to a second aspect of the embodiments of this application, a visual defect detection system for capsule materials based on a hybrid neural network is provided, comprising: Image acquisition module, used to acquire visual imaging images of capsule material; The parallel feature extraction module includes parallel local feature extraction branches and global feature extraction branches. It is used to extract local features of the image through a lightweight convolutional neural network in the local feature extraction branch to generate a local feature map; at the same time, it extracts global features of the image through a visual Transformer model in the global feature extraction branch to generate a global feature vector. The feature fusion module employs a gated cross-attention interactive fusion module, which is used to input local feature maps and global feature vectors into the gated cross-attention interactive fusion module for bidirectional interaction and fusion, generating a fused feature vector. The classification and recognition module is used to input the fused feature vector into the classification head for classification and recognition, and output the defect category of the capsule material.

[0022] According to a third aspect of the embodiments of this application, a device for detecting visual defects in capsule materials based on a hybrid neural network is provided, comprising: a storage unit for storing executable instructions; and a processing unit for connecting to the storage unit to execute the executable instructions to complete the method for detecting visual defects in capsule materials based on a hybrid neural network.

[0023] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon; the computer program is executed by a processor to implement a method for visual defect detection of capsule materials based on a hybrid neural network.

[0024] The present application discloses a method and system for visual defect detection of capsule materials based on hybrid neural networks. The method includes simultaneously inputting visual images of the capsule material into parallel local feature extraction branches and global feature extraction branches; extracting local features of the image using a lightweight convolutional neural network in the local feature extraction branch to generate a local feature map; extracting global features of the image using a visual Transformer model in the global feature extraction branch to generate a global feature vector; inputting the local feature map and global feature vector into a gated cross-attention interaction fusion module for bidirectional interaction and fusion to generate a fused feature vector; and inputting the fused feature vector into a classification head for classification and recognition, outputting the defect category of the capsule material. This application significantly improves the recognition accuracy of complex defects (especially structural wrinkles) with diverse shapes and scales by constructing a parallel hybrid model of CNN and ViT, taking into account both local and global features, and then significantly improving the recognition accuracy of complex defects (especially structural wrinkles) with diverse shapes and scales through feature complementarity and intelligent fusion. This solves the core problems of low accuracy, poor generalization ability, and difficulty in real-time deployment faced by surface defect detection in the automated production process of capsule materials.

[0025] This application addresses the fundamental differences in the characteristics of two types of defects on the surface of capsule materials: “damage” and “wrinkles.” It designs a parallel hybrid model that can take into account both local and global characteristics and can perform feature complementarity and intelligent fusion. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The figure shows a schematic diagram of the steps of a visual defect detection method for capsule materials based on a hybrid neural network according to an embodiment of this application; Figure 2 The diagram illustrates the steps of feature fusion according to an embodiment of this application; Figure 3 The diagram shows an overall flowchart of a visual defect detection method for capsule materials based on a hybrid neural network according to an embodiment of this application; Figure 4 The diagram shows a principle flowchart of a gated cross-attention interaction fusion module according to an embodiment of this application; Figure 5 The diagram shows a schematic representation of a visual defect detection system for capsule materials based on a hybrid neural network according to an embodiment of this application. Figure 6 The diagram shows a structural schematic of a cyst material visual defect detection device 400 based on a hybrid neural network according to an embodiment of this application. Detailed Implementation

[0027] Regarding this application, for the specific industrial visual classification task of detecting surface defects in capsule materials, with the rise of deep learning, models represented by convolutional neural networks (CNNs) have been applied to defect detection, significantly improving accuracy. CNNs, through their inherent inductive bias, excel at extracting local features of images (such as edges, textures, and corners), making them very suitable for identifying local defects such as cracks and scratches. However, for defects that require understanding the overall structure of the image for judgment, such as large-area slight wrinkles, overall material deformation, or uneven tension, the local receptive field of CNNs limits their ability to capture long-range dependencies and global contextual information.

[0028] Meanwhile, the Vision Transformer (ViT) model has made breakthroughs in the field of image recognition. ViT, through its self-attention mechanism, can effectively capture global features and long-range dependencies in images, compensating for the shortcomings of CNNs. However, the standard ViT model may ignore the lack of fine-grained local details, which is crucial for defect detection, and its computational complexity is high.

[0029] Therefore, how to effectively combine the advantages of CNN in local feature extraction with the global information modeling capabilities of ViT, while ensuring the lightweight nature of the model to adapt to industrial deployment, is a technical challenge that urgently needs to be solved in this field.

[0030] In summary, this application aims to solve the following technical problems existing in the visual inspection technology of capsule materials: 1. Limitations of single feature representation: Existing methods typically rely on only one of CNN or ViT, resulting in incomplete feature representation. Using CNN alone is prone to ignoring global and structural defects such as wrinkles; using ViT alone may lose local texture and edge details that are crucial for minor damage, leading to missed or false detections.

[0031] 2. Insufficient fusion of local and global information: Simple feature concatenation or addition methods cannot intelligently and adaptively weigh the importance of local and global features in the final decision. For example, for the "damaged" category, local features should have higher weights; while for the "wrinkled" category, global features are more critical. Existing methods lack a dynamic and discriminative feature fusion mechanism.

[0032] 3. Difficulty in balancing model performance and efficiency: In industrial online inspection scenarios, models not only need high accuracy but also must have low-latency real-time inference capabilities. Many high-precision models have complex structures and a large number of parameters, making them difficult to deploy efficiently on edge computing devices.

[0033] Based on this, this application provides a method and system for visual defect detection of capsule materials based on hybrid neural networks. The method includes simultaneously inputting visual images of the capsule material into parallel local feature extraction branches and global feature extraction branches; extracting local features of the image using a lightweight convolutional neural network in the local feature extraction branch to generate a local feature map; extracting global features of the image using a visual Transformer model in the global feature extraction branch to generate a global feature vector; inputting the local feature map and global feature vector into a gated cross-attention interaction fusion module for bidirectional interaction and fusion to generate a fused feature vector; and inputting the fused feature vector into a classification head for classification and recognition, outputting the defect category of the capsule material. This application significantly improves the recognition accuracy of complex defects (especially structural wrinkles) with diverse shapes and scales by constructing a parallel hybrid model of CNN and ViT, taking into account both local and global features, and then significantly improving the recognition accuracy of complex defects (especially structural wrinkles) with diverse shapes and scales through feature complementarity and intelligent fusion. This solves the core problems of low accuracy, poor generalization ability, and difficulty in real-time deployment faced by surface defect detection in the automated production process of capsule materials.

[0034] Compared with existing technologies, this application has the following technical advantages: 1. Improve detection accuracy: By complementing and dynamically fusing local and global features, the ability to identify complex defects (such as wrinkles and damage) is significantly improved.

[0035] 2. Enhance model generalization ability: By combining data augmentation, transfer learning and other strategies, the model has better adaptability to different lighting, backgrounds and defect morphologies.

[0036] 3. Achieve efficient real-time detection: The model structure is lightweight and suitable for deployment on edge devices, meeting the real-time requirements of industrial online detection.

[0037] 4. Adaptive feature fusion: The gating mechanism enables the model to dynamically adjust feature weights according to different defect types, thereby improving classification accuracy.

[0038] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0039] Example 1 Figure 1 The diagram illustrates the steps of a method for visual defect detection of capsule materials based on a hybrid neural network according to an embodiment of this application.

[0040] like Figure 1 As shown, a method for visual defect detection of capsule materials based on hybrid neural networks is provided, including the following steps: S1: Acquire visual imaging images of the capsule material; S2: Simultaneously input the image into the parallel local feature extraction branch and the global feature extraction branch; S3: Extract local features of the image using a lightweight convolutional neural network in the local feature extraction branch to generate a local feature map; extract global features of the image using a visual Transformer model in the global feature extraction branch to generate a global feature vector. S4: Input the local feature map and the global feature vector into the gated cross-attention interaction fusion module for bidirectional interaction and fusion to generate a fused feature vector; S5: Input the fused feature vector into the classification head for classification and recognition, and output the defect category of the capsule material.

[0041] By constructing a parallel hybrid model of CNN and ViT, taking into account both local and global features, and then through feature complementarity and intelligent fusion, the accuracy of identifying complex defects (especially structural wrinkles) with diverse shapes and scales is significantly improved. This solves the core problems of low accuracy, poor generalization ability, and difficulty in real-time deployment faced by surface defect detection in the automated production process of capsule materials.

[0042] Figure 2 The diagram illustrates the feature fusion steps according to an embodiment of this application.

[0043] Further description, such as Figure 2 As shown, in S4, the local feature map and global feature vector are input into the gated cross-attention interaction fusion module for bidirectional interaction and fusion, generating a fused feature vector, including: S41: Global to Local Guided Enhancement Stage: Using global features as queries and local features as keys and values, a context vector weighted by global information is calculated through a cross-attention mechanism; S42: Local to Global Feedback Modulation Stage: The context vector and local features are added and aggregated to generate a local information summary vector; and a modulation gate is dynamically generated by a gating unit based on the local information summary vector and global features to modulate the global features; S43: Final Feature Aggregation Stage: The modulated global features and local information summary vectors are concatenated and linearly transformed to generate a fused feature vector.

[0044] Specifically, in S41, a context vector weighted by global information is calculated through a cross-attention mechanism, which includes: First, the local feature map is flattened into a sequence of local feature tokens in the spatial dimension; Then, the local feature token sequence and the global feature vector are mapped to a unified interaction dimension through linear projection; Secondly, using the projected global features as the query and the projected local feature sequence as the key and value, cross-attention computation is performed to obtain the context vector.

[0045] In the preferred implementation, the gating unit adopts the Sigmoid activation function, dynamically generates a modulation gate with a range of (0,1) based on the local information digest vector and global features, and modulates the global features through element-wise multiplication.

[0046] In the preferred implementation, the local feature extraction branch uses MobileNetV3 as a lightweight convolutional neural network to output a two-dimensional local feature map that preserves spatial structure information.

[0047] The classification head is a multilayer perceptron, which includes one or more fully connected layers. Finally, it outputs the probability distribution corresponding to each defect category through a linear layer with K output units and a Softmax function, where K is the number of defect categories.

[0048] Defect categories include acceptable, broken, and wrinkled.

[0049] Figure 3 The diagram shows an overall flowchart of a visual defect detection method for capsule materials based on a hybrid neural network according to an embodiment of this application.

[0050] like Figure 3 As shown, this application presents a parallel, lightweight deep learning detection method that integrates local and global feature attention in its overall process.

[0051] Input: Obtain a visual image of the capsule material.

[0052] Parallel Feature Extraction: The input image is simultaneously fed into two parallel feature extraction branches. Local feature extraction branch: A lightweight convolutional neural network (CNN) is used to efficiently extract fine-grained features such as local texture and edges from images.

[0053] Global Feature Extraction Branch: A Visual Transformer (ViT) model for capturing long-range dependencies and global structural features of images.

[0054] Hybrid Feature Fusion: The local feature maps and global feature vectors extracted from the two branches are input into a HybridLocal-Global Feature Attention Fusion Module designed in this invention. This module dynamically enhances and filters key local features using global contextual information, generating a more informative and expressive fused feature vector.

[0055] Classification: The fused feature vectors are fed into a multi-layer perceptron (MLP) classification head, which passes through fully connected layers and a softmax activation function to finally output the classification result of the image: "qualified", "damaged" or "wrinkled".

[0056] This method achieves efficient extraction of local and global features through a parallel structure, and realizes intelligent complementarity of the two features through the core attention fusion module. Ultimately, it significantly improves the recognition accuracy of various complex defects while ensuring lightweight design.

[0057] The overall architecture of the neural network model proposed in this application for the visual defect detection method of cystic materials based on hybrid neural networks consists of an input module, parallel feature extraction branches (local and global), a gated cross-attention interaction fusion module, and a final classification head.

[0058] The following section provides a detailed explanation of the principles and mathematical methods used in each part.

[0059] 1) Regarding system input: A visual imaging image of a capsule material .

[0060] Before being fed into the model, the images are preprocessed to a uniform size. And convert it as needed. Each color channel (e.g., for RGB, (for Grayscale).

[0061] Therefore, the input tensor can be represented as: .

[0062] 2) Regarding the local feature extraction branch (Lightweight CNN Branch); The local feature extraction branch aims to capture high-frequency local details in images in a computationally efficient manner, such as edges with minor damage and fine textures on material surfaces.

[0063] Local feature extraction branch structure: A lightweight convolutional neural network is used. (such as MobileNetV3).

[0064] Process: Input Image It goes through a series of convolutional layers, activation functions, and max pooling layers.

[0065] Output: Generate a two-dimensional local feature map. This map preserves key spatial structure information, as shown below: ; in, It is the spatial dimension after downsampling. It represents the number of channels for local features.

[0066] 3) Regarding the global feature extraction branch (Vision Transformer Branch); The global feature extraction branch is responsible for capturing long-range dependencies and global contextual information in an image, which is crucial for identifying large-scale structural defects such as wrinkles.

[0067] Global feature extraction branch structure: Employing the visual Transformer model .

[0068] Process: Image patching: Transforming the input image into patches... Divided into There are 3 non-overlapping image patches, each of size 1. .

[0069] Linear Embedding: Flattens each image patch and maps it through a linear projection layer to... A dimensional vector.

[0070] [CLS] Token and Position Encoding: A learnable classification token [CLS] is added before the sequence, and position encoding is added to all tokens to preserve spatial information.

[0071] Transformer encoder: The processed token sequence is fed into a multi-layer Transformer encoder, each layer of which consists of a multi-head self-attention (MHSA) and a feed-forward network (FFN).

[0072] Output: Extract the output vector corresponding to the [CLS] token as the global feature representation of the entire image. : .

[0073] 4) Regarding the Gated Cross-Attention Interactive Fusion Module (GCA-IFM); The core innovation of this invention is the gated cross-attention interaction fusion module. This module aims to break the unidirectional information flow and achieve full, bidirectional interaction and enhancement between local and global features, rather than simply splicing them together.

[0074] Figure 4 The diagram shows a schematic flowchart of a gated cross-attention interaction fusion module according to an embodiment of this application.

[0075] like Figure 4 As shown, the gated cross-attention interaction fusion module comprises three key stages: Phase 1: Global-to-Local Guidance; This stage utilizes global contextual information to dynamically enhance the local feature regions that are most meaningful for the final classification.

[0076] First, prepare the features; Local feature maps Flattening it in the spatial dimension yields a local feature token sequence. ,in .

[0077] Through linear projection layer Map local and global features to a unified interaction dimension : ; .

[0078] Then, construct a cross-attention mechanism; Construct a cross-attention module where global features serve as the query and local feature sequences serve as the key and value, as shown below: ; ; ; Attention calculation yields a context vector weighted by global information. : .

[0079] Phase Two: Local-to-Global Feedback.

[0080] This stage utilizes the enhanced local information, guided by global guidance, to refine and modulate the global features, forming an information loop.

[0081] First, local information aggregation; The context vector obtained in the previous stage Compared with the original local feature sequence The sequences are added (or linearly transformed after concatenation) and then normalized at the application layer (LayerNorm) to generate enhanced local feature sequences. : ; Then, global average pooling is performed on the enhanced local feature sequences to generate a concise local information summary vector. : .

[0082] Next, we will design a gated modulation mechanism; Design a gating unit that, based on local summaries... and original global features Dynamically generate a modulation gate as follows: ; in, It is the Sigmoid activation function, with an output range of (0, 1). These are learnable parameters.

[0083] The door The original global features are multiplied element-wise to achieve modulation: ; in This represents element-wise multiplication (Hadamard Product).

[0084] Phase 3: Final Feature Aggregation; The local and global information, which has undergone two-way interaction and optimization, will be finally integrated.

[0085] Modulated global features Summary vector representing local information Perform concatenation; The concatenated vectors are then deeply fused through a linear layer with an activation function (such as GELU) to generate the final, highly condensed feature vector. .

[0086] .

[0087] 5) Finally, regarding the Classification Head.

[0088] Classification head structure: A standard multilayer perceptron (MLP). .

[0089] Process: Receive the final fused feature vector. After passing through one or more fully connected layers, it finally passes through a layer with... One output unit (in this invention) The linear layer is used to obtain the logits corresponding to each category.

[0090] Output: Logical values ​​are processed by the Softmax function to transform them into the final class probability distribution. as follows: ; The category with the highest probability is the model's prediction.

[0091] To verify the effectiveness and superiority of the visual defect detection method for capsule materials based on hybrid neural networks proposed in this invention, a series of comparative experiments and ablation experiments were designed and conducted in the embodiments of this application.

[0092] 1) Dataset and experimental setup.

[0093] Dataset: This experiment constructed a dedicated dataset for capsule material defects, CM-Defect. This dataset contains 15,000 high-resolution images of capsule material surfaces collected in a real industrial production line environment, precisely labeled by senior quality control engineers. The dataset includes three categories: acceptable, damaged, and wrinkled, with a balanced sample distribution. It is randomly divided into a training set (12,000 images), a validation set (1,500 images), and a test set (1,500 images) in an 8:1:1 ratio.

[0094] Comparison Methods: To comprehensively evaluate the performance of this invention, the following representative methods were selected for comparison: A. ResNet-50: A classic deep convolutional neural network, serving as the benchmark for pure CNN models.

[0095] B. ViT-Base: The standard visual Transformer model, serving as a benchmark for pure Transformer models.

[0096] C. D1-SUMMNet: A serial hybrid model implemented according to the method described in Comparative Document 1 (CN117993442B).

[0097] D. Method of the present invention: The parallel CNN-ViT hybrid architecture and gated cross-attention interaction fusion module (GCA-IFM) proposed in this invention.

[0098] Experimental Environment and Parameters: All experiments were conducted on an Ubuntu 18.04 system using a single NVIDIA TITAN Xp GPU. Model training employed the AdamW optimizer with an initial learning rate of 1e-3, coupled with a cosine annealing scheduler. The batch size was set to 32. For fair comparison, all comparison models were fine-tuned on the same CM-Defect dataset using the same training strategy.

[0099] Evaluation metrics: Accuracy, macro average F1-Score, number of model parameters, and inference speed (FPS, Frames Per Second) are used as core evaluation metrics to comprehensively measure the model's classification accuracy, lightweight nature, and real-time performance.

[0100] 2) Overall performance comparison results and analysis.

[0101]

[0102] Table 1 Overall performance comparison (on the CM-Defect test set) The conclusions and analysis are as follows: Accuracy Advantage: The method of this invention (D) achieved the highest accuracy (95.4%) and F1-Score (0.949), significantly outperforming the pure CNN model (A), the pure Transformer model (B), and D1-SUMMNet (C), which represents a serial hybrid model. This fully demonstrates the effectiveness of the parallel hybrid architecture combined with the GCA-IFM module in the task of classifying defects in capsule materials.

[0103] Efficiency advantages: While achieving the highest accuracy, the method of this invention has only 4.2M parameters, far lower than other comparative models, and even an order of magnitude smaller than the lightweight ResNet-50. Its inference speed also reaches the fastest 62 FPS, perfectly balancing high accuracy and high efficiency, and meeting the real-time requirements of industrial online inspection.

[0104] Targeted problem solving: Further analysis of the recall rates for each category revealed that the present invention improved the recall rate for structural defects such as "wrinkles" by 8.5% compared to D1-SUMMNet, and improved the recall rate for local detail defects such as "damage" by 5.2%. This directly confirms that the present invention can effectively solve the two core problems mentioned in the background technology: "limitations of single feature representation" and "insufficient fusion of local and global information".

[0105] 3) Ablation Study.

[0106] To verify the contribution of each component in the core module of this invention—the Gated Cross-Attention Interaction Fusion Module (GCA-IFM)—the following ablation experiments were designed in this application embodiment, and the ablation experiments of the GCA-IFM module are analyzed with reference to Table 2.

[0107]

[0108] Table 2: Experimental Analysis of GCA-IFM Module Ablation The conclusions and analysis are as follows: Effectiveness of one-way interaction: The accuracy of variants 2 and 3 improved by 1.7% and 2.4% respectively compared to the baseline (variant 1), which proves that each one-way interaction mechanism in GCA-IFM is effective and can bring significant performance gains to the model.

[0109] Synergistic Effects of Two-Way Interaction (Unexpected Technical Results): This invention (Variant 4), which fully combines two one-way interaction mechanisms, achieved an accuracy of 95.4%. Its performance improvement (3.3% compared to the baseline) exceeded the simple sum of the performance improvements of Variant 2 and Variant 3 (1.7% + 2.4% = 4.1%). This phenomenon indicates a positive synergistic effect between global guidance and local feedback; they do not work independently but form an organic, two-way complementary information enhancement loop.

[0110] 4) Experiment summary.

[0111] Based on the above comparative and ablation experiments, the following conclusions can be drawn: 1. The visual defect detection method for capsule materials based on hybrid neural networks provided by this invention is significantly superior to existing mainstream models and hybrid models in terms of detection accuracy.

[0112] 2. This invention successfully achieves a balance between accuracy and efficiency, with a high degree of model lightweighting, meeting the needs of real-time industrial detection.

[0113] 3. The core innovation of this invention—the Gated Cross-Attention Interaction Fusion Module (GCA-IFM) and its bidirectional interaction mechanism—has been experimentally proven to be the key to performance improvement, and this mechanism has produced a synergistic enhancement effect that exceeds the expectations of those skilled in the art.

[0114] Regarding model training, the following training strategy is adopted to achieve optimal performance of the model proposed in this invention: Dataset: Construct a large-scale, high-quality visual image dataset of capsule materials. This dataset should include samples in three categories: "qualified," "damaged," and "wrinkled," ensuring a balanced data distribution. The dataset should be rigorously partitioned into training, validation, and test sets.

[0115] Data Augmentation: During training, a series of online data augmentation techniques are applied to improve the model's generalization ability, including but not limited to: random rotation, flipping, cropping, color jittering (brightness, contrast, saturation adjustment), Gaussian blur, and adding noise.

[0116] Loss Function: The Cross-Entropy Loss function, which is suitable for multi-class classification tasks, is adopted.

[0117] Optimizer: The AdamW optimizer is used, which improves upon Adam by handling weight decay, typically resulting in better training performance. An initial learning rate is set, and a learning rate decay strategy (such as a cosine annealing scheduler) is employed to reduce the learning rate later in training, helping the model converge to a better solution.

[0118] Transfer Learning: To accelerate convergence and improve performance, both the lightweight CNN for local feature extraction and the ViT model for global feature extraction can be initialized by loading pre-trained weights on large public datasets (such as ImageNet) and then fine-tuned on our own vesicle material dataset.

[0119] Training process: End-to-end training is performed using Mini-batch Stochastic Gradient Descent (SGD). At the end of each training epoch, the model's performance (e.g., accuracy, precision, recall) is evaluated on the validation set, and the weights of the best-performing model are saved. Early stopping can be used to prevent overfitting based on changes in validation set performance.

[0120] In summary, the visual defect detection method for capsule materials based on hybrid neural networks in this application has the following key innovations: (1) Parallel feature extraction architecture: At the same time, a lightweight CNN is used to extract local detail features (such as edges and textures); Use ViT to extract global structural features (such as long-range dependencies).

[0121] (2) Gated Cross-Attention Interaction Fusion Module (GCA-IFM): Global-to-local guided enhancement: dynamically enhancing key local features using global context; Local-to-global feedback modulation: Refining global features through local information to form two-way information interaction; (3) Gating mechanism: dynamically modulate global features to achieve adaptive feature fusion.

[0122] (4) Lightweight design and efficient classification: It employs lightweight CNNs (such as MobileNetV3) and ViT, combined with a high-efficiency fusion module; Finally, the MLP classification head outputs three categories of results: "qualified", "damaged", and "wrinkled".

[0123] Compared with existing technologies, it has the following technical advantages: 1. Improve detection accuracy: By complementing and dynamically fusing local and global features, the ability to identify complex defects (such as wrinkles and damage) is significantly improved.

[0124] 2. Enhance model generalization ability: By combining data augmentation, transfer learning and other strategies, the model has better adaptability to different lighting, backgrounds and defect morphologies.

[0125] 3. Achieve efficient real-time detection: The model structure is lightweight and suitable for deployment on edge devices, meeting the real-time requirements of industrial online detection.

[0126] 4. Adaptive feature fusion: The gating mechanism enables the model to dynamically adjust feature weights according to different defect types, thereby improving classification accuracy.

[0127] In some preferred embodiments, the feature fusion task employs an adaptive dynamic fusion network.

[0128] Gating mechanisms can not only fuse features, but also dynamically select the dominant branch based on the potential defect types of the input image. For example, the model can automatically determine that "this image is more likely to have wrinkles," thereby automatically increasing the weight of the global branch during fusion.

[0129] In practice, a lightweight "routing network" is added before the gated cross-attention interaction fusion module GCA-IFM to pre-classify the input and guide the subsequent fusion strategy.

[0130] In some preferred embodiments, a regression subnetwork oriented towards defect severity is added, extending a regression head parallel to the classification head to predict the severity of defects (such as the percentage of wrinkled area, the pixel value of the damage length).

[0131] This allows the output to be upgraded from "whether there are defects" to "how severe the defects are", which is more instructive for industrial quality rating.

[0132] In some preferred embodiments, this application also proposes a hybrid pre-training strategy.

[0133] Training the CNN and ViT branches for collaborative pre-training tasks targeting industrial defects, such as using unlabeled industrial images for contrastive learning, forces both branches to learn to focus on complementary information during the pre-training phase.

[0134] This enables the model to improve its feature extraction capabilities in specific domains and accelerates the convergence of downstream tasks.

[0135] Therefore, the visual defect detection method for capsule materials based on hybrid neural networks proposed in this application includes simultaneously inputting the visual imaging image of the capsule material into parallel local feature extraction branches and global feature extraction branches; extracting local features of the image through a lightweight convolutional neural network in the local feature extraction branch to generate a local feature map; extracting global features of the image through a visual Transformer model in the global feature extraction branch to generate a global feature vector; inputting the local feature map and the global feature vector into a gated cross-attention interaction fusion module for bidirectional interaction and fusion to generate a fused feature vector; and inputting the fused feature vector into a classification head for classification and recognition to output the defect category of the capsule material.

[0136] This application constructs a parallel hybrid model of CNN and ViT, taking into account both local and global features. Then, through feature complementarity and intelligent fusion, it significantly improves the recognition accuracy of complex defects (especially structural wrinkles) with diverse shapes and scales. It solves the core problems of low accuracy, poor generalization ability, and difficulty in real-time deployment faced by surface defect detection in the automated production process of capsule materials.

[0137] The core objective of this application is to design a parallel hybrid model that can take into account both local and global features and perform feature complementarity and intelligent fusion, addressing the fundamental differences in the characteristics of "damage" and "wrinkles" on the surface of capsule materials.

[0138] Specifically, this invention uses parallel lightweight CNN branches and ViT branches to accurately capture the microscopic texture and macroscopic structure of images, respectively. Furthermore, through an innovative gated cross-attention interaction fusion module (GCA-IFM), deep bidirectional interaction between the two branches is achieved: global features guide local features to focus on key regions related to structural defects, while the enhanced local information, in turn, feeds back and refines the global features.

[0139] This architecture ensures that the model relies more heavily on global contextual information when dealing with suspected "wrinkles," while amplifying the contribution of local detailed features when judging "damage." Ultimately, without significantly increasing computational overhead, it significantly improves the accuracy and robustness of identifying various complex defects such as "wrinkles" and "damage," thereby meeting the stringent requirements of industrial automated production for detection accuracy and efficiency.

[0140] Example 2 This embodiment provides a visual defect detection system for capsule materials based on a hybrid neural network. For details not disclosed in this embodiment, please refer to the specific implementation details of the visual defect detection scheme for capsule materials based on a hybrid neural network in other embodiments.

[0141] Figure 5 The diagram shows a structural schematic of a visual defect detection system for capsule materials based on a hybrid neural network, according to an embodiment of this application.

[0142] like Figure 5 As shown, the visual defect detection system for capsule materials based on hybrid neural networks includes: Image acquisition module 10 is used to acquire visual imaging images of the capsule material; The parallel feature extraction module 20 includes parallel local feature extraction branches and global feature extraction branches. It is used to extract local features of the image through a lightweight convolutional neural network in the local feature extraction branch and generate a local feature map. At the same time, it extracts global features of the image through a visual Transformer model in the global feature extraction branch and generates a global feature vector. The feature fusion module 30 adopts a gated cross-attention interactive fusion module, which is used to input local feature maps and global feature vectors into the gated cross-attention interactive fusion module for bidirectional interaction and fusion to generate fused feature vectors. The classification and recognition module 40 is used to input the fused feature vector into the classification head for classification and recognition, and output the defect category of the capsule material.

[0143] The visual defect detection system for capsule materials based on hybrid neural networks according to this application embodiment uses a parallel feature extraction module 20 to simultaneously input the visual imaging image of the capsule material into parallel local feature extraction branches and global feature extraction branches. A lightweight convolutional neural network in the local feature extraction branch extracts local features of the image to generate a local feature map. A visual Transformer model in the global feature extraction branch extracts global features of the image to generate a global feature vector. A feature fusion module 30 inputs the local feature map and the global feature vector into a gated cross-attention interactive fusion module for bidirectional interaction and fusion to generate a fused feature vector. A classification and recognition module 40 inputs the fused feature vector into a classification head for classification and recognition, outputting the defect category of the capsule material.

[0144] This application constructs a parallel hybrid model of CNN and ViT, taking into account both local and global features. Then, through feature complementarity and intelligent fusion, it significantly improves the recognition accuracy of complex defects (especially structural wrinkles) with diverse shapes and scales. It solves the core problems of low accuracy, poor generalization ability, and difficulty in real-time deployment faced by surface defect detection in the automated production process of capsule materials.

[0145] Example 3 This embodiment provides a device for detecting visual defects in capsule materials based on a hybrid neural network. For details not disclosed in this embodiment, please refer to the specific implementation details of the methods or systems for detecting visual defects in capsule materials based on hybrid neural networks in other embodiments.

[0146] Figure 6 The diagram shows a structural schematic of a cyst material visual defect detection device 400 based on a hybrid neural network according to an embodiment of this application.

[0147] like Figure 6 As shown, the capsule material visual defect detection device 400 based on hybrid neural network includes: a storage unit 402 for storing executable instructions; and a processing unit 401 for connecting to the storage unit 402 to execute the executable instructions to complete the capsule material visual defect detection method based on hybrid neural network.

[0148] Those skilled in the art will understand that the illustration Figure 6 This is merely an example of a hybrid neural network-based visual defect detection device 400 for capsule materials and does not constitute a limitation on the device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the hybrid neural network-based visual defect detection device 400 for capsule materials may also include input / output devices, network access devices, buses, etc.

[0149] The processing unit 401 (Central Processing Unit, CPU) can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processing unit 401 can be any conventional processor. The processing unit 401 is the control center of the hybrid neural network-based cyst material visual defect detection device 400, connecting all parts of the device 400 using various interfaces and lines.

[0150] Storage unit 402 can be used to store computer-readable instructions. Processing unit 401 implements various functions of the capsule material visual defect detection device 400 based on hybrid neural networks by running or executing the computer-readable instructions or modules stored in storage unit 402 and calling the data stored in storage unit 402. Storage unit 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the capsule material visual defect detection device 400 based on hybrid neural networks, etc. In addition, storage unit 402 may include hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, read-only memory (ROM), random access memory (RAM), or other non-volatile / volatile storage devices.

[0151] If the integrated module of the cyst material visual defect detection device 400 based on a hybrid neural network is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, they can implement the steps of the various method embodiments described above.

[0152] Example 5 This embodiment provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method for visual defect detection of capsule materials based on hybrid neural networks in other embodiments.

[0153] The device and storage medium for visual defect detection of capsule materials based on hybrid neural networks according to this application include: simultaneously inputting visual images of the capsule material into parallel local feature extraction branches and global feature extraction branches; extracting local features of the image through a lightweight convolutional neural network in the local feature extraction branch to generate a local feature map; extracting global features of the image through a visual Transformer model in the global feature extraction branch to generate a global feature vector; inputting the local feature map and global feature vector into a gated cross-attention interactive fusion module for bidirectional interaction and fusion to generate a fused feature vector; and inputting the fused feature vector into a classification head for classification and recognition to output the defect category of the capsule material.

[0154] This application constructs a parallel hybrid model of CNN and ViT, taking into account both local and global features. Then, through feature complementarity and intelligent fusion, it significantly improves the recognition accuracy of complex defects (especially structural wrinkles) with diverse shapes and scales. It solves the core problems of low accuracy, poor generalization ability, and difficulty in real-time deployment faced by surface defect detection in the automated production process of capsule materials.

[0155] Those skilled in the art will understand that the terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” as used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0156] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0157] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0158] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for visual defect detection of capsule materials based on hybrid neural networks, characterized in that, Includes the following steps: Acquire visual imaging images of the capsule material; The image is simultaneously input into a parallel local feature extraction branch and a global feature extraction branch; Local features of the image are extracted using a lightweight convolutional neural network in the local feature extraction branch, generating a local feature map. Global features of the image are extracted using the visual Transformer model in the global feature extraction branch, and a global feature vector is generated. The local feature map and global feature vector are input into the gated cross-attention interaction fusion module for bidirectional interaction and fusion to generate a fused feature vector. The fused feature vector is input into the classification head for classification and identification, and the defect category of the capsule material is output.

2. The method for visual defect detection of capsule materials based on hybrid neural networks according to claim 1, characterized in that, The step of inputting the local feature map and global feature vector into the gated cross-attention interaction fusion module for bidirectional interaction and fusion to generate a fused feature vector includes: The global-to-local guided enhancement stage: using global features as queries and local features as keys and values, a context vector weighted by global information is calculated through a cross-attention mechanism; Local-to-global feedback modulation stage: The context vector is added to and aggregated with local features to generate a local information summary vector; and a modulation gate is dynamically generated by a gating unit based on the local information summary vector and global features to modulate the global features; Final feature aggregation stage: The modulated global features and local information summary vectors are concatenated and linearly transformed to generate the fused feature vector.

3. The method for visual defect detection of capsule materials based on hybrid neural networks according to claim 2, characterized in that, The context vector calculated through the cross-attention mechanism and weighted by global information includes: Flatten the local feature map in the spatial dimension into a sequence of local feature tokens; The local feature token sequence and global feature vector are mapped to a unified interaction dimension through linear projection; Using the projected global features as the query and the projected local feature sequence as the key and value, cross-attention computation is performed to obtain the context vector.

4. The method for visual defect detection of capsule materials based on hybrid neural networks according to claim 2, characterized in that, The gating unit uses the Sigmoid activation function to dynamically generate a modulation gate with a range of (0,1) based on the local information digest vector and global features, and modulates the global features through element-wise multiplication.

5. The method for visual defect detection of capsule materials based on hybrid neural networks according to claim 1, characterized in that, The local feature extraction branch uses MobileNetV3 as a lightweight convolutional neural network to output a two-dimensional local feature map that preserves spatial structure information.

6. The method for visual defect detection of capsule materials based on hybrid neural networks according to claim 1, characterized in that, The classification head is a multilayer perceptron, which includes one or more fully connected layers. Finally, it outputs the probability distribution corresponding to each defect category through a linear layer with K output units and a Softmax function, where K is the number of defect categories.

7. The method for visual defect detection of capsule materials based on hybrid neural networks according to claim 1, characterized in that, The defect categories include acceptable, broken, and wrinkled.

8. A visual defect detection system for capsule materials based on hybrid neural networks, characterized in that, include: Image acquisition module, used to acquire visual imaging images of capsule material; The parallel feature extraction module includes a parallel local feature extraction branch and a global feature extraction branch. It is used to extract local features of the image through a lightweight convolutional neural network in the local feature extraction branch to generate a local feature map; at the same time, it extracts global features of the image through a visual Transformer model in the global feature extraction branch to generate a global feature vector. The feature fusion module employs a gated cross-attention interactive fusion module, which is used to input the local feature map and global feature vector into the gated cross-attention interactive fusion module for bidirectional interaction and fusion to generate a fused feature vector. The classification and recognition module is used to input the fused feature vector into the classification head for classification and recognition, and output the defect category of the capsule material.

9. A device for visual defect detection of capsule materials based on hybrid neural networks, characterized in that, include: Storage unit, used to store executable instructions; as well as A processing unit is configured to be connected to a memory to execute executable instructions to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon; the computer program is executed by a processor to implement the method as described in any one of claims 1-7.

Images