A small target detection device and method based on a lightweight backbone network GhostoneNet and a feature fusion network EfficientRepGFPN-EEP

Abstract

The application provides a small target detection device and method based on a lightweight backbone network GhostoneNet and a feature fusion network EfficientRepGFPN-EEP, taking YOLOv8 as a benchmark model, including a backbone network side, a feature fusion network side and a detection output side; in the backbone network side, a Mobileone module based on a depth separable convolution structure is introduced; in the feature fusion network side, channel reduction and scale adjustment are performed to match the GhostoneNet; in the detection output side, an OTA distribution strategy is introduced to replace the static label distribution strategy in the original model. The technical scheme can realize high-precision and low-delay defect identification on a low-power embedded platform, effectively improving production efficiency and product yield.

Description

Technical Field

[0001] This invention relates to the field of industrial target detection technology, and in particular to a small target detection device and method based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP. Background Technology

[0002] In current industrial scenarios, embedded platforms generally suffer from limited computing power and scarce memory resources, making it difficult for traditional object detection algorithms to achieve low-cost deployment and efficient operation on low-computing-power devices. In particular, industrial micro-defect detection tasks (such as broken solder joints in electronic components or scratches on backlight panels) face the following core challenges: defect sizes are typically less than 0.5mm and are easily obscured by light fluctuations and metal reflections in industrial environments; traditional methods lack sensitivity to latent defects against regular textured backgrounds; existing feature fusion networks (such as FPN and PANet) often struggle to balance accuracy and efficiency in multi-scale feature fusion due to redundant computation or structural complexity, especially in lightweight models; static label allocation strategies (such as anchor-based methods) are difficult to adapt to the dynamic target distribution in complex industrial scenarios during the training phase, limiting model generalization ability; simultaneously, traditional attention mechanisms (such as SE and CBAM) are difficult to run in real-time on low-power devices due to high global computational costs. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a small target detection device and method based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP, which can achieve high-precision and low-latency defect identification on a low-computing-power embedded platform, effectively improving production efficiency and product yield.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a small target detection device based on a lightweight backbone network GhostoneNet and a feature fusion network EfficientRepGFPN-EEP, using YOLOv8 as the baseline model, including a backbone network side, a feature fusion network side, and a detection output side; on the backbone network side, a depthwise separable convolutional structure-based Mobileone module is introduced, which is composed of a 3×3 lightweight neural network module MobileoneBlock convolutional module containing 3×3 depthwise convolutions and a 1×1 lightweight neural network module Mobileoneblock convolutional module containing 1×1 pointwise convolutions; the backbone network... On the feature fusion network side, the bottleneck layer Bottleneck structure from the lightweight network GhostNetV2 is introduced to improve the original feature extraction module C2fBlock, and combined with structural reparameterization to form a cross-stage ghost fusion module C2f-ghostoneBlock. Then, the lightweight backbone network GhostoneNet is constructed by stacking the cross-stage ghost fusion module C2f-ghostoneBlock and the Mobileone module based on the depthwise separable convolutional structure. On the feature fusion network side, channel reduction and scale adjustment are performed to match the lightweight backbone network GhostoneNet. On the detection output side, the OTA allocation strategy based on optimal transport theory is introduced to replace the static label allocation strategy in the original model.

[0005] In a preferred embodiment, in the lightweight backbone network GhostoneNet, the lightweight neural network module Mobilconeblock convolutional module is first used as the first layer of the network to achieve fast downsampling of images; then, the lightweight backbone network GhostoneNet builds the main body of the network by cross-repeatedly stacking the depth-separable convolutional structure Mobilcone module and the cross-stage ghost fusion module C2f-ghostoneBlock.

[0006] In the lightweight backbone network GhostoneNet, the Mobilcone module, based on a depthwise separable convolutional structure, replaces the ordinary convolutional modules in YOLOv8. The Mobilcone module is based on the depthwise separable convolutional structure found in the MobileNet-V1 network module, and consists of 3×3 depthwise convolutions (DWConv). Convolution employs a 3×3 MobilconeBlock convolutional module and a 1×3 Mobilconeblock convolutional module containing a 1×1 pointwise convolution PWConv. During training, the 3×3 MobilconeBlock module introduces reparameterizable skip connections with only batch normalized (BN) layers and a feature extraction branch containing a 1×1 convolutional layer, building upon the 3×3 depthwise convolution. The 1×3 lightweight neural network module, Mobilconeblock, introduces skip connections with only BN layers on top of the 1×1 pointwise convolution. During inference, multiple branches of each lightweight MobilconeBlock module are merged into a single convolutional branch. The lightweight MobilconeBlock module introduces hyperparameters to adjust the number of branches in the convolutional branches, with values ​​ranging from 1 to 5. When the hyperparameter value is 2 or higher, the lightweight MobilconeBlock module will have multiple parallel and repeating convolutional branches.

[0007] In a preferred embodiment, the cross-stage ghost fusion module C2f-ghostoneBlock replaces the feature extraction module, i.e., the cross-stage partial fusion module C2fBlock, in YOLOv8. The cross-stage partial fusion module C2fBlock consists of a feature extraction stage and a cross-stage connection stage. The feature extraction stage contains n YOLOv8 Bottleneck modules, i.e., YOLOv8 bottleneck residual modules. The cross-stage ghost fusion module C2f-ghostoneBlock is an improvement on the cross-stage partial fusion module C2fBlock, replacing the YOLOv8 Bottleneck with the ghost single-branch bottleneck module GhostoneBottlenec. k, combined with the decoupled fully connected DFC attention mechanism, reduces the number of model parameters; the GhostoneBottleneck module is based on the residual structure of the lightweight network GhostNetV2, and integrates structural reparameterization to optimize feature extraction capabilities, divided into two stages: feature extraction and fusion; the feature extraction stage adopts the improved GhostoneModule, replacing the 1×1 and 5×5 ordinary convolutional units with the 1×1 and 3×3 lightweight neural network modules MobileoneBlock, and adjusting the channel dimension, implementing depthwise separable convolution and concatenating the output; the feature fusion stage adds the input features and high-level semantic features through skip connections.

[0008] In a preferred embodiment, a decoupled fully connected DFC attention mechanism is used to provide feature extraction capability for the GhostoneBottleneck module, a bottleneck module with a single branch; a fully connected layer FC with fixed weights is used to generate an attention map with a global receptive field; the decoupled fully connected DFC attention mechanism decouples the fully connected layer along the horizontal and vertical directions, and uses these two types of fully connected layers to aggregate long-distance spatial information in these two directions; the calculation formulas of these two types of fully connected layers are shown in Equation (1) and Equation (2) respectively, which are implemented directly through convolution by sharing some weights;

[0009] (1)

[0010] (2)

[0011] Where ⊙ represents element-wise multiplication; The value of the intermediate output feature of the horizontal decoupling layer at position (h, w); The horizontal weight parameter is connected to the input height. With output height h; This represents the input feature map at position h'×w; This represents the numerical value of the output feature map at spatial location (h, w). The weight parameter represents the vertical direction, and the width of the intermediate output is connected. The final output width w; H represents the feature map height; W represents the feature map width; This represents the value of the intermediate output of the horizontal decoupling layer at position h,w'.

[0012] The image information processing process of the decoupled fully connected DFC attention mechanism is as follows: a focused image feature block of the input image will be connected to the horizontal and vertical feature blocks simultaneously through two fully connected layers. At the same time, the image feature block will also be connected to other feature blocks in both directions. Thus, it can be seen that the focused image feature block will be connected to all feature blocks in the world in a direct or indirect manner.

[0013] In a preferred embodiment, a sub-millimeter-level mesh refinement is implemented in densely populated regions of small target detection points using an efficient extended dot product attention mechanism (EEP). The specific implementation steps are as follows:

[0014] (1) Segmentation: When the input feature map is input into the lower branch, the lower branch uses a sliding window of size K×K to segment the input feature map into K×K regional feature maps. At this time, the size of each regional feature map is C×K×K.

[0015] (2) Expansion and attention operation: When the input feature map is input to the upper branch, the 1×1 convolution will expand its channels to K. 2 Ultimately, the attention filter will be generated through the compression excitation module SE; the coordinate attention mechanism will be applied to the intermediate of the efficient extended dot product attention mechanism EEP.

[0016] (3) Dot product: Perform Hadamard product operation on the outputs of the upper branch and the lower branch, and broadcast it; the dot product operation is equivalent to orderly expanding the filter in the channel dimension, and then multiplying the same number and size of attention filters with the elements at the same position in the feature map to be weighted; the dot product operation between the attention filter output of the upper branch and the segmented feature map output of the lower branch is equivalent to first orderly expanding the attention filter in the channel dimension and dividing the segmented feature map into C groups according to the channel order, and then multiplying the attention filter with the segmented feature map of each group, that is, multiplying the elements at the same position of the two, and finally outputting the weighted feature map, thereby achieving the filtering of feature maps representing the features of each region by the attention filter through the dot product operation;

[0017] (4) Feature map addition: The size generated by the dot product operation in the previous step is C×K 2The feature maps ×H×W are added together for normalization, and then added to the input feature map to obtain an output feature map with the same shape as the input feature map. Here, K represents the size of the sliding window used to segment the feature map, C represents the number of channels, H represents the spatial dimension of the feature map in the vertical direction, and W represents the spatial dimension of the feature map in the horizontal direction.

[0018] In a preferred embodiment, the feature fusion network Efficient-RepGFPN-EEP introduces a lightweight convolutional module GhostConv to generate more feature maps from existing feature maps using linear transformations. When the input feature map is input to GhostConv, it first undergoes channel transformation through regular convolution to generate inherent feature maps, and then obtains more feature maps. Finally, the obtained feature maps are concatenated with the feature maps obtained through regular convolution in the first step to obtain the final feature map output.

[0019] In a preferred embodiment, a weighted feature fusion method, Weighted Concat, is introduced to perform discriminative fusion of feature maps containing defect information at different scales; as shown in Equation (3), Fastnormalized fusion shrinks the value range of each normalization weight to between [0, 1].

[0020] (3)

[0021] Where O represents the output feature after fusion, which is the final result obtained through weighted fusion; I i W represents the i-th input feature. i Represents the learnable weights, ε represents a small constant to avoid numerical instability, and e is an exponential function. Representative of all ownership weight W j The exponent and summation, each The ReLU function will be applied to ensure that it is greater than or equal to 0; ε is equal to 0.0001.

[0022] This invention also provides a small target detection method based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP, which operates based on the aforementioned small target detection device based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP.

[0023] Compared with existing technologies, this invention has the following advantages: It not only innovates in the structure of target detection algorithms but also closely integrates with the needs of small defect detection in industrial quality inspection, demonstrating high adaptability in typical manufacturing fields such as steel, electronics, and ceramics. Through experimental verification on low-computing-power embedded devices, this model achieves a good balance between accuracy, speed, and deployment cost, providing a practical technical path and engineering support for promoting intelligent industrial inspection. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the lightweight target detection algorithm structure of a preferred embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the GhostoneNet backbone network structure according to a preferred embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the Mobileone network module according to a preferred embodiment of the present invention;

[0027] Figure 4 This is a comparative schematic diagram of the feature extraction modules in a preferred embodiment of the present invention, wherein (a) is the cross-stage partial fusion module C2f Block, and (b) is the cross-stage ghost fusion module C2fghostone Block;

[0028] Figure 5 This is a comparative schematic diagram of the Bottleneck residual module in a preferred embodiment of the present invention, wherein (a) is the lightweight network GhostNetV2Bottleneck, and (b) is the ghost single-branch bottleneck module GhostoneBottleneck.

[0029] Figure 6 This is a schematic diagram of the image information processing process of the DFC attention module in a preferred embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram of the EEP attention mechanism of a preferred embodiment of the present invention;

[0031] Figure 8 This is a schematic diagram of the dot product process according to a preferred embodiment of the present invention;

[0032] Figure 9 This is a schematic diagram of the overall structure of the novel feature fusion network according to a preferred embodiment of the present invention;

[0033] Figure 10 This is a schematic diagram of the overall workflow of the feature fusion network according to a preferred embodiment of the present invention;

[0034] Figure 11This is a schematic diagram of the model visualization results for backlight panel defect detection according to a preferred embodiment of the present invention, wherein (a) is the label and (b) is the optimized model No. 4;

[0035] Figure 12 This is a schematic diagram of an industrial defect detection system based on the Jetson embedded platform according to a preferred embodiment of the present invention;

[0036] Figure 13 This is a schematic diagram of the structure of an industrial defect detection system based on the NVIDIA Jetson AGX Orin embedded platform, which is a preferred embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0040] refer to Figure 1-13 This invention addresses the detection challenges, edge hardware resource configuration, and operational environment constraints in the aforementioned practical industrial scenarios by proposing a novel lightweight target detection algorithm—GH-YOLO-Efficient—to enable low-cost deployment and efficient operation of target detection algorithms on existing low-computing-power embedded platforms (such as Jetson and RK series chips). GH-YOLO-Efficient is an algorithm improvement based on YOLOv8, and its algorithm structure is as follows: Figure 1As shown. On the backbone network side, GH-YOLO-Efficient introduces a reparameterization module based on a depthwise separable convolutional structure—the Mobileone module based on a depthwise separable convolutional structure. This module consists of a 3×3 lightweight neural network module MobileoneBlock convolutional module containing 3×3 depthwise convolutions and a 1×1 lightweight neural network module Mobileoneblock convolutional module containing 1×1 pointwise convolutions. In addition, on the backbone network side, the bottleneck layer Bottleneck structure from the lightweight network GhostNetV2 is introduced to improve the original feature extraction module, the cross-stage partial fusion module C2f Block (corresponding to the CSP module in YOLOv8). Combined with structural reparameterization technology, a new feature extraction module—the cross-stage ghost fusion module C2f-ghostoneBlock—is proposed. Then, by stacking the cross-stage ghost fusion module C2f-ghostoneBlock and the depthwise separable convolutional structure-based Mobileone module, a novel lightweight backbone network—GhostoneNet—is constructed. On the feature fusion network side, the proposed novel feature fusion network, Efficient-RepGFPN-EEP, undergoes channel reduction and size adjustment to match the larger-scale lightweight backbone network, GhostoneNet. On the detection output side, an optimal transport theory-based assignment strategy, OTA (Optimal Transport Assignment), is introduced to replace the static label assignment strategy in the original algorithm, thereby improving the model's training performance.

[0041] I. GhostoneNet New Backbone Network

[0042] This design uses the YOLOv8 backbone network as its architecture and proposes a novel lightweight backbone network—GhostoneNet—based on a new reparameterized feature extraction module and a cross-stage ghost fusion module, C2f-ghostoneBlock. The network structure of the lightweight backbone network GhostoneNet is as follows: Figure 2As shown, the network first uses the lightweight neural network module Mobilconeblock convolutional module as the first layer to achieve fast image downsampling. Then, the lightweight backbone network GhostoneNet constructs the main network by repeatedly stacking the depthwise separable convolutional structure Mobilcone module and the novel cross-stage ghost fusion module C2f-ghostoneBlock feature extraction module proposed in this application. Furthermore, this application inserts a Spatial Pyramid Pooling (SPP) module into the penultimate layer of the lightweight backbone network GhostoneNet to improve the model's generalization ability and detection performance. This module can perform multi-scale pooling operations on feature maps to fully capture spatial information at different scales. The specific structure and parameter configuration of the GhostoneNet backbone network are shown in Table 1.

[0043] Table 1 GhostoneNet Backbone Network Parameter Configuration Table

[0044]

[0045] To meet the real-time inspection requirements of embedded intelligent vision systems in industrial production, the GhostoneNet backbone network is optimized for lightweighting based on YOLOv8. This not only significantly reduces the computational load and memory consumption of the model on embedded processors, but also demonstrates excellent real-time response capabilities and robustness in the detection of minute defects on the surface of workpieces during actual high-speed transmission.

[0046] The GhostoneNet backbone network follows a lightweight improvement method based on the YOLOv8 benchmark model. It is also built upon the YOLOv8 backbone network structure, with the following main improvements compared to YOLOv8:

[0047] (1) Replace the ordinary convolutional modules in YOLOv8 with the Mobilcone module. For example... Figure 3As shown, the Mobilcone module is designed based on the depthwise separable convolution structure in the MobileNet-V1 network module. It consists of a 3×3 MobilconeBlock convolution module containing 3×3 depthwise convolution (DWConv) and a 1×3 Mobilconeblock convolution module containing 1×1 pointwise convolution (PWConv). The Mobilconeblock convolutional module differs from traditional convolutional modules by introducing structural reparameterization. During training, the 3×3 MobilconeBlock convolutional module adds reparameterizable skip connections with only batch normalized (BN) layers and a feature extraction branch containing a 1×1 convolutional layer, based on 3×3 depthwise convolutions. The 1×3 Mobilconeblock convolutional module adds skip connections with only BN layers to 1×1 pointwise convolutions. During inference, the multiple branches of each MobilconeBlock module are merged into a single convolutional branch. This allows the model to achieve high training accuracy with a multi-branch structure during training, and high inference efficiency with a streamlined single-branch structure while maintaining high accuracy. This design significantly reduces the computational complexity and inference latency of the model on edge devices, making it crucial for real-time detection in industrial settings. In addition, the MobilconeBlock convolution module introduces hyperparameters to adjust the number of branches in the convolution (the hyperparameter can be set from 1 to 5). When the hyperparameter is 2 or higher, there will be multiple parallel and repeated convolution branches in the middle of the module, thereby achieving higher training accuracy. During inference, the structure will be uniformly integrated into the single-branch structure through structural reparameterization technology, thereby achieving higher detection accuracy with the same inference efficiency. This is especially important for identifying tiny defects that are easily drowned out by background noise (such as fine cracks on ceramic substrates).

[0048] (2) A novel reparameter-based feature extraction module, C2f-ghostoneBlock, is proposed to replace the feature extraction module C2f Block (CSP feature extraction module) in YOLOv8. For example... Figure 4 As shown in (a), the C2f Block is the core network module in the YOLOv8 backbone network. It consists of a feature extraction stage and a cross-stage connection stage. The feature extraction stage mainly consists of n residual modules YOLOv8Bottleneck, which are mainly responsible for extracting features with high-level semantic information from the input feature map. The cross-stage connection stage is used to fuse the output feature map of the feature extraction stage with the module input feature map to enhance the network's information transmission and feature expression capabilities; Figure 4As shown in (b), C2f-ghostoneBlock is an improvement on the C2f Block feature extraction module in YOLOv8. It is mainly improved by replacing the YOLOv8Bottleneck module in the C2f Block feature extraction stage with a more lightweight and reparameterizable residual module—GhostoneBottleneck. Since the feature extraction module is the main part of the backbone network and accounts for most of the network's computation and parameters, the lightweight improvement in this application significantly reduces the overall network's model parameters, computational complexity, and memory usage, making it more suitable for deployment on resource-constrained embedded platforms. Furthermore, due to the introduction of structural parameterization technology and a more scientific network structure in the novel feature extraction module proposed in this application, it can effectively capture key defect features such as broken electronic solder joints and tiny metal scratches while achieving lightweight design, maintaining a high detection accuracy.

[0049] GhostoneBottleneck is based on the GhostNetV2Bottleneck residual module in the GhostNetV2 backbone network, and is improved through fusion structure reparameterization technology. For example... Figure 5 As shown in (a), the GhostNetV2 Bottleneck module has a similar structure to the YOLOv8 Bottleneck module, which can be divided into a feature extraction stage and a feature fusion stage. The feature extraction stage mainly consists of the GhostModule module and the DFC attention module, used to extract high-level semantic information from the input features. The feature fusion stage fuses the high-level semantic features output from the feature extraction stage with the module input features through skip connections and feature map addition operations, thereby effectively alleviating the problems of gradient vanishing and exploding in the network and accelerating the network convergence speed. Figure 5As shown in (b), the GhostoneBottleneck module has a basically the same structure as the GhostNetV2Bottleneck module. It is improved by introducing structural reparameterization technology into the main module GhostoneModule of the feature extraction stage. Specifically, the 1×1 ordinary convolution and 5×5 ordinary convolution in GhostoneModule are replaced by 1×1 MobilconeBlock convolution modules and 3×3 MobilconeBlock convolution modules, respectively. The feature extraction operation of GhostoneModule can be divided into three stages: the first stage performs channel adjustment through the 1×1 MobilconeBlock convolution module; the second stage performs channel-wise convolution operation through the 3×3 MobilconeBlock convolution module. In the 3×3 MobilconeBlock convolution module, one convolution kernel of the 3×3 depthwise convolution is only responsible for one channel of the input feature map, which can effectively reduce the computational cost and parameter count of the model compared with ordinary convolution; the third stage fuses the output feature maps of the first and second stages through feature map concatenation operation. GhostoneBottleneck maintains the superior lightweight module structure of GhostNetV2Bottleneck, and further improves the feature extraction capability of the modules while maintaining the lightweight nature of the modules by introducing structural reparameterization technology, thereby effectively improving the inference efficiency of the overall network on embedded devices.

[0050] Furthermore, to improve the inference speed of the model on embedded platforms, Ghostone Bottleneck adopts the Decoupled Fully Connected (DFC) attention mechanism, specifically designed for lightweight small models in GhostNetV2 Bottleneck. This mechanism achieves full capture of long-range spatial information with an efficient modular structure, thereby effectively improving Ghostone Bottleneck's feature extraction capabilities with relatively low computational cost. The DFC attention mechanism differs from traditional, inefficient self-attention mechanisms by using fully connected layers (FC) with fixed weights to efficiently generate attention maps with a global receptive field. While fully connected layers can effectively capture the global receptive field of the input feature map, the direct connection between input and output would impose an unbearable computational burden on ordinary fully connected layers. Therefore, the DFC attention mechanism decouples the fully connected layers along both the horizontal and vertical directions, using these two types of fully connected layers respectively to aggregate long-range spatial information in these two directions, thereby significantly reducing computational cost and parameter count. The calculation formulas for these two fully connected layers are shown in Equation (1) and Equation (2) respectively. By sharing some weights, they can be directly implemented through convolution, thus saving time-consuming operations such as matrix reshaping and transpose.

[0051] (1)

[0052] (2)

[0053] Where ⊙ represents element-wise multiplication; The value of the intermediate output feature of the horizontal decoupling layer at position (h, w); The horizontal weight parameter is connected to the input height. With output height h; This represents the input feature map at position h'×w; This represents the numerical value of the output feature map at spatial location (h, w). The weight parameter represents the vertical direction, and the width of the intermediate output is connected. The final output width w; H represents the feature map height; W represents the feature map width; This represents the value of the intermediate output of the horizontal decoupling layer at position h,w'.

[0054] The image information processing process of the DFC attention mechanism is as follows: Figure 6As shown, a focused image feature block of the input image will be connected to both horizontal and vertical feature blocks simultaneously through two fully connected layers. At the same time, the image feature block will also be connected to other feature blocks in both directions. Thus, it can be seen that the focused image feature block will be connected to all feature blocks in the world directly or indirectly. Therefore, the DFC attention mechanism can model global spatial information by decoupling the fully connected layers, which is an efficient operation. This has a significant advantage in locating small, isolated defects (such as scratches and missing solder joints) in a background with large areas of regular texture (such as metal plates and integrated circuits).

[0055] II. EEP Attention

[0056] In the field of industrial product surface defect detection, the accurate location of minute defects (such as broken solder joints in electronic components and scratches on metal parts) faces a core challenge: the defect size is usually less than 0.5 mm and is mostly concentrated in specific functional areas (such as chip pin gaps or bearing contact surfaces). At the same time, it is easily obscured by the processing texture due to interference from industrial ambient light fluctuations and metal reflections. Traditional attention mechanisms adopt a global feature fusion strategy, which is easily dominated by the regular structural features of the product during the calculation process, resulting in insufficient sensitivity to micron-level defects at the edges of functional areas.

[0057] The efficient extended dot product (EEP) attention mechanism proposed in this application addresses the characteristics of the aforementioned industrial inspection scenarios by dynamically dividing the input feature map into function-oriented perceptual units (such as...) through a multi-granularity grid segmentation strategy. Figure 7 (As shown). In electronic component inspection tasks, this mechanism implements sub-millimeter-level mesh refinement for densely packed solder joint areas. Within each cell, the three elements of industrial defect analysis are performed in parallel: channel attention filters metal melting features related to welding anomalies, spatial attention enhances the microscopic deformation response of pin gaps, and finally, cross-cell feature comparison identifies consistency deviations in batch production. For metal part inspection, the algorithm automatically identifies key areas such as bearing contact surfaces and thread grooves, constructs reflection suppression modules within local cells, and eliminates specular reflection interference from metal surfaces through multi-exposure feature fusion. This mechanism enables the model to significantly improve its ability to capture latent defects against regular texture backgrounds while maintaining industrial-grade real-time computational efficiency, providing reliable technical support for intelligent manufacturing quality control.

[0058] The EEP attention mechanism first segments the input feature map into multiple regional feature maps with different spatial locations. Then, it assigns different weights to each regional feature map, thereby filtering out important target features in different spatial locations. Finally, it outputs the result through a feature map addition operation. Figure 7 For example, the input feature map to the EEP attention mechanism module is of size C×W×H, and the parameters in the figure are set as follows: =3, H=3, C=3, K=3, G=1, where K represents the size of the sliding window used to segment the feature map, and G represents the number of groups, i.e., the number of slice groups of the input feature map in the channel dimension. This mechanism performs the same attention operation on each group of segmented feature maps. The specific implementation steps of this attention mechanism are as follows:

[0059] (1) Segmentation: When the input feature map is input into the lower branch, the lower branch uses a sliding window of size K×K to segment the input feature map into K×K regional feature maps. At this time, the size of each regional feature map is C×K×K.

[0060] (2) Expand and Attention Operation: When the input feature map is input to the upper branch, the 1×1 convolution will expand its channels to K. 2 Finally, an attention filter will be generated through a squeeze-and-excitation (SE) module. This application applies a coordinate attention mechanism to the middle of the EEP attention mechanism, realizing attention operations on regional feature maps representing different spatial locations with low computational overhead, thereby enabling the model to effectively capture defect feature information clustered in local regions.

[0061] (3) Dot Product: Perform a Hadamard product operation on the outputs of the upper and lower branches, and broadcast C times. The dot product operation can be equivalent to performing an ordered expansion of the channel-dimensional filters, and then multiplying the same number and size of attention filters with the elements at the same position in the feature map to be weighted. For example Figure 8 As shown, the dot product operation between the attention filter output from the upper branch and the segmented feature maps output from the lower branch can be equivalent to first unfolding the attention filter in an ordered manner along the channel dimension and dividing the segmented feature maps into C groups according to the channel order, then multiplying the attention filter with the segmented feature maps of each group, that is, multiplying the elements at the same position of the two, and finally outputting the weighted feature maps. Thus, the dot product operation realizes the selection of feature maps representing the features of each region by the attention filter.

[0062] (4) Feature map addition: The size generated by the dot product operation in the previous step is C×K. 2The ×H×W feature maps are added together for normalization, and then added back to the input feature map to obtain an output feature map with the same shape as the input feature map. This local focusing strategy is particularly suitable for processing industrial images where the defect locations are relatively fixed or concentrated in specific areas (such as bearing raceways or solder pads).

[0063] The computational and parameter capacity of the EEP attention mechanism is limited by hyperparameters, and the model size can be adjusted according to the actual embedded hardware resources deployed. The scale of the mechanism's parameters and FLOPs (floating-point operations) mainly depends on the hyperparameters G (number of groups) and K (sliding window size). In network design, by adjusting G, K, and the insertion position of the attention module, the size of the attention module can be effectively changed, thereby better balancing the trade-off between accuracy improvement and computational burden. This attention mechanism can achieve comprehensive and effective extraction of local image features while balancing accuracy, parameter capacity, and computational burden. Therefore, compared with most other attention mechanisms, it is more suitable for lightweight industrial defect detection models deployed on resource-constrained edge devices.

[0064] III. Novel Feature Fusion Network – Efficient-RepGFPN-EEP

[0065] In industrial micro-defect target detection tasks, feature fusion networks are widely used to improve the performance of target detection models. They are typically used as the bottleneck stage of the model, fusing feature maps from different levels to obtain rich semantic and localization information. This invention proposes a more efficient feature fusion network—Efficient-RepGFPN-EEP—to serve as the bottleneck stage of the detection model, meeting the need for efficient feature fusion networks in lightweight target detection networks in industrial scenarios. The overall structure of the network is as follows: Figure 9 As shown. Advantages of Efficient-RepGFPN-EEP:

[0066] (1) The lightweight convolutional module GhostConv is introduced to replace the original ordinary convolutional module, thereby achieving efficient extraction of image features at different resolutions and reducing computational overhead on edge devices. The lightweight convolutional module GhostConv introduced in this invention generates more feature maps from existing feature maps by using linear transformations, maximizing the use of available computational and memory resources. Without increasing the number of model parameters, GhostConv can effectively reduce the amount of computation while maintaining the model's feature extraction capability. By replacing ordinary convolution with GhostConv, the model's performance in detecting small defects (such as fine cracks) can be maintained while significantly reducing the computational burden on embedded platforms, improving the model's efficiency and adaptability.

[0067] like Figure 10 As shown, when the input feature map is input into the GhostConv convolution, it first undergoes channel transformation through regular convolution to generate inherent feature maps. Then, a series of simple linear operations (simultaneous unit mapping and linear transformation) are used to obtain more feature maps. Finally, the different feature maps are concatenated with the feature map obtained through regular convolution in the first step, and the final feature map is the output. In object detection models, the input image undergoes a process of gradually transferring spatial information to channels in the backbone network, accompanied by spatial compression and channel expansion of the feature maps, which inevitably leads to the partial loss of semantic information. To address this, the GhostConv convolution tries to preserve the hidden connections between each channel as much as possible to reduce the loss of semantic information. However, if GhostConv convolution is used in all stages of the object detection model, the network layers will be deepened, significantly increasing the inference time. Therefore, this invention only uses GhostConv convolution in the bottleneck stage. This is because the input feature map in the bottleneck stage has less redundant information and does not require further spatial compression. This selective use ensures lightweight design while maximizing the preservation of key defect information.

[0068] (3) A weighted feature fusion method, Fastnormalized fusion, is introduced to enable the network to learn the importance of different input features more efficiently and to perform discriminative fusion of feature maps containing defect information at different scales. As shown in Equation (3), similar to the Softmax operation, Fastnormalized fusion will shrink the range of each normalized weight to [0, 1], but it has a faster training speed and ultimately improves the fusion effect of the model in complex industrial backgrounds.

[0069] (3)

[0070] Where O represents the output feature after fusion, which is the final result obtained through weighted fusion; I i W represents the i-th input feature. i Represents the learnable weights, ε represents a small constant to avoid numerical instability, and e is an exponential function. Representative of all ownership weight W j The exponent and summation, each The ReLU function will be applied to ensure that it is greater than or equal to 0; ε is equal to 0.0001.

[0071] (3) Insert a self-designed lightweight attention mechanism—EEP attention mechanism—near the input of the feature fusion network to achieve efficient fusion of channel information and spatial information of the input feature map, thereby effectively compensating for the accuracy loss caused by the lightweight design of the network, especially when dealing with surface defects of materials such as metal and ceramic that are susceptible to light and background interference.

[0072] Furthermore, the Efficient-RepGFPN-EEP feature fusion network retains the basic structure of the original Efficient-RepGFPN fusion block. This module introduces structural reparameterization technology and efficient layer aggregation networks (ELAN) to effectively improve model accuracy with a lighter computational burden. The core of the Fusion Block is the CSPStage Block, an improvement on the CSPNet (Cross Stage Partial Network) network module. It compresses the input feature map from two 1×1 convolutional features into two parts through partial connections and outputs them to two branches respectively. This design effectively reduces the computational complexity of the module. One branch of the feature map input is a skip connection branch, which is directly concatenated with the output of the other branch. The other branch is the feature extraction branch, which is mainly composed of N 3×3 convolutions and residual combinations of the Rep-Block structural reparameterization module. It achieves full interaction of input features between different stages through skip connections between different stages, thereby improving feature reusability. This will help improve the feature extraction capability of the object detection model. Furthermore, this structural design fully considers the flow path of features in the network, not only avoiding gradient vanishing or exploding but also improving the model's generalization ability, resulting in good performance on both general datasets and various defect datasets. The Rep-Block module embedded in the feature extraction branch combines structural reparameterization techniques. During the training phase, it achieves higher training accuracy through a dual-branch approach, while during the inference phase, it merges the two branches into one through reparameterization. This maintains high detection accuracy while improving the model's inference efficiency, ultimately enhancing the balance between detection performance and inference efficiency on embedded devices.

[0073] In summary, this invention not only innovates in the structure of target detection algorithms but also closely integrates with the needs of small defect detection in industrial quality inspection, demonstrating high adaptability in typical manufacturing fields such as steel, electronics, and ceramics. Experimental verification on low-computing-power embedded devices shows that this model achieves a good balance between accuracy, speed, and deployment cost, providing a practical technical path and engineering support for promoting intelligent industrial inspection.

[0074] IV. Algorithm Deployment and System Design

[0075] In practical industrial inspection scenarios, considering factors such as real-time performance and network bandwidth, most industrial defect detection applications require the target detection model to be directly deployed on embedded, mobile, and edge devices to achieve local offline detection and edge processing. Therefore, this invention selects the NVIDIA Jetson AGX Orin embedded platform for algorithm optimization and deployment to achieve edge detection of industrial defects, and evaluates the detection performance of the deployment platform. A corresponding industrial defect detection system is also designed.

[0076] During the algorithm optimization and deployment phase, this invention will generate multiple different optimized models by selecting different deep learning inference frameworks and adjusting parameter settings. As shown in Table 2, these models can be divided into four types, numbered 1-4, representing four optimized models generated by GH-YOLO-Lite under different frameworks and parameter settings. These are the dynamic graph and static graph models output under the ONNX framework, and the models generated under the TensorRT framework with weight precisions of single-precision floating-point number FP32 and half-precision floating-point number FP16, respectively.

[0077] As shown in Table 3, in the backlight panel defect detection task, models 3 and 4 optimized using the TensorRT framework significantly improved detection speed compared to models 1 and 2 optimized using the Open Neural Network Exchange (ONNX) framework, while maintaining the same level of accuracy. Specifically, among the optimized models using the TensorRT framework, model 4 with a weight precision of FP16 improved detection speed by 20% compared to model 3 with a weight precision of FP32, while maintaining essentially the same accuracy. The optimized model 4 using the GH-YOLO-Efficient algorithm achieved the highest detection speed, reaching 51.5 FPS.

[0078] Table 2

[0079]

[0080] Table 3

[0081]

[0082] Visualization results of the backlight panel defect detection model are as follows: Figure 11As shown, the detection task is quite challenging due to the small size of the backlight panel defects and their relatively low proportion of the image area. Comparing the original labels on the dataset, the models can accurately locate all target positions in this detection task. Among them, the optimized model 4 exhibits superior robustness and a relatively lower false negative rate.

[0083] After optimizing and deploying a lightweight object detection algorithm on the NVIDIA Jetson AGX Orin embedded platform, this application proposes a design scheme for an industrial defect detection system based on the NVIDIA Jetson AGX Orin embedded platform. The overall system structure is as follows: Figure 12 As shown, it can be divided into two parts: a defect acquisition unit and a defect processing unit.

[0084] The defect acquisition unit integrates a light source system and an industrial camera, jointly undertaking the crucial task of efficiently and accurately acquiring images of product surface defects. The design and configuration of the light source system are core elements ensuring the generation of high-quality, high-contrast images. It eliminates the effects of shadows and glare by carefully controlling lighting conditions, achieving sufficient illumination of the target area and effectively highlighting the structure and characteristics of product surface defects. Simultaneously, the industrial camera, with its superior image capture capabilities, records the product surface condition in real time after optimized illumination by the light source system and transmits the captured defect images to the NVIDIA Jetson AGX Orin embedded platform for further processing. The collaborative operation between the two provides a robust and reliable data input source for subsequent image processing.

[0085] The defect detection unit, primarily composed of the NVIDIA Jetson AGX Orin embedded platform, is tasked with preprocessing images output from an industrial camera and predicting defect targets. First, the defect detection unit preprocesses the input image using a single-scale adjustment strategy combined with an adaptive image scaling algorithm to adjust the input image to the required detection scale. During preprocessing, when the industrial camera continuously acquires multiple defect images, the defect detection unit can process multiple images simultaneously, uniformly scaling them before inputting them into the detection model in parallel. Subsequently, the defect detection unit inputs the preprocessed image into the detection model for further processing. During processing, the convolutional network on the backbone network side extracts features from the input image, converting it into high-dimensional abstract semantic information, and outputs three-scale feature maps to the feature fusion network side for further processing. Then, the feature pyramid structure on the feature fusion side fuses the input feature maps, also outputting three-scale feature maps. Finally, the detection model's output side performs target prediction on each feature map, predicting and outputting the confidence information, category information, and location (regression) information of the defect target. The defect detection unit acquires the target information output by the detection model, uses this information to draw a detection box on the original input image, and feeds the detection results and target information back to the visualization terminal for staff to view.

[0086] As a crucial component of the inspection system, the visualization terminal consists of a main display and input devices such as a keyboard. It connects directly to an industrial control computer or embedded platform, providing operators with an interactive interface for real-time observation of inspection results and comprehensive control of the inspection system. Specifically, the visualization terminal offers an intuitive interface, enabling operators to easily view defect detection results and promptly identify and analyze potential problems. Furthermore, the visualization terminal is responsible for summarizing and displaying inspection information acquired from various defect acquisition units, facilitating comprehensive data analysis and decision-making. Simultaneously, the visualization terminal also features the ability to dynamically retrieve and replace model weights in the image processing unit, enabling flexible application and optimization updates of the system in different scenarios. Through these functions, the visualization terminal achieves effective monitoring and management of the entire system, providing operators with a convenient and efficient operating experience.

[0087] Based on the above solution, this invention utilizes existing backlight panel defect detection platforms to build an industrial defect detection system based on the NVIDIA Jetson AGX Orin embedded platform. For example... Figure 13As shown, the system is also divided into two parts: a defect acquisition unit and a defect processing unit. The defect acquisition unit consists of a backlight panel defect detection platform, which includes a conveyor belt for placing the backlight panel and an industrial camera for capturing product images. In actual operation, the backlight panel is first placed at position 1 of the conveyor belt, then transported to position 2 for dust removal, and then to position 3 for image acquisition. During the acquisition process, the backlight panel is placed in an opaque, dark area, and images are captured by illuminating the backlight panel. Finally, the backlight panel is moved to position 4 for unloading. The defect processing unit mainly consists of an NVIDIA Jetson AGX Orin embedded platform, which is responsible for processing and inspecting the product images captured by the industrial camera.

Claims

1. A small target detection device based on a lightweight backbone network GhostoneNet and a feature fusion network EfficientRepGFPN-EEP, characterized in that, YOLOv8 as the benchmark model, including the backbone network side, the feature fusion network side and the detection output side; in the backbone network side, the Mobileone module based on the depth separable convolution structure is introduced, which is composed of the 3*3 lightweight neural network module MobileoneBlock convolution module containing 3*3 depth convolution and the 1*1 lightweight neural network module Mobileoneblock convolution module containing 1*1 point convolution; the bottleneck structure Bottleneck in the lightweight network GhostNetV2 is introduced to improve the original feature extraction module C2fBlock in the backbone network side, and the cross-stage ghost fusion module C2f-ghostoneBlock is formed by combining the structure reparameterization, then the lightweight backbone network GhostoneNet is constructed by stacking the cross-stage ghost fusion module C2f-ghostoneBlock and the Mobileone module based on the depth separable convolution structure; in the feature fusion network side, the channel reduction and scale adjustment are carried out to match the lightweight backbone network GhostoneNet; in the detection output side, the distribution strategy OTA distribution strategy based on the optimal transport theory is introduced to replace the static label distribution strategy in the original model.

2. The small target detection device based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP according to claim 1, characterized in that, In the lightweight backbone network GhostoneNet, first, the lightweight neural network module Mobilconeblock convolution module is used as the first layer of the network to realize fast down-sampling of the image; then the lightweight backbone network GhostoneNet builds the network main body by cross-repeating stacking the Mobilcone module based on the depth separable convolution structure and the cross-stage ghost fusion module C2f-ghostoneBlock; In the lightweight backbone network GhostoneNet, the Mobilcone module based on the depth separable convolution structure is used to replace the ordinary convolution module in YOLOv8, which is based on the depth separable convolution structure in the MobileNet-V1 network module, and is divided into the 3*3 MobilconeBlock convolution module containing the 3*3 depth convolution DWConv and the 1*3 Mobilconeblock convolution module containing the 1*1 point convolution PWConv; during training, the 3*3 MobilconeBlock convolution module introduces the reparameterizable skip connection with only the batch normalization BN layer and the feature extraction branch containing the 1*1 convolution layer based on the 3*3 depth convolution, and the 1*3 lightweight neural network module Mobilconeblock convolution module introduces the skip connection with only the batch normalization BN layer based on the 1*1 point convolution; during inference, the branches of each lightweight neural network module MobilconeBlock are unified into a single convolution branch; The light-weight neural network module MobilconeBlock introduces a hyperparameter to adjust the number of branches of the convolution branch, the hyperparameter takes a value of 1 to 5, when the hyperparameter takes a value of 2 or above, there will be multiple parallel repeated convolution branches in the light-weight neural network module MobilconeBlock.

3. The small target detection device based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP according to claim 1, characterized in that, The cross-stage ghost fusion module C2f-ghostoneBlock replaces the feature extraction module in YOLOv8, that is, the cross-stage partial fusion module C2fBlock, the cross-stage partial fusion module C2fBlock is composed of a feature extraction stage and a cross-stage connection stage, the feature extraction stage contains n YOLOv8Bottleneck, that is, YOLOv8 bottleneck residual module; The cross-stage ghost fusion module C2f-ghostoneBlock is improved based on the cross-stage partial fusion module C2fBlock, the YOLOv8Bottleneck therein is replaced by the ghost one branch bottleneck module GhostoneBottleneck, combined with the decoupled fully connected DFC attention mechanism, the model parameter amount is reduced; the ghost one branch bottleneck module GhostoneBottleneck is based on the residual structure of the light-weight network GhostNetV2, and the feature extraction capability is optimized by fusing structure reparameterization, which is divided into two stages of feature extraction and fusion; the feature extraction stage adopts the improved ghost one branch module GhostoneModule, and the 1×1 and 5×5 ordinary convolution is replaced by the 1×1 and 3×3 light-weight neural network module MobileoneBlock, the channel dimension is adjusted, the depth separable convolution is implemented, and the output is spliced and fused; the feature fusion stage adds the input features and high-level semantic features through the jump connection.

4. The small target detection device based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP according to claim 3, characterized in that, The decoupled fully connected DFC attention mechanism is used to provide the feature extraction capability of the ghost one branch bottleneck module GhostoneBottleneck; a fully connected layer FC with fixed weights is used to generate an attention map with a global receptive field; the decoupled fully connected DFC attention mechanism decouples the fully connected layer along the horizontal and vertical directions, and uses the two fully connected layers to aggregate the long-distance spatial information in the two directions respectively; the calculation formulas of the two fully connected layers are shown in formulas (1) and (2), which are realized by convolution through sharing part of the weights; （1） （2） where represents element-wise multiplication; represents the value of the intermediate output of the horizontal decoupling layer at position (h, w); represents the weight parameter in the horizontal direction, connecting the input height and the output height h; represents the input feature map at position h' x w; represents the value of the output feature map at spatial position (h, w), represents the weight parameter in the vertical direction, connecting the width of the intermediate output and the width w of the final output; H represents the feature map height; W represents the feature map width; represents the value of the intermediate output of the horizontal decoupling layer at position h, w'; The image information processing process of the decoupled fully connected DFC attention mechanism is as follows: the focused image feature block of the input image will be connected with the horizontal and vertical feature blocks through the two fully connected layers at the same time, and at the same time, the image feature block will be connected with other feature blocks in the two directions, so the focused image feature block will be connected with all global feature blocks in a direct or indirect manner.

5. The small target detection device based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP according to claim 1, characterized in that, The efficient expansion point multiplication attention mechanism EEP is used to implement sub-millimeter grid encryption division on small target detection point dense areas, and the specific implementation steps are as follows: (1) segmentation: when the input feature map is input into the lower branch, the lower branch divides the input feature map into KxK regional feature maps by using a sliding window of KxK size, and at this time the size of each regional feature map is CxKxK; (2) Expansion and attention operation: when the input feature map is input to the upper branch, in the upper branch, the 1x1 convolution will expand its channel to K 2 Finally, the attention filter will be generated by the compression excitation module SE; the coordinate attention mechanism is applied to the efficient expansion point multiplication attention mechanism EEP (3) point multiplication: the outputs of the upper branch and the lower branch are subjected to Hadamard product operation and broadcasting; the point multiplication operation is equivalent to sequentially expanding the filter in the channel dimension, and then multiplying the same number and size of attention filters and the elements at the same position in the feature map to be weighted; the point multiplication operation between the attention filter output by the upper branch and the segmented feature map output by the lower branch is equivalent to first sequentially expanding the attention filter in the channel dimension and dividing the segmented feature map into C groups in channel order, then multiplying the attention filter and the segmented feature map in each group, i.e. multiplying the elements at the same position, and finally outputting the weighted feature map, so as to realize the screening of the feature map representing each regional feature by the attention filter through the point multiplication operation; (4) Feature map addition: The size generated by the dot product operation in the previous step is C×K 2 The feature maps ×H×W are added together for normalization, and then added to the input feature map to obtain an output feature map with the same shape as the input feature map. Here, K represents the size of the sliding window used to segment the feature map, C represents the number of channels, H represents the spatial dimension of the feature map in the vertical direction, and W represents the spatial dimension of the feature map in the horizontal direction.

6. The small target detection device based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP according to claim 1, characterized in that, In the feature fusion network Efficient-RepGFPN-EEP, a lightweight convolution module GhostConv is introduced to generate more feature maps from existing feature maps by using linear transformation; when the input feature map is input into the ghost convolution GhostConv, first, the channel transformation is performed by the conventional convolution to generate the inherent feature mapping, then more feature maps are obtained; finally, the obtained feature maps and the feature maps obtained by the conventional convolution in the first step are spliced to obtain the final feature map output.

7. The small target detection device based on the lightweight backbone network GhostoneNet and the feature fusion network EfficientRepGFPN-EEP according to claim 6, characterized in that, A weighted feature fusion method Weighted Concat is introduced to distinguishably fuse the feature maps containing defect information of different scales; as shown in equation (3), the fast normalized fusion Fast normalized fusion puts the value range of each normalized weight into [0, 1]; （3） where O represents the output feature after fusion, and the final result is obtained through weighted fusion; I i represents the i-th input feature, W i represents a learnable weight, and ε represents a small numerical constant to avoid numerical instability, and e is an exponential function, represents the sum of the exponentials of all weights W j , and each will apply the Relu function to ensure that it is greater than or equal to 0; ε is equal to 0.0001.

8. A small target detection method based on a lightweight backbone network GhostoneNet and a feature fusion network EfficientRepGFPN-EEP, characterized in that, A small target detection device based on the lightweight backbone network GhostoneNet and the feature fusion network Efficient-RepGFPN-EEP according to any one of claims 1-7 is run.

Images