An efficient detection system and detection method based on SD network full-level perception

Abstract

An efficient detection system and detection method based on the full-level perception of an S-D network and a detection method thereof, the system comprises a backbone network, a neck network and a head network, wherein the neck network comprises a full-level feature sharing module (FLIS) and a detailed information extraction module (DFE); the FLIS module designs branches at the first and second levels for different features, and uses structural reparameterization and the idea of ​​Transformer to align and fuse information at each level and then inject it so that each layer obtains global information and strengthens cross-layer information flow; the DFE module uses a three-feature fusion mapping to extract detailed information, and then uses a channel and position attention mechanism to fuse multi-scale information with the detailed information for output; the S-D neck network of the present invention improves the multi-scale feature fusion capability and achieves an ideal balance between rate and accuracy at all model scales; experimental results in the VOC dataset show that the accuracy is 3.1% higher than that of the original SOTA model, and the model computational complexity is reduced by 7%, while also having optimal characteristics in the generalization verification visdrone dataset.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to an efficient detection system and a detection method based on full-level perception of SD networks. Background Art

[0002] Currently, the neck network of real-time object detectors mainly uses feature pyramid networks (FPNs) and their derivatives. The FPN has the ability to connect and exchange information across scales, making it a basic module for processing multi-scale features. In the paper "Path Aggregation Network for Instance Segmentation" published by Shu Liu of the Chinese University of Hong Kong, a path aggregation network (PANet) was proposed. This method proposed a method for connecting top-down paths, diversifying the connection methods of the neck network. Meanwhile, in the paper "EfficientDet: Scalable and Efficient Object Detection" published by Tan Mingxing et al. of the Google Artificial Intelligence Research Group, a weighted fusion method (BiFPN) was proposed. This method proposed a skip link method for information exchange in long transmission networks.

[0003] The FPN network provides an effective solution to the problem of multi-scale feature fusion, enabling the fusion of multi-scale adjacent layer information. Therefore, the most commonly used network for multi-scale feature fusion is FPN and its derivative networks. However, FPN and its derivative networks have the following problems:

[0004] (1) The current layer can only obtain the information of the interval layer by fusing the adjacent layer with the interval layer. Therefore, the information of the interval layer is not comprehensive and much information is discarded during the first fusion.

[0005] (2) From a global perspective, the effective fusion range of single-layer information is only between adjacent layers, and the global fusion effect will be limited.

[0006] (3) Upsampling the small-scale feature layer, disassembling or adding features, and ignoring the detailed information of the large-scale feature layer.

[0007] Therefore, the network lacks the ability to fuse global information and has weak perception of feature details. Summary of the Invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an efficient detection system and detection method based on full-level perception of the SD network. By aligning and fusing multi-scale features in the first and second branches, the fused information after centralized processing is obtained, and then the fused features are injected separately according to the scale of the features of each layer. Then, the three-feature processing method and the multi-scale attention fusion method are used to finally realize the function of cross-layer detailed information fusion. The present invention has a more powerful neck fusion capability.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] An efficient detection system based on full-level perception of SD network includes a backbone network, a neck network and a head network. The neck network includes: a full-level feature sharing module (FLIS) and a detailed information extraction module (DFE);

[0011] The full-level feature sharing module (FLIS) consists of a first branch and a second branch. Each branch includes a feature alignment module (FAS), a feature fusion module (FFS), and a feature injection module. The input is first input to the feature alignment module (FAS) to make the input features of the same size. The aligned output is then input to the feature fusion module (FFS) for feature fusion. The fused features are then input to the feature injection module. Finally, the output of the feature injection module is input to each level.

[0012] The detailed information extraction module (DFE) includes a multi-scale attention fusion module (Msaf) and a triple feature processing module (SSFF). The multi-scale attention fusion module (Msaf) is used to obtain detailed information of features of different scales; the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (TFE) and a channel and position attention mechanism (CPAM). Among them, the triple feature encoding module (TFE) is used to distinguish features of different sizes to obtain classification features, and the channel and position attention mechanism (CPAM) is used to combine detailed information with scale information to obtain more accurate output information; the triple feature processing module (SSFF) consists of stack, 3D Conv and BN / SiLU. The triple feature processing module (SSFF) is used to enhance the network's multi-scale information extraction capability.

[0013] The feature alignment module F-FAS of the first branch takes the S2, S3, S4, and S5 layers of the backbone network as input for fusion, selects the S4 layer as the base layer, and uses the average pooling (Avgpool) and bilinear (Bilinear) methods in the feature alignment module F-FAS of the first branch to merge {R S2 ,R S3 ,R S5} and {R S4}Perform feature alignment R S4 =1 / 4R, and obtain the input of the feature fusion (Fuse) module F-FFS adapted to the first branch;

[0014] The feature alignment module S-FAS of the second branch takes the M3, M4, and M5 layers as input for fusion, selects the M5 layer as the base layer, and uses the average pooling method in the feature fusion module S-FFS of the second branch to combine {R M3 ,R M4} and {R M5}Perform feature alignment R M5 =1 / 8R, and obtain the input of the feature fusion (Fuse) module S-FFS adapted to the second branch.

[0015] The feature fusion (Fuse) module F-FFS of the first branch includes a multi-layer online reparameterized convolution block (OREPA) and a separation module; the multi-layer online reparameterized convolution block (OREPA) removes all nonlinear layers and replaces them with linear scaling layers, and adds a BN layer at the end of the structure. The simplified sequential and parallel structure formulas of the multi-layer online reparameterized convolution block (OREPA) are as follows:

[0016] Y=(W N (W N-1 *…W2*W1))*W=W e *X

[0017]

[0018] Among them, W j represents the weight of the jth layer, W m represents the weight of the mth branch, is a unified weight;

[0019] The input of the multi-layer online reparameterized convolution block (OREPA) is F align Provided by the output of the feature alignment module F-FAS of the first branch, the output of the multi-layer online re-parameterized convolution block (OREPA) is Ffuse , and then pass through the separation module to obtain the final output of the feature fusion (Fuse) module F-FFS of the first branch;

[0020] The feature fusion (Fuse) module S-FFS of the second branch consists of a stacked GAU module and a separation module. The stacked GAU module is based on the transformer, which treats the self-attention and linear gating units as a unified layer and shares the calculation. The optimization formula of the stacked GAU module for the linear gating unit is as follows:

[0021]

[0022]

[0023]

[0024]

[0025] in, Contains token-token attention weights, where Z represents (s<<d) 4 , and κ represent two simple transformations, and b represents the relative position deviation;

[0026] The structure of the feature fusion (Fuse) module S-FFS of the second branch includes: (1) first, obtaining the output of the alignment end from the feature alignment (Alignment) module S-FAS of the second branch; (2) inputting the output of the alignment end into the stacked GAU module to obtain the output of the fusion end; (3) finally inputting the output of the fusion end into the separation module, and obtaining the final output of the feature fusion (Fuse) module F-FFS of the second branch through the separation module.

[0027] The input of the feature injection module includes the output of the feature fusion module F-FFS of the first branch, the output of the feature fusion module S-FFS of the second branch, and the input of the multi-scale features of each layer.

[0028] The output of the feature fusion module of the first branch is combined with the information of each layer. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection module. The double convolution alignment method is used to process the multi-scale feature input of each layer. After the attention fusion is completed, the GAU is used to extract information again.

[0029] The information of each layer is fused with the output of the feature fusion (Fuse) module of the second branch. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection (Inject) module. The double convolution alignment method is used to process the information in the input part of the multi-scale features of each layer. After the attention fusion is completed, the GAU is used to extract the information again.

[0030] The three-feature processing module (SSFF) organically links the high-dimensional information in the deep feature map with the detail information in the shallow feature map, constructs a scale space along the scale axis, and obtains images of the same resolution and different scales. The input formula of the three-feature processing module (SSFF) is:

[0031] F σ (w,h)=G σ (w,h)×f(w,h)

[0032]

[0033] Among them, f(w,h) represents the two-dimensional input end of the image, F σ (w,h) is the sum of G σ (w,h) obtained by two-dimensional Gaussian filtering and smoothing;

[0034] After unifying the resolution, the images are spliced ​​together, and feature maps of different scales are superimposed. 3D convolution is used to extract the scale sequence features. The S3, M3, and P3 layers with detail information and key information are used as the base layers of the three-feature processing module (SSFF). The specific structure includes:

[0035] (1) Use single convolution to unify the number of channels of M4 and M5;

[0036] (2) Use the nearest neighbor interpolation method to align the input features and perform multi-dimensional transformation on the input feature map;

[0037] (3) Use the unsqueeze operation to convert the input three-dimensional feature map into a four-dimensional feature map;

[0038] (4) Perform three-dimensional stitching of the four-dimensional feature map based on the depth information to effectively retain the acquired features;

[0039] (5) Use three-dimensional convolution, normalization / SiLU to extract scale sequence features and output the results.

[0040] The terminal outputs of the full-level feature sharing module (FLIS) are P3, P4 and P5, which are input into the multi-scale attention fusion module (Msaf); the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (TFE) and a channel and position attention mechanism (CPAM), and uses a cross-stage local module (CSP) and a linked cross-stage local module (CCSP) as the interaction structure of the triple feature encoding module (TFE) and the channel and position attention mechanism (CPAM); the specific fusion process of the multi-scale attention fusion module (Msaf) is as follows:

[0041] (1) Input the outputs of P4 and P3 into the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer, and then input the input of P3 and the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer into the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer to obtain the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer;

[0042] (2) The output of P5, the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer, and the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer are input to the fifth-layer linked cross-stage local module (CCSP), the fourth-layer linked cross-stage local module (CCSP) and the channel and position attention mechanism (CPAM) respectively;

[0043] (3) The outputs of the 5th layer linked cross-stage local module (CCSP), the 4th layer linked cross-stage local module (CCSP) and the channel and position attention mechanism (CPAM) are input into the head part and detected in the head at the same time.

[0044] The structural functions of the triple feature encoding module (TFE) include: first, unifying the number of channels and keeping them consistent, and uniformly performing ConvBNSiLU operations on the three-scale feature maps; second, downsampling the large-scale feature map using Avg+MaxPooling and performing ConvBNSiLU operations to obtain a large-scale feature map output, directly performing ConvBNSiLU operations on the medium-scale feature map to obtain a medium-scale feature map output, and downsampling the small-scale feature map using the Nearest method and performing ConvBNSiLU operations to obtain a small-scale feature map output; finally, the large-scale feature map output, the medium-scale feature map output and the small-scale feature map are uniformly convolved and spliced ​​to obtain the final output part of the triple feature encoding module (TFE)).

[0045] The structural functions of the channel and position attention mechanism (CPAM) include: for different feature information in multiple channels, the channel and position attention mechanism (CPAM) combines detailed information with scale information; the input end of the channel and position attention mechanism (CPAM) is composed of the position attention information, i.e., the output end of the three-feature processing module (SSFF) and the channel attention information, i.e., the output end of the P3 layer TFE+CSP module; the channel attention information is used as the bottom layer input after pooling and convolution, and then enters the next layer together with the position information as the upper layer input, and finally the output of the channel and position attention mechanism (CPAM) enters the Head part.

[0046] The backbone network uses EfficientRep, and the head network uses a three-output structure.

[0047] An efficient detection method based on full-layer perception of SD networks includes the following steps:

[0048] Step 1: Input multiple features in the backbone network into the full-level feature sharing module (FLIS). The full-level feature sharing module (FLIS) includes a first branch and a second branch. Each branch includes a feature alignment module F-FAS, a feature fusion module F-FFS, and a feature injection module. The feature alignment module F-FAS and the feature fusion module F-FFS of the first branch and the second branch are used to centrally process the features of each level. Then, the feature injection modules of the first branch and the second branch are used to inject the centrally processed features back to each level, so that each level obtains global features. At the same time, the outputs of the first branch and the second branch are obtained respectively.

[0049] Step 2: The output of the first branch after fusion with each layer is input to the three-feature processing module (SSFF) in the detailed information extraction module (DFE). Then, the output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each layer are input to the multi-scale attention fusion module (Msaf).

[0050] Step 3: The output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each layer are input into the multi-scale attention fusion module (Msaf). In the multi-scale attention fusion module (Msaf), the triple feature encoding module (TFE) is used to save the detailed information in the features. Then, the channel and position attention mechanism (CPAM) is used to divide the output features of different sizes. The three-output mode of the channel and position attention mechanism (CPAM) enables features of different sizes to be processed separately in the head to obtain efficient prediction results.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. This paper designs a full-level feature sharing module (FLIS) to uniformly manage and redistribute information at different layers, significantly enhancing the information fusion capability of the neck without significantly increasing the delay, and improving the detection performance of the model at multiple scales.

[0053] 2. The present invention uses a detailed information extraction module (DFE) to amplify and obtain detailed information on the detailed information lost in the large-scale feature layer, thereby obtaining more detailed information in the large-scale feature layer and significantly improving the overall detection effect.

[0054] 3. The present invention combines the full-level feature sharing module (FLIS) and the detailed information extraction module (DFS) to preserve the detailed information ignored in multi-scale fusion, re-assign these detailed information to each level, and differentiate the prediction process from the training process, effectively reducing the model prediction time and enhancing the model prediction efficiency.

[0055] In summary, this paper effectively improves the traditional neck network through the full-level feature sharing module (FLIS) and detailed information extraction module (DFE), greatly enhancing the cross-layer information exchange capability of the neck network. It also recovers the detailed information of the large feature layer lost in the traditional neck network, thereby enhancing the information carrying capacity of the output channel. Experiments conducted on the Voc and Visdrone datasets show that the new neck network proposed in this paper surpasses the existing state-of-the-art neck networks, effectively demonstrating the effectiveness of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is the overall framework diagram of the present invention.

[0057] Figure 2 is a structural diagram of the first branch and the second branch of the full-level feature sharing module (FLIS) of the present invention; wherein, Figure 2(a) is a structural diagram of the first branch of the present invention, and Figure 2(b) is a structural diagram of the second branch of the present invention.

[0058] Figure 3 is a structural diagram of the injection part in the full-level feature sharing module (FLIS) of the present invention; wherein, Figure 3(a) is a structural diagram of the first branch injection part of the present invention, and Figure 3(b) is a structural diagram of the second branch injection part of the present invention.

[0059] Figure 4 is a block diagram of the important structures of the detailed information extraction module (DFE) of the present invention; among them, Figure 4(a) is the structure diagram of the triple feature encoding module (TFE), and Figure 4(b) is the structure diagram of the channel and position attention mechanism (CPAM). DETAILED DESCRIPTION

[0060] The technical solution adopted by the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0061] This paper proposes a new SD neck network, which is tested on multiple datasets. The experimental results show that this method can better balance the relationship between accuracy and computational complexity, and has significant advantages over existing methods. The main contributions of this paper are summarized as follows:

[0062] (1) To address the problems of cross-layer information interaction blocking and feature mapping missing, the present invention designs a full-level feature sharing module (FLIS) for different features, which significantly enhances the information fusion capability of the neck without significantly increasing the delay, and improves the detection performance of the model at multiple scales.

[0063] (2) To address the problem of missing information in large-scale feature layers, the present invention uses a detailed information extraction module (DFE) to amplify the lost detailed information and obtain detailed information.

[0064] The present invention conducts experiments on VOC and Visdrone datasets, and the experiments prove that the new neck network proposed by the present invention surpasses all existing neck networks, effectively demonstrating the effectiveness of the method proposed by the present invention.

[0065] like Figure 1 As shown, an efficient detection system based on full-level perception of SD network includes a backbone network, a neck network and a head network, wherein the neck network includes: a full-level feature sharing module (FLIS) and a detailed information extraction module (DFE);

[0066] The full-level feature sharing module (FLIS) consists of a first branch and a second branch. Each branch includes a feature alignment module (FAS), a feature fusion module (FFS), and a feature injection module. The input is first input to the feature alignment module (FAS) to make the input features of the same size. The aligned output is then input to the feature fusion module (FFS) for feature fusion. The fused features are then input to the feature injection module. Finally, the output of the feature injection module is input to each level.

[0067] The detailed information extraction module (DFE) includes a multi-scale attention fusion module (Msaf) and a triple feature processing module (SSFF). The multi-scale attention fusion module (Msaf) is used to obtain detailed information of features of different scales; the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (TFE) and a channel and position attention mechanism (CPAM). Among them, the triple feature encoding module (TFE) is used to distinguish features of different sizes to obtain classification features, and the channel and position attention mechanism (CPAM) is used to combine detailed information with scale information to obtain more accurate output information; the triple feature processing module (SSFF) consists of stack, 3D Conv and BN / SiLU. The triple feature processing module (SSFF) is used to enhance the network's multi-scale information extraction capability.

[0068] As shown in Figure 2, the feature alignment module:

[0069] The base layer of the feature alignment module should consider the larger feature layer to retain more underlying information, and also pay attention to the computational complexity brought by different layers. In order to balance the contradiction between underlying information and computational delay, the feature alignment module F-FAS of the first branch of the present invention takes the S2, S3, S4, and S5 layers of the backbone network as input for fusion, selects the S4 layer as the base layer, and uses the average pooling (Avgpool) and bilinear (Bilinear) methods in the feature alignment module F-FAS of the first branch to fuse {R S2 ,R S3 ,R S5} and {R S4}Perform feature alignment R S4 = 1 / 4R, obtaining the input of the feature fusion (F-FFS) module adapted to the first branch. The feature alignment (F-FAS) module of the first branch ensures efficient aggregation of information from each layer while taking into account the computational complexity of the neck network.

[0070] Consistent with the requirements of the feature alignment module of the first branch, it is necessary to consider retaining more underlying information and also to pay attention to the computational complexity brought by different layers. The feature alignment module S-FAS of the second branch of the present invention takes the M3, M4, and M5 layers as input for fusion, selects the M5 layer as the base layer, and uses the average pooling method in the feature fusion module S-FFS of the second branch to combine {R M3 ,R M4} and {R M5}Perform feature alignment R M5 =1 / 8R, and obtain the input of the feature fusion (Fuse) module S-FFS adapted to the second branch. This ensures the efficient aggregation of information at each layer and reduces the computational complexity of the GAU module.

[0071] Feature fusion (Fuse) module:

[0072] The feature fusion (Fuse) module F-FFS of the first branch includes a multi-layer online reparameterized convolution block (OREPA) and a separation module; in order to improve the performance of the deep model without increasing any inference time cost, the idea of ​​structural reparameterization is applied to the fusion end. However, the accuracy of structural reparameterization depends on the complex training-time block, and the training cost is inevitably increased. The multi-layer online reparameterized convolution block (OREPA) removes all nonlinear layers and replaces them with linear scaling layers, and adds a BN layer at the end of the structure. This improvement compresses the complex training-time block into a single convolution, greatly reducing the training cost. The simplified sequential structure and parallel structure formulas of the multi-layer online reparameterized convolution block (OREPA) are as follows:

[0073] Y=(W N (W N-1 *…W2*W1))*W=W e *X

[0074]

[0075] Among them, W j represents the weight of the jth layer, W m represents the weight of the mth branch, is a unified weight;

[0076] The input of the multi-layer online reparameterized convolution block (OREPA) is F align Provided by the output of the feature alignment module F-FAS of the first branch, the output of the multi-layer online re-parameterized convolution block (OREPA) is F fuse , and then the final output of the feature fusion (Fuse) module F-FFS of the first branch is obtained through the separation module, and the structure is shown in Figure 2(a).

[0077] The feature fusion (Fuse) module S-FFS of the second branch consists of a stacked GAU module and a separation module. The stacked GAU module is based on the transformer. It treats the self-attention and linear gating units as a unified layer and shares calculations, improving computational efficiency while reducing the number of parameters. The optimization formula of the stacked GAU module for the linear gating unit is as follows:

[0078]

[0079]

[0080]

[0081]

[0082] in, Contains token-token attention weights, where Z represents (s<<d) 4 , and κ represent two simple transformations, and b represents the relative position deviation;

[0083] The structure of the feature fusion (Fuse) module S-FFS of the second branch includes: (1) first, obtaining the output of the alignment end from the feature alignment (Alignment) module S-FAS of the second branch; (2) inputting the output of the alignment end into the stacked GAU module to obtain the output of the fusion end; (3) finally inputting the output of the fusion end into the separation module, and obtaining the final output of the feature fusion (Fuse) module F-FFS of the second branch through the separation module; the structure is shown in Figure 2(b).

[0084] Feature Injection Module:

[0085] The input end of the feature injection module includes the output part of the feature fusion (Fuse) module F-FFS of the first branch, the output part of the feature fusion (Fuse) module S-FFS of the second branch, and the input part of the multi-scale features of each layer. In order to inject the output information of the feature fusion (Fuse) module F-FFS of the first branch into each layer, the present invention fuses the information of each layer with the output part of the feature fusion (Fuse) module of the first branch, introduces the LAF module at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection module, and uses double convolution alignment to process the information in the input part of the multi-scale features of each layer. After the attention fusion is completed, the GAU is used to extract information again.

[0086] To inject the output information of the S-FFS end of the feature fusion (Fuse) module of the second branch into each layer, the present invention fuses the information of each layer with the output of the feature fusion (Fuse) module of the second branch. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection (Inject) module. The multi-scale feature input of each layer is processed by double convolution alignment. After the attention fusion is completed, the GAU is used to extract the information again. The structure is shown in Figure 3.

[0087] Detailed Information Extraction Module (DFE):

[0088] The three-feature processing module (SSFF) addresses the problem of missing pyramid feature maps by organically linking the high-dimensional information in the deep feature map with the detailed information in the shallow feature map. Since the image size changes during the scaling process, while the scale feature remains constant, a scale space is constructed along the scale axis to obtain images of the same resolution but different scales. The input formula of the three-feature processing module (SSFF) is:

[0089] F σ (w,h)=G σ (w,h)×f(w,h)

[0090]

[0091] Among them, f(w,h) represents the two-dimensional input end of the image, F σ (w,h) is the sum of G σ (w,h) obtained by two-dimensional Gaussian filtering and smoothing;

[0092] After unifying the resolution, the images are spliced ​​together, and the feature maps of different scales are superimposed. The scale sequence features are extracted using three-dimensional convolution. The present invention uses the S3, M3, and P3 layers with detail information and key information as the base layers of the three-feature processing module (SSFF), as shown in Figure 4. The specific structure includes:

[0093] (1) Use single convolution to unify the number of channels of M4 and M5 to facilitate feature alignment;

[0094] (2) Using the nearest neighbor interpolation method to align the input features, the input feature map can be effectively converted into multiple dimensions;

[0095] (3) Use the unsqueeze operation to convert the input three-dimensional feature map into a four-dimensional feature map (adding depth information) to obtain more information;

[0096] (4) Perform three-dimensional stitching of the four-dimensional feature map based on the depth information to effectively retain the acquired features;

[0097] (5) Use three-dimensional convolution, normalization / SiLU to extract scale sequence features and output the results.

[0098] The terminal outputs of the full-level feature sharing module (FLIS) are P3, P4 and P5, which are input into the multi-scale attention fusion module (Msaf); the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (TFE) and a channel and position attention mechanism (CPAM), and uses a cross-stage local module (CSP) and a linked cross-stage local module (CCSP) as the interaction structure of the triple feature encoding module (TFE) and the channel and position attention mechanism (CPAM); as shown in Figure 4, the specific fusion process of the multi-scale attention fusion module (Msaf) is as follows:

[0099] (1) Input the three-feature encoding module (TFE) + CSP module from the P4 and P3 layers, and then combine the output with the two parallel outputs obtained by inputting the three-feature encoding module (TFE) + CSP module from the P3 layer;

[0100] (2) The P5 layer and two parallel outputs are fed into the CCSP layer (CONCAT+CSP), and the CCSP layer and the channel and position attention mechanism (CPAM);

[0101] (3) The outputs of the CCSP layer (CONCAT+CSP) and the CCSP layer with the channel and position attention mechanism (CPAM) are input into the Head part, and the three parts are detected in the Head at the same time.

[0102] The structure of the multi-scale attention fusion module (Msaf) contains two core functional parts: the triple feature encoding module (TFE) and the channel and position attention mechanism (CPAM). The triple feature encoding module (TFE) aims to distinguish features of different sizes to obtain classification features, and the channel and position attention mechanism (CPAM) combines detailed information with scale information to obtain more accurate output information.

[0103] Triple feature encoding module (TFE): Due to the pyramid structure, the top feature layer is only downsampled, which loses a lot of detailed information. TFE addresses this problem by splitting features into three categories: large, medium, and small, and performing feature scaling to capture more detailed information. The structure of TFE is shown in Figure 4(a). The structural functions of the triple feature encoding module (TFE) include: first, unifying the number of channels and keeping them consistent, and uniformly performing ConvBNSiLU operations on the three-scale feature maps; second, the large-scale feature map is downsampled using Avg+MaxPooling and subjected to ConvBNSiLU operations to obtain a large-scale feature map output, which helps to maintain feature diversity; the medium-scale feature map is directly subjected to ConvBNSiLU operations to obtain a medium-scale feature map output; the small-scale feature map is downsampled using the Nearest method and subjected to ConvBNSiLU operations to obtain a small-scale feature map output, which helps to prevent information loss; finally, the large-scale feature map output, the medium-scale feature map output and the small-scale feature map are uniformly convolved and spliced ​​to obtain the final output part of the triple feature encoding module (TFE), the structure of which is shown in Figure 4(a).

[0104] The structural functions of the channel and position attention mechanism (CPAM) include: for different feature information in multiple channels, the channel and position attention mechanism (CPAM) combines detailed information with scale information; the input of the channel and position attention mechanism (CPAM) is composed of the position attention information, that is, the output of the three-feature processing module (SSFF) and the channel attention information, that is, the output of the P3 layer TFE+CSP module; the channel attention information is used as the bottom-level input after pooling and convolution, and then enters the next layer together with the position information as the upper-level input. Finally, the output of the channel and position attention mechanism (CPAM) enters the head part, which divides the multi-scale features into three parts: large, medium and small, and adapts to different detection heads to make the detection effect of the head part more accurate. The structure is shown in Figure 4(b).

[0105] An efficient detection method based on full-layer perception of SD networks includes the following steps:

[0106] Step 1: Input multiple features in the backbone network into the full-level feature sharing module (FLIS). The full-level feature sharing module (FLIS) includes a first branch and a second branch. Each branch includes a feature alignment module F-FAS, a feature fusion module F-FFS, and a feature injection module. The feature alignment module F-FAS and the feature fusion module F-FFS of the first branch and the second branch are used to centrally process the features of each level. Then, the feature injection modules of the first branch and the second branch are used to inject the centrally processed features back to each level, so that each level obtains global features. At the same time, the outputs of the first branch and the second branch are obtained respectively.

[0107] Step 2: The output of the first branch after fusion with each layer is input to the three-feature processing module (SSFF) in the detailed information extraction module (DFE). The three-feature processing module (SSFF) enhances the multi-scale information extraction capability of the network. The output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each layer are then input to the multi-scale attention fusion module (Multi scale attention fusion (Msaf)).

[0108] Step 3: The output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each layer are input into the multi-scale attention fusion module (Msaf). In the multi-scale attention fusion module (Msaf), the triple feature encoding module (TFE) is used to save a large amount of detailed information in the features. Then, the channel and position attention mechanism (CPAM) is used to divide the output features of different sizes. The three-output mode of the channel and position attention mechanism (CPAM) enables features of different sizes to be processed separately in the head to obtain efficient prediction results.

[0109] Experiment and analysis

[0110] 1. Experimental Design

[0111] The basic data sets used in the present invention are part of VOC2007 and VOC2012. All experiments in the present invention do not perform pre-training, and the initial seed is fixed. The backbone network uses EfficientRep, and the head uses three layers of BOX output and one layer of Mask output. The optimizer uses stochastic gradient descent (SGD), momentum and cosine decay with learning rate and other methods. Data enhancement uses Mosaic and Mixup methods. Except for special statements, the epochs trained in the present invention are all 150, and all data in the present invention do not participate in pruning, distillation, and acceleration. The data set used for generalization verification is the visdrone data set. All models of the present invention are trained on 4 1080 GPUs.

[0112] 2. Comparative Analysis

[0113] This paper focuses on evaluating the relationship between model computational overhead (FLOPs) and detection accuracy (AP). This aims to assess the overall performance of the model after deployment on a mobile device or in the cloud. The detector studied in this paper is compared with other state-of-the-art detectors, and FLOPs are compared on the same machine. The experimental comparison is shown in the table:

[0114] As can be seen from the table, OURS-1 showed significant progress in the experiment, achieving 6.6%, 4.4%, 4.6%, and 2.6% improvements in detection accuracy compared to YOLOv5-1, YOLOv6-1, EfficientDet-D7x, and YOLOv8-1, respectively. In terms of model computational complexity, OURS-1's model computational complexity is almost the same as YOLOv6-1, reduced by about 63% compared to EfficientDet-D7x, and reduced by 7% compared to YOLOv8-1. Therefore, the algorithm of the present invention improves detection accuracy and model computational complexity by 2.6% and 7%, respectively, compared to the current SOTA model (YOLOv8-1). Compared with mainstream algorithm models, the algorithm model of the present invention has the dual advantages of accuracy and model lightweightness.

[0115] Table 1 Comparison of the algorithm of the present invention with other algorithms

[0116]

[0117]

[0118] 3. Ablation Experiment

[0119] To verify the effectiveness of the feature pyramid analysis presented in this paper and to evaluate the effectiveness of the algorithm as a whole, this experiment independently tested each module mentioned in this paper and analyzed the role each module plays in the overall network. The experimental results are shown in the table below. The SD network and the SSFF+Msaf module in the first and second branches have a significant impact on the performance of the overall network. The experiment is shown in the table:

[0120] Table 2 Impact of each module on overall performance (S)

[0121]

[0122] The neck network proposed in this invention is compared with other neck networks, and the adaptability of the neck network of this invention under different backbone networks is verified. The experimental results are shown below:

[0123] Table 3 Neck comparison under EfficientRep backbone network

[0124]

[0125] Table 4 Neck comparison under ResNet backbone network

[0126]

[0127] 4. Generalization Verification

[0128] In order to verify the generalization performance of the SD network, the present invention sets the data set used in the experiment to visdrone and re-runs the experiment. The experimental data are shown below. The experiment shows that the algorithm of the present invention also has the highest detection accuracy in the new data set. The experiment proves that the algorithm is highly practical and has strong performance under multiple data. The experimental results are shown below:

[0129] Table 5 Generalization experimental verification

[0130]

[0131] This paper carefully analyzes the traditional Feature Pyramid Network (FPN) architecture and critically examines its constraints from both the perspectives of information transmission and information extraction. Based on this, the present invention proposes a SD Neck Network for target detection tasks, achieving optimal results across multiple datasets. Within the SD Neck Network, the present invention introduces a novel information interaction mechanism that strategically improves the effectiveness and efficiency of information fusion and transmission, avoiding unnecessary losses. It also introduces a detailed information acquisition mechanism, which synergistically significantly improves the model's detection capabilities.

Claims

1. An efficient detection system based on full-level perception of SD network, including backbone network, neck network and head network, characterized by: The neck network includes: a full-level feature sharing module (FLIS) and a detailed information extraction module (DFE); The full-level feature sharing module (FLIS) consists of a first branch and a second branch. Each branch includes a feature alignment module (FAS), a feature fusion module (FFS), and a feature injection module. The input is first input to the feature alignment module (FAS) to make the input features of the same size. The aligned output is then input to the feature fusion module (FFS) for feature fusion. The fused features are then input to the feature injection module. Finally, the output of the feature injection module is input to each level. The detailed information extraction module (DFE) includes a multi-scale attention fusion module (Multi scale attention fusion (Msaf)) and a triple feature processing module (SSFF). The triple feature processing module (SSFF) consists of stack, 3D Conv and BN / SiLU. The triple feature processing module (SSFF) is used to enhance the multi-scale information extraction capability of the network. The output after the fusion of the first branch and each layer is input into the triple feature processing module (SSFF), and then the output of the triple feature processing module (SSFF) and the output after the fusion of the second branch and each layer are input into the multi-scale attention fusion module (Multi scale attention fusion (Msaf)); the multi-scale attention fusion module (Multi scale attention fusion (Msaf)) is used to obtain detailed information of features of different scales; the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (Triple feature encoding module (TFE)) and a channel and position attention mechanism (CPAM), among which the triple feature encoding module (Triple feature encoding The Translated Feathering (TFE) module is used to distinguish features of different sizes, obtain classification features, and preserve detailed information in the features. The Channel and Position Attention Mechanism (CPAM) is used to combine detailed information with scale information to obtain more accurate output information. The three-output mode of the Channel and Position Attention Mechanism (CPAM) enables features of different sizes to be processed separately in the head to obtain efficient prediction results. The three-feature processing module (SSFF) organically links the high-dimensional information in the deep feature map with the detail information in the shallow feature map, constructs a scale space along the scale axis, and obtains images of the same resolution and different scales. The input formula of the three-feature processing module (SSFF) is: F σ (w,h)=G σ (w,h)×f(w,h) Among them, f(w,h) represents the two-dimensional input end of the image, F σ (w,h) is the sum of G σ (w,h) obtained by two-dimensional Gaussian filtering and smoothing; After unifying the resolution, the images are spliced ​​together, and feature maps of different scales are superimposed. 3D convolution is used to extract the scale sequence features. The S3, M3, and P3 layers with detail information and key information are used as the base layers of the three-feature processing module (SSFF). The specific structure includes: (1) Use single convolution to unify the number of channels of M4 and M5; (2) Use the nearest neighbor interpolation method to align the input features and perform multi-dimensional transformation on the input feature map; (3) Use the unsqueeze operation to convert the input three-dimensional feature map into a four-dimensional feature map; (4) Perform three-dimensional stitching of the four-dimensional feature map based on the depth information to effectively retain the acquired features; (5) Use three-dimensional convolution, normalization / SiLU to extract scale sequence features and output the results.

2. The efficient detection system based on SD network full-level perception according to claim 1 is characterized in that: The feature alignment module F-FAS of the first branch takes the S2, S3, S4, and S5 layers of the backbone network as input for fusion, selects the S4 layer as the base layer, and uses the average pooling (Avgpool) and bilinear (Bilinear) methods in the feature alignment module F-FAS of the first branch to merge {R S2 ,R S3 ,R S5 } and {R S4 }Perform feature alignment R S4 =1 / 4R, and obtain the input of the feature fusion (Fuse) module F-FFS adapted to the first branch; The feature alignment module S-FAS of the second branch takes the M3, M4, and M5 layers as input for fusion, selects the M5 layer as the base layer, and uses the average pooling method in the feature fusion module S-FFS of the second branch to combine {R M3 ,R M4 } and {R M5 }Perform feature alignment R M5 =1 / 8R, and obtain the input of the feature fusion (Fuse) module S-FFS adapted to the second branch.

3. The efficient detection system based on SD network full-level perception according to claim 1 is characterized in that: The feature fusion (Fuse) module F-FFS of the first branch includes a multi-layer online reparameterized convolution block (OREPA) and a separation module; the multi-layer online reparameterized convolution block (OREPA) removes all nonlinear layers and replaces them with linear scaling layers, and adds a BN layer at the end of the structure. The simplified sequential and parallel structure formulas of the multi-layer online reparameterized convolution block (OREPA) are as follows: Y=(W N (W N-1 *…W2*W1))*W=W e *X Among them, W j represents the weight of the jth layer, W m represents the weight of the mth branch, is a unified weight; The input of the multi-layer online reparameterized convolution block (OREPA) is F align Provided by the output of the feature alignment module F-FAS of the first branch, the output of the multi-layer online re-parameterized convolution block (OREPA) is F fuse , and then pass through the separation module to obtain the final output of the feature fusion (Fuse) module F-FFS of the first branch; The feature fusion (Fuse) module S-FFS of the second branch consists of a stacked GAU module and a separation module. The stacked GAU module is based on the transformer, which treats the self-attention and linear gating units as a unified layer and shares the calculation. The optimization formula of the stacked GAU module for the linear gating unit is as follows: in, Contains token-token attention weights, where Z represents (s<<d) 4 , and κ represent two simple transformations, and b represents the relative position deviation; The structure of the feature fusion (Fuse) module S-FFS of the second branch includes: (1) first, obtaining the output of the alignment end from the feature alignment (Alignment) module S-FAS of the second branch; (2) inputting the output of the alignment end into the stacked GAU module to obtain the output of the fusion end; (3) finally inputting the output of the fusion end into the separation module, and obtaining the final output of the feature fusion (Fuse) module F-FFS of the second branch through the separation module.

4. The efficient detection system based on SD network full-level perception according to claim 1 is characterized in that: The input of the feature injection module includes the output of the feature fusion module F-FFS of the first branch, the output of the feature fusion module S-FFS of the second branch, and the input of the multi-scale features of each layer. The output of the feature fusion module of the first branch is combined with the information of each layer. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection module. The double convolution alignment method is used to process the multi-scale feature input of each layer. After the attention fusion is completed, the GAU is used to extract information again. The information of each layer is fused with the output of the feature fusion (Fuse) module of the second branch. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection (Inject) module. The double convolution alignment method is used to process the information in the input part of the multi-scale features of each layer. After the attention fusion is completed, the GAU is used to extract the information again.

5. The efficient detection system based on SD network full-level perception according to claim 1 is characterized in that: The terminal outputs of the full-level feature sharing module (FLIS) are P3, P4 and P5, which are input into the multi-scale attention fusion module (Msaf); the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (TFE) and a channel and position attention mechanism (CPAM), and uses a cross-stage local module (CSP) and a linked cross-stage local module (CCSP) as the interaction structure of the triple feature encoding module (TFE) and the channel and position attention mechanism (CPAM); the specific fusion process of the multi-scale attention fusion module (Msaf) is as follows: (1) Input the outputs of P4 and P3 into the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer, and then input the input of P3 and the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer into the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer to obtain the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer; (2) The output of P5, the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer, and the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer are input to the fifth-layer linked cross-stage local module (CCSP), the fourth-layer linked cross-stage local module (CCSP) and the channel and position attention mechanism (CPAM) respectively; (3) The outputs of the 5th layer linked cross-stage local module (CCSP), the 4th layer linked cross-stage local module (CCSP) and the channel and position attention mechanism (CPAM) are input into the head part and detected in the head at the same time.

6. The efficient detection system based on SD network full-level perception according to claim 1 or 5, characterized in that: The structural functions of the triple feature encoding module (TFE) include: first, unifying the number of channels and keeping them consistent, and uniformly performing ConvBNSiLU operations on the three-scale feature maps; second, downsampling the large-scale feature map using Avg+MaxPooling and performing ConvBNSiLU operations to obtain a large-scale feature map output, directly performing ConvBNSiLU operations on the medium-scale feature map to obtain a medium-scale feature map output, and downsampling the small-scale feature map using the Nearest method and performing ConvBNSiLU operations to obtain a small-scale feature map output; finally, the large-scale feature map output, the medium-scale feature map output and the small-scale feature map are uniformly convolved and spliced ​​to obtain the final output part of the triple feature encoding module (TFE)).

7. The efficient detection system based on SD network full-level perception according to claim 1 or 5, characterized in that: The structural functions of the channel and position attention mechanism (CPAM) include: for different feature information in multiple channels, the channel and position attention mechanism (CPAM) combines detailed information with scale information; the input end of the channel and position attention mechanism (CPAM) is composed of the position attention information, i.e., the output end of the three-feature processing module (SSFF) and the channel attention information, i.e., the output end of the P3 layer TFE+CSP module; the channel attention information is used as the bottom layer input after pooling and convolution, and then enters the next layer together with the position information as the upper layer input, and finally the output of the channel and position attention mechanism (CPAM) enters the Head part.

8. The efficient detection system based on SD network full-level perception according to claim 1 is characterized in that: The backbone network uses EfficientRep, and the head network uses a three-output structure.

9. An efficient detection method based on full-layer perception of SD networks, comprising the following steps: Step 1: Input multiple features in the backbone network into the full-level feature sharing module (FLIS). The full-level feature sharing module (FLIS) includes a first branch and a second branch. Each branch includes a feature alignment module F-FAS, a feature fusion module F-FFS, and a feature injection module. The feature alignment module F-FAS and the feature fusion module F-FFS of the first branch and the second branch are used to centrally process the features of each level. Then, the feature injection modules of the first branch and the second branch are used to inject the centrally processed features back to each level, so that each level obtains global features. At the same time, the outputs of the first branch and the second branch are obtained respectively. Step 2: The output of the first branch after fusion with each layer is input to the three-feature processing module (SSFF) in the detailed information extraction module (DFE). Then, the output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each layer are input to the multi-scale attention fusion module (Msaf). Step 3: The output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each layer are input to the multi-scale attention fusion module (Msaf). In the multi-scale attention fusion module (Msaf), the triple feature encoding module (TFE) is used to save the detailed information in the features. Then, the channel and position attention mechanism (CPAM) is used to divide the output features of different sizes. The three-output mode of the channel and position attention mechanism (CPAM) enables features of different sizes to be processed separately in the head to obtain efficient prediction results. The three-feature processing module (SSFF) organically links the high-dimensional information in the deep feature map with the detail information in the shallow feature map, constructs a scale space along the scale axis, and obtains images of the same resolution and different scales. The input formula of the three-feature processing module (SSFF) is: F σ (w,h)=G σ (w,h)×f(w,h) Among them, f(w,h) represents the two-dimensional input end of the image, F σ (w,h) is the sum of G σ (w,h) obtained by two-dimensional Gaussian filtering and smoothing; After unifying the resolution, the images are spliced ​​together, and feature maps of different scales are superimposed. 3D convolution is used to extract the scale sequence features. The S3, M3, and P3 layers with detail information and key information are used as the base layers of the three-feature processing module (SSFF). The specific structure includes: (1) Use single convolution to unify the number of channels of M4 and M5; (2) Use the nearest neighbor interpolation method to align the input features and perform multi-dimensional transformation on the input feature map; (3) Use the unsqueeze operation to convert the input three-dimensional feature map into a four-dimensional feature map; (4) Perform three-dimensional stitching of the four-dimensional feature map based on the depth information to effectively retain the acquired features; (5) Use three-dimensional convolution, normalization / SiLU to extract scale sequence features and output the results.

Images