Efficient detection system based on S-D network full-level perception and detection method thereof

Abstract

The invention discloses an efficient detection system and method based on S-D network full-level perception, the system comprises a backbone network, a neck network and a head network, the neck network comprises a full-level feature sharing module (FLIS) and a detailed information extraction module (DFE); the FLIS module designs branches of a first hierarchy and a second hierarchy according to different features, information of all the hierarchies is aligned, fused and then injected by using the thought of structural reparameterization and Transformer to enable all the hierarchies to obtain global information, and cross-layer information flow is enhanced. The DFE module extracts detailed information by using three-feature fusion mapping, and then fuses and outputs the multi-scale information and the detailed information by using a channel and position attention mechanism; according to the S-D neck network, the multi-scale feature fusion capability is improved, and the ideal balance between the rate and the accuracy is realized on all model scales; the precision of an experimental result in a VOC data set is improved by 3.1% compared with that of an original SOTA model, the calculated amount of the model is reduced by 7%, and meanwhile the optimal characteristic is achieved in a generalization verification vidrone data set.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to an efficient detection system and a detection method based on full-level perception of SD network. Background Art

[0002] At present, the neck network of real-time object detectors mainly uses feature pyramid (FPN) and its derivatives. Feature pyramid network (FPN) has the ability to connect and exchange information across scales, and has become a basic module for processing multi-scale features. The Path Aggregation Network for Instance Segmentation published by Shu Liu of the Chinese University of Hong Kong proposed a path aggregation network (PANet), which proposed a method to connect top-down paths, making the connection method of the neck network diversified; at the same time, the EfficientDet: Scalable and Efficient Object Detection published by Tan Mingxing et al. of the Google Artificial Intelligence Research Group proposed a weighted fusion method (BiFPN), which proposed a skip connection method for information interaction in long transmission networks.

[0003] The FPN network proposes an effective solution to the problem of multi-scale feature fusion, which enables the fusion of multi-scale adjacent layer information. Therefore, the most commonly used network for multi-scale feature fusion is FPN and its derivative networks. However, FPN and its derivative networks have the following problems:

[0004] (1) The current layer can only obtain the information of the interval layer by fusing the adjacent layer with the interval layer. Therefore, the information of the interval layer is not comprehensive, and much information has been discarded during the first fusion.

[0005] (2) From a global perspective, the effective fusion range of single-layer information is only between adjacent layers, and the global fusion effect will be limited.

[0006] (3) Upsampling the small-scale feature layer, disassembling or adding features, while ignoring the detailed information of the large-scale feature layer.

[0007] Therefore, the network lacks the ability to fuse global information and has a weak perception of feature details. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an efficient detection system and a detection method based on full-level perception of the SD network. By aligning and fusing multi-scale features in the first and second branches, the fused information after centralized processing is obtained, and then the fused features are injected separately according to the scale of the features of each layer. Then, the three-feature processing method and the multi-scale attention fusion method are used to finally realize the function of cross-layer detailed information fusion. The present invention has a more powerful neck fusion capability.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] An efficient detection system based on full-level perception of SD network, comprising a backbone network, a neck network and a head network, wherein the neck network comprises: a full-level feature sharing module (FLIS) and a detailed information extraction module (DFE);

[0011] The full-level feature sharing module (FLIS) includes a first branch and a second branch, each of which includes a feature alignment module FAS, a feature fusion module FFS and a feature injection module; the input is first input into the feature alignment module FAS to make the input features of the same size, then the aligned output is input into the feature fusion module FFS for feature fusion, then the fused features are input into the feature injection module, and finally the output of the feature injection module is input into each level;

[0012] The detailed information extraction module (DFE) includes a multi-scale attention fusion module (Multi scale attention fusion (Msaf)) and a triple feature processing module (SSFF). The multi-scale attention fusion module (Multi scale attention fusion (Msaf)) is used to obtain detailed information of features of different scales; the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (Triple feature encoding module (TFE)) and a channel and position attention mechanism (CPAM). Among them, the triple feature encoding module (Triple feature encoding module (TFE)) is used to distinguish features of different sizes and obtain classification features, and the channel and position attention mechanism (CPAM) is used to combine detailed information with scale information to obtain more accurate output information; the triple feature processing module (SSFF) is composed of stack, 3D Conv and BN / SiLU. The triple feature processing module (SSFF) is used to enhance the network's multi-scale information extraction capability.

[0013] The feature alignment module F-FAS of the first branch takes the S2, S3, S4, and S5 layers of the backbone network as input for fusion, selects the S4 layer as the base layer, and uses the average pooling (Avgpool) and bilinear (Bilinear) methods in the feature alignment module F-FAS of the first branch to merge {R S2 ,R S3 ,R S5} and {R S4}Perform feature alignment R S4 =1 / 4R, and obtain the input of the feature fusion (Fuse) module F-FFS adapted to the first branch;

[0014] The feature alignment module S-FAS of the second branch takes the M3, M4, and M5 layers as input for fusion, selects the M5 layer as the base layer, and uses the average pooling method in the feature fusion module S-FFS of the second branch to combine {R M3 ,R M4} and {R M5}Perform feature alignment R M5 =1 / 8R, and obtain the input of the feature fusion (Fuse) module S-FFS adapted to the second branch.

[0015] The feature fusion (Fuse) module F-FFS of the first branch includes a multi-layer online reparameterized convolution block (OREPA) and a separation module; the multi-layer online reparameterized convolution block (OREPA) removes all nonlinear layers and replaces them with linear scaling layers, and adds a BN layer at the end of the structure. The simplified sequential structure and parallel structure formulas of the multi-layer online reparameterized convolution block (OREPA) are as follows:

[0016] Y=(W N (W N-1 *…W2*W1))*W=W e *X

[0017]

[0018] Among them, W j represents the weight of the jth layer, W m represents the weight of the mth branch, is the uniform weight;

[0019] The input of the multi-layer online reparameterized convolution block (OREPA) is F align Provided by the output of the feature alignment module F-FAS of the first branch, the output of the multi-layer online re-parameterized convolution block (OREPA) is Ffuse , and then the final output of the feature fusion (Fuse) module F-FFS of the first branch is obtained through the separation module;

[0020] The feature fusion (Fuse) module S-FFS of the second branch consists of a stacked GAU module and a separation module. The stacked GAU module is based on transformer, which treats self-attention and linear gating units as a unified layer and shares calculations. The optimization formula of the stacked GAU module for the linear gating unit is as follows:

[0021]

[0022]

[0023]

[0024]

[0025] in, Contains token-token attention weights, Z represents (s<<d) 4 , and κ represent two simple transformations, and b represents the relative position deviation;

[0026] The structure of the feature fusion (Fuse) module S-FFS of the second branch includes: (1) first, obtaining the output of the alignment end from the feature alignment (Alignment) module S-FAS of the second branch; (2) inputting the output of the alignment end into the stacked GAU module to obtain the output of the fusion end; (3) finally, inputting the output of the fusion end into the separation module, and obtaining the final output of the feature fusion (Fuse) module F-FFS of the second branch through the separation module.

[0027] The input end of the feature injection module includes the output part of the feature fusion module F-FFS of the first branch, the feature fusion module S-FFS of the second branch, and the input part of the multi-scale features of each layer;

[0028] The output part of the feature fusion (Fuse) module of the first branch is fused with the information of each layer. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection (Inject) module. The double convolution alignment method is used to process the information in the input part of the multi-scale features of each layer. After the attention fusion is completed, the GAU is used to extract the information again.

[0029] The output part of the feature fusion (Fuse) module of each layer is fused with the information of the second branch. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection (Inject) module. The double convolution alignment method is used to process the information in the input part of the multi-scale features of each layer. After the attention fusion is completed, GAU is used to extract the information again.

[0030] The three-feature processing module (SSFF) organically links the high-dimensional information in the deep feature map with the detail information in the shallow feature map, constructs a scale space along the scale axis, and obtains images of the same resolution and different scales. The input formula of the three-feature processing module (SSFF) is:

[0031] F σ (w,h)=G σ (w,h)×f(w,h)

[0032]

[0033] Among them, f(w,h) represents the two-dimensional input end of the image, F σ (w,h) is the sum of G σ (w,h) obtained by two-dimensional Gaussian filtering and smoothing;

[0034] After unifying the resolution, the images are spliced, and the feature maps of different scales are superimposed. The scale sequence features are extracted using 3D convolution. The S3, M3, and P3 layers with detail information and key information are used as the reference layers of the three-feature processing module (SSFF). The specific structure includes:

[0035] (1) Use single convolution to unify the number of channels of M4 and M5;

[0036] (2) The nearest neighbor interpolation method is used to align the input features and perform multi-dimensional transformation on the input feature map;

[0037] (3) Use the unsqueeze operation to convert the input three-dimensional feature map into a four-dimensional feature map;

[0038] (4) Perform three-dimensional stitching of the four-dimensional feature map based on the depth information to effectively retain the acquired features;

[0039] (5) Use three-dimensional convolution, normalization / SiLU to extract scale sequence features and output the results.

[0040] The terminal outputs of the full-level feature sharing module (FLIS) are P3, P4 and P5, which are input into the multi-scale attention fusion module (Multi scale attention fusion (Msaf)); the multi-scale attention fusion module (Multi scale attention fusion (Msaf)) includes a triple feature encoding module (Triple feature encoding module (TFE) and a channel and position attention mechanism (CPAM), and uses a cross-stage local module (CSP) and a linked cross-stage local module (CCSP) as the triple feature encoding module (Triple feature encoding module (TFE) and channel and position attention mechanism (CPAM) interaction structure; the specific fusion process of the multi-scale attention fusion module (Multi scale attention fusion (Msaf)) is as follows:

[0041] (1) Input the outputs of P4 and P3 into the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer, and then input the input of P3 and the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer into the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer to obtain the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer;

[0042] (2) The output of P5, the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer, and the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer are respectively input to the fifth-layer linked cross-stage local module (CCSP), the fourth-layer linked cross-stage local module (CCSP) and the channel and position attention mechanism (CPAM);

[0043] (3) The outputs of the fifth-layer linked cross-stage local module (CCSP), the fourth-layer linked cross-stage local module (CCSP) and the channel and position attention mechanism (CPAM) are input into the head part and detected in the head at the same time.

[0044] The structural functions of the triple feature encoding module (TFE) include: first, unifying the number of channels and keeping them consistent, and uniformly performing ConvBNSiLU operations on the three-scale feature maps; second, downsampling the large-scale feature map using Avg+MaxPooling and performing ConvBNSiLU operations to obtain a large-scale feature map output, directly performing ConvBNSiLU operations on the medium-scale feature map to obtain a medium-scale feature map output, and downsampling the small-scale feature map using the Nearest method and performing ConvBNSiLU operations to obtain a small-scale feature map output; finally, the large-scale feature map output, the medium-scale feature map output and the small-scale feature map are uniformly convolved and spliced ​​to obtain the final output part of the triple feature encoding module (TFE)).

[0045] The structural functions of the channel and position attention mechanism (CPAM) include: for different feature information in multiple channels, the channel and position attention mechanism (CPAM) combines detailed information with scale information; the input end of the channel and position attention mechanism (CPAM) is composed of the position attention information, that is, the output end of the three-feature processing module (SSFF) and the channel attention information, that is, the output end of the P3 layer TFE+CSP module; the channel attention information is used as the bottom layer input after pooling and convolution, and then enters the next layer together with the position information as the upper layer input, and finally the output of the channel and position attention mechanism (CPAM) enters the Head part.

[0046] The backbone network uses EfficientRep, and the head network uses a three-output structure.

[0047] An efficient detection method based on full-level perception of SD network includes the following steps:

[0048] Step 1: Input multiple features in the backbone network into the full-level feature sharing module (FLIS). The full-level feature sharing module (FLIS) includes a first branch and a second branch. Each branch includes a feature alignment module F-FAS, a feature fusion module F-FFS and a feature injection module. The feature alignment module F-FAS and the feature fusion module F-FFS of the first branch and the second branch are used to centrally process the features of each level, and then the feature injection modules of the first branch and the second branch are used to inject the centrally processed features back to each level, so that each level obtains global features, and the outputs of the first branch and the second branch are obtained respectively.

[0049] Step 2: The output after the fusion of the first branch and each level is input into the three-feature processing module (SSFF) in the detailed information extraction module (DFE), and then the output of the three-feature processing module (SSFF) and the output after the fusion of the second branch and each level are input into the multi-scale attention fusion module (Multi scale attention fusion (Msaf));

[0050] Step 3: The output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each level are input into the multi-scale attention fusion module (Msaf). In the multi-scale attention fusion module (Msaf), the triple feature encoding module (TFE) is used to save the detailed information in the features, and then the channel and position attention mechanism (CPAM) is used to divide the output features of different sizes. The three-output mode of the channel and position attention mechanism (CPAM) enables features of different sizes to be processed separately in the Head to obtain efficient prediction results.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention designs a full-level feature sharing module (FLIS) to uniformly manage and redistribute information at different layers, significantly enhancing the information fusion capability of the neck without significantly increasing the delay, and improving the detection performance of the model at multiple scales.

[0053] 2. The present invention uses a detailed information extraction module (DFE) to amplify the detailed information lost in the large-scale feature layer and obtain detailed information, thereby obtaining more detailed information in the large-scale feature layer and significantly improving the overall detection effect.

[0054] 3. The present invention combines the full-level feature sharing module (FLIS) and the detailed information extraction module (DFS) to preserve the detailed information ignored in multi-scale fusion, re-assign these detailed information to each level, and differentiate the prediction process from the training process, which effectively reduces the model prediction time and enhances the model prediction efficiency.

[0055] In summary, the present invention effectively improves the traditional neck network through the full-level feature sharing module (FLIS) and the detailed information extraction module (DFE), which greatly improves the cross-layer information interaction capability in the neck network, and reacquires the detailed information of the large feature layer lost in the traditional neck network, thereby enhancing the information carrying capacity of the output channel. The present invention conducts experiments in the Voc and Visdrone datasets, and the experiments prove that the new neck network proposed in this paper surpasses the existing most advanced neck network, effectively proving the effectiveness of the method proposed in this paper. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is the overall framework diagram of the present invention.

[0057] Figure 2 is a structural diagram of the first branch and the second branch of the full-level feature sharing module (FLIS) of the present invention; wherein, Figure 2(a) is a structural diagram of the first branch of the present invention, and Figure 2(b) is a structural diagram of the second branch of the present invention.

[0058] Figure 3 is a structural diagram of the injection part in the full-level feature sharing module (FLIS) of the present invention; wherein, Figure 3(a) is a structural diagram of the first branch injection part of the present invention, and Figure 3(b) is a structural diagram of the second branch injection part of the present invention.

[0059] FIG4 is a block diagram of the important structures of the detailed information extraction module (DFE) of the present invention; wherein FIG4(a) is a structural diagram of the triple feature encoding module (TFE), and FIG4(b) is a structural diagram of the channel and position attention mechanism (CPAM). DETAILED DESCRIPTION

[0060] The technical solution adopted by the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0061] The present invention proposes a new SD neck network, which is experimented on multiple data sets. The experimental results show that this method can better balance the relationship between accuracy and computational complexity, and has great advantages over existing methods. The main contributions of this invention are summarized as follows:

[0062] (1) To address the problems of cross-layer information interaction blocking and feature mapping missing, the present invention designs a full-level feature sharing module (FLIS) for different features, which significantly enhances the information fusion capability of the neck without significantly increasing the delay, and improves the detection performance of the model at multiple scales.

[0063] (2) To address the problem of missing information in large-scale feature layers, the present invention uses a detailed information extraction module (DFE) to amplify the features of the lost detailed information and obtain detailed information.

[0064] The present invention conducts experiments in VOC and Visdrone datasets, and the experiments prove that the new neck network proposed by the present invention surpasses all existing neck networks, effectively proving the effectiveness of the method proposed by the present invention.

[0065] like Figure 1 As shown, an efficient detection system based on full-level perception of SD network includes a backbone network, a neck network and a head network, wherein the neck network includes: a full-level feature sharing module (FLIS) and a detailed information extraction module (DFE);

[0066] The full-level feature sharing module (FLIS) includes a first branch and a second branch, each of which includes a feature alignment module FAS, a feature fusion module FFS and a feature injection module; the input is first input into the feature alignment module FAS to make the input features of the same size, then the aligned output is input into the feature fusion module FFS for feature fusion, then the fused features are input into the feature injection module, and finally the output of the feature injection module is input into each level;

[0067] The detailed information extraction module (DFE) includes a multi-scale attention fusion module (Multi scale attention fusion (Msaf)) and a triple feature processing module (SSFF). The multi-scale attention fusion module (Multi scale attention fusion (Msaf)) is used to obtain detailed information of features of different scales; the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (Triple feature encoding module (TFE)) and a channel and position attention mechanism (CPAM). Among them, the triple feature encoding module (Triple feature encoding module (TFE)) is used to distinguish features of different sizes and obtain classification features, and the channel and position attention mechanism (CPAM) is used to combine detailed information with scale information to obtain more accurate output information; the triple feature processing module (SSFF) is composed of stack, 3D Conv and BN / SiLU. The triple feature processing module (SSFF) is used to enhance the network's multi-scale information extraction capability.

[0068] As shown in Figure 2, the feature alignment module:

[0069] The reference layer of the feature alignment module should consider the larger feature layer to retain more underlying information, and also pay attention to the computational complexity brought by different layers. In order to balance the contradiction between underlying information and computational delay, the feature alignment module F-FAS of the first branch of the present invention takes the S2, S3, S4, and S5 layers of the backbone network as input for fusion, selects the S4 layer as the basic layer, and uses the average pooling (Avgpool) and bilinear (Bilinear) methods in the feature alignment module F-FAS of the first branch to merge {R S2 ,R S3 ,R S5} and {R S4}Perform feature alignment R S4 =1 / 4R, and the input of the feature fusion module F-FFS adapted to the first branch is obtained. The feature alignment module F-FAS of the first branch ensures the efficient aggregation of information at each layer while taking into account the computational complexity of the neck network.

[0070] Consistent with the requirements of the feature alignment module of the first branch, it is necessary to consider retaining more underlying information and also pay attention to the computational complexity brought by different layers. The feature alignment module S-FAS of the second branch of the present invention takes the M3, M4, and M5 layers as input for fusion, selects the M5 layer as the basic layer, and uses the average pooling method in the feature fusion module S-FFS of the second branch to merge {R M3 ,R M4} and {R M5}Perform feature alignment R M5 =1 / 8R, and obtain the input of the feature fusion (Fuse) module S-FFS adapted to the second branch. This ensures efficient aggregation of information at each layer and reduces the computational complexity of the GAU module.

[0071] Feature fusion (Fuse) module:

[0072] The feature fusion (Fuse) module F-FFS of the first branch includes a multi-layer online reparameterized convolution block (OREPA) and a separation module; in order to improve the performance of the deep model without increasing any inference time cost, the idea of ​​structural reparameterization is applied to the fusion end. However, the accuracy of structural reparameterization depends on the complex training-time block, and the training cost is inevitably increased. The multi-layer online reparameterized convolution block (OREPA) removes all nonlinear layers and replaces them with linear scaling layers, and adds a BN layer at the end of the structure. This improvement compresses the complex training-time block into a single convolution, greatly reducing the training cost. The simplified sequential structure and parallel structure formulas of the multi-layer online reparameterized convolution block (OREPA) are as follows:

[0073] Y=(W N (W N-1 *…W2*W1))*W=W e *X

[0074]

[0075] Among them, W j represents the weight of the jth layer, W m represents the weight of the mth branch, is the uniform weight;

[0076] The input of the multi-layer online reparameterized convolution block (OREPA) is F align Provided by the output of the feature alignment module F-FAS of the first branch, the output of the multi-layer online re-parameterized convolution block (OREPA) is F fuse , and then the final output of the feature fusion (Fuse) module F-FFS of the first branch is obtained through the separation module, and the structure is shown in Figure 2(a).

[0077] The feature fusion (Fuse) module S-FFS of the second branch consists of a stacked GAU module and a separation module. The stacked GAU module is based on transformer, which treats self-attention and linear gating units as a unified layer and shares calculations, thereby improving computational efficiency while reducing the number of parameters. The optimization formula of the stacked GAU module for the linear gating unit is as follows:

[0078]

[0079]

[0080]

[0081]

[0082] in, Contains token-token attention weights, Z represents (s<<d) 4 , and κ represent two simple transformations, and b represents the relative position deviation;

[0083] The structure of the feature fusion (Fuse) module S-FFS of the second branch includes: (1) first, obtaining the output of the alignment end from the feature alignment (Alignment) module S-FAS of the second branch; (2) inputting the output of the alignment end into the stacked GAU module to obtain the output of the fusion end; (3) finally, inputting the output of the fusion end into the separation module, and obtaining the final output of the feature fusion (Fuse) module F-FFS of the second branch through the separation module; the structure is shown in Figure 2(b).

[0084] Feature injection module:

[0085] The input end of the feature injection module includes the output part of the feature fusion (Fuse) module F-FFS of the first branch, the feature fusion (Fuse) module S-FFS of the second branch, and the input part of each layer of multi-scale features. In order to inject the output information of the feature fusion (Fuse) module F-FFS of the first branch into each level, the present invention fuses the information of each layer with the output part of the feature fusion (Fuse) module of the first branch, introduces the LAF module at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection module, uses the double convolution alignment method to process the information in the input part of each layer of multi-scale features, and uses GAU to extract information again after the attention fusion is completed.

[0086] In order to inject the output information of the feature fusion (Fuse) module S-FFS of the second branch into each level, the present invention fuses the information of each layer with the output part of the feature fusion (Fuse) module of the second branch, introduces the LAF module at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection (Inject) module, and uses the double convolution alignment method to process the information in the input part of the multi-scale features of each layer, and uses GAU to extract information again after the attention fusion is completed. The structure is shown in Figure 3.

[0087] Detailed Information Extraction (DFE):

[0088] The three-feature processing module (SSFF) addresses the problem of missing pyramid feature maps by organically linking the high-dimensional information in the deep feature map with the detail information in the shallow feature map. Since the image size of the feature map changes during the scaling process, and the scale feature is constant, the scale space is constructed along the scale axis to obtain images of the same resolution and different scales. The input formula of the three-feature processing module (SSFF) is:

[0089] F σ (w,h)=G σ (w,h)×f(w,h)

[0090]

[0091] Among them, f(w,h) represents the two-dimensional input end of the image, F σ (w,h) is the sum of G σ (w,h) obtained by two-dimensional Gaussian filtering and smoothing;

[0092] After unifying the resolution, splicing is performed, and the different scale feature maps are superimposed, and the scale sequence features are extracted by three-dimensional convolution. The present invention uses the S3, M3, and P3 layers with detail information and key information as the reference layer of the three-feature processing module (SSFF), as shown in Figure 4. The specific structure includes:

[0093] (1) Use single convolution to unify the number of channels of M4 and M5 to facilitate feature alignment;

[0094] (2) The nearest neighbor interpolation method is used to align the input features, which can effectively perform multi-dimensional transformation on the input feature map;

[0095] (3) Use the unsqueeze operation to convert the input three-dimensional feature map into a four-dimensional feature map (adding depth information) to obtain more information;

[0096] (4) Perform three-dimensional stitching of the four-dimensional feature map based on the depth information to effectively retain the acquired features;

[0097] (5) Use three-dimensional convolution, normalization / SiLU to extract scale sequence features and output the results.

[0098] The terminal outputs of the full-level feature sharing module (FLIS) are P3, P4 and P5, which are input into the multi-scale attention fusion module (Multi scale attention fusion (Msaf)); the multi-scale attention fusion module (Multi scale attention fusion (Msaf)) includes a triple feature encoding module (Triple feature encoding module (TFE) and a channel and position attention mechanism (CPAM), and uses a cross-stage local module (CSP) and a linked cross-stage local module (CCSP) as the triple feature encoding module (Triple feature encoding module (TFE) and the channel and position attention mechanism (CPAM) interaction structure; as shown in Figure 4, the specific fusion process of the multi-scale attention fusion module (Multiscale attention fusion (Msaf)) is as follows:

[0099] (1) Input the three-feature encoding module (TFE) + CSP module from the P4 and P3 layers, and then input the output and the two parallel outputs from the three-feature encoding module (TFE) + CSP module from the P3 layer;

[0100] (2) The P5 layer and two parallel outputs are input into the CCSP layer (CONCAT+CSP), and the CCSP layer and the channel and position attention mechanism (CPAM) respectively;

[0101] (3) The outputs of the CCSP layer (CONCAT+CSP) and the CCSP layer with the channel and position attention mechanism (CPAM) are input into the Head part, and the three parts are detected in the Head at the same time.

[0102] The structure of the multi-scale attention fusion module (Msaf) contains two core functional parts: triple feature encoding module (TFE) and channel and position attention mechanism (CPAM). The triple feature encoding module (TFE) aims to distinguish features of different sizes to obtain classification features, and the channel and position attention mechanism (CPAM) combines detailed information with scale information to obtain more accurate output information.

[0103] Triple feature encoding module (TFE): Due to the structure of the pyramid, the top feature layer is only processed by downsampling, which loses a lot of detailed information. TFE solves this problem by splitting the features into three categories: large, medium and small, and performing feature scaling to obtain a lot of detailed information. The structure of TFE is shown in Figure 4(a). The structural functions of the triple feature encoding module (TFE) include: first, unify the number of channels and keep them consistent, and uniformly perform ConvBNSiLU operations on the three-scale feature maps; second, the large-scale feature map is downsampled using Avg+MaxPooling and the ConvBNSiLU operation is performed to obtain a large-scale feature map output, which helps to maintain feature diversity, the medium-scale feature map is directly subjected to the ConvBNSiLU operation to obtain a medium-scale feature map output, and the small-scale feature map is downsampled using the Nearest method and the ConvBNSiLU operation is performed to obtain a small-scale feature map output, which helps to prevent information loss; finally, the large-scale feature map output, the medium-scale feature map output and the small-scale feature map are uniformly convolved and spliced ​​to obtain the final output part of the triple feature encoding module (TFE), and the structure is shown in Figure 4(a).

[0104] The structural functions of the channel and position attention mechanism (CPAM) include: for different feature information in multiple channels, the channel and position attention mechanism (CPAM) combines detailed information with scale information; the input of the channel and position attention mechanism (CPAM) is composed of the position attention information, that is, the output of the three-feature processing module (SSFF) and the channel attention information, that is, the output of the P3 layer TFE+CSP module; the channel attention information is used as the bottom layer input after pooling and convolution, and then enters the next layer together with the position information as the upper layer input, and finally the output of the channel and position attention mechanism (CPAM) enters the Head part, which divides the multi-scale features into three parts: large, medium and small, and adapts to different detection heads to make the detection effect of the Head part more accurate. The structure is shown in Figure 4(b).

[0105] An efficient detection method based on full-level perception of SD network includes the following steps:

[0106] Step 1: Input multiple features in the backbone network into the full-level feature sharing module (FLIS). The full-level feature sharing module (FLIS) includes a first branch and a second branch. Each branch includes a feature alignment module F-FAS, a feature fusion module F-FFS and a feature injection module. The feature alignment module F-FAS and the feature fusion module F-FFS of the first branch and the second branch are used to centrally process the features of each level, and then the feature injection modules of the first branch and the second branch are used to inject the centrally processed features back to each level, so that each level obtains global features, and the outputs of the first branch and the second branch are obtained respectively.

[0107] Step 2: The output after the fusion of the first branch and each level is input into the three-feature processing module (SSFF) in the detailed information extraction module (DFE), and the multi-scale information extraction capability of the network is enhanced through the three-feature processing module (SSFF). Then, the output of the three-feature processing module (SSFF) and the output after the fusion of the second branch and each level are input into the multi-scale attention fusion module (Multi scale attention fusion (Msaf));

[0108] Step 3: The output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each level are input into the multi-scale attention fusion module (Msaf). In the multi-scale attention fusion module (Msaf), the triple feature encoding module (TFE) is used to save a large amount of detailed information in the features. Then, the channel and position attention mechanism (CPAM) is used to divide the output features of different sizes. The three-output mode of the channel and position attention mechanism (CPAM) enables features of different sizes to be processed separately in the Head to obtain efficient prediction results.

[0109] Experiment and analysis

[0110] 1. Experimental design

[0111] The basic data sets used in the present invention are part of VOC2007 and VOC2012. All experiments in the present invention do not perform pre-training, and the initial seed is fixed. The backbone network uses EfficientRep, and the head uses three layers of BOX output and one layer of Mask output. The optimizer uses methods such as stochastic gradient descent (SGD), momentum and cosine decay with learning rate. Data enhancement uses Mosaic and Mixup methods. Except for special statements, the epochs trained in the present invention are all 150, and all data in the present invention do not participate in pruning, distillation, and acceleration. The data set used for generalization verification is the visdrone data set. All models of the present invention are trained on 4 1080 GPUs.

[0112] 2. Comparative analysis

[0113] This paper focuses on evaluating the relationship between the model computational load (FLOPs) and the detection accuracy (AP). It aims to evaluate the comprehensive performance of the model after it is deployed on a mobile terminal or in the cloud. The detector studied in this paper is compared with other state-of-the-art detectors, and the FLOPs are compared on the same machine. The experimental comparison is shown in the table:

[0114] It can be seen from the table that OURS-1 shows obvious progress in the experiment. Compared with YOLOv5-1, YOLOv6-1, EfficientDet-D7x, and YOLOv8-1, it has achieved 6.6%, 4.4%, 4.6%, and 2.6% progress in detection accuracy, respectively; in terms of model calculation amount, the model calculation amount of OURS-1 is almost the same as that of YOLOv6-1, the calculation amount is reduced by about 63% compared with EfficientDet-D7x, and the calculation amount is reduced by 7% compared with YOLOv8-1. Therefore, the research algorithm of the present invention improves the detection accuracy and model calculation amount by 2.6% and 7% respectively compared with the current SOTA model (YOLOv8-1). Compared with the mainstream algorithm model, the algorithm model of the present invention has dual advantages in accuracy and model lightweight.

[0115] Table 1 Comparison of the algorithm of the present invention with other algorithms

[0116]

[0117]

[0118] 3. Ablation experiment

[0119] In order to verify the effectiveness of the feature pyramid analysis of the present invention and evaluate the effectiveness of the algorithm of the present invention as a whole, this experiment independently detects each module mentioned in the present invention and analyzes the role of each module in the overall network. The experimental results are shown in the following table. The SD network of the first and second branches and the SSFF+Msaf module have an important impact on the performance of the overall network. The experiment is shown in the table:

[0120] Table 2 Impact of each module on overall performance (S)

[0121]

[0122] The neck network proposed in the present invention is compared with other neck networks, and the adaptability of the neck network of the present invention under different backbones is verified. The experimental results are as follows:

[0123] Table 3 Neck comparison under EfficientRep backbone network

[0124]

[0125] Table 4 Neck comparison under ResNet backbone network

[0126]

[0127] 4. Generalization Verification

[0128] In order to verify the generalization performance of the SD network, the present invention sets the data set used in the experiment as visdrone, and re-experiments. The experimental data are shown below. The experiment shows that the algorithm of the present invention also has the highest detection accuracy in the new data set. The experiment proves that the algorithm is highly practical and has strong performance under multiple data. The experimental results are shown below:

[0129] Table 5 Generalization experimental verification

[0130]

[0131] The present invention carefully analyzes the traditional feature pyramid network (FPN) architecture and critically analyzes its constraints from the perspective of information transmission and information extraction. Based on this, the present invention proposes the SD neck network for target detection tasks and achieves optimal results in multiple data sets. In the SD neck network, the present invention introduces a new information interaction mechanism, which strategically improves the effectiveness and efficiency of information fusion and transmission, avoids unnecessary losses, and introduces a detailed information acquisition mechanism. The two work together to significantly improve the detection capability of the model.

Claims

1. An efficient detection system based on full-level perception of SD network, including backbone network, neck network and head network, characterized in that: The neck network includes: a full-level feature sharing module (FLIS) and a detailed information extraction module (DFE); The full-level feature sharing module (FLIS) includes a first branch and a second branch, each of which includes a feature alignment module FAS, a feature fusion module FFS and a feature injection module; the input is first input into the feature alignment module FAS to make the input features of the same size, then the aligned output is input into the feature fusion module FFS for feature fusion, then the fused features are input into the feature injection module, and finally the output of the feature injection module is input into each level; The detailed information extraction module (DFE) includes a multi-scale attention fusion module (Multi scale attention fusion (Msaf)) and a triple feature processing module (SSFF). The multi-scale attention fusion module (Multi scale attention fusion (Msaf)) is used to obtain detailed information of features of different scales; the multi-scale attention fusion module (Msaf) includes a triple feature encoding module (Triple feature encoding module (TFE)) and a channel and position attention mechanism (CPAM). Among them, the triple feature encoding module (Triple feature encoding module (TFE)) is used to distinguish features of different sizes and obtain classification features, and the channel and position attention mechanism (CPAM) is used to combine detailed information with scale information to obtain more accurate output information; the triple feature processing module (SSFF) is composed of stack, 3D Conv and BN / SiLU. The triple feature processing module (SSFF) is used to enhance the network's multi-scale information extraction capability.

2. According to claim 1, an efficient detection system based on SD network full-level perception is characterized in that: The feature alignment module F-FAS of the first branch takes the S2, S3, S4, and S5 layers of the backbone network as input for fusion, selects the S4 layer as the base layer, and uses the average pooling (Avgpool) and bilinear (Bilinear) methods in the feature alignment module F-FAS of the first branch to merge {R S2 ,R S3 ,R S5 } and {R S4 }Perform feature alignment R S4 =1 / 4R, and obtain the input of the feature fusion (Fuse) module F-FFS adapted to the first branch; The feature alignment module S-FAS of the second branch takes the M3, M4, and M5 layers as input for fusion, selects the M5 layer as the base layer, and uses the average pooling method in the feature fusion module S-FFS of the second branch to combine {R M3 ,R M4 } and {R M5 }Perform feature alignment R M5 =1 / 8R, and obtain the input of the feature fusion (Fuse) module S-FFS adapted to the second branch.

3. According to claim 1, an efficient detection system based on SD network full-level perception is characterized in that: The feature fusion (Fuse) module F-FFS of the first branch includes a multi-layer online reparameterized convolution block (OREPA) and a separation module; the multi-layer online reparameterized convolution block (OREPA) removes all nonlinear layers and replaces them with linear scaling layers, and adds a BN layer at the end of the structure. The simplified sequential structure and parallel structure formulas of the multi-layer online reparameterized convolution block (OREPA) are as follows: Y=(W N (W N-1 *…W2*W1))*W=W e *X Among them, W j represents the weight of the jth layer, W m represents the weight of the mth branch, is the uniform weight; The input of the multi-layer online reparameterized convolution block (OREPA) is F align Provided by the output of the feature alignment module F-FAS of the first branch, the output of the multi-layer online re-parameterized convolution block (OREPA) is F fuse , and then the final output of the feature fusion (Fuse) module F-FFS of the first branch is obtained through the separation module; The feature fusion (Fuse) module S-FFS of the second branch consists of a stacked GAU module and a separation module. The stacked GAU module is based on transformer, which treats self-attention and linear gating units as a unified layer and shares calculations. The optimization formula of the stacked GAU module for the linear gating unit is as follows: in, Contains token-token attention weights, Z represents (s<<d) 4 , and κ represent two simple transformations, and b represents the relative position deviation; The structure of the feature fusion (Fuse) module S-FFS of the second branch includes: (1) first, obtaining the output of the alignment end from the feature alignment (Alignment) module S-FAS of the second branch; (2) inputting the output of the alignment end into the stacked GAU module to obtain the output of the fusion end; (3) finally, inputting the output of the fusion end into the separation module, and obtaining the final output of the feature fusion (Fuse) module F-FFS of the second branch through the separation module.

4. According to claim 1, an efficient detection system based on SD network full-level perception is characterized in that: The input end of the feature injection module includes the output part of the feature fusion module F-FFS of the first branch, the feature fusion module S-FFS of the second branch, and the input part of the multi-scale features of each layer; The output part of the feature fusion (Fuse) module of the first branch is fused with the information of each layer. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection (Inject) module. The double convolution alignment method is used to process the information in the input part of the multi-scale features of each layer. After the attention fusion is completed, the GAU is used to extract the information again. The output part of the feature fusion (Fuse) module of each layer is fused with the information of the second branch. The LAF module is introduced at the input end to align the input features through Avgpool and Bilinear operations as the final input of the feature injection (Inject) module. The double convolution alignment method is used to process the information in the input part of the multi-scale features of each layer. After the attention fusion is completed, GAU is used to extract the information again.

5. According to claim 1, an efficient detection system based on SD network full-level perception is characterized in that: The three-feature processing module (SSFF) organically links the high-dimensional information in the deep feature map with the detail information in the shallow feature map, constructs a scale space along the scale axis, and obtains images of the same resolution and different scales. The input formula of the three-feature processing module (SSFF) is: F σ (w,h)=G σ (w,h)×f(w,h) Among them, f(w,h) represents the two-dimensional input end of the image, F σ (w,h) is the sum of G σ (w,h) obtained by two-dimensional Gaussian filtering and smoothing; After unifying the resolution, the images are spliced, and the feature maps of different scales are superimposed. The scale sequence features are extracted using 3D convolution. The S3, M3, and P3 layers with detail information and key information are used as the reference layers of the three-feature processing module (SSFF). The specific structure includes: (1) Use single convolution to unify the number of channels of M4 and M5; (2) Use the nearest neighbor interpolation method to align the input features and perform multi-dimensional transformation on the input feature map; (3) Use the unsqueeze operation to convert the input three-dimensional feature map into a four-dimensional feature map; (4) Perform three-dimensional stitching of the four-dimensional feature map based on the depth information to effectively retain the acquired features; (5) Use three-dimensional convolution, normalization / SiLU to extract scale sequence features and output the results.

6. According to claim 1, an efficient detection system based on SD network full-level perception is characterized in that: The terminal outputs of the full-level feature sharing module (FLIS) are P3, P4 and P5, which are input into the multi-scale attention fusion module (Multi scale attention fusion (Msaf)); the multi-scale attention fusion module (Multi scale attention fusion (Msaf)) includes a triple feature encoding module (Triple feature encoding module (TFE) and a channel and position attention mechanism (CPAM), and uses a cross-stage local module (CSP) and a linked cross-stage local module (CCSP) as the triple feature encoding module (Triple feature encoding module (TFE) and channel and position attention mechanism (CPAM) interaction structure; the specific fusion process of the multi-scale attention fusion module (Multi scale attention fusion (Msaf)) is as follows: (1) Input the outputs of P4 and P3 into the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer, and then input the input of P3 and the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer into the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer to obtain the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer; (2) The output of P5, the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the fourth layer, and the output of the three-feature encoding module (TFE) and the cross-stage local module (CSP) of the third layer are respectively input to the fifth-layer linked cross-stage local module (CCSP), the fourth-layer linked cross-stage local module (CCSP) and the channel and position attention mechanism (CPAM); (3) The outputs of the fifth-layer linked cross-stage local module (CCSP), the fourth-layer linked cross-stage local module (CCSP) and the channel and position attention mechanism (CPAM) are input into the head part and detected in the head at the same time.

7. An efficient detection system based on SD network full-level perception according to claim 1 or 6, characterized in that: The structural functions of the triple feature encoding module (TFE) include: first, unifying the number of channels and keeping them consistent, and uniformly performing ConvBNSiLU operations on the three-scale feature maps; second, downsampling the large-scale feature map using Avg+MaxPooling and performing ConvBNSiLU operations to obtain a large-scale feature map output, directly performing ConvBNSiLU operations on the medium-scale feature map to obtain a medium-scale feature map output, and downsampling the small-scale feature map using the Nearest method and performing ConvBNSiLU operations to obtain a small-scale feature map output; finally, the large-scale feature map output, the medium-scale feature map output and the small-scale feature map are uniformly convolved and spliced ​​to obtain the final output part of the triple feature encoding module (Triple feature encoding module (TFE)).

8. An efficient detection system based on SD network full-level perception according to claim 1 or 6, characterized in that: The structural functions of the channel and position attention mechanism (CPAM) include: for different feature information in multiple channels, the channel and position attention mechanism (CPAM) combines detailed information with scale information; the input end of the channel and position attention mechanism (CPAM) is composed of the position attention information, that is, the output end of the three-feature processing module (SSFF) and the channel attention information, that is, the output end of the P3 layer TFE+CSP module; the channel attention information is used as the bottom layer input after pooling and convolution, and then enters the next layer together with the position information as the upper layer input, and finally the output of the channel and position attention mechanism (CPAM) enters the Head part.

9. According to claim 1, an efficient detection system based on SD network full-level perception is characterized in that: The backbone network uses EfficientRep, and the head network uses a three-output structure.

10. An efficient detection method based on full-level perception of SD network, comprising the following steps: Step 1: Input multiple features in the backbone network into the full-level feature sharing module (FLIS). The full-level feature sharing module (FLIS) includes a first branch and a second branch. Each branch includes a feature alignment module F-FAS, a feature fusion module F-FFS and a feature injection module. The feature alignment module F-FAS and the feature fusion module F-FFS of the first branch and the second branch are used to centrally process the features of each level, and then the feature injection modules of the first branch and the second branch are used to inject the centrally processed features back to each level, so that each level obtains global features, and the outputs of the first branch and the second branch are obtained respectively. Step 2: The output after the fusion of the first branch and each level is input into the three-feature processing module (SSFF) in the detailed information extraction module (DFE), and then the output of the three-feature processing module (SSFF) and the output after the fusion of the second branch and each level are input into the multi-scale attention fusion module (Multi scale attention fusion (Msaf)); Step 3: The output of the three-feature processing module (SSFF) and the output of the second branch after fusion with each level are input into the multi-scale attention fusion module (Msaf). In the multi-scale attention fusion module (Msaf), the triple feature encoding module (TFE) is used to save the detailed information in the features, and then the channel and position attention mechanism (CPAM) is used to divide the output features of different sizes. The three-output mode of the channel and position attention mechanism (CPAM) enables features of different sizes to be processed separately in the Head to obtain efficient prediction results.

Images