A lightweight object detection method and system based on structure reparameterization

Abstract

The application provides a lightweight target detection method and system based on structure reparameterization, comprising: obtaining a data sample to be detected and preprocessing; inputting the preprocessed data sample into a network inference model to output a target detection result, wherein the network inference model is a single-path network model converted from a multi-branch network training module by structure reparameterization, and the structure reparameterization mode is branch merging and convolution sequence merging. The training and inference stages are decoupled by the structure reparameterization mode, more semantic information and feature representation are obtained by using the multi-branch structure in the training stage, the weight parameters are iteratively updated, and the single-path structure with the training weight information is used in the inference stage, so that the inference speed of the model is accelerated and the model accuracy is ensured.

Description

Technical Field

[0001] This invention belongs to the field of deep learning computer vision technology, and in particular relates to a lightweight object detection method and system based on structure reparameterization. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The rapid development of deep learning technology has yielded remarkable achievements in computer vision, leveraging the generalization and feature extraction capabilities of neural networks. Object detection, as a computer vision task, plays a crucial role in fields such as intelligent security, industrial inspection, and medical diagnosis. While academia largely pursues high-precision models, these models typically involve large numbers of parameters and high computational demands, making them difficult to meet practical needs. From academia to industry, the deployment of neural network models is often constrained by computational resources and memory, leading to difficulties in deployment. Therefore, designing lightweight models for deployment on resource-constrained mobile and edge devices remains a significant research challenge.

[0004] Object detection is mainly divided into single-order and two-order detection methods. Among them, the SSD algorithm is a popular single-order object detection method. The original SSD algorithm uses the VGG16 network as its backbone network. However, the VGG16 network has a large number of model parameters, making it difficult to apply to lightweight object detection tasks. The RepVGG model is an improved algorithm proposed for VGG16. It combines the advantages of VGGNet and ResNet networks and uses a structure reparameterization approach to decouple the training and inference phases. It focuses more on the model's accuracy during training and more on the model's inference speed during inference. However, RepVGG pays less attention to the number of model parameters and theoretical computational cost. Therefore, to better suit lightweight detection tasks, the RepVGG network needs further improvement. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a lightweight target detection method and system based on structural reparameterization. It uses RWNet as the backbone network and extracts data feature information through an end-to-end training mechanism to achieve lightweight target detection.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solution: a lightweight target detection method based on structure reparameterization, comprising:

[0007] Acquire data samples of the target to be detected and perform preprocessing;

[0008] The preprocessed data samples are input into the network inference model to output the target detection results. The network inference model is a single-path network model converted from a trained multi-branch network training module using structural reparameter conversion. The structural reparameter conversion method is branch merging and convolution sequence merging.

[0009] A second aspect of the present invention provides a lightweight target detection system based on structure reparameterization, comprising:

[0010] Data acquisition module: Acquires data samples of the target to be detected and performs preprocessing;

[0011] Detection output module: Inputs the preprocessed data samples into the network inference model and outputs the target detection results. The network inference model is a single-path network model converted by the trained multi-branch network training module using structural reparameter conversion. The structural reparameter conversion method is branch merging and convolution sequence merging.

[0012] A third aspect of the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps described in the above method.

[0013] A fourth aspect of the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps described in the above method.

[0014] The above one or more technical solutions have the following beneficial effects:

[0015] In this invention, a multi-scale hybrid structure and a multi-scale receptive field module are used to enhance the feature extraction capability of the neural network and expand the model's receptive field to adapt to changes in image size, respectively. The multi-branch structure efficiently expresses the sparsity of features, aggregates local features of the model, and uses dilated convolutions with different dilation rates to expand the model's receptive field while keeping the number of parameters almost constant. A bottleneck structure is used to achieve dimensionality changes in the model, thereby reducing model parameters.

[0016] In this invention, the training and inference phases are decoupled by structural reparameterization. During the training phase, a multi-branch structure is used to obtain more semantic information and feature representations and iteratively update the weight parameters. At the same time, a single-path structure with training weight information is used during the inference phase, which accelerates the inference speed of the model while ensuring the model accuracy.

[0017] This invention can be effectively applied to target detection tasks. At the same time, the lightweight network structure designed is more convenient to be applied to embedded and mobile devices, has good robustness, and avoids the problems of complex models and large number of parameters that make it difficult to apply and deploy.

[0018] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is a flowchart of the detection process according to Embodiment 1 of the present invention;

[0021] Figure 2 This is a structural diagram of the RWNet-SSD model in Embodiment 1 of the present invention;

[0022] Figure 3 This is a structural diagram of some model components in Embodiment 1 of the present invention;

[0023] Figure 4 This is a structural diagram of the RWNet-SSD model after structural reparameterization in Embodiment 1 of the present invention;

[0024] Figure 5 This is an example diagram of model detection and machine grasping when applied to an assembly line in Embodiment 1 of the present invention;

[0025] Figure 6 This is an example diagram showing the training accuracy and loss of the model in Embodiment 1 of the present invention. Detailed Implementation

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0028] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0029] Example 1

[0030] like Figure 1 As shown, this embodiment discloses a lightweight target detection method based on structure reparameterization, including:

[0031] Acquire data samples of the target to be detected and perform preprocessing;

[0032] Extract features from the preprocessed data samples;

[0033] The extracted features are input into the network inference model to output the target detection result. The network inference model is a single-path network model transformed from a trained multi-branch network training module by using branch merging and convolution sequence merging and structural parameter refactoring.

[0034] In this embodiment, the collected data samples are labeled using the LabelImg software for manual labeling and formatted as VOC data. After acquiring the data sample set, the data is divided to construct training and testing sample sets. Through data preprocessing, the data sample set is expanded and data augmentation operations are performed. Different rotation angles are set, and data sample expansion is achieved through rotation angles (such as 30°, 270°, 330°), horizontal mirroring, and other operations. Then, data augmentation methods are used to adjust and enhance the brightness, color gamut, etc. of the image, employing methods such as photometric distortion and geometric distortion.

[0035] like Figure 2 As shown, this embodiment establishes a multi-branch network training module, namely the RWNet backbone network and SSD, using training and test samples. The RWNet backbone network is used for feature extraction and image category classification; the SSD (Single Shot MultiBox Detector) detector is used to locate the image samples, and a training sample set is used to obtain a trained deep learning model. Then, through structural reparameterization, training and inference are decoupled, and the network inference model is used as input to detect the test sample set to verify the category and location of the data. The RWNet model is placed within the SSD detector, configuring the deep learning model framework to achieve end-to-end model training and optimization iteration.

[0036] In this embodiment, RWNet includes five sequentially connected stage layers (stage0, stage1, stage2, stage3, and stage4) and two extras layers. Stage0 and stage1 adopt the structure of the RepVGG network (block_3×3), and the block_3×3 structure consists of a 3×3 convolution, a 1×1 convolution branch, and an identity residual branch. Stage2 and stage3 each consist of a multi-scale hybrid structure (MHS) and a multi-scale receptive field module (MRFM) structure. Stage4 contains a block_3×3 structure, and the SSD auxiliary convolutional layer contains two extras layers, all of which are composed of a block_3×3 structure.

[0037] The multi-scale hybrid structure (MHS) includes a block_1×1 structure, a bottleneck_1×3 structure, and a block_3×3 structure connected in sequence; the multi-scale receptive field module (MRFM) includes two dilated convolutional modules with different dilation rates and a block_1×1 structure.

[0038] Specifically, the input image size is 300×300×3. In stage 0, the block_3×3 module is used for convolutional feature extraction, and the number of channels changes from 3 to 48. At this time, the feature map size becomes 150×150. Then, in stage 1, it goes through two consecutive block_3×3 layers. In this stage, the number of channels remains unchanged at 48, but the feature map is modified to 75×75. The feature matrix at this time is [-1,48,75,75]. In stage 2, the feature map first passes through a block_1×1 structure, increasing the number of channels from 48 to 96, with a stride of 1 and a size of 75×75. Then it passes through a bottleneck_1×3 module, which is a bottleneck structure. Within the bottleneck_1×3 module, it first undergoes a convolution with a kernel of 1, which includes a residual branch and a 1×1 convolution branch, achieving dimensionality reduction. The feature map size is 75×75 with 48 channels. Next, it undergoes a convolution with a kernel of 3, which also includes a residual branch and a 1×1 convolution branch, achieving dimensionality increase. The feature map size is 75×75 with 96 channels. Then, it passes through dilated convolution modules with dilation rates of 3 and 5, and finally passes through another block_1×1 structure, resulting in a stage 2 output feature map of 38×38 with 96 channels, denoted as feature map f1.

[0039] In stage 3, the same operation as in stage 2 is used, resulting in a 19×19 feature map with 192 channels. The resulting feature map is denoted as f2.

[0040] In stage 4, a block 3×3 structure is used, increasing the number of channels from 192 to 1280. The feature map is now 10×10 with 1280 channels, denoted as feature map f3. In the extras layer, it first passes through a first block 3×3 module, resulting in a 5×5 feature map with 256 channels, denoted as feature map f4; then through another block 3×3 module, resulting in a 3×3 feature map with 256 channels, denoted as feature map f5. Similarly, in extras2, it first passes through a block 3×3 module, resulting in a 2×2 feature map with 256 channels, followed by a second block 3×3 module, resulting in a 1×1 feature map with 256 channels, denoted as feature map f6. The feature information extracted from stage2, stage3 and stage4 in the backbone network (i.e. f1-f3) is combined with SSD auxiliary convolutional layers (f4-f6) to form a total of 6 feature maps of different sizes. Candidate boxes are set in the 6 feature maps of different scales to achieve model prediction.

[0041] like Figure 3 As shown, the block_1×1 structure includes three branches: residual branch, Conv1×1 branch, and Conv1×1 branch. The outputs of the three branches are summed and then activated by the ReLU function.

[0042] The block_3×3 structure includes three branches: the residual branch, the Conv3×3 branch, and the Conv1×1 branch. The outputs of the three branches are summed and then activated by the ReLU function.

[0043] In this example, RWNet has K convolutional blocks (such as Conv1×1, Conv3×3, etc.), each of which consists of convolution, Batch Normalization (BN), and the ReLU activation function. Furthermore, based on the homogeneity of convolutions, a fusion of convolution (Conv(x)) and BN is used to accelerate the process. The BN layer does not require parameter updates during the inference phase, which helps to speed up model inference.

[0044] It should be noted that the convolutional structure blocks mentioned in this embodiment all adopt the acceleration method of fusion of convolution and BN.

[0045] Specifically, convolutional neural networks are stacked layer by layer, and the weight (ω) information is iteratively updated. The prediction data is then fitted using a bias b to obtain the final prediction vector. The convolution can be represented by the following formula:

[0046] Conv(x)=ω(x)+b (1)

[0047] In the Batch Normalization (BN) layer, to avoid affecting the feature distribution learned by the network, learnable transformation and reconstruction parameters γ and β are introduced to restore the feature distribution learned by the original network and achieve normalization of the network model. The formula is shown below:

[0048]

[0049] Where mean refers to the average value and var is the variance.

[0050] The formula for accelerating convolution and batch normalization is as follows:

[0051]

[0052] Where, ω fused for b fused for

[0053] like Figure 6 As shown, in this embodiment, candidate boxes are established in six feature maps f1-f6 to achieve feature map prediction at different scales, resulting in classification and regression predictions of the candidate boxes. The obtained predicted boxes are then sorted by score and filtered using non-maximum suppression (NMS). Simultaneously, the training process employs a weighted sum of location error (locatization loss, loc) and confidence error (conf) to define a loss function, as shown in the following formula:

[0054]

[0055] Among them, L conf (x,c) represents the confidence loss, L loc (x,l,g) represents the localization loss, N is the number of candidate boxes in the Ground Truth, α is the weighting coefficient that adjusts the ratio of confidence loss to localization loss, c is the confidence level, and l and g refer to the position parameters of the candidate boxes and the ground truth boxes, respectively.

[0056] Using the labeled candidate bounding boxes (prior bounding boxes) as the baseline, N is the number of candidate bounding boxes that match the ground truth, which is used to compare the error between the predicted situation and the actual situation.

[0057] Where L loc (x,l,g) can be expressed by the formula:

[0058]

[0059] Where i represents the i-th prior box (candidate box), and j represents the j-th ground truth box (real box); The value is 0 or 1, used to determine whether the i-th predicted box matches the j-th ground truth box with respect to category k; smooth L1 This is the regression loss, used to calculate the regression loss. For the prediction box, Refers to the actual frame.

[0060] Where L conf (x,c) can be expressed by the formula:

[0061]

[0062] in, This indicates the match between the predicted bounding box i and the true bounding box j with respect to category p. This represents the predicted probability of category p corresponding to the i-th candidate box.

[0063] In model prediction, an end-to-end model training approach is adopted, with the SGD model optimizer set as the training optimizer and the iterative learning rate set. The initial learning rate is set to 1e-4, and the learning rate is reduced to 1e-5 at 60k. Simultaneously, default boxes are generated for each pixel in feature maps at six different scales, and NMS is used to filter them, obtaining positive and negative samples and the training model loss, thus achieving the localization of data samples.

[0064] In this embodiment, the approach of branch merging and convolution sequence merging is adopted, and the structural reparameterization method is used to decouple the training stage and the inference stage. The training stage adopts a multi-branch structure to obtain high accuracy, while the inference stage adopts a single-path structure to accelerate the model inference speed.

[0065] like Figure 4 As shown, specifically, branch merging leverages the additivity of convolution. When two convolutional kernels have the same size and shape, the additivity of convolution is satisfied. Therefore, block_1×1 can be transformed into a Conv1×1 structure. Simultaneously, in the block_3×3 structure, a 1×1 convolution can be padded with zeros to become a convolution with a kernel size of 3. Therefore, block_3×3 can use this method to achieve convolution addition. During the inference phase, the multi-branch structure of block_3×3 is transformed into a Conv3×3 structure.

[0066] The convolution sequence merging is specifically as follows: In a convolution with kernel F, if the input feature map I, the output feature map O can be represented as:

[0067]

[0068] Where b is the bias.

[0069] If, after two consecutive convolutions, the input feature map is I, the kernel size is F1, and the bias is b1, then the output feature map is O1. After the second consecutive convolution, the input feature map becomes O1, the kernel size is F2, and the bias is b2, resulting in an output feature map of O2, as shown in the formula:

[0070]

[0071]

[0072] Therefore, after combining two sets of consecutive convolutions, it can be represented as:

[0073]

[0074] It can then be called: This allows for the merging of convolutional sequences.

[0075] like Figure 5 The image shown is an example of model detection and machine capture when the model of this embodiment is applied to the pipeline. After experimental verification, the average detection accuracy (mAP) of this solution is 94.61%, while the number of parameters is 8.71M, the computational load is 3.21GFLOPs, and the number of frames transmitted per second (FPS) reaches 101FPS.

[0076] Example 2

[0077] The purpose of this embodiment is to provide a lightweight target detection system based on structure reparameterization, including:

[0078] Data acquisition module: Acquires data samples of the target to be detected and performs preprocessing;

[0079] Detection output module: Inputs the preprocessed data samples into the network inference model and outputs the target detection results. The network inference model is a single-path network model converted by the trained multi-branch network training module using structural reparameter conversion. The structural reparameter conversion method is branch merging and convolution sequence merging.

[0080] Example 3

[0081] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0082] Example 4

[0083] The purpose of this embodiment is to provide a computer-readable storage medium.

[0084] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0085] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0086] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0087] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A lightweight target detection method based on structural reparameterization, characterized in that, include: Acquire data samples of the target to be detected and perform preprocessing; The preprocessed data samples are input into the network inference model to output the target detection results. The network inference model is a network model with a single-path structure that is converted from a trained multi-branch network training module by structural reparameter conversion. The structural reparameter conversion method is branch merging and convolution sequence merging. The multi-branch network training module, namely the RWNet backbone network and SSD, is established through training and test samples. The RWNet backbone network is used for feature extraction and image category classification. The SSD detector is used to locate the image samples and train the sample set to obtain the trained deep learning model. Then, through structural reparameterization, the training and inference are decoupled. The network inference model is used as input to detect the test sample set to verify the category and location of the data. The multi-branch network training module includes five stage layers and an SSD-assisted convolutional layer. Specifically, stage 0, stage 1, and stage 4 adopt the RepVGG network structure blocks; stage 2 and stage 3 are both composed of sequentially connected multi-scale hybrid structures and multi-scale receptive field modules. The multi-scale hybrid structure includes a block_1×1 structure, a bottleneck_1×3 module, and a block_3×3 structure; the multi-scale receptive field module includes dilated convolutions with different dilation rates and a block_1×1 structure; the SSD-assisted convolutional layer includes two extras layers, and the extras layer includes two block_3×3 structures. The branch merging method described above is as follows: The block_1×1 in the trained multi-branch network training module can be converted into a Conv1×1 structure; In the trained multi-branch network training module, the block_3×3 is converted into a Conv3×3 structure; The 1×1 convolution in block_3×3 of the trained multi-branch network training module is padded with zeros to become a convolution with a kernel size of 3, thus achieving convolution addition.

2. The lightweight target detection method based on structural reparameterization as described in claim 1, characterized in that, The block_1×1 structure consists of three branches: 1×1 convolution, residual structure, and 1×1 convolution; the block_3×3 structure consists of three branches: 1×1 convolution, 3×3 convolution, and residual structure.

3. The lightweight target detection method based on structure reparameterization as described in claim 1, characterized in that, The method of merging convolutional sequences after two consecutive convolutions is as follows: Where I is the input feature map of the first convolution, F1 is the convolution kernel of the first convolution, b1 is the bias of the first convolution, F2 is the convolution kernel of the second convolution, b2 is the bias of the second convolution, and is the output feature map of the second convolution.

4. The lightweight target detection method based on structure reparameterization as described in claim 1, characterized in that, The feature information extracted from stage 2, stage 3, and stage 4 is combined with the output of each extras layer in the SSD auxiliary convolutional layer as the input to the SSD detector to obtain the predicted category and location.

5. A lightweight target detection system based on structural reparameterization, characterized in that, include: Data acquisition module: Acquires data samples of the target to be detected and performs preprocessing; Detection output module: Inputs the preprocessed data samples into the network inference model and outputs the target detection results. The network inference model is a single-path network model converted by the trained multi-branch network training module using structural reparameter conversion. The structural reparameter conversion method is branch merging and convolution sequence merging. The multi-branch network training module, namely the RWNet backbone network and SSD, is established through training and test samples. The RWNet backbone network is used for feature extraction and image category classification. The SSD detector is used to locate the image samples and train the sample set to obtain the trained deep learning model. Then, through structural reparameterization, the training and inference are decoupled. The network inference model is used as input to detect the test sample set to verify the category and location of the data. The multi-branch network training module includes five stage layers and an SSD-assisted convolutional layer. Specifically, stage 0, stage 1, and stage 4 adopt the RepVGG network structure blocks; stage 2 and stage 3 are both composed of sequentially connected multi-scale hybrid structures and multi-scale receptive field modules. The multi-scale hybrid structure includes a block_1×1 structure, a bottleneck_1×3 module, and a block_3×3 structure; the multi-scale receptive field module includes dilated convolutions with different dilation rates and a block_1×1 structure; the SSD-assisted convolutional layer includes two extras layers, and the extras layer includes two block_3×3 structures. The branch merging method described above is as follows: The block_1×1 in the trained multi-branch network training module can be converted into a Conv1×1 structure; In the trained multi-branch network training module, the block_3×3 is converted into a Conv3×3 structure; The 1×1 convolution in block_3×3 of the trained multi-branch network training module is padded with zeros to become a convolution with a kernel size of 3, thus achieving convolution addition.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the lightweight target detection method based on structure reparameterization as described in any one of claims 1-4.

7. A processing apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the lightweight target detection method based on structure reparameterization as described in any one of claims 1-4.

Images