A target detection model training method, a target detection method, a device, and a medium
By reparameterizing and pruning convolutional layers according to channel importance criteria, a new model is constructed, which solves the problem of high computational cost of convolutional neural networks and achieves efficient object detection on mobile devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG INSPUR SCI RES INST CO LTD
- Filing Date
- 2023-02-16
- Publication Date
- 2026-06-16
AI Technical Summary
Existing convolutional neural networks are computationally expensive for object detection tasks, exceeding the capabilities of mobile devices, and cannot be delegated to cloud servers, resulting in a waste of computing and storage resources.
By reparameterizing and pre-defined channel importance criteria, a new model is constructed to identify channels to be pruned in the standard pruned convolutional layer, retaining important channels and reducing computational and storage resource consumption.
While ensuring the same number of input and output channels in the deep convolutional layer, it improves target detection efficiency, saves computing and storage resources, and is suitable for AIoT edge devices.
Smart Images
Figure CN116051961B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of target detection technology, and in particular to a target detection model training method, target detection method, device and medium. Background Technology
[0002] Since deep learning algorithms achieved remarkable success in image tasks, deep convolutional neural networks have gradually replaced traditional statistical learning as the mainstream framework and method in computer vision, and have been increasingly applied to various scenarios such as mobile phones, autonomous driving, and smart hospitals. However, many convolutional neural networks require high computational costs, and the hardware resources in many scenarios are strictly limited by form factor, battery capacity, and heat dissipation. These computational costs far exceed the capabilities of most mobile devices, such as vehicles, mobile phones, IoT devices, and AIoT edge devices. Moreover, these computational tasks cannot be delegated to cloud servers because they may be highly sensitive to latency (e.g., autonomous driving) or privacy (e.g., healthcare).
[0003] Currently, for object detection tasks, how to reduce the unnecessary computation, parameter count, and memory consumption of network models during task execution, so that the models can perform object detection on images more efficiently while generating less memory consumption, thereby effectively saving computational and storage resources, is the problem that baseband solutions are currently addressing. Summary of the Invention
[0004] In view of this, the purpose of this application is to provide a training method for an object detection model, an object detection method, an apparatus, and a medium that, while ensuring the consistent number of input and output channels in the deep convolutional layers, avoids pruning important channels in the pointwise convolutional layers, thereby improving the efficiency of object detection and effectively saving computational and storage resources. The specific solution is as follows:
[0005] Firstly, this application discloses a method for training an object detection model, including:
[0006] The initial model is trained using the first image sample training set to obtain the trained model, and the trained model is then reparameterized to obtain the reparameterized model; the reparameterized model includes a depthwise convolutional layer and a pointwise convolutional layer.
[0007] The channels to be pruned in each convolutional layer of the reparameterized model are determined by using a preset channel importance discrimination criterion, and the reparameterized model is pruned based on the channels to be pruned to obtain the pruned model.
[0008] The first output channel of the deep convolutional layer and the second output channel of the pointwise convolutional layer are determined respectively; the first output channel is an output channel other than the first target channel, the first target channel is the output channel of itself and the input channel with the same channel number as itself, which are both channels to be pruned; the second output channel is an output channel other than the second target channel, and the second target channel has the same channel number as the first target channel.
[0009] A new model is constructed based on the pruned model to obtain a target detection model; wherein the new model retains the first output channel and the second output channel.
[0010] Optionally, after pruning the reparameterized model based on the channels to be pruned to obtain the pruned model, the method further includes:
[0011] The pruned model is trained using the second image sample training set to obtain the target pruned model;
[0012] Accordingly, the step of constructing a new model based on the pruned model to obtain the target detection model includes:
[0013] A new model is constructed based on the target-pruned model to obtain the final target detection model.
[0014] Optionally, the step of constructing a new model based on the pruned model to obtain the target detection model includes:
[0015] The target pruned model and the new model are tested using an image sample test set to obtain their accuracy.
[0016] If the accuracy of the pruned target model is consistent with that of the new model, then repeat the pruning and model building process until the new model meets the preset performance indicators, and then use the new model as the final target detection model.
[0017] Optionally, before training the pruned model using the second image sample training set to obtain the target pruned model, the method further includes:
[0018] Add a batch normalization layer to the pruned model.
[0019] Optionally, the initial model includes a convolutional block structure; the convolutional block structure includes a depthwise convolutional layer, a pointwise convolutional layer, a batch normalization layer, and a compression excitation module;
[0020] Accordingly, the preset channel importance discrimination criterion is: the absolute value of the channel weight / the absolute value of the weight parameter of the batch normalization layer / the size of the first-order Taylor term of the channel minus the size of the Flops regularization term of each layer / the size of the output of the compression excitation module.
[0021] Optionally, before training the initial model using the first image sample training set to obtain the trained model, the method further includes:
[0022] Add a regularization factor to the gradient of the weight parameters of the batch normalized layer.
[0023] Optionally, determining the channels to be pruned corresponding to each convolutional layer in the reparameterized model using a preset channel importance criterion includes:
[0024] The channel importance of each channel in each convolutional layer is determined using a preset channel importance discrimination criterion;
[0025] The channel importance of all the depthwise convolutional layers and all the pointwise convolutional layers is sorted to obtain a first sorting result for all the depthwise convolutional layers and a second sorting result for all the pointwise convolutional layers.
[0026] Based on the first sorting result and the second sorting result, the channels to be pruned for the depthwise convolutional layer and the pointwise convolutional layer are determined respectively.
[0027] Secondly, this application discloses a target detection method, including:
[0028] Acquire the image to be detected;
[0029] The image to be detected is input into the target detection model to obtain the target detection result; wherein, the target detection model is obtained based on the target detection model training method described above.
[0030] Thirdly, this application discloses a target detection model training device, comprising:
[0031] The model training module is used to train the initial model using the first image sample training set to obtain the trained model;
[0032] The reparameterization module is used to reparameterize the trained model to obtain a reparameterized model; the reparameterized model includes depthwise convolutional layers and pointwise convolutional layers.
[0033] The model pruning module is used to determine the channels to be pruned corresponding to each convolutional layer in the reparameterized model using a preset channel importance discrimination criterion, and to prune the reparameterized model based on the channels to be pruned, so as to obtain the pruned model.
[0034] The channel determination module is used to determine the first output channel of the depthwise convolutional layer and the second output channel of the pointwise convolutional layer respectively; the first output channel is an output channel other than the first target channel, the first target channel is the output channel of itself and the input channel with the same channel number as itself, which are all channels to be pruned; the second output channel is an output channel other than the second target channel, and the second target channel has the same channel number as the first target channel.
[0035] The model building module is used to build a new model based on the pruned model to obtain a target detection model; wherein the new model retains the first output channel and the second output channel.
[0036] Fourthly, this application discloses an electronic device, including a memory and a processor, wherein:
[0037] The memory is used to store computer programs;
[0038] The processor is configured to execute the computer program to implement the aforementioned target detection model training method, and / or the aforementioned target detection method.
[0039] Fifthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned target detection model training method, and / or the aforementioned target detection method.
[0040] As can be seen, this application first trains an initial model using a first image sample training set to obtain a trained model, and then reparameterizes the trained model to obtain a reparameterized model. The reparameterized model includes a depthwise convolutional layer and a pointwise convolutional layer. Then, a preset channel importance discrimination criterion is used to determine the channels to be pruned corresponding to each convolutional layer in the reparameterized model, and the reparameterized model is pruned based on the channels to be pruned to obtain a pruned model. Furthermore, the first output channel of the depthwise convolutional layer and the second output channel of the pointwise convolutional layer are determined respectively. The first output channel is an output channel other than the first target channel. The first target channel is itself and the input channel with the same channel number as itself, which are both output channels of the channels to be pruned. The second output channel is an output channel other than the second target channel. The second target channel has the same channel number as the first target channel. Then, a new model is constructed based on the pruned model to obtain a target detection model. The new model retains the first output channel and the second output channel. In other words, this application determines the model to be pruned based on the reparameterized model, performs pruning, and constructs a new model based on the pruned model to obtain the target detection model. For deep convolutional layers, the final pruned layers are only the layers themselves and the output channels of the input channels with the same channel number as the layers themselves, which are all channels to be pruned. While ensuring that the number of input and output channels of the deep convolutional layers is consistent, this avoids pruning important channels in the pointwise convolutional layers, thereby improving the efficiency of target detection and effectively saving computational and storage resources. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0042] Figure 1 A flowchart of a target detection model training method provided in this application;
[0043] Figure 2 A schematic diagram of a reparameterized network structure provided in this application;
[0044] Figure 3 A schematic diagram illustrating the construction of a specific new model provided in this application;
[0045] Figure 4 A flowchart illustrating a specific object detection model training method provided in this application;
[0046] Figure 5 A schematic diagram of a specific initial model structure provided for this application;
[0047] Figure 6 This application provides a schematic diagram of a specific convolutional block structure;
[0048] Figure 7 A flowchart of a target detection method provided in this application;
[0049] Figure 8 This application provides a schematic diagram of the structure of a target detection model training device.
[0050] Figure 9 This application provides a structural diagram of an electronic device. Detailed Implementation
[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0052] Currently, for object detection tasks, reducing unnecessary computation, parameter count, and memory consumption during network model execution, enabling more efficient object detection while minimizing memory usage, and thus effectively saving computational and storage resources, is a key challenge for baseband processors. To address this, this application provides an object detection model training scheme that, while maintaining a consistent number of input and output channels in the deep convolutional layers, avoids pruning important channels in the pointwise convolutional layers, thereby improving object detection efficiency and effectively saving computational and storage resources.
[0053] See Figure 1 As shown in the figure, this application discloses a method for training an object detection model, including:
[0054] Step S11: Train the initial model using the first image sample training set to obtain the trained model, and reparameterize the trained model to obtain the reparameterized model; the reparameterized model includes a depthwise convolutional layer and a pointwise convolutional layer.
[0055] It is understood that the initial model is a network model built based on the object detection task, and is a multi-branch parallel network suitable for object detection tasks. The initial model includes a convolutional block structure; in one embodiment, the convolutional block structure includes depthwise convolutional layers, pointwise convolutional layers, and batch normalization layers. Furthermore, it may also include a compressed excitation module and a shortcut (i.e., direct connection) structure.
[0056] It's important to note that reparameterized network architectures are a type of efficient neural network. A reparameterized network architecture uses different network structures for inference and training, but outputs the same network result. In real-world scenarios, training resources are generally relatively abundant, so multi-branch parallel network structures are used. However, when deploying the network for inference, memory size and performance must be considered. Therefore, reparameterized network architectures have emerged. RepVGG and MobileOne are both reparameterized network architectures. RepVGG's training structure consists of a 3x3 convolutional layer with a batch normalization (BN) layer, a 1x1 convolutional layer with a BN layer, and a separate BN layer. During inference, a reparameterization algorithm merges these four layers into a single 3x3 convolutional layer. Building upon this, the proposed MobileOne network architecture differs from RepVGG. MobileOne employs depthwise separable convolutions, resulting in a two-layer structure during training. The upper layer uses four 3x3 convolutional layers with Batch Normalization (BN), one 1x1 convolutional layer with BN, and a separate BN layer. The lower layer uses four 1x1 convolutional layers with BN and a separate BN layer. During inference, the upper layer becomes a 3x3 convolutional layer, and the lower layer becomes a 1x1 convolutional layer. For example, see [link to documentation]. Figure 2 As shown, Figure 2 This is a schematic diagram of a reparameterized network structure provided in an embodiment of this application. Figure 2 In the middle (a), the RepVGG structure is shown. Figure 2 (b) shows the MobileOne structure. The convolutional block structure in the initial model of this application can adopt either the RepVGG structure or the MobileOne structure. Furthermore, improvements can be made by adding compressed excitation modules and shortcut (i.e., direct connection) structures to enhance model performance.
[0057] Furthermore, embodiments of this application can add a regularization factor to the gradient of the weight parameters of the batch normalization layer. That is, this application trains the initial multi-branch parallel network to achieve optimal accuracy, inserting a regularization factor for the BN layer during training to make the weight parameter γ of the BN layer closer to 0. After training, a reparameterization process is performed to merge the parameters. Moreover, embodiments of this application can also add an L2 regularization factor to the channel weights.
[0058] Step S12: Determine the channels to be pruned corresponding to each convolutional layer in the reparameterized model using a preset channel importance discrimination criterion, and prune the reparameterized model based on the channels to be pruned to obtain the pruned model.
[0059] In the first embodiment, the preset channel importance discrimination criterion is the absolute value of the channel weight.
[0060] In the second embodiment, the preset channel importance discrimination criterion is the absolute value of the weight parameters of the batch normalization layer.
[0061] In the third implementation, the preset channel importance criterion is the size of the channel's first-order Taylor term minus the size of each layer's Flops regularization term.
[0062] In the fourth embodiment, the preset channel importance criterion is the size of the output of the compression excitation module.
[0063] Furthermore, in a specific implementation, the channel importance of each channel in each convolutional layer can be determined using a preset channel importance discrimination criterion; the channel importance of all the depth convolutional layers and all the pointwise convolutional layers are sorted to obtain a first sorting result for all the depth convolutional layers and a second sorting result for all the pointwise convolutional layers; and the channels to be pruned for the depth convolutional layers and the pointwise convolutional layers are determined based on the first sorting result and the second sorting result, respectively.
[0064] In one implementation, for a deep convolutional layer, the output channel with the lowest channel importance at a first preset ratio is determined as the channel to be pruned. For a pointwise convolutional layer, the output channel with the lowest channel importance at a second preset ratio is determined as the channel to be pruned. The first preset ratio may be the same as or different from the second preset ratio.
[0065] In another implementation, for deep convolutional layers, output channels with channel importance below a first preset threshold are identified as channels to be pruned. For pointwise convolutional layers, output channels with channel importance below a second preset threshold are identified as channels to be pruned. The first preset threshold may be the same as or different from the second preset threshold.
[0066] Furthermore, the number of channels to be pruned in a deep convolutional layer can be less than the number of channels to be pruned in a pointwise convolutional layer. It should be noted that deep convolutional layers are more important than pointwise convolutional layers, and the embodiments of this application retain more deep convolutional layers, resulting in better model performance.
[0067] In other words, this embodiment uses preset channel importance criteria to prune the trained network structure. These criteria can be: the absolute value of each channel weight, the absolute value of the weight parameter γ in each channel's BN layer, the size of the first-order Taylor term minus the size of the Flops regularization term in each layer, and the size of the output of the compression activation module in the model. Each of these four criteria can be used for pruning, and the optimal model is selected. Furthermore, after reparameterization, the BN layer is merged into the convolutional layer, so the first γ value obtained is the γ value in the BN layer before reparameterization. Additionally, the four criteria for each layer need to be normalized to ensure that each layer is pruned according to the same standard. Each standard is normalized and arranged according to the size of the 3*3 and 1*1 layers in the network structure. Because the 3*3 depthwise convolutional layer is mainly used to extract information from the image while the 1*1 pointwise convolutional layer is mainly used to change the channels, the two have different importance. The smaller the ranking of the channel, the more likely it is to be pruned. A certain proportion or a certain value can be selected to select the threshold to determine the channels to be pruned. Then, the parameters of the channels to be pruned are all set to 0.
[0068] In other embodiments, the importance of each convolutional layer can also be determined. The number of channels to be pruned in the first convolutional layer is higher than that in the second convolutional layer, and the importance of the first convolutional layer is lower than that in the second convolutional layer. That is, the number of channels to be pruned is determined based on the principle that the higher the importance of the convolutional layer, the fewer channels are removed.
[0069] Furthermore, in a specific implementation, the pruned model can be trained using a second image sample training set to obtain the target pruned model.
[0070] In addition, before training the pruned model using the second image sample training set to obtain the target pruned model, the method further includes adding a batch normalization layer to the pruned model.
[0071] In other words, the embodiments of this application train and fine-tune the model after pruning to improve accuracy. The model after the first pruning does not have a BN layer, so a BN layer needs to be added and fine-tuned until the model reaches optimal accuracy.
[0072] Step S13: Determine the first output channel of the depth convolutional layer and the second output channel of the pointwise convolutional layer respectively; the first output channel is an output channel other than the first target channel, the first target channel is itself and the input channel with the same channel number as itself are both output channels to be pruned, the second output channel is an output channel other than the second target channel, the second target channel has the same channel number as the first target channel.
[0073] Step S14: Construct a new model based on the pruned model to obtain a target detection model; wherein the new model retains the first output channel and the second output channel.
[0074] In a specific implementation, a new model can be constructed based on the pruned target model to obtain the final target detection model. That is, the corresponding parameters of the pruned target model are obtained to construct a new model.
[0075] Furthermore, in this embodiment, the target pruned model and the new model can be tested using an image sample test set to obtain the accuracy of the target pruned model and the new model. If the accuracy of the target pruned model is consistent with that of the new model, the pruning and model construction are repeated until the constructed new model meets the preset performance index, and the new model is used as the final target detection model.
[0076] It should be noted that the pruned model becomes a channel-sparsed model, requiring the construction of a new model that does not include the channels whose channel parameters were previously set to 0. The first 3x3 convolution of a depthwise separable convolution layer needs to adhere to the alignment principle, meaning that the input channel that is pruned, along with the input channel with the same channel index, becomes the output channel of the channel to be pruned. For example, see... Figure 3 As shown, Figure 3 This is a schematic diagram illustrating a specific new model construction disclosed in this application embodiment. Arabic numerals represent channel numbers, and grayscale squares represent channels to be pruned. Channels 3, 6, and 7 are pruned. Channels 1, 2, 4, and 5 do not require pruning due to alignment principles and are therefore retained. Furthermore, the 1x1 convolutional layers in the new model do not require this process; channel pruning can be performed directly. This application embodiment performs iterative pruning: after testing the accuracy of the new model, one pruning cycle is completed. After one pruning cycle, pruning, model fine-tuning, new model construction, and testing can be repeated until the model's parameter count, computational load, memory consumption, and accuracy all meet preset performance indicators, at which point pruning stops. The final pruned new model is deployed to an AIoT edge device for object detection.
[0077] It should be noted that the purpose of this application's embodiments is to accelerate neural network inference speed by reducing the computational load, parameter count, and memory consumption of the model, thereby enabling the model to be deployed in AIoT edge devices for object detection. A structured pruning scheme based on a reparameterized network structure is proposed. This embodiment can also be applied to other image tasks, such as image classification and object segmentation. By pruning unimportant parameters in the network model, unnecessary computational load, parameter count, and memory consumption during image tasks are reduced. This allows the pruned network model to perform image classification, object detection, or object segmentation more efficiently, while generating less memory consumption, thus effectively saving computational and storage resources and enabling the model to be better applied to AIoT edge devices.
[0078] For example, see Figure 4 As shown, Figure 4 A flowchart illustrating a specific object detection model training method provided in this application embodiment. It specifically includes the following steps:
[0079] First, construct the initial model: Utilize a multi-branch network structure to build an object detection model. See [link / reference] Figure 5 As shown, Figure 5 This is a schematic diagram of a specific initial model structure provided in an embodiment of this application. Figure 5(a) shows the original SSD structure, and (b) shows the improved initial model structure of this application embodiment. The SSD network mainly consists of VGG16 as the backbone network, plus four additional convolutional layers. Each convolutional layer consists of a 1*1 convolutional layer (used to change the number of channels) and a 3*3 convolutional layer (used to extract features). Classification and regression extract a total of 6 layers from the SSD network structure, namely the 512*38*38 feature output map of convolutional layer 4-3, the 1024*19*19 feature map of convolutional layer 7, and the second layer of the four additional convolutional layers, namely the 512*10*10 feature map of convolutional layer 8-2, the 256*5*5 feature map of convolutional layer 9-2, the 256*3*3 feature map of convolutional layer 10-2, and the 256*1*1 feature map of convolutional layer 11-2. Shallow feature maps are used to extract small targets, while deep feature maps are used to extract larger targets. This application first replaces the entire VGG16 with the MobileOne-S0 structure, that is, replaces the SSD backbone network with the MobileOne-S0 structure. In MobileOne-S0, stage 7 is an Avgpool layer, so only stage 6 is extracted. Additionally, the four extra convolutional layers are replaced with the reparameterized structure from MobileOne, consisting of four depthwise separable convolutions, each of which can be divided into a 3x3 depthwise convolution and a 1x1 pointwise convolution. Furthermore, since the model is applied to pedestrian object detection, the camera targets large objects such as pedestrians. Therefore, considering the specificity of this task, in the classification and regression extraction of the six layers in the SSD network structure, the 38x38 feature output map is first removed, and a 19x19 map is used as the first feature output layer. Since the second layer in the original SSD structure has 1024 channels, the number of channels in stage 6 of MobileOne-S0 is changed from 512 to 1024. The first feature output layer becomes the 256*19*19 feature map of stage 5 of MobileOne-S0, and the second layer becomes the 1024*19*19 feature map of stage 6 of MobileOne-S0. The following four layers are all feature output maps of the second layer of the extra layers, namely the 512*5*5 feature map of extra layer 7-2, the 256*3*3 feature map of extra layer 8-2, the 256*2*2 feature map of extra layer 9-2, and the 256*1*1 feature map of extra layer 10-2. Figure 5 In this diagram, X1 to X6 represent feature maps. Taking convolutional layer 7-2 as an example, 7-2 means that this layer takes the output feature map of the second convolutional layer. Furthermore, the structure is modified for each convolutional block in a multi-branch structure as follows: Figure 6 Each convolutional block adds a compression excitation module and a shortcut structure. Figure 6 (Plus sign) The shortcut structure is triggered only when the shapes of the preceding and following feature maps are exactly the same.
[0080] Further, the model is trained: the multi-branch parallel network before reparameterization is trained. During training, a regularization factor is inserted into the BN layer. Specifically, a λ*sign(γ) (λ defaults to 0.0001) is added to the gradient of the weight parameter γ in the BN layer, so that γ approaches 0 when updating parameters. After training, the reparameterization process is performed. Parameters are then merged.
[0081] Furthermore, the criteria for determining channel importance are as follows: There are four criteria for determining channel importance through pruning: the absolute value of the weight of each channel, the absolute value of the weight parameter γ of each channel's BN layer, the output of the compressed excitation module, and the size of the first-order Taylor term of each channel minus the size of the Flops regularization term of each convolutional layer. The first three criteria can be directly extracted. The formula for the fourth criterion, the first-order Taylor term of the channel, is:
[0082]
[0083] Among them, h i Let C represent the output of the i-th convolutional layer, and let θC / θh represent the number of channels. i The gradients output by the convolutional layer can only be obtained during backpropagation, so an additional training epoch without parameter updates is required. Furthermore, the FLOPS regularization term is formulated as: FLOPS = C in *C out *W*H*k*k*λ; where C in The number of input channels for the convolutional layer, C out Here, represents the number of output channels for the convolutional layer, W and H represent the width and height of the output image during network operation, k represents the size of the convolutional kernel, and λ is set to 0.001. The absolute value of the weight parameter γ for each channel's BN layer needs to be calculated before reparameterization. Each channel can be pruned using its own four criteria, and the optimal model is selected. After calculating the size of each criterion, normalization is performed for each layer using the following formula:
[0084]
[0085] Among them, Z k Z represents the value of any standard for the channel to be normalized, n represents the total number of channels, and Z represents the value of any standard. i This represents the value of this standard for the i-th channel.
[0086] Furthermore, model pruning: All four criteria are normalized based on the 3x3 and 1x1 layers in the network structure, and then arranged by size. Channels with smaller rankings are considered pruningable. A certain percentage can be chosen; for example, to prune 10% of the channels, the total number of rankings multiplied by 10% is selected as the pruningable channels, or a fixed value such as 10 can be chosen.-5 As a threshold, the standards for the two convolutional layers are calculated separately. The weights and biases of the channels that can be pruned are set to 0. The mask of each layer is saved. The mask and the number of channels are the same, consisting of 1 and 0. 1 represents unpruned channels and 0 represents pruned channels.
[0087] Further, model fine-tuning: The pruned model is trained and fine-tuned to improve accuracy. At this stage, the model is a reparameterized structure with either 3x3 convolutional layers or a combination of 3x3 and 1x1 convolutional layers, without a batch normalization (BN) layer. Therefore, the model after the first pruning needs a BN layer added before further fine-tuning until optimal accuracy is achieved. The formula for the BN layer is:
[0088]
[0089] Here, M is the input to the previous layer, W is the weight of the convolutional layer, μ, γ, σ, and β are the mean, weight, standard deviation, and bias of the BN layer, respectively, and i represents the i-th channel. To ensure that the output of the BN layer is consistent before and after insertion, μ, γ, and σ are set to 0, 1, and 1, respectively, and β is set as the bias of the convolutional layer, with the convolutional layer bias set to none. This will make the output consistent with the previous result.
[0090] Furthermore, a new model is constructed: Since the first 3x3 convolutional layer of the depthwise separable convolution is a depthwise convolution, the number of input channels and the number of output channels must be consistent. Therefore, an alignment principle is implemented when constructing the channels. Figure 3 For example, the specific algorithm is as follows:
[0091] ① First, find the indices of the reserved channels in the input and output channels to obtain two index sets, namely 0,2,5,8,9 and 0,1,4,8,9.
[0092] ② Find the union of the two index sets mentioned above, i.e., 0, 1, 2, 4, 5, 8, 9. This is the channel that needs to be constructed.
[0093] ③ Find the difference between this union and the two index sets mentioned above, namely 1,4 and 2,5. These two difference sets are the indices of the channels whose masks will be set to 0 after the new model is built.
[0094] ④ Find the indices of the new model corresponding to the two difference sets, namely 1,3 and 2,4.
[0095] ⑤ Construct a new model based on ②, and then set the masks of the corresponding channels, namely 1, 3 and 2, 4, to 0 based on the results of ④.
[0096] In addition, the 1*1 convolutional layer in the new model does not need to consider this process; the channels can be pruned directly.
[0097] The specific algorithm for constructing the new model is as follows:
[0098] 1) Set the mask corresponding to the index in step ③ above to 1 in each layer of the mask;
[0099] 2) Keep the channels with 1s in each mask layer and remove the channels with 0s;
[0100] 3) Set the mask corresponding to the index in step ④ above to 0 in each layer of the mask.
[0101] Furthermore, this application employs iterative pruning: The accuracy of the new model is tested; if it matches the accuracy after fine-tuning, pruning is completed; otherwise, previous steps are checked. After each pruning cycle, pruning, model fine-tuning, and the construction and testing of the new model are repeated until the model's parameter count, computational cost, memory consumption, and accuracy all meet preset performance indicators, at which point pruning stops. Four pruning criteria can be applied simultaneously, and the best-performing criterion is selected.
[0102] Finally, the newly constructed model is deployed to AIoT edge devices.
[0103] It is understood that this application provides a pruning scheme for depthwise separable convolutions in the MobileOne structure based on the aforementioned alignment principle. Compared to directly pruning 3*3 depthwise convolution layers and then removing the channels corresponding to the upper 1*1 pointwise convolution, this application can prune both types of layers simultaneously. This avoids pruning important layers in the pointwise convolutions and the need for subsequent parameter fine-tuning. The pruned network can be directly deployed in hardware, with the same output as the original network without building a new model. Furthermore, since different layers in depthwise separable convolutions have different importance, this application prunes them separately, improving model performance.
[0104] See Figure 7 As shown in the embodiment of this application, a target detection method includes:
[0105] Step S21: Obtain the image to be detected.
[0106] Step S22: Input the image to be detected into the target detection model to obtain the target detection result; wherein, the target detection model is obtained based on the target detection model training method disclosed in the foregoing embodiments.
[0107] In a specific implementation, the target detection model obtained in the aforementioned embodiments can be deployed in an AIoT edge device to collect pedestrian images, obtain images to be detected, input the images to be detected into the target detection model, and obtain the target detection results.
[0108] As can be seen, the target detection model obtained by pruning in this application can improve the efficiency of target detection and effectively save computing and storage resources.
[0109] See Figure 8 As shown, Figure 8 An object detection model training apparatus disclosed in this application includes:
[0110] Model training module 11 is used to train the initial model using the first image sample training set to obtain the trained model;
[0111] The reparameterization module 12 is used to reparameterize the trained model to obtain a reparameterized model; the reparameterized model includes a depthwise convolutional layer and a pointwise convolutional layer.
[0112] The model pruning module 13 is used to determine the channels to be pruned corresponding to each convolutional layer in the reparameterized model using a preset channel importance discrimination criterion, and to prune the reparameterized model based on the channels to be pruned, so as to obtain the pruned model.
[0113] The channel retention determination module 14 is used to determine the first output channel of the depth convolutional layer and the second output channel of the pointwise convolutional layer respectively; the first output channel is an output channel other than the first target channel, the first target channel is the output channel of itself and the input channel with the same channel number as itself, which are both channels to be pruned; the second output channel is an output channel other than the second target channel, and the second target channel has the same channel number as the first target channel.
[0114] The model building module 15 is used to build a new model based on the pruned model to obtain a target detection model; wherein the new model retains the first output channel and the second output channel.
[0115] As can be seen, in this embodiment, the initial model is first trained using a first image sample training set to obtain a trained model, and then the trained model is reparameterized to obtain a reparameterized model. The reparameterized model includes a depthwise convolutional layer and a pointwise convolutional layer. Then, a preset channel importance discrimination criterion is used to determine the channels to be pruned corresponding to each convolutional layer in the reparameterized model, and the reparameterized model is pruned based on the channels to be pruned to obtain a pruned model. Furthermore, the first output channel of the depthwise convolutional layer and the second output channel of the pointwise convolutional layer are determined respectively. The first output channel is an output channel other than the first target channel. The first target channel is itself and the input channel with the same channel number as itself, which are both output channels of the channels to be pruned. The second output channel is an output channel other than the second target channel. The second target channel has the same channel number as the first target channel. Then, a new model is constructed based on the pruned model to obtain a target detection model. The new model retains the first output channel and the second output channel. In other words, this application determines the model to be pruned based on the reparameterized model, performs pruning, and constructs a new model based on the pruned model to obtain the target detection model. For deep convolutional layers, the final pruned layers are only the layers themselves and the output channels of the input channels with the same channel number as the layers themselves, which are all channels to be pruned. While ensuring that the number of input and output channels of the deep convolutional layers is consistent, this avoids pruning important channels in the pointwise convolutional layers, thereby improving the efficiency of target detection and effectively saving computational and storage resources.
[0116] See Figure 9 As shown in the figure, this application discloses an electronic device 20, including a processor 21 and a memory 22; wherein, the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, the target detection model training method disclosed in the foregoing embodiments, and / or, the target detection method.
[0117] Regarding the above-mentioned object detection model training method, and / or the specific process of the object detection method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.
[0118] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.
[0119] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26; wherein, the power supply 23 is used to provide operating voltage for the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0120] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the target detection model training method and / or the target detection method disclosed in the foregoing embodiments.
[0121] Regarding the above-mentioned object detection model training method, and / or the specific process of the object detection method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.
[0122] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0123] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0124] The above provides a detailed description of the target detection model training method, target detection method, device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for training an object detection model, characterized in that, include: The initial model is trained using the first image sample training set to obtain the trained model, and the trained model is then reparameterized to obtain the reparameterized model; the reparameterized model includes a depthwise convolutional layer and a pointwise convolutional layer. The channels to be pruned in each convolutional layer of the reparameterized model are determined by using a preset channel importance discrimination criterion, and the reparameterized model is pruned based on the channels to be pruned to obtain the pruned model; wherein, the importance of each convolutional layer is determined, and the channels to be pruned are determined based on the principle that the higher the importance of the convolutional layer, the fewer channels are removed. The first output channel of the deep convolutional layer and the second output channel of the pointwise convolutional layer are determined respectively; the first output channel is an output channel other than the first target channel, the first target channel is the output channel of itself and the input channel with the same channel number as itself, which are both channels to be pruned; the second output channel is an output channel other than the second target channel, and the second target channel has the same channel number as the first target channel. A new model is constructed based on the pruned model to obtain a target detection model; wherein the first output channel and the second output channel are retained in the new model; The process of pruning the reparameterized model based on the channels to be pruned to obtain the pruned model further includes: training the pruned model using a second image sample training set to obtain a target pruned model; correspondingly, the process of constructing a new model based on the pruned model to obtain a target detection model includes: constructing a new model based on the target pruned model to obtain the final target detection model.
2. The target detection model training method according to claim 1, characterized in that, The step of constructing a new model based on the pruned model to obtain the target detection model includes: The target pruned model and the new model are tested using an image sample test set to obtain their accuracy. If the accuracy of the pruned target model is consistent with that of the new model, then repeat the pruning and model building process until the new model meets the preset performance indicators, and then use the new model as the final target detection model.
3. The target detection model training method according to claim 1, characterized in that, Before training the pruned model using the second image sample training set to obtain the target pruned model, the method further includes: Add a batch normalization layer to the pruned model.
4. The target detection model training method according to claim 3, characterized in that, The initial model includes a convolutional block structure; the convolutional block structure includes a depthwise convolutional layer, a pointwise convolutional layer, a batch normalization layer, and a compression excitation module. Accordingly, the preset channel importance discrimination criterion is: the absolute value of the channel weight / the absolute value of the weight parameter of the batch normalization layer / the size of the first-order Taylor term of the channel minus the size of the Flops regularization term of each layer / the size of the output of the compression excitation module.
5. The target detection model training method according to claim 4, characterized in that, Before training the initial model using the first image sample training set to obtain the trained model, the method further includes: Add a regularization factor to the gradient of the weight parameters of the batch normalized layer.
6. The target detection model training method according to any one of claims 1 to 5, characterized in that, The step of determining the channels to be pruned for each convolutional layer in the reparameterized model using a preset channel importance criterion includes: The channel importance of each channel in each convolutional layer is determined using a preset channel importance discrimination criterion; The channel importance of all the depthwise convolutional layers and all the pointwise convolutional layers is sorted to obtain a first sorting result for all the depthwise convolutional layers and a second sorting result for all the pointwise convolutional layers. Based on the first sorting result and the second sorting result, the channels to be pruned for the depthwise convolutional layer and the pointwise convolutional layer are determined respectively.
7. A target detection method, characterized in that, include: Acquire the image to be detected; The image to be detected is input into the target detection model to obtain the target detection result; wherein the target detection model is obtained based on the target detection model training method as described in any one of claims 1 to 6.
8. An electronic device, characterized in that, Includes memory and processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the target detection model training method as described in any one of claims 1 to 6, and / or the target detection method as described in any one of claims 7.
9. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the object detection model training method as described in any one of claims 1 to 6, and / or the object detection method as described in any one of claims 7.