Compressive pruning ship detection method and system based on complex model
By modifying and pruning the convolution kernel, channel and senet structure of the ship detection model, the contradiction between ship detection speed and accuracy in the prior art is solved, and efficient ship detection is achieved.
Patent Information
- Application Number
- CN202411458275.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-10-18
AI Technical Summary
Existing ship inspection technology is difficult to speed up the inspection speed while ensuring accuracy, and the model has high hardware requirements and many missed and missed inspections.
By modifying the size of the convolution kernel, the number of convolutional channels and the number of convolution kernels of the backbone network, multiple rounds of training and fine-tuning are carried out, combining senet structure and similarity analysis, pruning and freezing redundant weights, reducing the amount of model parameters, and improving the model Complexity and receptive field.
It improves the accuracy and speed of ship detection, reduces the hardware computing power requirements, and achieves the acceleration of inference while ensuring the detection rate.
Smart Images

Figure CN118982801A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of ship detection, and in particular to a ship detection method and system based on compression pruning of a complex model. Background Art
[0002] With the rapid development of the transportation industry, including water transportation, the tasks of water transportation operations have become increasingly heavy, among which ship detection is one of the important tasks. Therefore, the use of deep learning to achieve automated ship detection is widely used in the industry, but deep learning currently has the following problems: ① Models with a small number of parameters and simple structures, although they can achieve real-time detection, have biased model detection capabilities, and have more missed detections and false detections: ② Models with a large number of parameters and complex structures, although the model detection capabilities are enhanced, missed detections and false detections are reduced, cannot achieve real-time detection, and have high hardware requirements. Therefore, based on this problem, we explored it and found that the weights of some feature layers or feature maps are redundant parameters, which not only increase the amount of calculations, reduce the reasoning speed, but also increase the number of false detection targets.
[0003] With regard to the problem that the existing ship detection technology cannot ensure accuracy while speeding up the detection, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiment of the present invention provides a ship detection method and system based on compression pruning of a complex model to solve the problem in the prior art that ship detection cannot speed up the detection while ensuring the accuracy.
[0005] To achieve the above-mentioned purpose, on the one hand, the present invention provides a ship detection method based on compression pruning of complex models, the method comprising: S1, modifying the convolution kernel size, the number of convolution channels, and the number of convolution kernels of each module of the backbone network of the original ship detection model to obtain a first modified ship detection model; S2, inputting the training set into the first modified ship detection model for multiple rounds of training to obtain a first updated ship detection model and the weight of each layer of convolution in the first updated ship detection model; S3, resetting the weight of the current layer of convolution with low weight to 0 and freezing it; inputting the training set into the first updated ship detection model after the weight is modified for fine-tuning to obtain a second updated ship detection model; S4, adding a senet structure to the second updated ship detection model to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; according to the similarity of the weight of each convolution channel and the feature map output by each convolution channel S5, according to the similarity of the feature map of each convolution kernel in each module and the similarity of the feature map of each convolution channel, the weights of the convolution kernels with high similarity and low weight are reset to 0 and skipped, and the training set is input into the fourth updated ship detection model after the weight is modified for fine-tuning to obtain the fifth updated ship detection model; S6, the convolution channels with weight 0 and the convolution kernels with weight 0 in the fifth updated ship detection model are deleted to obtain the target ship detection model; S7, the ship image to be detected is input into the target ship detection model for detection to obtain the position and category of the ship.
[0006] Optionally, the S3 includes: S31, performing a sigmoid operation on the weight of each layer of convolution to obtain a normalized weight of each layer of convolution; S32, comparing the normalized weight of each layer of convolution with a first preset threshold, resetting the normalized weight of the current layer of convolution that is less than the first preset threshold to 0 and freezing it; S33, inputting the training set into the first updated ship detection model after the weight is modified for fine-tuning to obtain a second updated ship detection model.
[0007] Optionally, the S4 includes: S41, adding a senet structure for calculating the weights of convolution channels to the second updated ship detection model to obtain a second modified ship detection model; inputting the training set into the second modified ship detection model for fine-tuning to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; S42, comparing the weight of each convolution channel with a second preset threshold, resetting the weight of the current convolution channels that are less than the second preset threshold to 0, resetting all parameters corresponding to the current convolution channels whose weights are reset to 0 to 0 and freezing them; saving the weight of each convolution channel; S43, removing the weight of each convolution channel, and inputting the training set into the removed convolution channel. S44, calculate the feature maps of any two convolution channels using cosine similarity to obtain a first similarity value. If the first similarity value is greater than a third preset threshold, compare the weights of the two convolution channels, reset the weight of the convolution channel with a low weight to 0 and freeze it; S45, input the training set into the third modified ship detection model after the weight modification for fine-tuning, and obtain the fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel.
[0008] Optionally, the S5 includes: S51, calculating the similarity of the feature maps of any two convolution kernels in the current module to obtain a second similarity value; S52, calculating the similarity of the feature maps of any two convolution channels in the same convolution kernel and calculating the similarity of the feature maps of any two convolution channels in different convolution kernels in the feature maps of the two selected convolution kernels, and summing the similarities of the feature maps of any two convolution channels to obtain a third similarity value; S53, obtaining a total similarity value according to the second similarity value and the third similarity value; if the total similarity value is greater than a fourth preset threshold, resetting the weight of the subsequent convolution kernel to 0 and skipping it; S54, inputting the training set into the fourth updated ship detection model after modifying the weights for fine-tuning to obtain a fifth updated ship detection model.
[0009] Optionally, the first similarity value is calculated according to the following formula:
[0010] in, is the first similarity value, is the feature map of one of the convolution channels in any two convolution channels, is the feature map of the other convolution channel in any two convolution channels.
[0011] Optionally, the second similarity value is calculated according to the following formula:
[0012] in, is the second similarity value, is the feature map of one of the convolution kernels in any two convolution kernels, is the feature map of the other convolution kernel in any two convolution kernels.
[0013] On the other hand, the present invention provides a ship detection system based on compression pruning of complex models, the system comprising: a modification unit, used to modify the convolution kernel size, the number of convolution channels, and the number of convolution kernels of each module of the backbone network of the original ship detection model to obtain a first modified ship detection model; a training unit, used to input the training set into the modified ship detection model for multiple rounds of training to obtain a first updated ship detection model and the weight of each layer of convolution in the first updated ship detection model; a first compression pruning unit, used to reset the weight of the current layer of convolution with low weight to 0 and freeze it; input the training set into the first updated ship detection model after the weight is modified for fine-tuning to obtain a second updated ship detection model; a second compression pruning unit, used to add a senet structure to the second updated ship detection model to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; according to the similarity of the weight of each convolution channel and the feature map output by each convolution channel The weights of the convolution channels with high similarity and low weight are reset to 0 and frozen, and the training set is input into the third updated ship detection model after the weight is modified for fine-tuning to obtain the fourth updated ship detection model, the weights of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel; the third compression pruning unit is used to reset the weights of the convolution kernels with high similarity and arranged at the back to 0 and skip them according to the similarity of the feature map of each convolution kernel in each module and the similarity of the feature map of each convolution channel, and input the training set into the fourth updated ship detection model after the weight is modified for fine-tuning to obtain the fifth updated ship detection model; the deletion unit is used to delete the convolution channels with a weight of 0 and the convolution kernels with a weight of 0 in the fifth updated ship detection model to obtain the target ship detection model; the detection unit is used to input the ship image to be detected into the target ship detection model for detection to obtain the ship position and category.
[0014] Optionally, the first compression pruning unit includes: a first calculation subunit, used to perform a sigmoid operation on the weight of each layer of convolution to obtain a normalized weight of each layer of convolution; a first reset subunit, used to compare the normalized weight of each layer of convolution with a first preset threshold, reset the normalized weight of the current layer of convolution that is less than the first preset threshold to 0 and freeze it; a first fine-tuning subunit, used to input the training set into the first updated ship detection model after the weight is modified for fine-tuning, to obtain a second updated ship detection model.
[0015] Optionally, the second compression pruning unit includes: a structure adding subunit, used to add a senet structure for calculating the weights of the convolution channels in the second updated ship detection model to obtain a second modified ship detection model; inputting the training set into the second modified ship detection model for fine-tuning to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; a second resetting subunit, used to compare the weight of each convolution channel with a second preset threshold, reset the weight of the current convolution channels that are less than the second preset threshold to 0, reset all parameters corresponding to the current convolution channels whose weights are reset to 0 to 0 and freeze them; save the weight of each convolution channel; a second fine-tuning subunit, used to remove the weight of each convolution channel, input the training set into The third updated ship detection model after removing the weights of the convolution channels is fine-tuned to obtain the third modified ship detection model and the feature map of each convolution channel in the third modified ship detection model; the third resetting subunit is used to calculate the feature maps of any two convolution channels using cosine similarity to obtain a first similarity value. If the first similarity value is greater than a third preset threshold, the weights of the two convolution channels are compared, and the weight of the convolution channel with a low weight is reset to 0 and frozen; the third fine-tuning subunit is used to input the training set into the third modified ship detection model after modifying the weights for fine-tuning to obtain the fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel.
[0016] Optionally, the third compression pruning unit includes: a first calculation subunit, used to calculate the similarity of feature maps of any two convolution kernels in the current module to obtain a second similarity value; a second calculation subunit, used to calculate the similarity of feature maps of any two convolution channels in the same convolution kernel, and calculate the similarity of feature maps of any two convolution channels in different convolution kernels, and sum up the similarities of the feature maps of all any two convolution channels to obtain a third similarity value; a fourth resetting subunit, used to obtain a total similarity value according to the second similarity value and the third similarity value; if the total similarity value is greater than a fourth preset threshold, the weight of the convolution kernel arranged behind is reset to 0 and skipped; a fourth fine-tuning subunit, used to input the training set into the fourth updated ship detection model after the weight is modified for fine-tuning to obtain a fifth updated ship detection model.
[0017] Beneficial effects of the present invention: The present invention provides a ship detection method and system based on compression pruning of complex models, wherein the method effectively improves the complexity and receptive field of the original ship detection model by modifying the convolution kernel size, the number of convolution channels, and the number of convolution kernels of each module of the backbone network, thereby improving the accuracy of ship detection; by resetting the lower weights of layer convolution, convolution channels, and convolution kernels to 0, the number of model parameters is reduced, thereby reducing the model's demand for hardware computing power and accelerating the ship detection speed; adopting staged model training, fine-tuning the model when modifying it, thereby achieving the speed of inference of the ship model while ensuring the ship detection rate; deleting the convolution channels and convolution kernels with a weight of 0, thereby reducing the model's demand for hardware computing power and further accelerating the ship detection speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a flow chart of a ship detection method based on compression pruning of a complex model provided by an embodiment of the present invention; Figure 2 is a flow chart of a complex model compression and pruning provided by an embodiment of the present invention to obtain a second updated ship detection model; Figure 3 is a flowchart of a complex model compression and pruning provided by an embodiment of the present invention to obtain a fourth updated ship detection model; Figure 4 is a flowchart of a complex model compression and pruning provided by an embodiment of the present invention to obtain a fifth updated ship detection model; Figure 5 It is a structural schematic diagram of a ship detection system based on compression pruning of a complex model provided by an embodiment of the present invention; Figure 6 is a structural schematic diagram of a first compression pruning unit provided in an embodiment of the present invention; Figure 7 is a schematic structural diagram of a second compression pruning unit provided in an embodiment of the present invention; Figure 8 4 is a schematic diagram of the structure of a third compression pruning unit provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0020] Figure 1 A ship detection method based on compression pruning of a complex model is provided in an embodiment of the present invention. Figure 1 As shown, the method includes: S1. Modify the convolution kernel size, the number of convolution channels, and the number of convolution kernels of each module of the backbone network of the original ship detection model to obtain a first modified ship detection model; In the original ship detection model, the backbone network, i.e., the convolutional neural network, is the core component of feature extraction and is responsible for extracting multi-level features from the input image. This embodiment optimizes the backbone network structure to improve the accuracy of feature extraction and enhances the generalization and expression capabilities of the original ship detection model without significantly increasing the amount of calculation.
[0021] The parameters of the convolution kernel include its size (width and height), the number of convolution channels, and the number of convolution kernels.
[0022] (1) Expand the convolution kernel size; In the convolutional neural network of the original ship detection model, a 3×3 convolution kernel is often used for feature extraction. Although the 3×3 convolution kernel is relatively small in computational complexity, its receptive field is limited and it is difficult to capture global features. In this embodiment, the convolution kernel size is expanded from 3×3 to 5×5 to expand the receptive field of each convolution operation, so that the model can capture more contextual information in a single convolution. This convolution kernel expansion method significantly improves the performance of the model in complex backgrounds or dense feature scenes. It should be noted that the present invention does not limit the size of the convolution kernel expansion, and it can also be expanded to 7×7, 9×9, etc.
[0023] (2) Increase the number of convolution channels In a convolutional neural network, the number of convolution channels determines the richness of feature representation. In the original ship detection model, the number of convolution channels is relatively small, resulting in limited feature extraction capabilities. This embodiment increases the number of convolution channels by 1.6 times on the original basis, significantly improving the feature extraction and expression capabilities of the convolutional neural network. By increasing the number of convolution channels, the model can capture richer feature dimensions, thereby improving the accuracy of ship detection, especially in the detection tasks of complex backgrounds or multi-scale ship targets.
[0024] (3) Increase the number of convolution kernels The depth of the backbone network plays an important role in capturing multi-level and complex features. Shallow networks have few parameters and low computational complexity, so it is difficult to extract high-order semantic features, resulting in a high missed detection rate. This embodiment increases the number of convolution kernels in each module of the backbone network, for example, the number of convolution kernels in each module of the ResNet-18 network is expanded to the depth of ResNet-34, wherein the ResNet-18 network has 18 convolution kernels and the ResNet-34 network has 34 convolution kernels. In a specific embodiment, the two convolution kernels in a module of the ResNet-18 network are increased to three convolution kernels, and the number of convolution kernels in other modules is also increased accordingly until the number of all convolution kernels is increased to 34. The feature extraction capability of the backbone network is significantly improved by increasing the number of convolution kernels. By increasing the depth, the network can learn more semantic information at a deeper level, enhance the expressive power of the model, and reduce missed detection.
[0025] S2. Inputting the training set into the modified ship detection model for multiple rounds of training to obtain the first updated ship detection model and the weight of each convolution layer in the first updated ship detection model; In an optional embodiment, a camera may be used to photograph ships passing within a preset range to obtain 1,000 ship images, which are used as a training set. It should be noted that these 1,000 ship images include different ship types, different ship shooting angles, and different backgrounds, and each ship image may contain one or more ships. Before model training, the images in the training set need to be preprocessed to improve the training efficiency and generalization ability of the model. Common preprocessing methods include image normalization, random cropping, horizontal flipping, color adjustment, etc. These preprocessing operations can make the model more robust during the training process, thereby improving the performance of the model in the test data.
[0026] The preprocessed training set is input into the first modified ship detection model for forward propagation and back propagation training. In the forward propagation process, the preprocessed training set undergoes multi-layer convolution and pooling operations to generate feature maps and performs classification or regression through the fully connected layer; the loss function measures the difference between the model prediction result and the true label. Back propagation updates the model weights according to the gradient of the loss function and gradually optimizes the performance of the model.
[0027] During the training process, with multiple iterations, the loss function of the model gradually decreases, and finally fluctuates within a preset range to reach a convergence state. In this embodiment, by adjusting the learning rate reasonably, it is ensured that the model can converge quickly within a certain training round and achieve the expected detection accuracy. When the model converges, the model training is no longer performed, and the weight of each layer of convolution in the first updated ship detection model and the first updated ship detection model is obtained. Among them, the weight of each layer of convolution is the weight corresponding to each face (face formed by width and height) in the convolution kernel.
[0028] S3, resetting the weight of the current layer convolution with low weight to 0 and freezing it; inputting the training set into the first updated ship detection model after the weight modification for fine-tuning to obtain the second updated ship detection model; Figure 2 1 is a flowchart of a complex model compression and pruning to obtain a second updated ship detection model provided by an embodiment of the present invention; Figure 2 As shown, the S3 includes: S31, perform sigmoid operation on the weight of each convolution layer to obtain the normalized weight of each convolution layer; Perform a sigmoid operation on the weights of each layer of convolution, map the weight values to between 0 and 1, and obtain the normalized weights of each layer of convolution for subsequent processing.
[0029] S32, comparing the normalized weight of each layer of convolution with a first preset threshold, resetting the normalized weight of the current layer of convolution that is less than the first preset threshold to 0 and freezing it; For the current layer convolution whose weight value after the sigmoid operation is less than the first preset threshold (set to 0.01 in the present invention), its weight is directly reset to 0. All these layer convolutions set to 0 are frozen and will no longer participate in subsequent back propagation and training; this operation can significantly reduce the amount of calculation while retaining the parameters that have little impact on the model accuracy, that is, non-zero weights.
[0030] S33, inputting the training set into the first updated ship detection model after the weights are modified for fine-tuning to obtain a second updated ship detection model.
[0031] After freezing the weights set to 0, the remaining non-zero weights are fine-tuned, that is, the training set is input into the first updated ship detection model after the weights are modified for training, and the second updated ship detection model is obtained; the training process further fine-tunes (optimizes) the parameters of the model (i.e., non-zero weights) by reducing the learning rate to ensure that the model can still maintain a high detection accuracy under the pruned structure (i.e., freezing the weights set to 0).
[0032] S4. Adding a senet structure to the second updated ship detection model to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; according to the weight of each convolution channel and the similarity of the feature map output by each convolution channel, the weight of the convolution channel with high similarity and low weight is reset to 0 and frozen, and the training set is input into the third updated ship detection model after the weight is modified for fine-tuning to obtain a fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel; After preliminary pruning and weight sparsification, the convolutional channels are evaluated for importance and feature map similarity is analyzed to further reduce model redundancy.
[0033] Figure 3 1 is a flowchart of a complex model compression and pruning provided by an embodiment of the present invention to obtain a fourth updated ship detection model; Figure 3 As shown, the S4 includes: S41, adding a senet structure for calculating the convolution channel weights to the second updated ship detection model to obtain a second modified ship detection model; inputting the training set into the second modified ship detection model for fine-tuning to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; In an optional embodiment, a SENET structure is introduced into the second updated ship detection model to calculate the weight of each convolution channel; the SENET structure assigns a weight to each convolution channel through an attention mechanism, thereby helping the model to identify and highlight the contribution of a specific channel to the ship detection task.
[0034] The weight of each convolution channel is calculated according to the following formula:
[0035] in, Represents the weight of each convolution channel, sigmoid represents the S-type function (sigmoid activation function), FC represents the full convolution function, RELU represents the RELU activation function, GP represents the global pooling operation, and feature_map represents the output feature map; The training set is input into the model with the senet structure added (called the second modified ship detection model) for training. The purpose is to fine-tune the model parameters under the senet weighting condition so that the model maintains a high detection accuracy. When the model converges, no more training is performed to obtain the weight of each convolution channel. Among them, feature_map represents the feature map output by the training set in the second modified ship detection model. After calculating it through the senet structure, the weight of each convolution channel is obtained.
[0036] S42, comparing the weight of each convolution channel with a second preset threshold, resetting the weight of the current convolution channel that is less than the second preset threshold to 0, resetting all parameters corresponding to the current convolution channel whose weight is reset to 0 to 0 and freezing them; saving the weight of each convolution channel; The weight of each convolution channel is compared with the second preset threshold (set to 0.01 in the present invention) to identify unimportant convolution channels; for convolution channels with weights less than the second preset threshold, their weights are reset to 0, and all parameters corresponding to the convolution channel are reset to 0. This means that these convolution channels contribute little to the final detection results and are redundant channels.
[0037] The convolutional channels whose weights are frozen are reset to 0 and no longer back-propagated, which reduces the amount of model calculation and training complexity. The frozen convolutional channels will not be involved in the calculation in subsequent training and reasoning, thereby speeding up the reasoning speed of the model.
[0038] Save the weights of each convolution channel so that you can continue to use these weights for filtering in subsequent steps.
[0039] S43, removing the weight of each convolution channel, inputting the training set into the third updated ship detection model after removing the weight of the convolution channel, and performing fine-tuning to obtain a third modified ship detection model and a feature map of each convolution channel in the third modified ship detection model; The model is further simplified by removing the weights of each convolution channel. This step aims to eliminate the extra complexity of the senet structure and make the model structure more concise.
[0040] After saving the weight information of each convolution channel in S42, these weights are removed to return the model to the convolutional neural network structure without senet. After removing the senet structure, the model only uses convolution operations and no longer performs weighted adjustment on each convolution channel.
[0041] The training set is input into the third updated ship detection model after removing the weights of the convolutional channels, and fine-tuning training is performed to make the model converge again without relying on weighted weighting. This fine-tuning process ensures that the model gradually adapts to the unweighted structure while maintaining high detection accuracy by reducing the learning rate and extending the number of training rounds; when the model converges, no more training is performed to obtain the third modified ship detection model, and the feature map of each convolutional channel generated during the last training process is recorded.
[0042] S44, calculating the feature maps of any two convolution channels by using cosine similarity to obtain a first similarity value, and if the first similarity value is greater than a third preset threshold, comparing the weights of the two convolution channels, resetting the weight of the convolution channel with a lower weight to 0 and freezing it; The similarity between the feature maps of any two convolution channels is calculated using the cosine similarity formula to obtain a first similarity value, which is calculated according to the following formula:
[0043] in, is the first similarity value, is the feature map of one of the convolution channels in any two convolution channels, is the feature map of the other convolution channel in any two convolution channels.
[0044] If the first similarity value of two convolution channels is greater than a third preset threshold (set to 0.8 in the present invention), the two convolution channels are considered to be redundant, and only one of the channels needs to be retained.
[0045] The weights of similar convolution channels are further compared, the convolution channels with larger weights are retained, the weights of the convolution channels with smaller weights are reset to 0, and the convolution channels are frozen. This can retain important convolution channels while eliminating redundant calculations and reducing model parameters.
[0046] S45. Input the training set into the third modified ship detection model after weight modification for fine-tuning to obtain the fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel.
[0047] After pruning and freezing the redundant convolution channels, the training set is input into the third modified ship detection model after the weights are modified for fine-tuning training to obtain the fourth updated ship detection model; this step ensures that the pruned model improves the detection speed while the fine-tuning model parameters maintain a high detection accuracy.
[0048] When the model converges, no more training is performed. The weights of each convolution kernel in each module, the feature maps of each convolution kernel, and the feature maps of each convolution channel in each convolution kernel generated during the last training process are recorded to prepare for the similarity calculation between subsequent convolutions.
[0049] S5. According to the similarity of the feature map of each convolution kernel in each module and the similarity of the feature map of each convolution channel, the weights of the convolution kernels with high similarity and arranged at the back are reset to 0 and skipped, and the training set is input into the fourth updated ship detection model after the weight is modified for fine-tuning to obtain the fifth updated ship detection model; The similarity of feature maps of convolutions within the same module is further calculated, and deeper pruning and optimization are performed on the redundancy of different convolutions.
[0050] Figure 4 is a flowchart of a complex model compression and pruning provided by an embodiment of the present invention to obtain a fifth updated ship detection model; Figure 4 As shown, the S5 includes: S51, performing similarity calculation on the feature maps of any two convolution kernels in the current module to obtain a second similarity value; Select any two convolution kernels in the same module and calculate the similarity (second similarity value) between the feature maps they output to identify redundant convolutions in the same module.
[0051] The second similarity value is calculated according to the following formula:
[0052] in, is the second similarity value, is the feature map of one of the convolution kernels in any two convolution kernels, is the feature map of the other convolution kernel in any two convolution kernels.
[0053] S52, in the feature maps of the two selected convolution kernels, calculating the similarity of the feature maps of any two convolution channels in the same convolution kernel, and calculating the similarity of the feature maps of any two convolution channels in different convolution kernels, and summing the similarities of the feature maps of all any two convolution channels to obtain a third similarity value; For two selected convolution kernels, for example: convolution kernel 1 and convolution kernel 2, if convolution kernel 1 has three convolution channels, namely: convolution channel 11, convolution channel 12, convolution channel 13; convolution kernel 2 also has three convolution channels, namely: convolution channel 21, convolution channel 22, convolution channel 23; It is necessary to calculate the similarity between the feature map of convolution channel 11 and the feature map of convolution channel 12, calculate the similarity between the feature map of convolution channel 11 and the feature map of convolution channel 13, calculate the similarity between the feature map of convolution channel 12 and the feature map of convolution channel 13, calculate the similarity between the feature map of convolution channel 21 and the feature map of convolution channel 22, calculate the similarity between the feature map of convolution channel 21 and the feature map of convolution channel 23, and calculate the similarity between the feature map of convolution channel 22 and the feature map of convolution channel 23; And the similarity between the feature map of convolution channel 11 and the feature map of convolution channel 21 is required, the similarity between the feature map of convolution channel 11 and the feature map of convolution channel 22 is calculated, the similarity between the feature map of convolution channel 11 and the feature map of convolution channel 23 is calculated, the similarity between the feature map of convolution channel 12 and the feature map of convolution channel 21 is calculated, the similarity between the feature map of convolution channel 12 and the feature map of convolution channel 22 is calculated, and the similarity between the feature map of convolution channel 12 and the feature map of convolution channel 23 is calculated; the similarity between the feature map of convolution channel 13 and the feature map of convolution channel 21, the similarity between the feature map of convolution channel 13 and the feature map of convolution channel 22, and the similarity between the feature map of convolution channel 13 and the feature map of convolution channel 23 is calculated.
[0054] Among them, the similarity calculation formula of the feature maps of any two convolution channels is the same as the similarity calculation formula of the feature maps of any two convolution kernels mentioned above, and will not be repeated here; The third similarity value is obtained by summing the similarities of the feature maps of all any two convolution channels in the two selected convolution kernels.
[0055] S53, obtaining a total similarity value according to the second similarity value and the third similarity value; if the total similarity value is greater than a fourth preset threshold, resetting the weight of the subsequent convolution kernel to 0 and skipping it; The second similarity value and the third similarity value are summed and divided by 2 to obtain a total similarity value; if the total similarity value is greater than a fourth preset threshold (set to 0.8 in the present invention), it is considered that there is redundancy between the two convolution kernels.
[0056] For convolution kernels with high similarity, the weights of the convolution kernels that follow are reset to 0 and skipped, and they do not participate in model training.
[0057] S54, inputting the training set into the fourth updated ship detection model after the weights are modified for fine-tuning to obtain a fifth updated ship detection model.
[0058] After pruning and skipping convolution, the model is fine-tuned to ensure that the model improves the detection speed while fine-tuning the model parameters to maintain high detection accuracy. By reducing the learning rate and conducting multiple rounds of training, the model is gradually converged, and finally the optimized ship detection model is obtained, that is, the fifth updated ship detection model.
[0059] S6. Delete the convolution channel with a weight of 0 and the convolution kernel with a weight of 0 in the fifth updated ship detection model to obtain a target ship detection model; Delete the convolution channel with weight 0 in the fifth updated ship detection model to reduce hardware computing power and floating point numbers, and speed up inference; Delete the convolution kernel with weight 0 in the fifth updated ship detection model to reduce hardware computing power and floating point numbers, and speed up inference.
[0060] S7. Input the image of the ship to be detected into the target ship detection model for detection to obtain the position and category of the ship.
[0061] The above method can obtain an efficient and lightweight ship detection model, so that when the subsequent images of ships to be detected taken by the camera are input into the model for detection, the amount of calculation and reasoning time are reduced while maintaining a high detection accuracy.
[0062] Figure 5 is a structural schematic diagram of a ship detection system based on compression pruning of a complex model provided by an embodiment of the present invention; Figure 5 As shown, the system includes: The modification unit 201 is used to modify the convolution kernel size, the number of convolution channels, and the number of convolution kernels of each module of the backbone network of the original ship detection model to obtain a first modified ship detection model; A training unit 202 is used to input the training set into the modified ship detection model for multiple rounds of training to obtain the first updated ship detection model and the weight of each convolution layer in the first updated ship detection model; The first compression pruning unit 203 is used to reset the weight of the current layer convolution with low weight to 0 and freeze it; input the training set into the first updated ship detection model after the weight is modified for fine-tuning to obtain a second updated ship detection model; Figure 6 is a schematic diagram of the structure of the first compression pruning unit provided in an embodiment of the present invention, such as Figure 6 As shown, the first compression pruning unit 203 includes: The first calculation subunit 2031 is used to perform a sigmoid operation on the weight of each convolution layer to obtain a normalized weight of each convolution layer; A first resetting subunit 2032 is used to compare the normalized weight of each convolution layer with a first preset threshold, and reset the normalized weight of the current convolution layer that is less than the first preset threshold to 0 and freeze it; The first fine-tuning subunit 2033 is used to input the training set into the first updated ship detection model after the weight is modified for fine-tuning to obtain a second updated ship detection model.
[0063] The second compression pruning unit 204 is used to add a senet structure to the second updated ship detection model to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; according to the weight of each convolution channel and the similarity of the feature map output by each convolution channel, the weight of the convolution channel with high similarity and low weight is reset to 0 and frozen, and the training set is input into the third updated ship detection model after the weight is modified for fine-tuning to obtain a fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel; Figure 7 is a schematic diagram of the structure of the second compression pruning unit provided in an embodiment of the present invention, such as Figure 7 As shown, the second compression pruning unit 204 includes: The structure adding subunit 2041 is used to add a senet structure for calculating the convolution channel weights in the second updated ship detection model to obtain a second modified ship detection model; input the training set into the second modified ship detection model for fine-tuning to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; The second resetting subunit 2042 is used to compare the weight of each convolution channel with the second preset threshold, reset the weight of the current convolution channel that is less than the second preset threshold to 0, reset all parameters corresponding to the current convolution channel whose weight is reset to 0 to 0 and freeze them; save the weight of each convolution channel; A second fine-tuning subunit 2043 is used to remove the weight of each convolution channel, input the training set into the third updated ship detection model after the weight of the convolution channel is removed for fine-tuning, and obtain a third modified ship detection model and a feature map of each convolution channel in the third modified ship detection model; The third resetting subunit 2044 is used to calculate the feature maps of any two convolution channels by using cosine similarity to obtain a first similarity value. If the first similarity value is greater than a third preset threshold, the weights of the two convolution channels are compared, and the weight of the convolution channel with a lower weight is reset to 0 and frozen. The third fine-tuning subunit 2045 is used to input the training set into the third modified ship detection model after modifying the weights for fine-tuning, so as to obtain the fourth updated ship detection model, the weights of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel.
[0064] The third compression pruning unit 205 is used to reset the weights of the convolution kernels with high similarity and arranged at the back to 0 and skip them according to the similarity of the feature map of each convolution kernel in each module and the similarity of the feature map of each convolution channel, and input the training set into the fourth updated ship detection model after the weight is modified for fine-tuning to obtain a fifth updated ship detection model; Figure 8 is a schematic diagram of the structure of the third compression pruning unit provided in an embodiment of the present invention, such as Figure 8 As shown, the third compression pruning unit 205 includes: The first calculation subunit 2051 is used to calculate the similarity of the feature maps of any two convolution kernels in the current module to obtain a second similarity value; The second calculation subunit 2052 is used to calculate the similarity of the feature maps of any two convolution channels in the same convolution kernel in the feature maps of the two selected convolution kernels, and calculate the similarity of the feature maps of any two convolution channels in different convolution kernels, and sum the similarities of the feature maps of all any two convolution channels to obtain a third similarity value; The fourth resetting subunit 2053 is used to obtain a total similarity value according to the second similarity value and the third similarity value; if the total similarity value is greater than a fourth preset threshold, the weight of the convolution kernel arranged behind is reset to 0 and skipped; The fourth fine-tuning subunit 2054 is used to input the training set into the fourth updated ship detection model with modified weights for fine-tuning to obtain a fifth updated ship detection model.
[0065] A deleting unit 206 is used to delete the convolution channels with a weight of 0 and the convolution kernels with a weight of 0 in the fifth updated ship detection model to obtain a target ship detection model; The detection unit 207 is used to input the image of the ship to be detected into the target ship detection model for detection to obtain the position and category of the ship.
[0066] The system provided by the present invention corresponds to the above method, and the relevant contents have been fully described in the above method embodiments, which will not be repeated here.
[0067] The method and system of the present invention effectively improve the complexity and receptive field of the original ship detection model by increasing the convolution kernel size, the number of convolution channels, and the number of convolution kernels of each module of the backbone network, thereby improving the accuracy of ship detection; by resetting the lower weights of the layer convolution, convolution channel and convolution kernel to 0, the number of parameters of the model is reduced, thereby reducing the model's demand for hardware computing power and accelerating the ship detection speed; adopting staged model training and fine-tuning the model when modifying it, thereby achieving the speed of inference of the ship model while ensuring the ship detection rate; deleting the convolution channels and convolution kernels with a weight of 0, thereby reducing the model's demand for hardware computing power and further accelerating the ship detection speed.
[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A ship detection method based on compression pruning of complex models, characterized in that: include: S1. Modify the convolution kernel size, the number of convolution channels, and the number of convolution kernels of each module of the backbone network of the original ship detection model to obtain a first modified ship detection model; S2. Inputting the training set into the first modified ship detection model for multiple rounds of training to obtain the first updated ship detection model and the weight of each convolution layer in the first updated ship detection model; S3, reset the weight of the current layer convolution with low weight to 0 and freeze it; The training set is input into the first updated ship detection model after the weight is modified for fine-tuning to obtain a second updated ship detection model; S4. Adding a senet structure to the second updated ship detection model to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; according to the weight of each convolution channel and the similarity of the feature map output by each convolution channel, the weight of the convolution channel with high similarity and low weight is reset to 0 and frozen, and the training set is input into the third updated ship detection model after the weight is modified for fine-tuning to obtain a fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel; S5. According to the similarity of the feature map of each convolution kernel in each module and the similarity of the feature map of each convolution channel, the weights of the convolution kernels with high similarity and arranged at the back are reset to 0 and skipped, and the training set is input into the fourth updated ship detection model after the weight is modified for fine-tuning to obtain the fifth updated ship detection model; S6. Delete the convolution channel with a weight of 0 and the convolution kernel with a weight of 0 in the fifth updated ship detection model to obtain a target ship detection model; S7. Input the image of the ship to be detected into the target ship detection model for detection to obtain the position and category of the ship.
2. The method according to claim 1, characterized in that The S3 includes: S31, perform sigmoid operation on the weight of each convolution layer to obtain the normalized weight of each convolution layer; S32, comparing the normalized weight of each layer of convolution with a first preset threshold, resetting the normalized weight of the current layer of convolution that is less than the first preset threshold to 0 and freezing it; S33, inputting the training set into the first updated ship detection model after the weights are modified for fine-tuning to obtain a second updated ship detection model.
3. The method according to claim 1, characterized in that The S4 includes: S41, adding a senet structure for calculating the convolution channel weights to the second updated ship detection model to obtain a second modified ship detection model; inputting the training set into the second modified ship detection model for fine-tuning to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; S42, comparing the weight of each convolution channel with a second preset threshold, resetting the weight of the current convolution channel that is less than the second preset threshold to 0, resetting all parameters corresponding to the current convolution channel whose weight is reset to 0 to 0 and freezing them; saving the weight of each convolution channel; S43, removing the weight of each convolution channel, inputting the training set into the third updated ship detection model after removing the weight of the convolution channel, and performing fine-tuning to obtain a third modified ship detection model and a feature map of each convolution channel in the third modified ship detection model; S44, calculating the feature maps of any two convolution channels by using cosine similarity to obtain a first similarity value, and if the first similarity value is greater than a third preset threshold, comparing the weights of the two convolution channels, resetting the weight of the convolution channel with a lower weight to 0 and freezing it; S45. Input the training set into the third modified ship detection model after weight modification for fine-tuning to obtain the fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel.
4. The method according to claim 1, characterized in that: The S5 includes: S51, performing similarity calculation on the feature maps of any two convolution kernels in the current module to obtain a second similarity value; S52, in the feature maps of the two selected convolution kernels, calculating the similarity of the feature maps of any two convolution channels in the same convolution kernel, and calculating the similarity of the feature maps of any two convolution channels in different convolution kernels, and summing the similarities of the feature maps of all any two convolution channels to obtain a third similarity value; S53, obtaining a total similarity value according to the second similarity value and the third similarity value; if the total similarity value is greater than a fourth preset threshold, resetting the weight of the subsequent convolution kernel to 0 and skipping it; S54, inputting the training set into the fourth updated ship detection model after the weights are modified for fine-tuning to obtain a fifth updated ship detection model.
5. The method according to claim 3, characterized in that: The first similarity value is calculated according to the following formula: ;in, is the first similarity value, is the feature map of one of the convolution channels in any two convolution channels, is the feature map of the other convolution channel in any two convolution channels.
6. The method according to claim 4, characterized in that The second similarity value is calculated according to the following formula: ;in, is the second similarity value, is the feature map of one of the convolution kernels in any two convolution kernels, is the feature map of the other convolution kernel in any two convolution kernels.
7. A ship detection system based on compression pruning of complex models, characterized in that: include: A modification unit, used to modify the convolution kernel size, the number of convolution channels, and the number of convolution kernels of each module of the backbone network of the original ship detection model to obtain a first modified ship detection model; A training unit, used for inputting the training set into the modified ship detection model for multiple rounds of training to obtain the first updated ship detection model and the weight of each convolution layer in the first updated ship detection model; The first compression pruning unit is used to reset the weight of the current layer convolution with low weight to 0 and freeze it; The training set is input into the first updated ship detection model after the weight is modified for fine-tuning to obtain a second updated ship detection model; A second compression pruning unit is used to add a senet structure to the second updated ship detection model to obtain a third updated ship detection model and a weight of each convolution channel in the third updated ship detection model; according to the weight of each convolution channel and the similarity of the feature map output by each convolution channel, the weight of the convolution channel with high similarity and low weight is reset to 0 and frozen, and the training set is input into the third updated ship detection model after the weight is modified for fine-tuning to obtain a fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel; a third compression pruning unit, configured to reset the weights of the convolution kernels with high similarity and arranged at the back to 0 and skip them according to the similarity of the feature maps of each convolution kernel in each module and the similarity of the feature maps of each convolution channel, and input the training set into the fourth updated ship detection model after the weight modification for fine-tuning to obtain a fifth updated ship detection model; A deleting unit, used to delete the convolution channel with a weight of 0 and the convolution kernel with a weight of 0 in the fifth updated ship detection model to obtain a target ship detection model; The detection unit is used to input the image of the ship to be detected into the target ship detection model for detection to obtain the position and category of the ship.
8. The system according to claim 7, characterized in that The first compression pruning unit comprises: The first calculation subunit is used to perform a sigmoid operation on the weight of each convolution layer to obtain a normalized weight of each convolution layer; A first resetting subunit, used to compare the normalized weight of each convolution layer with a first preset threshold, and reset the normalized weight of the current convolution layer that is less than the first preset threshold to 0 and freeze it; The first fine-tuning subunit is used to input the training set into the first updated ship detection model after the weight is modified for fine-tuning to obtain a second updated ship detection model.
9. The system according to claim 7, characterized in that The second compression pruning unit includes: A structure adding subunit is used to add a senet structure for calculating the convolution channel weights in the second updated ship detection model to obtain a second modified ship detection model; the training set is input into the second modified ship detection model for fine-tuning to obtain a third updated ship detection model and the weight of each convolution channel in the third updated ship detection model; A second resetting subunit is used to compare the weight of each convolution channel with a second preset threshold, reset the weight of the current convolution channel that is less than the second preset threshold to 0, reset all parameters corresponding to the current convolution channel whose weight is reset to 0 to 0 and freeze them; save the weight of each convolution channel; a second fine-tuning subunit, configured to remove the weight of each convolution channel, input the training set into the third updated ship detection model after the weight of the convolution channel is removed for fine-tuning, and obtain a third modified ship detection model and a feature map of each convolution channel in the third modified ship detection model; A third resetting subunit is used to calculate the feature maps of any two convolution channels by using cosine similarity to obtain a first similarity value. If the first similarity value is greater than a third preset threshold, the weights of the two convolution channels are compared, and the weight of the convolution channel with a lower weight is reset to 0 and frozen; The third fine-tuning subunit is used to input the training set into the third modified ship detection model after modifying the weights for fine-tuning, so as to obtain the fourth updated ship detection model, the weight of each convolution kernel in each module in the fourth updated ship detection model, the feature map of each convolution kernel, and the feature map of each convolution channel in each convolution kernel.
10. The system according to claim 7, characterized in that The third compression pruning unit comprises: The first calculation subunit is used to calculate the similarity of the feature maps of any two convolution kernels in the current module to obtain a second similarity value; The second calculation subunit is used to calculate the similarity of the feature maps of any two convolution channels in the same convolution kernel in the feature maps of the two selected convolution kernels, and calculate the similarity of the feature maps of any two convolution channels in different convolution kernels, and sum the similarities of the feature maps of all any two convolution channels to obtain a third similarity value; A fourth resetting subunit is used to obtain a total similarity value according to the second similarity value and the third similarity value; if the total similarity value is greater than a fourth preset threshold, the weight of the convolution kernel arranged behind is reset to 0 and skipped; The fourth fine-tuning subunit is used to input the training set into the fourth updated ship detection model with modified weights for fine-tuning to obtain a fifth updated ship detection model.
Citation Information
Patent Citations
Lightweight SAR image ship detection model and method based on strip pruning
CN114283331A
Ship category detection method and system based on DAL module optimization
CN116188753A
Infrared ship target detection method based on improved yolov7
CN117152691A
Pedestrian detection method based on knowledge migration pruning model
CN117315722A
Target detection model training method, target detection method, device, and medium
WO2024168972A1
