Lightweight convolutional neural network training method and system for water surface target detection

Abstract

The application discloses a lightweight convolutional neural network training method and system for water surface target detection. The method comprises the following steps: inputting a water surface floating garbage data set into an input end, and performing a pretreatment operation on image data input into a network; the pretreated image is input into a backbone network for feature extraction, the backbone network is a group of plug-and-play lightweight convolutional modules M-Conv encapsulated by two convolutional operations of channel shuffling and depth separable convolution; the image after feature extraction is input into a Neck end, the Neck end adopts an FPN+PAN structure, and is separated through a self-processing operation between the two; the image after feature fusion is input into a prediction end for final output; according to the network structure, an end-to-end training is established by using a loss function at an output end, so that the network converges. The network can meet the requirements of real-time performance and accuracy of small target detection of an edge end which is limited in storage and calculation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and particularly to a lightweight convolutional neural network training method and system for water surface target detection. BACKGROUND

[0002] Target detection is one of the core problems in the field of computer vision, and its task is to find all the targets of interest in the image and determine their categories and locations. Due to the different appearances, poses, and different degrees of occlusion of various objects, target detection has always been a challenging task. The target detection task can be mainly divided into four small tasks: (1) classification: given an image, know what categories of targets are contained in it. (2) location: given the location of the target in the image. (3) detection: locate the position of the target and determine the category of the target. (4) segmentation: determine which target or scene each pixel belongs to. The present application is more inclined to target detection in the water surface scene, especially for target cleaning of water surface floating garbage. Due to the accumulation of water surface floating garbage, the water quality is deteriorating, not only the aquatic organisms are decreasing, but also the river channel is more and more serious, which seriously affects the play of flood control function. In order to build a good circulating function of water ecological system, it is necessary to start from the management of small river channel, perfect the supporting facilities of water surface floating garbage treatment, and vigorously carry out the ecological regulation work of river channel.

[0003] The current research on water surface target detection is improved from the following aspects. First, the data set is expanded. Since the water surface target detection data set is not rich, most of the research work will first expand the data set to better train the model recognition accuracy, such as expanding based on the self-made data set using the Poisson fusion method. Second, the model structure is improved, such as improving the YOLOV5 algorithm by adding a feature map attention (FMA) layer at the end of the main stem to improve the network feature extraction capability, or replacing the original main network DarkNet-53 with a more simple and practical ResNet-50 network structure from the perspective of network universality and scene complexity, to facilitate the implementation of subsequent joint algorithms. In order to better deploy the neural network model on the edge, it can also be improved from the lightweight angle, such as improving the main network in YOLOv3, replacing part of the convolution operation with a smaller calculation linear operation without reducing the accuracy of the original model, so that the model has fewer parameters and consumes less computing power; or a water surface target detection model compression method based on deep learning is used: an improved network with depth separable convolution and lightweight attention model is used to replace the feature extraction network DarkNet, and model compression is performed through multi-scale feature fusion.

[0004] However, most of the existing work is improved from a single aspect, and a lightweight water surface target detection network is not designed as a whole. The difficulty of water surface target detection, that is, the problem of uneven image background illumination, has not been well improved, and the above network cannot meet the requirements of real-time and accuracy in specific application scenarios. SUMMARY

[0005] The present application provides a lightweight convolutional neural network training method and system for water surface target detection, which constructs a set of plug-and-play lightweight convolutional modules M-Conv that can be added to the backbone network for feature extraction or embedded into the Neck end for feature fusion, ensuring accuracy without loss, and making the network lightweight and easy to deploy on the edge.

[0006] Technical scheme: in order to achieve the above invention purpose, the technical scheme of the present application is as follows:

[0007] A lightweight convolutional neural network training method for water surface target detection, comprising the following steps:

[0008] Obtain a water surface floating garbage data set, which includes a training set image and a validation set image, and perform preprocessing operation on the image data input into the network;

[0009] The preprocessed image enters a convolutional neural network, and an end-to-end training is established at the output end of the network by using a loss function, so that the network converges, wherein the convolutional neural network structure comprises: an input end, a backbone network, a Neck end, and a prediction end, the input end inputs the preprocessed image; the backbone network is used for feature extraction of the input image, the backbone network is a group of plug-and-play lightweight convolution modules M-Conv encapsulated by two convolution operations of transposition operation uniform channel shuffling and depth separable convolution; the Neck end is used for feature fusion of the extracted feature map; and the prediction end is used for final detection result prediction and output according to the fused feature map.

[0010] The trained convolutional neural network model is verified by using the verification set image.

[0011] The M-Conv module penetrates the information generated by the channel-intensive convolution operation into each part of the information generated by the depth separable convolution by using the uniform channel shuffling operation, and the uniform channel shuffling operation refers to rearranging the original feature order and then recombining.

[0012] Preferably, the uniform channel shuffling operation comprises: dividing the input layer into g groups, the number of channels of each group of input feature maps is n, and the total number of channels is g*n, the feature map is unfolded into a four-dimensional matrix of g*n*w*h, wherein w and h represent the width and height of the input feature map, the g-axis and the n-axis of the matrix with the size of g*n*w*h are transposed, the g-axis and the n-axis are tiled to obtain the shuffled feature map, and finally 1*1 convolution within the group is performed.

[0013] Preferably, the Neck end of the convolutional neural network adopts a combination of FPN and PAN networks, the FPN layer conveys strong semantic features from top to bottom, the PAN conveys strong positioning features from bottom to top, the two are combined, and the two are separated by self-processing operations.

[0014] Preferably, the self-processing operation comprises: performing 1*1 convolution on the feature map, then performing 3*3 convolution, and then performing 1*1 convolution.

[0015] Preferably, in the convolutional neural network, the PAN and FPN networks are in residual connection with the M-Conv module, and the M-Conv module cooperates with the attention module.

[0016] Preferably, the loss function in the training process is defined as: Loss=a*lossobj+b*lossrect+c*lossclc, wherein lossrect is a bounding box loss, lossobj is a confidence loss, lossclc is a classification loss, a, b and c are respectively coefficients of corresponding loss terms, the rectangular box loss is calculated by using CIOU loss, the confidence loss and the classification loss are both calculated by using BCE loss, and the calculation formula is as follows:

[0017] CIOU loss:

[0018] wherein, IOU is a ratio of an intersection area of two boxes to an area of a union part, w1, h1 and w p , h p respectively represent the width, the height of the real box and the width, the height of the predicted box, wherein v is used to measure the consistency of the relative proportion of two rectangular boxes, alpha is a weight coefficient, (x p1 , y p1 ), (x p2 , y p2 ) are respectively the left upper corner coordinate and the right lower corner coordinate of the predicted rectangular box, (x l1 , y l1 ), (x l2 , y l2 ) are respectively the left upper corner coordinate and the right lower corner coordinate of the real rectangular box, rho represents the distance between the center point coordinate of the predicted box and the center point coordinate of the real box, and c refers to the length of the diagonal line of the minimum circumscribed rectangle of the predicted box and the real box.

[0019] BCE loss: loss BCE (z,x,y)=-L(z,x,y)*logP(z,x,y)-(1-L(z,x,y))*log(1-P(z,x,y))

[0020] wherein, L(z,x,y) and P(z,x,y) represent respectively a real class label and a predicted class label, that is, the real box confidence value of the corresponding position of the zth box with the coordinates (x, y) and the predicted box confidence value of the corresponding position.

[0021] The application also provides a lightweight convolutional neural network training system for water surface target detection, comprising:

[0022] An input end inputs a water surface floating garbage data set, the data set comprising a training set image and a verification set image, and performs a pretreatment operation on image data input into a network;

[0023] A backbone network is used to extract features from the preprocessed image, and the feature extraction mode is a set of plug-and-play lightweight convolution modules M-Conv encapsulated by two convolution operations of uniform channel shuffling and depth separable convolution through transposition operation;

[0024] A neck end is used to fuse features of the extracted feature map;

[0025] A prediction end is used to perform final detection result prediction and output according to the fused feature map;

[0026] A training control module is used to establish an end-to-end training at the output end by using a loss function according to the neural network structure composed of the input end, the backbone network, the neck end and the prediction end, so as to make the network converge, and the trained convolutional neural network model is verified by using a verification set image;

[0027] The M-Conv module penetrates information generated by a channel-intensive convolution operation into each part of information generated by a depth separable convolution by using a uniform channel shuffling operation, and the uniform channel shuffling operation refers to rearranging the original feature sequence.

[0028] The application further provides a computer device, comprising one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs are used to implement the steps of the lightweight convolutional neural network training method for water surface target detection.

[0029] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is used to implement the steps of the lightweight convolutional neural network training method for water surface target detection when executed by a processor.

[0030] Compared with the prior art, the application has the following advantages and beneficial effects: the application uses the lightweight convolution module M-Conv to extract features for the complex and changeable water surface scene, and can meet the requirements of real-time performance and accuracy of small target detection at the edge end with limited storage and calculation. The lightweight convolution module M-Conv is a plug-and-play lightweight convolution module encapsulated by a channel shuffling operation and a depth separable convolution operation, and its working principle is to reduce or directly discard unnecessary convolution operations and fully extract valuable information to realize the lightweight of the network model on the basis of ensuring the accuracy. In addition, the M-Conv module can cooperate with the attention mechanism and be connected in residual mode at the neck end and the prediction end of the network architecture, so as to further improve the recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a lightweight convolutional neural network training method flowchart for water surface target detection of the present application;

[0032] Figure 2 is a structure diagram of the plug-and-play lightweight convolutional module M-Conv of the present application;

[0033] Figure 3 is a processing process diagram of the Neck end of the present application;

[0034] Figure 4 is a schematic diagram of several residual connection types using M-Conv provided by the present application;

[0035] Figure 5 is a schematic diagram of the residual connection mode of M-Conv at the Neck end and the prediction end provided by the present application. DETAILED DESCRIPTION

[0036] The technical solutions of the present application will be further described below with reference to the accompanying drawings.

[0037] Referring to Figure 1 , the present application provides a lightweight convolutional neural network training method for water surface target detection, comprising the following steps:

[0038] Step (1), obtain a water surface floating garbage data set, the data set includes training set images and validation set images, and perform a preprocessing operation on the image data input into the network.

[0039] In the embodiment of the present application, the input end uses a water surface floating garbage data set with a first perspective of an unmanned aerial vehicle, the data set contains more than 1200 training set images and more than 800 validation set images. The data set should cover as many images as possible that can reflect the real water surface scene, such as water surface reflection, water surface fallen leaves covering, water surface floating plants, etc.

[0040] The pre-processing of the image in the application refers to the pre-processing adopted in YOLOv5, including Mosaic data enhancement, random scaling, cropping, and random arrangement for splicing, aiming to enhance the detection effect of small targets. Further, considering that the application is for water surface target detection, the natural factors of water surface ripples and wind will inevitably cause the blurring phenomenon of the photographed pictures in this scenario, so on the basis of retaining the original operation of YOLOv5, motion blur data enhancement processing is added. In the example embodiment, the motion blur data enhancement is realized by calling the MotionBlur function in the Albumentations library, setting the parameters blur_limit=[20, 25] and p=1. Wherein blur_limit can represent the degree of blur, the larger the value, the more blurred, and p represents the probability of triggering this function. The purpose is to simulate the images grabbed in the motion state, thereby enhancing the robustness of the model. Further, the pre-processing of YOLOv5 does not involve the processing of the target missing detection phenomenon caused by uneven light, considering that the application is for water surface target detection, the water surface reflection phenomenon will cause great interference to the identification of the target in this scenario. Based on this, on the basis of the above operation, the brightness of the input image is enhanced. First, the pixel value is normalized to the range of {0, 1}, then the new pixel value is calculated by taking gamma as the exponential value, and then the restored image brightness becomes very natural and not easy to distort.

[0041] Step (2), the image data at the input end enters the backbone network for feature extraction.

[0042] In the embodiment of the application, a set of plug-and-play lightweight convolution modules encapsulated by two convolution operations of channel shuffling and depthwise separable convolution (DSC) is adopted in the backbone network structure, which extracts the features of the pre-processed image.

[0043] Generally speaking, in order to accelerate the calculation speed of the final prediction, the images in the convolutional neural network (CNN) almost always undergo a similar conversion process in the backbone network: the information of the spatial dimension is gradually transferred to the channel. And each time the spatial (width and height) compression of the feature map and the expansion of the channel dimension will cause the loss of part of the semantics. Simply using the DSC operation can improve the speed of the detector, but these models expose the realistic problem of low accuracy in specific applications. MobileNets uses a large number of 1*1 dense convolution to fuse the channel information calculated independently; ShuffleNets uses "channel shuffle" to realize the interaction of channel information; GhostNet uses "halved" SC operation to retain the interaction information between channels. However, 1*1 dense convolution occupies more computing resources, and the effect of using "channel shuffle" still does not reach the result of standard convolution (SC). Many lightweight models only use depth separable convolution from beginning to end, but whether it is used for image classification or target detection, the defects of DSC are directly amplified in the backbone. Based on this, in the present application, the SC and DSC cooperate. Because the feature map generated by the output channel of DSC through only channel shuffle is still "depth separable". In order to make the output of DSC as close to SC as possible, a new convolution module-SC, DSC and channel shuffle hybrid convolution is introduced in the present application.

[0044] In the present application, the information (channel dense convolution operation) generated by SC is penetrated into each part of the information generated by DSC by using uniform channel shuffle operation (Shuffle). Here, uniform channel shuffle is to rearrange the original feature order: assuming that the input layer is divided into g groups, the number of channels of each group of input feature maps is n, the total number of channels is g*n, the feature map (Feature Map) is expanded into a four-dimensional matrix of g*n*w*h, w and h represent the width and height of the input feature map, transpose along the g axis and n axis of the matrix with a size of g*n*w*h, and the g axis and n axis are tiled to obtain the shuffled Feature Map, and finally the intra-group 1*1 convolution is performed. Figure 2As shown, assuming that the number of channels of the input feature map is C1channels, first, a 3*3 standard convolution is performed to obtain a feature map of C2 / 2channels. The output of this step is uniformly shuffled in the channel dimension, which only uniformly shuffles the channels and does not change the number of channels of the feature map. After the operation, the feature map is still C2 / 2channels. The above information is fused by using a DSC operation to obtain a feature map of C2 / 2channels. The input maps of the above two steps are spliced (Concat) in the channel dimension to obtain a feature map of C2channels. At this time, the channels are uniformly shuffled again, and down-sampling processing with a step of 2 is performed (the purpose is to reduce the dimension and reduce the number of picture channels to half of the original number). The obtained feature map is spliced in the channel dimension with the C2 / 2channels feature map obtained by the first down-sampling to obtain the final output feature map of C2 channels. Through the operation, the information from the SC is completely mixed into the output of the DSC, and the purpose of uniformly exchanging local information on different channels is achieved.

[0045] The lightweight convolution operation is encapsulated into a set of plug-and-play convolution modules M-Conv, which are used in the backbone network. The model has higher accuracy than the original model in detecting specific targets in water surface scenes, and the parameter amount is reduced by half compared with the original model. Overall, from the perspective of directly affecting the running speed of the model, the model reaches an optimal level in the balance between speed and accuracy, and is very suitable for the application of mobile end models.

[0046] Step (3), the image after feature extraction is input to the neck (Neck end), and parameters of different detection layers are aggregated from different backbone layers.

[0047] In the embodiment of the present application, the network structure of the Neck end is redesigned for the mainstream Feature Pyramid Network (FPN) and Path Aggregation Network (PANet). The mainstream FPN network mainly solves the multi-scale problem in object detection. Through simple network connection changes, the performance of small object detection is greatly improved without significantly increasing the original model calculation. The high-level features are up-sampled and the low-level features are connected from top to bottom, and each layer will be predicted. After the multi-layer network in FPN, the target information at the bottom is very fuzzy, so PANet adds a bottom-up route to make up for and strengthen the positioning information. The working principle of PANet is to add a pyramid from bottom to top after FPN to supplement FPN, and pass the low-level positioning features up, so that the pyramid formed combines semantic information and positioning information. As can be seen, the top-down and bottom-up operations are sequential, and there is no self-processing operation in both structures. Taking PANet as an example, the bottom-up operation is followed by the top-down operation, which is a suboptimal operation. Taking the operation of P4→P3 (the fourth layer of feature map to the third layer of fusion transmission semantic information) as an example, the operation of P3→P4 (the third layer of feature map to the fourth layer of fusion transmission semantic information) follows immediately, that is, P3 is updated by P4 immediately after updating P4 without feature digestion and refinement. Based on this, the network structure of the present application separates the top-down and bottom-up operations through self-processing operations, as shown in Figure 3 TD (top-down) represents top-down operation, BU (bottom-up) represents bottom-up operation, and S represents self-processing operation. The specific process includes C1→C3→C1, where C1 represents a convolution operation with a convolution kernel size of 1*1, and C3 represents a convolution operation with a convolution kernel size of 3*3. In this way, the features can have the opportunity to be processed by themselves before communication.

[0048] In addition, the PAN and FPN networks can be connected in residual connection with the M-Conv module. The residual module includes two or more M-Conv convolutions and a skip connection (SkipConnection). The skip connection adds the input directly to the output of the convolution layer to form a residual connection, as shown in Figure 4 (a), assuming that the input is (x) and the output of the M-Conv module is F(x), then the output of the residual module is H(x)=F(x)+x, where F(x) is a residual mapping and (x) is a skip connection. Figure 4(a) is a basic building block. (b), (c), and (d) refer to how this basic building block can be used to perform residual connections with other networks. These residual connection methods are different but feasible in the network. Here are some connection suggestions. Bottleneck refers to the bottleneck module. Since the input feature map size is C1 channels, the intermediate processed feature map size is C2 / 2 channels, and the final output is a C2 channel feature map, which visually resembles a bottleneck.

[0049] Furthermore, residual networks can also include pooling layers, fully connected layers, and activation functions. By introducing residual modules, residual networks solve the gradient vanishing problem during inference, enabling deeper network structures in real-world scenarios.

[0050] Step (4): The image after feature fusion is input into the prediction end for final output.

[0051] The output end uses three prediction heads of 76*76, 38*38, and 19*19 respectively to predict small, medium, and large targets.

[0052] Currently, mainstream attention mechanisms include SE (Squeeze & Excitation), CBAM (Convolutional Block Attention Module), and CA (Coordinate Attention). SE comprises two operations: squeezing and excitation. This module allows the network to focus more on feature channels with higher information content while neglecting those with lower information content. CBAM is a spatial channel attention mechanism module. The CA module is a novel solution to avoid the loss of positional information caused by global pooling 2D operations: it focuses attention on both the width and height dimensions to effectively utilize the spatial coordinate information of the input feature map. In this invention, the attention module is typically placed at the end of the backbone network for better performance. This is because shallow networks are filled with a large amount of low-level semantic information, resulting in minimal use of the information fusion function of the attention module; it is not necessary to fuse feature maps that already contain rich low-level semantic information. Furthermore, the M-Conv module can work synergistically with the CA module to... Figure 4 For example, before the prediction head outputs the final prediction result, the M-Conv module and the CA module can be used... Figure 4 The residual connection is performed in the manner of (b) or (c), which effectively corrects the problem that the previous architecture could not fully model the channel characteristic dependency. In this experimental scenario, it demonstrates full network representation capability and achieves state-of-the-art performance.

[0053] As Figure 5 shown, P3, P4, P5 refer to three feature maps of different sizes obtained by processing the original input feature map in the backbone network. Assuming that the size of the input image is 640*640*3, the number of channels of P3 obtained at the end of the neck is 512, the number of channels of P4 is 256, and the number of channels of P5 is 128, which are used to predict small, medium and large targets respectively. In the figure, upsample refers to the upsampling operation, which is to transmit the bottom layer voice information to the top layer. For example, after upsampling, the number of channels of P5 is changed from 128 to 256, which can be spliced with P4 in the channel dimension, and then the output feature map is input into the Res-M1 network for convolution operation, and the obtained feature map continues to transmit semantic information. The present application obtains a feature map after each semantic fusion, which is used as the input feature map for prediction in the head. Figure 5 In the example shown, CA is included in Res-M1. Since CA is also a plug-and-play attention mechanism module, the example uses Figure 4 the method of (b) in the CA connection.

[0054] At the output end, an end-to-end training is established by using a loss function. The role of the loss function is to measure the distance between the predicted information and the expected information (label). The closer the predicted information is to the expected information, the smaller the loss function value.

[0055] In the target detection algorithm, a large number of regions are usually sampled in the input image, and then it is judged whether the regions contain the target of interest, and the region edge position is adjusted to more accurately predict the real boundary box of the target, as shown on the right side of Figure 1 Different models may use different region sampling methods. Anchor refers to generating a plurality of boundary boxes of different sizes and aspect ratios centered on each pixel. Assuming that the height of the input image is h and the width is w. Different shapes of anchor boxes are generated centered on each pixel of the image. Let the size be s∈(0,1] and the aspect ratio be r>0. Here s represents the scale, i.e. the proportion of the anchor box to the picture size, and r represents the ratio. There are w’h’=whs and w’ / h’=r, so the width and height of the anchor box will be sqrt{shwr} and sqrt{shw / r} respectively. When the center position is given, the anchor box with known width and height is determined. A set of sizes s1,…,sn and a set of aspect ratios r1,…,rn are respectively set, and a set of anchor boxes is generated. The number of anchor boxes is equal to the product of the number of sizes and the number of aspect ratios, i.e. n*m. mIf all size and aspect ratio combinations are used, the input image will get whnm anchor boxes in total if centered on each pixel. Although these anchor boxes can cover all real bounding boxes, the computational complexity is easily too high. Therefore, usually only size and aspect ratio combinations containing s1 or r1 are of interest, that is, the number of anchor boxes centered on the same pixel is n+m-1. For the entire input image, a total of wh(n+m-1) anchor boxes will be generated, and it can be known that in the training set, the position of the real bounding box and the category of the contained target are labeled. After generating the anchor boxes, the anchor boxes are mainly labeled according to the position and category information of the real bounding box similar to the anchor box. If the loss function value of anchor box A and the target real bounding box is less than the threshold (the default is 0.5), anchor box A is labeled as the target category, and if the loss function value of anchor box A and the target real bounding box is greater than the threshold (the default is 0.5), anchor box A is labeled as background. Anchor boxes with a background category are usually referred to as negative anchor boxes, and the rest are referred to as positive anchor boxes.

[0056] The loss during training mainly includes three aspects: bounding box loss (lossrect), confidence loss (lossobj), and classification loss (lossclc). The loss function of the network is defined as: Loss = a*lossobj + b*lossrect + c*lossclc. CIOU loss is used to calculate the rectangular box loss, and BCE loss is used to calculate the confidence loss and the classification loss.

[0057] CIOU loss:

[0058] wherein, IOU is the ratio of the intersection area of two boxes to the union area, so it is also called intersection over union. The three terms of CIOU correspond to the calculation of IOU, center point distance, and aspect ratio. Alpha and v correspond to the aspect ratio, and w p , h p represent the width and height of the real box and the width and height of the predicted box, respectively. Wherein v is used to measure the consistency of the relative proportion of the two rectangular boxes, alpha is the weight coefficient, (x p1 , y p1 ), (x p2 , y p2 ) are the left upper corner and right lower corner coordinates of the predicted rectangular box, (x l1 , y l1 ), (x l2 , y l2 ) are the left upper corner and right lower corner coordinates of the real rectangular box. P represents the distance between the center point coordinates of the predicted box and the center point coordinates of the real box, and c refers to the length of the diagonal of the minimum enclosing rectangle of the predicted box and the real box.

[0059] BCE loss: loss BCE (z,x,y) = -L(z,x,y)*logP(z,x,y) - (1 - L(z,x,y))*log(1 - P(z,x,y))

[0060] BCE Loss is mainly used to create a standard that measures the binary cross-entropy between the target and the output. For an image segmented into an 80*80 grid, the neural network predicts three rectangular frames near each grid (referred to as predicted frames), and each predicted frame contains prediction information including center coordinates, width, height, confidence, and classification probability. Therefore, the neural network outputs a total of 3*80*80 prediction confidence values between 0 and 1, which correspond to 3*80*80 predicted frames. The confidence of each predicted frame represents the reliability of the predicted frame, and the greater the value, the more reliable the predicted frame, that is, the closer it is to the real minimum bounding box of the target. The label dimension should be consistent with the output dimension of the neural network, so the confidence label is also a 3*80*80 matrix. Assuming that the confidence label is matrix L and the predicted confidence is matrix P, then 0≤z<3, 0≤x<80, 0≤y<80. Where L(z,x,y) and P(z,x,y) represent the true class label and the predicted class label, respectively, that is, the true frame confidence value corresponding to the zth frame at position (x,y) and the predicted frame confidence value corresponding to the zth frame at position (x,y).

[0061] Step (5) starts the phase training to make the network converge.

[0062] In the first stage, the original YOLOv5s initial training weight is used to train the data set, and the model is converged in a short time; in the second stage, the training weight of the first stage is used, and the original CSP Darknet in the Backbone part is replaced by the lightweight network module M-Conv as described above; in the third stage, the CA attention mechanism is superimposed on the Neck end of the network structure, further improving the accuracy of network recognition. In actual training, the model reaches the fitting state after 300 epochs, so each stage is set to 300 epochs for training.

[0063] After the model is trained, the trained model is tested by using a cross-validation method: the original data set is divided into a training data set and a test data set, and the training data set is divided into k data sets; for each set of parameters, each data set is used as a validation data set, and the remaining data sets are used as a training data set to train k models; the performance indicators of the k models are calculated on the corresponding validation data set; the average value of the performance indicators of the k models is used as the final performance indicator of the model corresponding to the set of parameters. If the final indicator is not ideal, the parameters are adjusted again to obtain the average value of the performance indicators of the k models. The verified model can be used for detection of water surface floating objects, garbage and the like.

[0064] The application also provides a lightweight convolutional neural network training system for water surface target detection, comprising:

[0065] an input end for inputting a water surface floating garbage data set, the data set comprising training set images and validation set images, and performing a preprocessing operation on image data input into the network;

[0066] a backbone network for performing feature extraction on the preprocessed images, the feature extraction mode being a set of plug-and-play lightweight convolutional modules M-Conv encapsulated by two convolutional operations, namely channel shuffling and depth separable convolution;

[0067] a Neck end for performing feature fusion on the extracted feature maps;

[0068] a prediction end for performing final detection result prediction and output according to the fused feature maps;

[0069] a training control module for establishing an end-to-end training at the output end by using a loss function according to the network structures of the input end, the backbone network, the Neck end and the prediction end, making the network converge, and verifying the trained convolutional neural network model by using the validation set images;

[0070] The M-Conv module penetrates information generated by a channel-intensive convolutional operation into each part of information generated by a depth separable convolution by using a uniform channel shuffling operation, and the uniform channel shuffling operation refers to rearranging the original feature order.

[0071] It should be understood that the lightweight convolutional neural network training system in the embodiments of the application can realize all the technical solutions in the above method embodiments, and the functions of each functional module can be realized according to the methods in the above method embodiments, and the specific implementation process can be referred to the related description in the above embodiments, which will not be described here in detail.

[0072] The application further provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the program, when executed by the processor, implements the steps of the lightweight convolutional neural network training method for water surface target detection as described above.

[0073] The application further provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the lightweight convolutional neural network training method for water surface target detection as described above.

[0074] The above embodiments are only used to illustrate the technical solutions of the present application rather than limit the present application. Although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the specific embodiments of the present application can be modified or equivalent replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. A lightweight convolutional neural network training method for water surface target detection, characterized in that, Includes the following steps: Obtain a dataset of floating debris on the water surface, which includes training set images and validation set images, and perform preprocessing operations on the image data input to the network; The preprocessed image is fed into a convolutional neural network (CNN). End-to-end training is established at the network output using a loss function to achieve network convergence. The CNN structure includes an input terminal, a backbone network, a neck terminal, and a prediction terminal. The input terminal receives the preprocessed image. The backbone network extracts features from the input image and is a set of plug-and-play lightweight convolutional modules (M-Conv) encapsulated from channel shuffling and depthwise separable convolution operations. The neck terminal fuses the extracted feature maps. The prediction terminal predicts and outputs the final detection result based on the fused feature maps. The trained convolutional neural network model is validated using validation set images; The M-Conv module uses a uniform channel shuffling operation to permeate the information generated by channel-dense convolution operations into each part of the information generated by depthwise separable convolutions. The uniform channel shuffling operation refers to shuffling the original feature order and then recombining it. The uniform channel shuffling operation includes: dividing the input layer into g groups, with each group of input feature maps having n channels, so the total number of channels is g*n; unfolding the feature map into a four-dimensional matrix of size g*n*w*h, where w and h represent the width and height of the input feature map; transposing along the g-axis and n-axis of the matrix of size g*n*w*h; tiling the g-axis and n-axis to obtain the shuffled feature map; and finally performing 1*1 convolution within the group. The M-Conv module operates as follows: Assuming the input feature map has C1 channels, a standard 3x3 convolution is first performed to obtain a C2 / 2-channel feature map. The output of this step is then uniformly shuffled, which only shuffles the channels without changing the number of channels in the feature map. The resulting feature map is still C2 / 2 channels. A depthwise separable convolution is then used to fuse the information, resulting in another C2 / 2-channel feature map. The input maps from the previous two steps are then concatenated along the channel dimension to obtain a C2-channel feature map. At this point, the channels are again uniformly shuffled, and a downsampling process with a stride of 2 is performed. This resulting feature map is then concatenated along the channel dimension with the C2 / 2-channel feature map obtained from the first downsampling step to obtain the final C2-channel output feature map.

2. The method according to claim 1, characterized in that, The preprocessing of image data includes: performing Mosaic data enhancement, random scaling, cropping, and random arrangement of images for stitching, as well as motion blur data enhancement and Gamma brightness enhancement.

3. The method according to claim 1, characterized in that, The Neck layer of the convolutional neural network uses a combination of FPN and PAN networks. The FPN layer conveys strong semantic features from top to bottom, while the PAN conveys strong localization features from bottom to top. The two are combined and separated by self-processing operations.

4. The method according to claim 3, characterized in that, The self-processing operation includes: performing a 1*1 convolution on the feature map, followed by a 3*3 convolution, and then a 1*1 convolution.

5. The method according to claim 3, characterized in that, In convolutional neural networks, PAN and FPN networks are residually connected to the M-Conv module, and the M-Conv module works in conjunction with the attention module.

6. The method according to claim 1, characterized in that, The loss function during training is defined as: Loss = a * lossobj + b * lossrect + c * lossclc, where lossrect is the bounding box loss, lossobj is the confidence loss, lossclc is the classification loss, and a, b, and c are the coefficients of the corresponding loss terms. CIOU loss is used to calculate the bounding box loss, and BCE loss is used to calculate both the confidence loss and the classification loss. The calculation formulas are as follows: CIOU loss: in, IOU is the ratio of the area of ​​the intersection of the two boxes to the area of ​​their merged portion, where w1, h1, and w p h p represents the width and height of the ground truth bounding box and the width and height of the predicted bounding box, respectively. Here, v measures the consistency of the relative proportions of the two bounding boxes, and α is the weighting coefficient. p1 ,y p1 ), (x p2 ,y p2 ) represent the coordinates of the top-left and bottom-right corners of the predicted bounding box, respectively. l1 ,y l1 ), (x l2 ,y l2 ) are the coordinates of the top left and bottom right corners of the true bounding box, respectively; ρ represents the distance between the center point of the predicted bounding box and the center point of the true bounding box; and c refers to the length of the diagonal of the smallest bounding rectangle of the two rectangles, the predicted bounding box and the true bounding box. BCE loss:loss BCE (z,x,y)=-L(z,x,y) * logP(z,x,y)-(1-L(z,x,y))*log(1- P(z,x,y)) Where L(z,x,y) and P(z,x,y) represent the ground truth label and the predicted label, respectively, that is, the confidence value of the ground truth box at the z-th box with coordinates (x,y) and the confidence value of the predicted box at the same position.

7. A lightweight convolutional neural network training system for water surface target detection, characterized in that, include: The input consists of a dataset of floating debris on the water surface, which includes training and validation images. The image data input to the network is then preprocessed. The backbone network is used to extract features from the preprocessed image. The feature extraction mode is a set of plug-and-play lightweight convolutional modules M-Conv, which encapsulate two convolutional operations: channel shuffling and depthwise separable convolution. The Neck end is used for feature fusion of the extracted feature maps; The prediction end is used to predict and output the final detection result based on the fused feature map. The training control module is used to establish end-to-end training at the output end using a loss function based on the neural network structure consisting of the input end, backbone network, Neck end, and prediction end, so that the network converges, and to verify the trained convolutional neural network model using validation set images. The M-Conv module uses a uniform channel shuffling operation to permeate the information generated by channel-dense convolution operations into each part of the information generated by depthwise separable convolutions. The uniform channel shuffling operation refers to shuffling the original feature order and then recombining it. The uniform channel shuffling operation includes: dividing the input layer into g groups, with each group of input feature maps having n channels, so the total number of channels is g*n; unfolding the feature map into a four-dimensional matrix of size g*n*w*h, where w and h represent the width and height of the input feature map; transposing along the g-axis and n-axis of the matrix of size g*n*w*h; tiling the g-axis and n-axis to obtain the shuffled feature map; and finally performing 1*1 convolution within the group. The M-Conv module operates as follows: Assuming the input feature map has C1 channels, a standard 3x3 convolution is first performed to obtain a C2 / 2-channel feature map. The output of this step is then uniformly shuffled, which only shuffles the channels without changing the number of channels in the feature map. The resulting feature map is still C2 / 2 channels. A depthwise separable convolution is then used to fuse the information, resulting in another C2 / 2-channel feature map. The input maps from the previous two steps are then concatenated along the channel dimension to obtain a C2-channel feature map. At this point, the channels are again uniformly shuffled, and a downsampling process with a stride of 2 is performed. This resulting feature map is then concatenated along the channel dimension with the C2 / 2-channel feature map obtained from the first downsampling step to obtain the final C2-channel output feature map.

8. A computer device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the lightweight convolutional neural network training method for water surface target detection as described in any one of claims 1-6.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the lightweight convolutional neural network training method for water surface target detection as described in any one of claims 1-6.

Images