Target recognition method and system based on deep learning

Abstract

The present invention relates to image data processing in intelligent manufacturing, and discloses a target recognition method and system based on deep learning to improve the recognition accuracy of targets based on drone aerial images. The method includes: performing target recognition on drone aerial image data based on a deep learning model; wherein the architecture of the deep learning model includes a Backbone network for feature extraction, a Neck network based on FPN+PAN, and a Head network for corresponding targets of different scales, and a residual processing layer including a Z4 module is provided in the Backbone and Neck networks; the Z4 module is provided with two branches, the first branch passes through a standard convolution, and the second branch first passes through 4 parallel residual units and then a convolution layer in series, and the two branches are output after feature fusion and activation processing of a BN layer and a LeakyRLU in succession.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing in intelligent manufacturing, and in particular to a target recognition method and system based on deep learning. Background Art

[0002] Traditional target recognition based on drone aerial images mainly relies on manual interpretation, which has the disadvantages of slow recognition speed, easy misjudgment or missed targets. Therefore, it is of great practical significance to develop a target recognition method and system based on deep learning. Summary of the invention

[0003] The purpose of the present invention is to disclose a target recognition method and system based on deep learning to improve the recognition accuracy of targets based on drone aerial images.

[0004] To achieve the above object, the present invention discloses a target recognition method based on deep learning, comprising:

[0005] Acquire image data taken by drones;

[0006] Annotate the collected image data and construct a data set;

[0007] Determine the architecture of the deep learning model, train, verify and test relevant parameters in the architecture of the deep learning model based on the data set to obtain a final deep learning model; then perform target recognition according to the deep learning model;

[0008] Among them, the architecture of the deep learning model includes a Backbone network for feature extraction, a Neck network based on FPN+PAN, and a Head network for corresponding targets of different scales. The Backbone network includes 1 Focus layer, 1 SPP layer, and a number of convolutional layers at least equal to the number of target scale classifications plus 1, and a residual processing layer with at least 1 Z4 module, wherein 1 residual processing layer is located after the SPP, and the other residual processing layers are located between the two convolutional layers before the SPP layer; the Z4 module includes 3 convolutional layers, 4 residual units, and is provided with two branches, the first branch passes through a standard convolution, the second branch first passes through 4 parallel residual units and then a convolution layer in series, and the two branches are output after feature fusion and activation processing by the BN layer and LeakyRLU in succession;

[0009] In the Neck network, a residual processing layer including one Z4 module is also provided after each Concat layer for realizing feature fusion.

[0010] Preferably, each residual unit in the Z4 module is provided with two branches for inputting the add subunit; wherein, the first branch undergoes three layers of series convolution, and after each layer of convolution, it is processed by the BN layer and LeakyRLU before being output to the next layer of convolution; the second branch is directly connected to the input end; the add subunit fuses the features of the two branches and then outputs them after LeakyRLU activation processing.

[0011] Optionally, the method of the present invention is applied to blast identification in a fire-extinguishing bomb test range. In this scenario, the target scales include large-scale targets, medium-scale targets and small-scale targets, and the corresponding pixel sizes are 32*32, 16*16 and 8*8 respectively. Correspondingly, in the Head network, corresponding to large, medium and small scale targets, the sizes of the feature maps are 20*20, 40*40 and 80*80 respectively; among them, the feature maps of detection layers of different scales are expanded by 1×1 convolution, and the three detection layers correspond to feature maps of three different sizes respectively. The number of feature channels after expansion is: (number of categories + Z) × number of anchor boxes on each detection layer; the Z corresponds to the horizontal coordinate, vertical coordinate, width, height and confidence of the center point of the prediction box, and the value is 5; each grid on the feature map is preset with 3 anchor boxes of different aspect ratios, so that each prediction vector finally output by the model has a depth of 9, which includes 4 frame coordinate values, 1 frame confidence value, and 4 category probability values; among them, the 4 categories correspond to four combinations of whether to blast and whether there is a target.

[0012] Based on the above specific scenario, the loss function of the deep learning model described in the present invention during the training process includes the error in predicting the target position, the error in determining the target confidence, and the error in detecting the target category. The target position prediction loss function includes the center coordinate error and the width and height coordinate error; the target confidence error loss function consists of two parts, namely, with a target and without a target, and a weight coefficient is added to the loss part without a target to reduce the contribution weight of the calculation part without a target; the cross entropy function is selected as the loss function for the error in detecting the target category.

[0013] To achieve the above objectives, the present invention also discloses a target recognition system based on deep learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0014] The present invention has the following beneficial effects:

[0015] 1. Originally disclose a Z4 module and use it for residual processing in deep learning networks; its core function is to learn residual features, while maximizing the retention of specific feature information and preventing gradient disappearance. As the network depth increases, this structure effectively prevents the problem of gradient diffusion in gradient operations. Moreover, the deeper the network is, the more network layers there are, and the Z4 module can prevent the entire convolutional network from experiencing network degradation.

[0016] 2. The present invention can automatically identify targets in various complex environments, greatly improving the accuracy and real-time performance of target identification, especially the accuracy of identifying objects such as target missiles in drone aerial images.

[0017] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 It is a schematic diagram of the architecture of the deep learning model disclosed in an embodiment of the present invention.

[0020] Figure 2 It is a structural schematic diagram of the residual processing layer disclosed in an embodiment of the present invention.

[0021] Figure 3 yes Figure 2 Schematic diagram of the structure of the residual unit in . DETAILED DESCRIPTION

[0022] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, but the present invention can be implemented in a variety of different ways as defined and covered by the claims. It is worth noting that in the description of the following exemplary embodiments, Backbone is the backbone network, Neck is the neck network, and Head is the head network; Conv is the convolution layer, Upsample is the upsampling layer, Concat is the feature concatenation layer, SPP is the spatial pyramid pooling layer, BN is the batch normalization layer, add is the addition operation, LeakyRLU is the activation function, and Focus is a functional module for slicing images; the function box containing Z4 represents the residual processing layer, Z4*3 indicates that the corresponding residual processing layer contains three Z4 modules for feature fusion, and Z4*1 indicates that the corresponding residual processing layer contains only a single Z4 module.

[0023] Embodiment 1:

[0024] This embodiment discloses an intelligent forest fire extinguishing unexploded target bomb identification system based on deep learning technology, which includes the following parts:

[0025] Image acquisition part: This part is responsible for acquiring image data from the target area photographed by the drone.

[0026] Image annotation part: This part annotates the collected image data to obtain a dedicated target missile data set.

[0027] Deep learning model part: This part includes multiple deep learning models, which are used to identify targets and missiles based on preprocessed image data. The model structure is as follows: Figure 1 As shown, it mainly includes the Backbone network, the Neck network based on FPN (feature pyramid network) + PAN (Path Aggregation Network), and the Head network used to correspond to targets of different scales.

[0028] Among them, Backbone is the backbone network, or called the main network. Head is the network that obtains the output content of the network, using the previously extracted features, and the head uses these features to make predictions. Neck is placed between the backbone and the head to better utilize the features extracted by the backbone.

[0029] Backbone network is used as the feature vector extraction network of the input image, such as Figure 1 As shown in Figure 2, the Backbone network contains 1 Focus layer, 4 convolutional layers, 4 residual processing layers with at least 1 Z4 module, and 1 SPP layer.

[0030] In the backbone network, the Focus structure slices the image, which can reduce information loss during downsampling and reduce parameter calculation (its structure uses the same Focus structure as the yolov5 model); the convolution layer extracts features from the image, which includes convolution, normalization (Batch Normalization, BN) and Leaky ReLU activation function, which are collectively referred to as convolution layers Conv in this embodiment.

[0031] In the backbone network, the Z4 module is the main module for learning residual features. Its structure is as follows Figure 2As shown in the figure, it contains 3 standard convolution layers and 4 residual units. Its structure is divided into two branches, one of which only undergoes one standard convolution, and the other undergoes one layer of convolution and 4 residual units. Finally, the two branches are fused. While reducing the number of network parameters, the Z4 module can enhance the network's ability to learn and extract target features, and prevent the network from experiencing gradient vanishing or gradient explosion due to the increase in the number of layers.

[0032] Preferably, if Figure 3 As shown, each residual unit in the Z4 module of this embodiment is provided with two branches for inputting the add subunit; wherein, the first branch undergoes three layers of convolution in series, and each layer of convolution is processed by the BN layer and LeakyRLU before being output to the next layer of convolution; the second branch is directly connected to the input end; the add subunit fuses the features of the two branches and then outputs them after LeakyRLU activation processing. To some extent, the residual unit also has a Bottleneck effect.

[0033] Optionally, multiple Z4 modules can be set in the partial residual processing layer. Figure 1 Z4*3 means that three Z4 modules can be used for feature fusion. The convolution step size, convolution kernel size and other parameters in each residual unit can be flexibly set according to the implementation scenario and accuracy requirements. Optionally, some residual units can adopt the same series of parameters and adjust the parameters synchronously according to the user's settings.

[0034] The SPP (Spatial Pyramid Pooling) layer can achieve feature fusion of different scales through four maximum pooling layers with pooling kernel sizes of 1×1, 5×5, 9×9 and 13×13 respectively.

[0035] In this embodiment, the Neck network adopts the feature pyramid structure of FPN+PAN. FPN is a top-down network structure that gradually passes high-level features, while PAN is a bottom-up network that propagates low-level feature information upward. The two fuse features of different scales, which can effectively improve the model's detection ability for targets of different sizes. The Neck network structure extracts the feature hierarchy within the network on FPN. The top information flow in FPN needs to be passed down layer by layer through the backbone network (Backbone). Since the number of layers is relatively large, the amount of calculation is relatively large. The Neck network structure introduces a bottom-up path based on FPN. After the top-down feature fusion, the bottom-up feature fusion is performed, so that the bottom-level location information can also be passed to the deep layer, thereby enhancing the positioning capability at multiple scales.

[0036] Compared with FPN, the Neck network adds a bottom-up Neck network, which improves the network's multi-level feature fusion capability. At the same time, the Neck layer not only realizes the fusion of high-level and low-level multi-scale features, but also increases the network's receptive field.

[0037] The head part is mainly used to detect targets. For example, when the method of this embodiment is applied to blast recognition in a fire-extinguishing bomb test range, feature map sizes of 20*20, 40*40 and 80*80 can be output respectively, corresponding to targets of 32*32, 16*16 and 8*8 pixels.

[0038] For the specific scenario of blast recognition applied to the fire-extinguishing bomb test range, the Head of this embodiment can expand the number of channels of the feature maps of different scales obtained in the Neck by 1×1 convolution. The number of feature channels after expansion is (number of categories + 5) × the number of anchor boxes on each detection layer. Among them, 5 corresponds to the horizontal coordinate, vertical coordinate, width, height and confidence of the center point of the prediction box, and the confidence here indicates the credibility of the prediction box, and the value range is (0,1). The larger the value, the more likely it is that there is a target in the prediction box.

[0039] Among them, preferably, the three detection layers in the Head correspond to the three feature maps of different sizes obtained in the Neck. Each grid on the feature map is preset with three anchor boxes of different aspect ratios, and all the position information and classification information based on the anchor box prior box can be saved in the channel dimension of the feature map to predict and regress the target. In general, the depth of each prediction vector finally output by the model is 9, which contains 4 bounding box coordinate values, 1 bounding box confidence value, and 4 category probability values ​​(4 refers to the number of categories, for example: including four combinations of two types of ammunition and whether to explode). Its mathematical expression is (4+1+4)=9.

[0040] In the recognition result output part, each output detection frame vector can be counted by the set confidence threshold and category threshold, and then the detection target frame filtered by the threshold is output. In this way, the recognition result can be visualized and the user can be provided with intuitive recognition results. In this way, by training the deep learning model, this embodiment can accurately identify unexploded target bombs.

[0041] Preferably, the loss function of the deep learning model of this embodiment during the training process includes the error in predicting the target position, the error in determining the target confidence, and the error in detecting the target category. The target position prediction loss function includes the center coordinate error and the width and height coordinate error; the target confidence error loss function consists of two parts, namely, with a target and without a target, and a weight coefficient is added to the loss part without a target to reduce the contribution weight of the calculation part without a target.

[0042] Preferably, the target position loss function The specific calculation formula is:

[0043] ;

[0044] in, is the weight coefficient containing the target; , Respectively represent the width and height of the target box. , Represents the real coordinates of the target box.

[0045] Target confidence error loss function Variance can be used for calculation. Whether there is a certain category of target in the anchor box, the confidence error will be calculated. Because the confidence indicates the confidence that there is indeed a target in the box and the confidence that the box includes all the features of the entire target. It consists of two parts, namely, with a target and without a target. The loss without a target has a weight coefficient added The reason is that in most cases, the detected target of an image only occupies a small part of the image area. When performing target detection, most of the image area without targets will occupy a large contribution to the calculation, which will cause the image with the target to have a low contribution in the same image. This will eventually cause the convolutional network to determine that the target in the predicted cell is invalid. So we have to add a weight coefficient here. So we have to reduce the contribution weight of the part without the target calculation. The calculation formula is as follows:

[0046] ;

[0047] in, Indicates a situation with a goal, Indicates a situation where there is no goal; Represents the confidence of the prediction box, and the confidence of the real target box is Indicates that during the training process using the data set, the weight coefficient containing the target The value is 5, which will not include the weight coefficient of the target The value is 0.5.

[0048] Error in detecting target categories The cross entropy function can be selected as the loss function, and the calculation formula can be:

[0049] ;

[0050] The input image is divided into × grids, each grid generates B candidate boxes, each candidate box gets its corresponding detection box (bounding box) through the network, and the total will be × ×B detection frames; The value of is divided into two cases: if The first If a candidate box contains the predicted target, then The predicted value is set to 1. On the contrary, if Grid If a candidate box does not contain the predicted target, The predicted value is set to 0; Represents the elements in the category set classes, which includes the above 4 cases; Indicates grid target category prediction value, Indicates The true values ​​of the target categories in the grid.

[0051] Embodiment 2:

[0052] This embodiment discloses a target recognition system based on deep learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps based on the same subject as those in the above-mentioned embodiment 1 are implemented:

[0053] Step S1: Acquire image data taken by a drone.

[0054] Step S2: annotate the collected image data and construct a data set.

[0055] Step S3, determine the architecture of the deep learning model, train, verify and test the relevant parameters in the architecture of the deep learning model based on the data set to obtain the final deep learning model; then perform target recognition according to the deep learning model.

[0056] Among them, the architecture of the deep learning model includes a Backbone network for feature extraction, a Neck network based on FPN+PAN, and a Head network for corresponding targets of different scales. The Backbone network includes 1 Focus layer, 1 SPP layer, and a number of convolutional layers that is at least the number of target scale classifications plus 1, and a residual processing layer with at least 1 Z4 module, wherein 1 residual processing layer is located after the SPP, and the other residual processing layers are located between the two convolutional layers before the SPP layer; the Z4 module includes 3 convolutional layers, 4 residual units, and is provided with two branches, the first branch passes through a standard convolution, and the second branch first passes through 4 parallel residual units and then a layer of convolution in series, and the two branches are output after feature fusion and activation processing by the BN layer and LeakyRLU.

[0057] In the Neck network, a residual processing layer including one Z4 module is also provided after each Concat layer for realizing feature fusion.

[0058] Preferably, each residual unit in the Z4 module is provided with two branches for inputting the add subunit; wherein, the first branch undergoes three layers of series convolution, and after each layer of convolution, it is processed by the BN layer and LeakyRLU before being output to the next layer of convolution; the second branch is directly connected to the input end; the add subunit fuses the features of the two branches and then outputs them after LeakyRLU activation processing.

[0059] Similarly, the system can be applied to explosion identification in fire-extinguishing bomb test ranges.

[0060] In summary, the method and system disclosed in the present invention have at least the following beneficial effects:

[0061] 1. Originally disclose a Z4 module and use it for residual processing in deep learning networks; its core function is to learn residual features, while maximizing the retention of specific feature information and preventing gradient disappearance. As the network depth increases, this structure effectively prevents the problem of gradient diffusion in gradient operations. Moreover, the deeper the network is, the more network layers there are, and the Z4 module can prevent the entire convolutional network from experiencing network degradation.

[0062] 2. The present invention can automatically identify targets in various complex environments, greatly improving the accuracy and real-time performance of target identification, especially the accuracy of identifying objects such as target missiles in drone aerial images.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A target recognition method based on deep learning, characterized in that: The method is applied to blast identification at a fire-extinguishing bomb test range, and includes: Acquire image data taken by drones; Annotate the collected image data and build a data set; Determine the architecture of the deep learning model, and obtain a final deep learning model after training, verifying and testing relevant parameters in the architecture of the deep learning model based on the data set; and then perform target recognition according to the deep learning model; Among them, the architecture of the deep learning model includes a Backbone network for feature extraction, a Neck network based on FPN+PAN, and a Head network for corresponding targets of different scales. The Backbone network includes 1 Focus layer, 1 SPP layer, and a number of convolutional layers at least equal to the number of target scale classifications plus 1, and a residual processing layer with at least 1 Z4 module, wherein 1 residual processing layer is located after the SPP, and the other residual processing layers are located between the two convolutional layers before the SPP layer; the Z4 module includes 3 convolutional layers, 4 residual units, and is provided with two branches. During the processing process, the Z4 module first processes through the first convolutional layer, and the features processed by the first convolutional layer enter the two branches respectively. The first branch passes through a standard convolution, and the second branch first passes through 4 parallel residual units and then a layer of convolution in series. After the two branches are feature fused, they are successively processed by the BN layer and the LeakyRLU activation process and then output; In the Neck network, a residual processing layer including one Z4 module is also provided after each Concat layer for realizing feature fusion.

2. The target recognition method based on deep learning according to claim 1, characterized in that: Each residual unit in the Z4 module is provided with two branches for inputting the add subunit; wherein, the first branch undergoes three layers of series convolution, and after each layer of convolution, it is processed by the BN layer and LeakyRLU before being output to the next layer of convolution; the second branch is directly connected to the input end; the add subunit fuses the features of the two branches and then outputs them after LeakyRLU activation processing.

3. The target recognition method based on deep learning according to claim 1 or 2, characterized in that: The target scales include large-scale targets, medium-scale targets and small-scale targets, and the corresponding pixel sizes are 32*32, 16*16 and 8*8 respectively.

4. The target recognition method based on deep learning according to claim 3 is characterized in that: In the Head network, the sizes of feature maps for large, medium and small scale targets are 20*20, 40*40 and 80*80 respectively. The feature maps of detection layers of different scales are expanded by 1×1 convolution. The three detection layers correspond to feature maps of three different sizes respectively. The number of feature channels after expansion is: (number of categories + Z) × number of anchor boxes on each detection layer. The Z corresponds to the horizontal coordinate, vertical coordinate, width, height and confidence of the center point of the prediction box, and the value is 5; each grid on the feature map has three anchor boxes with different aspect ratios preset, so that the depth of each prediction vector finally output by the model is 9, which includes 4 box coordinate values, 1 box confidence value, and 4 category probability values; among them, the 4 categories correspond to four combinations of whether to blast and whether there is a target.

5. The target recognition method based on deep learning according to claim 4, characterized in that: The loss function of the deep learning model during the training process includes the error in predicting the target position, the error in determining the target confidence, and the error in detecting the target category. The target position prediction loss function includes the center coordinate error and the width and height coordinate error; the target confidence error loss function consists of two parts, namely, with a target and without a target, and a weight coefficient is added to the loss part without a target to reduce the contribution weight of the calculation part without a target.

6. A target recognition system based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

Images