A target recognition method and apparatus based on a MASK RCNN network model

Abstract

This invention discloses a target recognition method and apparatus based on a MASK R-CNN network model. The method includes: determining a multi-stage network as the base network; selecting at least one intermediate layer in the base network capable of extracting feature maps; inputting the feature maps output by the intermediate layers and the feature maps output by the terminal layers of the base network into corresponding MASK R-CNN recognition networks, thereby constructing a MASK R-CNN-based network model; training each MASK R-CNN recognition network using a dataset; and using the trained MASK R-CNN recognition networks to recognize targets. This scheme is well-suited for small target recognition in flying drones, avoiding interference from background information caused by manual target selection, providing a more precise tracking area for subsequent target tracking, and significantly improving the accuracy of small target recognition.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) target recognition, and more specifically to a target recognition method, apparatus, electronic device, and readable storage medium based on a MASK RCNN network model. Background Technology

[0002] The identification and tracking of moving ground targets by unmanned aerial vehicles (UAVs) has significant theoretical research value and application value, and is an important research direction in the field of autonomous control of unmanned aerial vehicle systems.

[0003] Currently, most drone tracking targets are selected using a bounding box method. The specific operation is as follows: the operator selects the target by dragging their finger on the mobile phone interface. When the drone is in flight, the target will move relative to the camera. When selecting the target on the mobile phone interface, marking errors are prone to occur, especially for small targets, which are more likely to be mixed with background information, making it easy for the target marking to be inaccurate and the marking to be offset. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a target recognition method, apparatus, electronic device and readable storage medium based on a MASK RCNN network model to overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present invention, a target recognition method based on a MASK RCNN network model is provided, the method comprising:

[0006] A multi-stage network was chosen as the basic network.

[0007] In the base network, at least one intermediate layer that can extract feature maps is selected. The feature maps output by the intermediate layer and the feature maps output by the end layer of the base network are respectively input into the corresponding MASK RCNN recognition network to construct a MASK RCNN-based network model. The feature maps output by the intermediate layer and the feature maps output by the end layer have different sizes.

[0008] The dataset is used to train each MASK RCNN recognition network until the preset training termination condition is met and training stops.

[0009] The trained MASK RCNN recognition networks are used to identify the target.

[0010] Optionally, at least one intermediate layer in the base network that can extract feature maps may be selected, including:

[0011] Two intermediate layers are selected, which can extract low-level feature maps and high-level feature maps of different sizes;

[0012] A first target recognition network constructed using low-level feature maps is used to identify the first target.

[0013] A second target recognition network constructed using high-level feature maps is used to identify the second target.

[0014] A third recognition network is constructed using the feature maps output from the terminal layer of the basic network to identify a third target.

[0015] The size of the first target is smaller than the size of the second target, and the size of the second target is smaller than the size of the third target.

[0016] Optionally, the intermediate layer includes a first type and a second type. The first type includes a convolutional layer and an activation function, and the second type includes a convolutional layer, an activation function, and a pooling layer.

[0017] Optionally, the first target recognition network constructed using low-level feature maps identifies the first target by including:

[0018] In the RPN network of the first target recognition network, the ratios parameter value and the scales parameter value are set so that each pixel in the low-level feature map corresponds to an anchor box region in the sample image input during the training of the first recognition network that is greater than a preset threshold, thereby covering the first target in the sample image.

[0019] Optionally, the first target recognition network constructed using low-level feature maps to recognize the first target also includes:

[0020] Determine whether the anchor frame region is the first target. If it is the first target, adaptively adjust the cross-union ratio parameter value in the first target recognition network according to the cross-union ratio of the anchor frame region.

[0021] Optionally, determining whether the anchor box region in the RPN network of the first target recognition network is the first target, and if it is the first target, adaptively adjusting the cross-union ratio (CUP) parameter value in the first target recognition network based on the CUP value of the anchor box region includes:

[0022] The area of ​​the preset standard region and the intersection-union ratio of the standard region are used. The standard region is the critical region between the second target and the first target.

[0023] Determine the area of ​​the anchor frame region, and determine the intersection ratio of the anchor frame region based on the product of the ratio of the area of ​​the anchor frame region to the area of ​​the standard region and the intersection ratio of the standard region.

[0024] The cross-union ratio of the anchor frame region is compared with that of the standard region. If the cross-union ratio of the anchor frame region is less than that of the standard region, and the target corresponding to the anchor frame region is determined to be the first target, then the cross-union ratio of the anchor frame region is set as the cross-union ratio parameter value of the first target recognition network, thereby realizing the adaptive adjustment of the cross-union ratio parameter value of the first target recognition network model.

[0025] Optionally, training the object recognition model using the preprocessed dataset includes:

[0026] Select or capture target images from different angles, distances, and sizes, normalize the target images, and generate a dataset; or,

[0027] Generate a dataset by selecting or capturing target images of the same size from different angles and distances.

[0028] According to another aspect of the present invention, a target recognition device based on a MASK RCNN network model is provided, the device comprising:

[0029] The network determination unit is suitable for determining a multi-stage network as the base network.

[0030] The model building unit is suitable for selecting at least one intermediate layer in the base network that can extract feature maps, and inputting the feature maps output by the intermediate layer and the feature maps output by the terminal layer of the base network into the corresponding MASK RCNN recognition network, thereby constructing a MASK RCNN-based network model, wherein the feature maps output by the intermediate layer and the feature maps output by the terminal layer have different sizes.

[0031] The model training unit is suitable for training each MASK RCNN recognition network using a dataset until the preset training termination condition is met and training stops.

[0032] The target recognition unit is suitable for recognizing targets using pre-trained MASK RCNN recognition networks.

[0033] Optionally, the model building unit is suitable for:

[0034] Two intermediate layers are selected, which can extract low-level feature maps and high-level feature maps of different sizes;

[0035] A first target recognition network constructed using low-level feature maps is used to identify the first target.

[0036] A second target recognition network constructed using high-level feature maps is used to identify the second target.

[0037] A third recognition network is constructed using the feature maps output from the terminal layer of the basic network to identify a third target.

[0038] The size of the first target is smaller than the size of the second target, and the size of the second target is smaller than the size of the third target.

[0039] Optionally, the model building blocks are also suitable for:

[0040] In the RPN network of the first target recognition network, the ratios parameter value and the scales parameter value are set so that each pixel in the low-level feature map corresponds to a region in the image input to the first recognition network that is greater than a preset threshold number, thereby covering the first target in the sample image.

[0041] Determine whether the anchor box region in the RPN network of the first target recognition network is the first target. If it is the first target, adaptively adjust the cross-union ratio parameter value in the first target recognition network according to the cross-union ratio of the anchor box region.

[0042] According to another aspect of the present invention, an electronic device is provided, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.

[0043] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, which, when executed by a processor, implement any of the methods described above.

[0044] As can be seen from the above, the technical solutions of the embodiments of the present invention can achieve the following technical effects:

[0045] The target recognition method based on the MASK R-CNN network model provided in this invention includes: firstly, selecting and determining a multi-stage network as the base network; selecting at least one intermediate layer in the base network that can extract feature maps, and inputting the feature maps output by the intermediate layer and the feature maps output by the terminal layer of the base network into the corresponding MASK R-CNN recognition networks, thereby constructing a MASK R-CNN network model, wherein the feature maps output by the intermediate layer and the feature maps output by the terminal layer have different sizes; then, training each MASK R-CNN recognition network using a dataset until training stops when a preset training termination condition is met; finally, using the trained MASK R-CNN recognition networks to recognize targets. The above technical solution automatically recognizes targets by using a trained network model, abandoning the traditional method that requires manual target selection by an operator, significantly reducing the impact of human error on recognition accuracy; furthermore, to achieve better recognition results for various targets, including small targets, this embodiment improves the MASK R-CNN network model by using multi-branch feature map acquisition, utilizing feature maps of different dimensions to recognize targets, avoiding the omission of tracked targets.

[0046] Experiments have shown that the above scheme can be well applied to the identification of small targets in flying drones, and avoids the interference of background information caused by human selection of targets, which greatly improves the accuracy of small target identification and provides a more precise tracking area for the next step of target tracking.

[0047] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0048] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0049] Figure 1 A flowchart illustrating a target recognition method based on a MASK RCNN network model according to an embodiment of the present invention is shown.

[0050] Figure 2 A schematic diagram of the structure of a target recognition device based on a MASK RCNN network model according to an embodiment of the present invention is shown.

[0051] Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown;

[0052] Figure 4 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention is shown;

[0053] Figure 5 A schematic diagram of a drone target identification and tracking process according to an embodiment of the present invention is shown;

[0054] Figure 6 A schematic diagram of the structure of a MASK RCNN network model according to an embodiment of the present invention is shown. Detailed Implementation

[0055] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0056] The technical field of this invention is unmanned aerial vehicle (UAV) target recognition and tracking, and its overall process is described in [link to documentation]. Figure 5 As shown, the process includes steps such as image data acquisition for small targets, image sample training, model convergence training, obtaining MASK R-CNN (MaskRegion-CNN, a network model based on convolutional networks for candidate region extraction masks) after training, using this network model to identify targets, and tracking targets. In the target identification stage, if no target is identified, the image is searched for and re-identified; if the target is lost, the image also needs to be searched again to obtain the identified target.

[0057] The embodiments of the present invention mainly involve improving the accuracy of target recognition by improving the network model of target recognition. The technical concept includes: abandoning the method of manually selecting target regions, constructing a neural network model to identify the target to be tracked, improving the recognition accuracy of small targets by increasing the number of branches of MASK RCNN to acquire feature maps, and acquiring feature maps through high-dimensional and low-dimensional layers respectively to avoid missing small targets, and optimizing the acquisition method of the intersection-union parameter value in the recognition network to address the inaccuracy of small target detection.

[0058] Figure 1 A flowchart illustrating a target recognition method based on a MASK RCNN network model according to an embodiment of the present invention is shown; the method includes:

[0059] Step S110: Determine a multi-stage network as the base network.

[0060] As the first step in building a network model, it is necessary to select a base network. Generally, base networks include Vgg network, Inception network, etc. In this embodiment, in order to identify targets of different sizes, feature maps are extracted from different network stages to construct recognition network branches. It is preferable to use a base network with multiple pooling layers. For example, low-level feature maps can be used to identify small targets.

[0061] In this embodiment, the base network can be determined as Vgg16, which includes 13 convolutional (conv) layers, 13 activation function (ReLU) layers, and 4 pooling layers. The activation function layers do not change the image size, while the pooling layers are feature extraction layers that can change the image size. The parameters of the 4 pooling layers are set as follows: kernel size = 2, stride = 2. After each pooling layer, the output image is half the size of the input image. After passing through the Vgg16 network, the size of the original input image becomes (M / 16)*(N / 16). Assuming the sample image has dimensions of 1000*600, the feature map sizes are 500*300*512, 250*150*512, 125*75*512, and 60*40*512, respectively, where 512 is the number of convolutional kernels in each convolutional layer, also known as the number of channels.

[0062] Step S120: Select at least one intermediate layer in the base network that can extract feature maps, and input the feature maps output by the intermediate layer and the feature maps output by the end layer of the base network into the corresponding MASK RCNN recognition network, thereby constructing a MASK RCNN-based network model, wherein the feature maps output by the intermediate layer and the feature maps output by the end layer have different sizes.

[0063] This step is the core of neural network model construction. To achieve the goal of recognizing targets of different sizes using feature maps of different sizes, this embodiment sets up at least two recognition networks to construct a network model based on MASK R-CNN. One recognition network acquires the feature map output from the final layer of the base network, while the other recognition networks select the feature map output from at least one intermediate layer of the base network. The recognition network in this embodiment is an improvement on the MASK R-CNN model.

[0064] It should be noted that, see Figure 6As shown, taking the MASK RCNN+Vgg16 network model as an example, the original network model only outputs a feature map at the back-end conv13. This processing has poor recognition performance for small targets (such as targets occupying a small area at the edge of an image). The improved model extracts feature maps at the front end (such as conv2 and conv9), i.e. Figure 6 This is a better extraction method.

[0065] Preferably, since each pooling operation changes the size of the feature map, for example... Figure 6 The image sizes at conv2 and conv3 are the same. To achieve better recognition results, multiple feature maps of different sizes can be extracted.

[0066] Step S130: Train each MASK RCNN recognition network using the dataset until the preset training termination condition is met and training stops.

[0067] Multiple generated MASK R-CNN recognition networks were trained separately to obtain recognition results for targets of different sizes. During training, pre-set conditions for termination can be set, such as stopping training when the target recognition accuracy reaches 95%.

[0068] Step S140: Use the trained MASK RCNN recognition networks to identify the target.

[0069] The trained model, which has now converged, can be used to identify moving ground targets in images obtained by UAVs. Experiments have shown that the accuracy of identifying moving ground targets, such as vehicles, using this model has been greatly improved.

[0070] In summary, this embodiment improves upon the existing MASK RCNN model, obtaining multiple MASK RCNN recognition networks, which greatly enhances the recognition effect of targets, especially small targets, avoids interference from background information, and provides a more accurate region for subsequent target tracking.

[0071] In one embodiment, selecting at least one intermediate layer in the base network capable of extracting feature maps in step S120 includes:

[0072] Two intermediate layers are selected, which can extract low-level feature maps and high-level feature maps of different sizes;

[0073] A first target recognition network constructed using low-level feature maps is used to identify the first target.

[0074] A second target recognition network constructed using high-level feature maps is used to identify the second target.

[0075] A third recognition network is constructed using the feature maps output from the terminal layer of the basic network to identify a third target.

[0076] The size of the first target is smaller than the size of the second target, and the size of the second target is smaller than the size of the third target.

[0077] See Figure 6 This embodiment describes a model capable of identifying targets of three sizes: large, medium, and small. Figure 6 The lower-level feature maps are obtained from the conv2 layer, and the higher-level feature maps are obtained from the conv9 layer. These, along with the feature maps output from the final layer of the base network, are used to construct the first, second, and third target recognition networks, respectively, to recognize small, medium, and large targets. Experimental results show that using the feature maps output from the conv2 and conv9 layers combined with the final conv13 layer for target recognition yields better recognition results.

[0078] In one embodiment, the intermediate layer includes a first type and a second type, the first type including a convolutional layer and an activation function, and the second type including a convolutional layer, an activation function, and a pooling layer.

[0079] See Figure 6 The intermediate layers in the base network can be divided into two types: the first type of intermediate layer only includes convolutional layers and activation functions, and the second type of intermediate layer can include convolutional layers, activation functions and pooling layers.

[0080] In addition, the MASK RCNN network model also includes an RPN network, a fully connected (FC) layer, classification, regression, and segmentation components. Figure 6 The RPN (Region Proposal Network) in the text is a network used to extract candidate bounding box regions; Figure 6 The fully connected layer (FC) in the convolutional neural network plays a "classification" role. If the operations of convolutional layers, pooling layers and activation function layers are to map the original data to the hidden feature space, then the fully connected layer maps the learned "distributed feature representation" to the sample label space. Figure 6 In this context, classification refers to the classifier, which uses functions such as softmax to classify data samples. For example, if an image is divided into 5 classes, the classifier can determine which class the image should be classified into, and the output is a discretized result. Regression, on the other hand, outputs a continuous result, generally an optimal result, including obtaining an optimal bounding box. Segmentation refers to the mask branch of the MASK RCNN network model, which generally identifies small targets by segmenting a three-dimensional region of interest from an image.

[0081] In one embodiment, identifying a first target using a first target recognition network constructed with low-level feature maps includes: in the RPN network of the first target recognition network, setting the ratios parameter value and the scales parameter value so that each pixel in the low-level feature map corresponds to an anchor box region in the image input to the first recognition network that is greater than a preset threshold number, thereby being able to cover the first target in the sample image.

[0082] After the convolutional layer, the size of the lowest layer feature image becomes 1 / 16 of the original image. When generating the anchor box, a base anchor box can be defined first. The base anchor box is a 16*16 bounding box. Since it is a point on the feature map (60*40), it can correspond to a 16*16 region on the original image (1000*600).

[0083] If the parameters ratios = [0.5, 1, 2] and scales = [0.5, 1, 2], the size of the corresponding region on the original image is obtained after parameter calculation. Here, ratio refers to the aspect ratio of the generated anchor box. One ratio value means that an anchor box with that aspect ratio is generated. Scale refers to the proportion of the area of ​​the generated anchor box, that is, generating anchor boxes with different area sizes. At the same time, ratio and scale determine the size of the region to be matched on the original image. The anchor box is obtained based on each pixel of the feature map. Each pixel will obtain the coordinate information of 9 anchor boxes in the original image. This method can avoid the position of the deepest feature map corresponding to the original image being offset due to downsampling, and at the same time, it can also correspond well to the region of the original image.

[0084] Based on the ratios and scales parameter values ​​in the recognition model, anchor box regions of different shapes and sizes can be generated. The more ratios and scales parameter values ​​there are, the more anchor box regions there are in the original input image, thus obtaining a finer-grained target region, which can cover the targets in the image, especially better identify small targets.

[0085] Preferably, in this embodiment, two elements, a and b, are added to the scales parameter. The specific values ​​of a and b are set according to the requirements, so the scales parameter is [a, 0.5, b, 1.0, 2.0]. The ratios parameter can remain unchanged, or one or more elements can be added to the ratios parameter. In this case, the target in the scene can be well covered, and diverse anchor boxes can be obtained.

[0086] In one embodiment, the first target recognition network constructed using low-level feature maps to identify the first target further includes: determining whether the anchor box region is the first target; if it is the first target, adaptively adjusting the cross-union ratio parameter value in the first target recognition network according to the cross-union ratio of the anchor box region.

[0087] Because ROI pooling involves two quantization operations and floating-point rounding, the regions in the original image corresponding to the feature maps are biased, especially for the first target. This causes a deviation in the Intersection-over-Union (IoU) ratio between the anchor boxes and the ground truth regions labeled in the original image. For the first target, although its region in the original image is small and the determined anchor box deviation is not significant, the IoU value may not meet a certain threshold due to the small target region, making it unsuitable for training and resulting in poor model robustness.

[0088] Intersection over Union (IoU) is a concept used in object detection. It represents the overlap ratio between the generated candidate bounding boxes and the original labeled ground truth bounding boxes; that is, the ratio of their intersection to their union. Ideally, they should overlap completely, meaning the ratio is 1.

[0089] To address this issue, this embodiment creates an IoUnet module in the RPN network. This IoUnet module can automatically identify whether the anchor frame is a region where deviation occurs and whether it is a small target region, and further creates conditions for adaptively adjusting the cross-union ratio parameter value in the identification network.

[0090] In one embodiment, determining whether an anchor box region in the RPN network of the first target recognition network is a first target, and if it is a first target, adaptively adjusting the cross-union ratio (CUP) parameter value in the first target recognition network based on the CUP value of the anchor box region includes:

[0091] The area of ​​the preset standard region and the intersection-union ratio of the standard region are used. The standard region is the critical region between the second target and the first target.

[0092] Determine the area of ​​the anchor frame region, and determine the intersection ratio of the anchor frame region based on the product of the ratio of the area of ​​the anchor frame region to the area of ​​the standard region and the intersection ratio of the standard region.

[0093] The cross-union ratio of the anchor frame region is compared with that of the standard region. If the cross-union ratio of the anchor frame region is less than that of the standard region, and the target corresponding to the anchor frame region is determined to be the first target, then the cross-union ratio of the anchor frame region is set as the cross-union ratio parameter value of the first target recognition network, thereby realizing the adaptive adjustment of the cross-union ratio parameter value of the first target recognition network model.

[0094] This embodiment discloses a method for adaptively adjusting the intersection-union ratio (IU) parameter value in a first target recognition network. Through the above steps, based on the critical region between the second target and the first target, a standard region area and a standard region IU value are set. Based on the size relationship between the standard region and the anchor frame region of the suspected first target, the IU value corresponding to the anchor frame region is calculated. During training, if the anchor frame region is smaller than the standard region, and the target frame defined by the anchor frame region is indeed a small target, the IU value corresponding to the anchor frame region is manually adjusted and set as the IU parameter value in the first target recognition network.

[0095] The test results on the test set show that the cross-union ratio parameter value in the first target recognition network can be adjusted automatically according to the size of the anchor box region, indicating that it has acquired the ability to adaptively adjust.

[0096] This embodiment performs supervised classification and regression of anchorboxes to determine the target category and region. It learns by comparing the ground truth box with the predicted anchorbox, thereby continuously updating the network weights of the RPN and RCNN networks to obtain a trained model.

[0097] In one embodiment, training a target recognition model using a preprocessed dataset includes: selecting or capturing target images of different angles, distances, and sizes; normalizing the target images; and generating a dataset.

[0098] Alternatively, select or capture target images of the same size but from different angles and distances to generate a dataset.

[0099] This embodiment provides a method for obtaining or generating samples in a dataset to be trained. One method is to select or capture target images of different angles, distances, and sizes, and then normalize the target images to generate a dataset. Another method is to select or capture target images of different angles and distances but the same size to directly generate a dataset.

[0100] Figure 2 A schematic diagram of a target recognition device based on a MASK RCNN network model according to an embodiment of the present invention is shown; the device includes:

[0101] The network determination unit 210 is suitable for determining a multi-stage network as the base network.

[0102] As the first step in building a network model, it is necessary to select a base network. Generally, base networks include Vgg network, Inception network, etc. In this embodiment, in order to identify targets of different sizes, feature maps are extracted from different network stages to construct recognition network branches. It is preferable to use a base network with multiple pooling layers. For example, low-level feature maps can be used to identify small targets.

[0103] The model building unit 220 is adapted to select at least one intermediate layer in the base network that can extract feature maps, and input the feature maps output by the intermediate layer and the feature maps output by the terminal layer of the base network into the corresponding MASK RCNN recognition network, thereby constructing a MASK RCNN-based network model, wherein the feature maps output by the intermediate layer and the feature maps output by the terminal layer have different sizes.

[0104] To achieve the goal of recognizing targets of different sizes using feature maps of different dimensions, this embodiment sets up at least two recognition networks to construct a network model based on MASK R-CNN. One recognition network acquires the feature map output from the final layer of the base network, while the other recognition networks select feature maps output from at least one intermediate layer of the base network. The recognition network in this embodiment is an improvement on the MASK R-CNN model and may include the following components: an RPN network, several fully connected layers, a classifier, a regressor, and a segmentation mask branch.

[0105] The model training unit 230 is suitable for training each MASK RCNN recognition network using a dataset until the preset training termination condition is met.

[0106] Multiple generated MASK R-CNN recognition networks were trained separately to obtain recognition results for targets of different sizes. During training, pre-set conditions for termination can be set, such as stopping training when the target recognition accuracy reaches 95%.

[0107] The target recognition unit 240 is adapted to recognize targets using pre-trained MASK RCNN recognition networks.

[0108] The trained model, which has now converged, can be used to identify moving ground targets in images obtained by UAVs. Experiments have shown that the accuracy of identifying moving ground targets, such as vehicles, using this model has been greatly improved.

[0109] In summary, this embodiment improves upon the existing MASK RCNN model, obtaining multiple MASK RCNN recognition networks, which greatly enhances the recognition effect of targets, especially small targets, avoids interference from background information, and provides a more accurate region for subsequent target tracking.

[0110] In one embodiment, the model building unit 220 is adapted to: select two intermediate layers, which are capable of extracting low-level feature maps and high-level feature maps of different sizes;

[0111] A first target recognition network constructed using low-level feature maps is used to identify the first target.

[0112] A second target recognition network constructed using high-level feature maps is used to identify the second target.

[0113] A third recognition network is constructed using the feature maps output from the terminal layer of the basic network to identify a third target.

[0114] The size of the first target is smaller than the size of the second target, and the size of the second target is smaller than the size of the third target.

[0115] In one embodiment, the intermediate layer includes a first type and a second type, the first type including a convolutional layer and an activation function, and the second type including a convolutional layer, an activation function, and a pooling layer.

[0116] In one embodiment, the model building unit 220 is further adapted to: in the RPN network of the first target recognition network, set the ratios parameter value and the scales parameter value such that each pixel in the low-level feature map corresponds to a region in the image input to the first recognition network that is greater than a preset threshold number, thereby being able to cover the first target in the sample image;

[0117] Determine whether the anchor box region in the RPN network of the first target recognition network is the first target. If it is the first target, adaptively adjust the cross-union ratio parameter value in the first target recognition network according to the cross-union ratio of the anchor box region.

[0118] In one embodiment, the model building unit 220 is further adapted to:

[0119] The area of ​​the preset standard region and the intersection-union ratio of the standard region are used. The standard region is the critical region between the second target and the first target.

[0120] Determine the area of ​​the anchor frame region, and determine the intersection ratio of the anchor frame region based on the product of the ratio of the area of ​​the anchor frame region to the area of ​​the standard region and the intersection ratio of the standard region.

[0121] The cross-union ratio of the anchor frame region is compared with that of the standard region. If the cross-union ratio of the anchor frame region is less than that of the standard region, and the target corresponding to the anchor frame region is determined to be the first target, then the cross-union ratio of the anchor frame region is set as the cross-union ratio parameter value of the first target recognition network, thereby realizing the adaptive adjustment of the cross-union ratio parameter value of the first target recognition network model.

[0122] In one embodiment, the model training unit 230 is adapted to:

[0123] Select or capture target images from different angles, distances, and sizes, normalize the target images, and generate a dataset; or,

[0124] Generate a dataset by selecting or capturing target images of the same size from different angles and distances.

[0125] In summary, the target recognition method based on the MASK R-CNN network model disclosed in this invention includes: determining a multi-stage network as the base network; selecting at least one intermediate layer in the base network capable of extracting feature maps; inputting the feature maps output by the intermediate layers and the feature maps output by the terminal layers of the base network into corresponding MASK R-CNN recognition networks, thereby constructing a MASK R-CNN network model; training each of the MASK R-CNN recognition networks using a dataset; and using the trained MASK R-CNN recognition networks to recognize the target. This scheme can be well used for small target recognition in flying drones, avoiding interference from background information caused by target selection, providing a more accurate tracking area for subsequent target tracking, and greatly improving the accuracy of small target recognition.

[0126] It should be noted that:

[0127] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0128] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0129] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0130] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0131] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0132] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the target recognition device based on the MASK RCNN network model according to embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0133] For example, Figure 3 A schematic diagram of an electronic device according to an embodiment of the present invention is shown. The electronic device 300 includes a processor 310 and a memory 320 arranged to store computer-executable instructions (computer-readable program code). The memory 320 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 320 has a storage space 330 for storing computer-readable program code 331 for performing any of the method steps described above. For example, the storage space 330 for storing computer-readable program code may include various computer-readable program codes 331 respectively for implementing the various steps in the above methods. The computer-readable program code 331 can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. Such computer program products are typically, for example... Figure 4 Computer-readable storage media. Figure 4 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown. The computer-readable storage medium 400 stores computer-readable program code 331 for performing the method steps according to the present invention, which can be read by the processor 310 of an electronic device 300. When the computer-readable program code 331 is executed by the electronic device 300, it causes the electronic device 300 to perform the various steps of the method described above. Specifically, the computer-readable program code 331 stored in the computer-readable storage medium can perform the methods shown in any of the above embodiments. The computer-readable program code 331 can be compressed in a suitable form.

[0134] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A target recognition method based on a MASK RCNN network model, characterized in that, The method is applied to small target tracking and identification of unmanned aerial vehicles (UAVs), and the method includes: A multi-stage network was chosen as the basic network. In the base network, at least one intermediate layer capable of extracting feature maps is selected. The feature maps output by the intermediate layer and the feature maps output by the terminal layer of the base network are respectively input into the corresponding MASK RCNN recognition network to construct a MASK RCNN network model. The feature maps output by the intermediate layer and the feature maps output by the terminal layer have different sizes. The dataset is used to train each of the MASK RCNN recognition networks until the preset training termination condition is met and training stops. When the UAV is in flight, it uses the trained MASK RCNN recognition networks to track and identify the target, which includes moving targets moving on the ground. The step of selecting at least one intermediate layer in the base network capable of extracting feature maps includes: Two intermediate layers are selected, which can extract low-level feature maps and high-level feature maps of different sizes; The first target recognition network constructed using the low-level feature maps identifies the first target; The second target recognition network constructed using the high-level feature map identifies the second target; A third recognition network is constructed using the feature maps output from the terminal layer of the basic network to identify a third target; The size of the first target is smaller than the size of the second target, and the size of the second target is smaller than the size of the third target. Determining whether the anchor frame region in the RPN network of the first target recognition network is the first target, and if it is the first target, adaptively adjusting the cross-union ratio (CUP) parameter value in the first target recognition network based on the CUP value of the anchor frame region includes: The area of ​​the preset standard region and the intersection-union ratio of the standard region are used, wherein the standard region is the critical region between the second target and the first target; Determine the area of ​​the anchor frame region, and determine the intersection-union ratio of the anchor frame region based on the product of the area ratio of the anchor frame region to the standard region and the intersection-union ratio of the standard region; The crossover ratio of the anchor frame region is compared with the crossover ratio of the standard region. If the crossover ratio of the anchor frame region is less than the crossover ratio of the standard region, and the target corresponding to the anchor frame region is determined to be the first target, then the crossover ratio of the anchor frame region is set as the crossover ratio parameter value of the first target recognition network.

2. The method as described in claim 1, characterized in that, The intermediate layer includes a first type and a second type. The first type includes a convolutional layer and an activation function, and the second type includes a convolutional layer, an activation function, and a pooling layer.

3. The method as described in claim 1, characterized in that, The first target recognition network constructed using the low-level feature map identifies the first target by: In the RPN network of the first target recognition network, the ratios parameter value and the scales parameter value are set so that each pixel in the low-level feature map corresponds to an anchor box region in the image input by the first target recognition network that is greater than a preset threshold, thereby covering the first target in the sample image.

4. The method as described in claim 3, characterized in that, The first target recognition network constructed using the low-level feature map further includes the following for recognizing the first target: Determine whether the anchor frame region is the first target. If it is the first target, adaptively adjust the cross-union ratio parameter value in the first target recognition network according to the cross-union ratio of the anchor frame region.

5. The method as described in claim 1, characterized in that, Select or capture target images of different angles, distances, and sizes, and normalize the target images to generate the dataset; or... The dataset is generated by selecting or capturing target images of the same size but from different angles and distances.

6. A target recognition device based on a MASK RCNN network model, characterized in that, The device for small target tracking and recognition in unmanned aerial vehicles includes: The network determination unit is suitable for determining a multi-stage network as the base network. The model building unit is adapted to select at least one intermediate layer in the base network that can extract feature maps, and input the feature maps output by the intermediate layer and the feature maps output by the terminal layer of the base network into the corresponding MASK RCNN recognition network, thereby constructing a MASK RCNN network model, wherein the feature maps output by the intermediate layer and the feature maps output by the terminal layer have different sizes. The model training unit is adapted to train each of the MASK RCNN recognition networks using the dataset until the preset training termination condition is met and training stops. The target recognition unit is adapted to track and recognize the target using the trained MASK RCNN recognition networks when the UAV is in flight, and the target includes moving targets moving on the ground; The model building unit is suitable for: Two intermediate layers are selected, which can extract low-level feature maps and high-level feature maps of different sizes; The first target recognition network constructed using the low-level feature maps identifies the first target; The second target recognition network constructed using the high-level feature map identifies the second target; A third recognition network is constructed using the feature maps output from the terminal layer of the basic network to identify a third target; The size of the first target is smaller than the size of the second target, and the size of the second target is smaller than the size of the third target. Determining whether the anchor frame region in the RPN network of the first target recognition network is the first target, and if it is the first target, adaptively adjusting the cross-union ratio (CUP) parameter value in the first target recognition network based on the CUP value of the anchor frame region includes: The area of ​​the preset standard region and the intersection-union ratio of the standard region are used, wherein the standard region is the critical region between the second target and the first target; Determine the area of ​​the anchor frame region, and determine the intersection-union ratio of the anchor frame region based on the product of the area ratio of the anchor frame region to the standard region and the intersection-union ratio of the standard region; The crossover ratio of the anchor frame region is compared with the crossover ratio of the standard region. If the crossover ratio of the anchor frame region is less than the crossover ratio of the standard region, and the target corresponding to the anchor frame region is determined to be the first target, then the crossover ratio of the anchor frame region is set as the crossover ratio parameter value of the first target recognition network.

7. The apparatus as claimed in claim 6, characterized in that, The model building unit is also suitable for: In the RPN network of the first target recognition network, the ratios parameter value and the scales parameter value are set so that each pixel in the low-level feature map corresponds to a region in the sample image input during the training of the first target recognition network that is greater than a preset threshold number, thereby covering the first target in the sample image; Determine whether the anchor frame region in the RPN network of the first target recognition network is the first target. If it is the first target, adaptively adjust the cross-union ratio parameter value in the first target recognition network according to the cross-union ratio of the anchor frame region.

Images