UAV aerial video content recognition method, device and system

Abstract

The present invention discloses a method for identifying content in drone aerial video based on deep learning, comprising: acquiring video images captured by a drone in real time; performing frame extraction processing on the video images to extract sample images, and preprocessing the sample images; labeling objects to be identified in the sample images to generate an object category database; expanding the object category database using data enhancement technology; using the expanded object category database to train a deep neural network model, wherein the deep neural network model is an SSD network model, and the SSD network model includes a multi-branch convolutional structure and a multi-scale feature map fusion structure; and using the trained deep neural network model to identify the video images to output the location and size information of each object category. The present invention also discloses a device and system for identifying content in drone aerial video based on deep learning. The present invention can effectively solve the problem of mutual influence between category recognition and position positioning in aerial video object detection.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device and system for identifying content in drone aerial video based on deep learning. Background Art

[0002] In recent years, with the continuous development of computer, multimedia, and network technologies, video capture devices such as mobile phones, cameras, and surveillance cameras have rapidly become ubiquitous, and video resources are becoming increasingly abundant. Quickly and accurately capturing information from videos is becoming increasingly important, and deep learning-based methods hold enormous potential. Their application in the field of computer vision is booming, becoming a trend today and in the future. Furthermore, accurate real-time detection and location of scene text and object recognition in videos have important applications in scenarios such as public security surveillance, drone flight, and autonomous driving.

[0003] In the past, tasks such as understanding and identifying video content were all done manually, which is undoubtedly a time-consuming, labor-intensive and inefficient method. In recent years, domestic and foreign scholars have proposed many deep neural network models for extracting visual features. At the same time, some scholars have begun to study the application of deep neural networks in the video field and proposed some networks for video action recognition and feature extraction. In terms of video content retrieval, deep neural networks have also been introduced to extract structured information from videos.

[0004] To date, deep convolutional neural networks (DCNNs) have become a universal approach for object detection. Current high-performance object detection algorithms and the latest research are all based on DCNNs. To improve the speed of object detection networks, Liu et al. proposed the SSD network. This network performs object category classification and position regression on feature maps of different sizes, utilizing them to detect objects of varying sizes, eliminating the need for the RPN network and significantly increasing network speed. Furthermore, Redmon et al. proposed the YOLO network, which divides the original image into 7x7 regions and directly performs object category classification and position regression on each region using a neural network, eliminating the need for classification and regression operations on feature maps of varying sizes and further improving the speed of the object detection network. Compared to Fast-RCNN, the detection accuracy of SSD and YOLO networks is slightly lower. In addition, many other object detection network models have been proposed based on the ideas of the above three models. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, device and system for identifying content in drone aerial videos based on deep learning, which can effectively solve the problem of mutual influence between category recognition and position positioning in aerial video object detection.

[0006] In order to solve the above technical problems, the present invention provides a method for identifying content in drone aerial videos based on deep learning, including: acquiring video images taken by a drone in real time; performing frame extraction processing on the video images to extract sample images, and preprocessing the sample images; marking objects to be identified in the sample images to generate an object category database; expanding the object category database using data enhancement technology; training a deep neural network model using the expanded object category database, wherein the deep neural network model is an SSD network model, and the SSD network model includes a multi-branch convolution structure and a multi-scale feature map fusion structure; and using the trained deep neural network model to identify the video images to output position information and size information of each object category.

[0007] As an improvement to the above scheme, the step of preprocessing the sample image includes: using a distortion correction algorithm to correct the sample image to form a regular planar sample image; and compressing the corrected sample image to make the sample image reach a target size that can be used for target recognition.

[0008] As an improvement to the above solution, the method for labeling the objects to be identified in the sample image includes a manual labeling method and / or an image target detection algorithm.

[0009] As an improvement to the above scheme, the step of expanding the object category database using data enhancement technology includes: performing data enhancement processing on the sample images in the object category database in a random probability superposition manner, and the data enhancement processing includes rotation processing, filling-type cropping processing and grayscale data processing.

[0010] As an improvement to the above scheme, the step of training the deep neural network model using the expanded object category database includes: inputting multiple sample images in the object category database into the deep neural network model; performing convolution processing on the multiple sample images respectively through a multi-branch convolution layer; normalizing the multiple sample images after convolution processing to generate scale feature maps; performing feature fusion processing on all feature maps; and convolution processing on the spliced ​​feature maps through a convolution layer to generate branched convolution feature maps.

[0011] As an improvement to the above scheme, the step of performing feature fusion processing on all feature maps includes: unifying the size of all feature maps; performing category identification and position positioning processing on each feature map after the unified size; and performing feature fusion processing on all feature maps after identification and positioning processing in a weighted manner.

[0012] Correspondingly, the present invention also provides a device for identifying content in drone aerial video based on deep learning, comprising: an acquisition module for acquiring video images taken by a drone in real time; a preprocessing module for performing frame extraction on the video images to extract sample images and preprocessing the sample images; a labeling module for labeling objects to be identified in the sample images and generating an object category database; an expansion module for expanding the object category database using data enhancement technology; a training module for training a deep neural network model using the expanded object category database, wherein the deep neural network model is an SSD network model, and the SSD network model includes a multi-branch convolution structure and a multi-scale feature map fusion structure; and a recognition module for using the trained deep neural network model to identify the video images to output the position information and size information of each object category.

[0013] As an improvement of the above scheme, the training module includes: an input unit for inputting multiple sample images in the object category database into the deep neural network model; a first convolution unit for performing convolution processing on the multiple sample images through a multi-branch convolution layer; a normalization unit for performing normalization processing on the multiple sample images after convolution processing to generate a scale feature map; a fusion unit for performing feature fusion processing on all feature maps; a second convolution unit for performing convolution processing on the feature map after splicing through a convolution layer to generate a branch convolution feature map.

[0014] As an improvement of the above scheme, the fusion unit includes: a size adjustment subunit, which is used to unify the sizes of all feature maps; an identification and positioning subunit, which is used to perform category identification and position positioning processing on each feature map after the unified size; and a feature fusion subunit, which is used to perform feature fusion processing on all feature maps after identification and positioning processing in a weighted manner.

[0015] Correspondingly, the present invention also provides a UAV aerial video content recognition system based on deep learning, including a UAV platform and a UAV aerial video content recognition device. The UAV platform is a platform equipped with a visible light camera and a thermal infrared camera, and performs multi-source image acquisition.

[0016] The implementation of the present invention has the following beneficial effects:

[0017] The present invention improves the existing SSD network model by adding a multi-branch convolutional structure to the SSD network model to improve the network's detection performance for small targets. A multi-scale feature map fusion structure is used to fuse feature maps of different scales on multi-scale feature maps. The expanded object category database is used to train a deep neural network model, solving the problem of mutual influence between category recognition and position positioning in aerial video object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of an embodiment of a method for identifying content in drone aerial video based on deep learning of the present invention;

[0019] Figure 2 is a flow chart of an embodiment of the present invention for training a deep neural network model using an expanded object category database;

[0020] Figure 3 Schematic diagram of the multi-branch convolution structure of the SSD network model in the present invention;

[0021] Figure 4 Schematic diagram of the multi-scale feature map fusion structure of the SSD network model in the present invention;

[0022] Figure 5 is a schematic diagram of the SSD network model in the present invention;

[0023] Figure 6 It is a structural diagram of the UAV aerial video content recognition system based on deep learning of the present invention;

[0024] Figure 7 It is a structural schematic diagram of the UAV aerial video content recognition device based on deep learning of the present invention. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be described in further detail below with reference to the accompanying drawings.

[0026] See also Figure 1 , Figure 1 The flowchart of an embodiment of the method for identifying content in drone aerial video based on deep learning of the present invention is shown, which includes:

[0027] S101, real-time acquisition of video images captured by the drone.

[0028] The present invention uses a drone to take a bird's-eye view shot at low altitude, and transmits the captured video images to a host (i.e., a drone aerial video content recognition device) in real time, so that the host can obtain the video images captured by the drone in real time.

[0029] S102: Perform frame extraction processing on the video image to extract a sample image, and perform preprocessing on the sample image.

[0030] The frame extraction strategy can be to extract the first frame and the last frame at a specified time interval, or to extract random frames in a video. There is no specific restriction here and you can choose as needed.

[0031] Specifically, the steps of preprocessing the sample image include:

[0032] (1) Use the distortion correction algorithm to correct the sample image to form a regular planar sample image.

[0033] During correction, the fisheye lens image of the drone can be processed using a distortion correction algorithm to obtain a regular and flat sample image.

[0034] (2) Compressing the corrected sample image to make it reach the target size that can be used for target recognition.

[0035] After the correction is completed, the sample image size needs to be compressed to obtain several target sizes that can be used for target recognition. For example, the target size can be set to 300mm×300mm.

[0036] S103: Label the objects to be identified in the sample image to generate an object category database.

[0037] Methods for labeling objects to be identified in sample images include manual labeling methods and / or image target detection algorithms.

[0038] It should be noted that object labeling in sample images can be done manually (in the absence of relevant data), with only the categories required for content recognition annotated. Alternatively, with relevant underlying data, other image object detection algorithms can be used for automated labeling. Once labeling is complete, a database of object categories is generated, which serves as the target space for video content recognition.

[0039] S104, using data enhancement technology to expand the object category database.

[0040] Specifically, the step of expanding the object category database using data enhancement technology includes: performing data enhancement processing on sample images in the object category database in a random probability superposition manner, and the data enhancement processing includes rotation processing, filling-type cropping processing and grayscale data processing.

[0041] Using data augmentation technology to expand the object category database in step S103 to increase the content diversity of the object category database, specifically by transforming a sample image in the object category database into a new sample image through augmentation operations including rotation, padding cropping, and grayscale conversion;

[0042] It should be noted that the enhancement operation is only used during the training process and is not required in testing or actual applications. All enhancement operations are applied to the original sample image with a certain probability to ensure the randomness of the enhancement operation results, which are then used as input data for this iterative training of the model.

[0043] S105, using the expanded object category database to train a deep neural network model.

[0044] The deep neural network model is an SSD network model, which includes a multi-branch convolution structure and a multi-scale feature map fusion structure.

[0045] Unlike the prior art, the present invention improves the existing SSD network model by adding a multi-branch convolution structure to the SSD network model to improve the network's detection performance for small targets, and adopts a multi-scale feature map fusion structure to fuse feature maps of different scales on multi-scale feature maps. The object category database expanded in step S104 is used to train the deep neural network model, solving the problem of mutual influence between category recognition and position positioning in aerial video object detection.

[0046] S106: Use the trained deep neural network model to recognize the video image to output the location information and size information of each object category.

[0047] The trained deep neural network model is used to identify the content in the video image transmitted by the drone, locate the position and size of each object category, and finally output the object content category and position information in the corresponding window.

[0048] Therefore, by improving the SSD network model, the present invention can effectively improve the recognition speed and efficiency of the model when processing video images, effectively solve the contradiction between category recognition and position positioning in aerial video content detection, and improve the accuracy of content recognition.

[0049] See also Figure 2 , Figure 2 A flowchart of an embodiment of the present invention for training a deep neural network model using an expanded object category database is shown, which includes:

[0050] S201, input multiple sample images in the object category database into a deep neural network model.

[0051] S202: Perform convolution processing on the multiple sample images through a multi-branch convolution layer.

[0052] S203 , normalizing the multiple sample images after the convolution processing to generate a scale feature map.

[0053] S204: Perform feature fusion processing on all feature maps.

[0054] Specifically, the steps of performing feature fusion processing on all feature maps include:

[0055] (1) Unify the size of all feature maps;

[0056] (2) Perform category recognition and position positioning on each feature map after unifying the size;

[0057] (3) All feature maps after identification and positioning are subjected to feature fusion processing in a weighted manner.

[0058] S205: Convolutionally process the concatenated feature map through a convolution layer to generate a branched convolutional feature map.

[0059] Therefore, the present invention improves the SSD network model by adding a convolution kernel that branches first and then splices. It also fuses the multi-scale feature maps generated by the convolution kernel for similar images.

[0060] The following combination Figure 3-Figure 5 Further details on the training process of the deep neural network model:

[0061] like Figure 3 As shown in the figure, in the multi-branch convolutional structure of the improved SSD network model, the sample image is convolved with convolution kernels of dimensions n and m, normalized, and fused, and then subjected to 1×1 convolution to obtain the branched convolution feature map. The multi-branch convolution operation can perform multiple convolution and pooling operations. Preferably, n = 1 and m = 3.

[0062] like Figure 4 As shown in the figure, in the multi-scale feature map fusion structure of the improved SSD network model, the number of fused feature maps is set to k=3, and the input is 3 consecutive feature maps; the m-1 layer performs a convolution, the m layer feature map does not change, and the m+1 layer performs a deconvolution. After the sizes of the three feature maps are unified, category recognition and position positioning are performed on the three maps respectively. Finally, the results obtained from the three feature maps are fused in a weighted manner. The feature fusion weight parameter is learned by the network and is initially set to 1 / 3.

[0063] Therefore, unlike the traditional SSD network that performs category recognition and position positioning from a single feature map, the present invention inputs k consecutive feature maps (where the features of deep feature maps have stronger representation capabilities, and the features of shallow feature maps are conducive to position positioning). After converting feature maps of different scales to the same size, category recognition and position positioning are performed on k maps respectively.

[0064] like Figure 5As shown, the improved SSD network model follows the basic structure of the one-step network model. The backbone network of the model uses VGG-16. Starting from the last convolutional layer of the network, a multi-branch convolution kernel structure is continuously used to perform convolution and pooling several times, resulting in a total of several feature maps of different scales. A sliding window is used to slide over the feature map sequence. The sliding window size is set to t. Feature fusion of feature maps of different scales is performed. The t feature maps within the sliding window are used as input to the feature fusion algorithm. The position regression and category recognition results of the objects within the corresponding window are output in sequence. The final recognition result is obtained after non-maximum suppression. For example, the number of convolution and pooling is set to five, resulting in a total of six feature maps of different scales. The sliding window size can be set to t = 3.

[0065] From the above, it can be seen that the present invention improves the SSD network model by adding a convolution kernel that first branches and then splices. The multi-scale feature maps generated by the convolution kernel for similar images are fused using multi-scale feature maps, which effectively solves the contradiction between category recognition and position positioning in aerial video content detection and improves the accuracy of content recognition.

[0066] See also Figure 6 , Figure 6 The specific structure of the UAV aerial video content recognition system based on deep learning of the present invention is shown, which includes a UAV platform 1 and a UAV aerial video content recognition device 2.

[0067] The drone platform 1 is equipped with a visible light camera and a thermal infrared camera for multi-source image acquisition. The present invention uses a drone to shoot low-altitude aerial photography and transmits the captured video images in real time to the drone aerial video content recognition device, allowing the drone aerial video content recognition device to obtain the video images captured by the drone in real time.

[0068] Specifically, the drone platform 1 includes a power supply, a computer motherboard, a ground monitoring client, a visible light camera, a thermal infrared camera, a camera mounting bracket, an image acquisition card, a 4G module, and a base station. The drone platform is equipped with a flight controller, a power system, GPS, a battery, and other components, and supports module expansion. The computer motherboard, visible light camera, and thermal infrared camera are all fixed to the drone platform. The image acquisition card is used to ensure that the computer motherboard can capture image data from the thermal infrared camera. The computer motherboard is equipped with an image acquisition card driver, which uses the SDK development structure provided by the image acquisition card to program and synchronously capture data from the visible light camera and thermal infrared camera. The 4G module is mounted on the computer motherboard and connects to the base station via automatic dialing. The ground monitoring client is connected to the base station, ensuring that the computer motherboard on the drone is connected to the ground monitoring client.

[0069] like Figure 6As shown, the UAV aerial video content recognition device 2 based on deep learning includes an acquisition module 21, a preprocessing module 22, a labeling module 23, an expansion module 24, a training module 25 and a recognition module 26. Specifically:

[0070] The acquisition module 21 is used to acquire video images taken by the drone in real time.

[0071] The preprocessing module 22 is used to extract sample images by extracting frames from the video image and preprocess the sample images. Specifically, the frame extraction strategy can be to extract the first and last frames at a specified time interval, or to extract random frames from a video segment. There are no specific restrictions here and the selection can be made as needed. In addition, after the preprocessing module 22 completes the frame extraction process, it uses a distortion correction algorithm to correct the sample image to form a regular planar sample image. Next, the sample image is compressed and resized to obtain several target sizes that can be used for target recognition. For example, the target size can be set to 300mm×300mm.

[0072] The labeling module 23 is used to label the objects to be identified in the sample images and generate an object category database. It should be noted that the labeling module 23 can use manual labeling methods and / or image target detection algorithms to label the objects to be identified in the sample images. In the absence of relevant data, the labeling of various objects in the sample images can be manual, and only the categories requiring content recognition are labeled. Based on relevant basic data, other image target detection algorithms can be used for machine automatic labeling. After labeling is completed, the object category database is obtained, which serves as the target space for video content recognition.

[0073] Expansion module 24 is used to expand the object category database using data augmentation techniques. This technique can be used to expand the object category database and increase its content diversity. Specifically, this technique transforms a sample image in the object category database into a new sample image through augmentation operations, including rotation, padding cropping, and grayscale conversion. These augmentation operations are only used during training and are not required for testing or actual applications. All augmentation operations are applied to the original sample image with a certain probability to ensure the randomness of the augmentation results, which are then used as input data for the current model iteration training.

[0074] The training module 25 is used to train a deep neural network model using the expanded object category database. The deep neural network model is an SSD network model, which includes a multi-branch convolutional structure and a multi-scale feature map fusion structure.

[0075] The recognition module 26 is used to use the trained deep neural network model to recognize the video image to output the location information and size information of each object category, and finally output the object content category and location information in the corresponding window.

[0076] Therefore, the present invention improves the existing SSD network model by adding a multi-branch convolution structure to the SSD network model to improve the network's detection performance for small targets, and adopts a multi-scale feature map fusion structure to fuse feature maps of different scales on multi-scale feature maps. The expanded object category database is used to train the deep neural network model, which solves the problem of mutual influence between category recognition and position positioning in aerial video object detection.

[0077] like Figure 7 As shown, the training module 25 includes:

[0078] An input unit 251 is used to input a plurality of sample images in the object category database into the deep neural network model;

[0079] A first convolution unit 252 is configured to perform convolution processing on the multiple sample images through a multi-branch convolution layer;

[0080] A normalization unit 253 is used to perform normalization on the multiple sample images after the convolution process to generate a scale feature map;

[0081] A fusion unit 254 is used to perform feature fusion processing on all feature maps;

[0082] The second convolution unit 255 is used to convolve the concatenated feature map through a convolution layer to generate a branched convolution feature map.

[0083] Furthermore, the fusion unit 254 includes:

[0084] The resizing subunit is used to unify the sizes of all feature maps;

[0085] The identification and positioning subunit is used to perform category identification and position positioning processing on each feature map after unified size;

[0086] The feature fusion subunit is used to perform feature fusion processing on all feature maps after recognition and positioning processing in a weighted manner.

[0087] From the above, it can be seen that the present invention improves the SSD network model by adding a convolution kernel that first branches and then splices. The multi-scale feature maps generated by the convolution kernel for similar images are fused using multi-scale feature maps, which effectively solves the contradiction between category recognition and position positioning in aerial video content detection and improves the accuracy of content recognition.

[0088] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for identifying content in drone aerial videos based on deep learning, characterized by: include: Real-time acquisition of video images taken by drones; Performing frame extraction processing on the video image to extract a sample image, and performing preprocessing on the sample image; Annotating the objects to be identified in the sample images to generate an object category database; Expanding the object category database using data augmentation technology; The expanded object category database is used to train a deep neural network model, wherein the deep neural network model is an SSD network model. The SSD network model follows the basic structure of the one-step network model, and the backbone network of the SSD network model adopts VGG-16. The SSD network model includes a multi-branch convolution structure and a multi-scale feature map fusion structure. Recognize the video image using a trained deep neural network model to output location information and size information for each object category; The step of using the expanded object category database to train the deep neural network model includes: Inputting a plurality of sample images in the object category database into the deep neural network model; Convolutionally processing the plurality of sample images through a multi-branch convolutional layer; Normalize the multiple sample images after convolution processing to generate scale feature maps; Unify the size of all feature maps; Perform category recognition and position positioning processing on each feature map after unification of size; All feature maps after identification and positioning are subjected to feature fusion processing in a weighted manner; The concatenated feature map is convolved through a convolutional layer to generate a branched convolutional feature map.

2. The method for identifying content in drone aerial video based on deep learning as claimed in claim 1 is characterized in that: The step of preprocessing the sample image comprises: Using a distortion correction algorithm to correct the sample image to form a regular planar sample image; The sample image after the correction processing is compressed so that the sample image reaches a target size capable of performing target recognition.

3. The method for identifying content in drone aerial video based on deep learning as claimed in claim 1 is characterized in that: The method for labeling the objects to be identified in the sample image includes a manual labeling method and / or an image target detection algorithm.

4. The method for identifying content in drone aerial video based on deep learning as claimed in claim 1 is characterized in that: The step of expanding the object category database using data enhancement technology includes: Data enhancement processing is performed on the sample images in the object category database in a random probability superposition manner, and the data enhancement processing includes rotation processing, filling-type cropping processing and grayscale data processing.

5. A device for identifying content in drone aerial video based on deep learning, characterized in that: include: Acquisition module, used to acquire video images taken by drones in real time; A preprocessing module, configured to perform frame extraction processing on the video image to extract a sample image, and preprocess the sample image; A labeling module, used to label the objects to be identified in the sample image and generate an object category database; An expansion module, configured to expand the object category database using data enhancement technology; A training module is used to train a deep neural network model using the expanded object category database. The deep neural network model is an SSD network model. The SSD network model follows the basic structure of the one-step network model. The backbone network of the SSD network model uses VGG-16. The SSD network model includes a multi-branch convolution structure and a multi-scale feature map fusion structure. a recognition module, configured to recognize the video image using a trained deep neural network model to output location and size information for each object category; The training module includes: an input unit for inputting a plurality of sample images in the object category database into the deep neural network model; a first convolution unit for performing convolution processing on the plurality of sample images through a multi-branch convolution layer; a normalization unit for performing normalization processing on the plurality of sample images after the convolution processing to generate a scale feature map; a fusion unit for performing feature fusion processing on all feature maps; a second convolution unit for performing convolution processing on the feature map after the splicing processing through a convolution layer to generate a branch convolution feature map; The fusion unit includes: a size adjustment subunit, which is used to unify the sizes of all feature maps; an identification and positioning subunit, which is used to perform category identification and position positioning processing on each feature map after the size is unified; and a feature fusion subunit, which is used to perform feature fusion processing on all feature maps after identification and positioning processing in a weighted manner.

6. A deep learning-based drone aerial video content recognition system, characterized by: It comprises a UAV platform and the UAV aerial video content recognition device as described in claim 5, wherein the UAV platform is a platform equipped with a visible light camera and a thermal infrared camera and performs multi-source image acquisition.

Images