Semantic segmentation method of main components of flying targets in video stream based on deep learning

Abstract

The present invention discloses a method for semantic segmentation of main components of a flying target in a video stream based on deep learning, and relates to the field of video semantic segmentation. The present invention includes the construction of a data set of main components of a flying target and the establishment of a video semantic segmentation model; wherein the data set is obtained by hardware simulation, a recording unit and a simulation unit are used to simulate and shoot the image of the flying target, and then the background of the flying target is simulated by post-processing; the video semantic segmentation model is a semantic segmentation network based on an improved residual block, which uses a local time attention mechanism to introduce the previous and next frame information of the current frame for prediction through delayed output, and trains, tests and adjusts the training parameters on the constructed data set to obtain better recognition effect. The present invention improves the recognition effect and speed on the data set of main components of a flying target, and mainly solves the problems of poor recognition effect of main components of a flying target in existing videos and insufficient utilization of timing information.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation methods, and in particular to a real-time video semantic segmentation method for key components of unmanned aerial vehicles based on deep learning. Background Art

[0002] Image classification, object detection and semantic segmentation are the three most widely studied key visual tasks in the field of computer vision. Object detection provides relatively coarse-grained object-level classification, but in some practical applications, it is necessary not only to detect the existence of the object, but also to obtain an accurate boundary of the object, and semantic segmentation provides a way to achieve pixel-level image classification. In the field of semantic segmentation, each pixel in an image will be classified as a specific class, so that a fine-grained classification of the image can be obtained. Before deep learning methods became mainstream, traditional semantic segmentation methods also achieved certain results, but with the development of deep learning, more and more deep learning-based methods have been applied in the field of semantic segmentation, especially the development of convolutional neural networks (CNN), which has significantly improved the effect of semantic segmentation algorithms.

[0003] With the development of photography technology and the popularization of video equipment, videos have gradually replaced static text and pictures and become one of the most common information media. Videos contain richer temporal information than images, and can provide more usable information for image analysis. At the same time, intelligent recognition technology for videos also faces the problems of continuity and real-time performance. Since videos involve the prediction of continuous frames, the amount of computation is quite large, and traditional deep image semantic segmentation networks will encounter real-time problems on videos. In addition, while videos provide the temporal information of continuous frames, the use of these previous and next frame temporal information will also bring additional computational complexity to the model, which also puts higher requirements on the efficiency of video target recognition. Therefore, the main difference between video semantic segmentation and image semantic segmentation lies in the use of temporal information and attempts to strike a balance between speed and effect. Summary of the invention

[0004] The present invention provides a semantic segmentation method for main components of flying targets in video streams based on deep learning, which solves the problems of lack of scene processing for segmentation of key components of unmanned aerial vehicles in existing neural networks, as well as low edge processing accuracy and slow processing speed.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A semantic segmentation method for main components of a flying target in a video stream based on deep learning, the method comprising the following steps:

[0007] Step 1, collect data:

[0008] Build a flight target simulation data acquisition system, use a physical model to simulate the actual flight target, and use a video recording device to capture video of the flight target model to collect video data containing the flight target;

[0009] Step 2: Build a flight target video dataset:

[0010] Perform image processing operations on the video data collected in step 1 to simulate the background and posture of the flying target in a real environment;

[0011] Labeling each to-be-identified component of the flying target, including the segmented component image area and the category label of the current component image area;

[0012] Extracting part of the data from the constructed flight target video dataset as video training data for training network model parameters;

[0013] Step 3: Build and train a flight target video semantic segmentation network based on a deep neural network:

[0014] The flight target video semantic segmentation network takes the video as input and is used to output the component segmentation results and component categories of each to-be-identified component of the flight target;

[0015] The video training data is input into the flight target video semantic segmentation network for training. The loss function of the flight target video semantic segmentation network is set based on the component segmentation results output by the flight target video semantic segmentation network (the predicted results of different components of the flight target) and the actual results to adjust the parameters of each layer of the network. The network is iterated repeatedly until the preset training end conditions are met to obtain the optimal video semantic segmentation network for the specified components of the flight target.

[0016] Preferably, in step 1, the constructed flight target simulation data acquisition system includes: a flight target model, a recording device (such as a camera), a support unit (a robotic arm or a stabilizer or a slide rail), a green screen and a fill light, and its purpose is to collect the flight target video image data simulated by the model.

[0017] The flying target model is used to support and drive the flying target model to move through the supporting unit, and simulate the posture of the flying target; the video recording device is used to collect the video image of the flying target model to obtain the video data containing the flying target.

[0018] Preferably, in step 2, image processing operations are performed on the video data collected in step 1 to simulate the background and posture of the flying target in a real environment, specifically:

[0019] The collected video data is processed by background replacement, and the green screen background in the collected video data is replaced with a background close to the actual scene of the real flight target.

[0020] Preferably, in step 2, the components to be identified of the flying target include: a fuselage, wings and mounts, and the mount object is divided into components (for example, a missile is divided into three parts: a warhead, a body and wings) to obtain the category of the components to be identified.

[0021] Preferably, in step 2, the component image region is an annotated image containing component annotation information generated based on an image annotation tool.

[0022] Preferably, step 2 also includes performing data enhancement processing on the original image and the annotated image of the video data to enrich the data samples, wherein the data enhancement processing methods include but are not limited to: flipping, rotation and displacement.

[0023] Preferably, the flight target video semantic segmentation network includes multiple convolutional layers, and uses a local time attention mechanism module to memorize timing information to assist model judgment.

[0024] Preferably, the network structure of the flight target video semantic segmentation network is specifically:

[0025] The flying target video semantic segmentation network includes an encoder, a decoder and a local time attention mechanism module;

[0026] In the encoder part, the input image first passes through two downsampling layers, then enters five 1D convolutional layers with bottleneck residual blocks, and then enters the Convolutional Block Attention Module (CBAM) after one downsampling layer. The output of the convolutional attention module is then passed through a feature extraction network consisting of eight 1D convolutional layers with bottleneck residual blocks to extract the image features, and the image coding features (i.e., the semantic graph output by the encoder) are obtained; in the decoder part, the image coding features are restored through two upsampling networks consisting of one upsampling layer and two 1D convolutional layers with bottleneck residual blocks to generate the final component segmentation results;

[0027] The local time attention mechanism module obtains the reading window by delayed output, extracts the information of the previous and next frames of the current frame in the video of the current reading window, and integrates the features of the previous and next frames into the current frame through the Non-local method, thereby assisting the model in judging the current frame.

[0028] Preferably, in step 3, the loss function of the flight target video semantic segmentation network is: the intersection-over-union ratio between the component segmentation result output by the flight target video semantic segmentation network and the true result.

[0029] The technical solution provided by the present invention brings at least the following beneficial effects:

[0030] The present invention solves the problems of poor recognition effect of main components of flying targets and insufficient use of timing information in existing videos, and improves the recognition effect and efficiency of main components of flying targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 is a flow chart of the overall method of the present invention;

[0033] Figure 2 Constructing a flow chart for the dataset of the present invention;

[0034] Figure 3 is a structural diagram of the network model of the present invention;

[0035] Figure 4 It is a schematic diagram of the component segmentation result diagram of the present invention, wherein (4-a) and (4-b) are schematic diagrams of the segmentation result diagrams of components of two different target objects. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0037] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0038] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0039] The embodiment of the present invention discloses a semantic segmentation method for the main components of a flying target in a video stream based on deep learning, and the method includes two parts: the first is the construction of a data set of the main components of a flying target. The data set simulates and shoots the flying target image by means of hardware simulation using a recording unit and a simulation unit, and then simulates the background of the flying target through post-processing; the second is to build a video semantic segmentation model, design a semantic segmentation network composed of improved residual blocks, and use the method of delayed output to introduce the previous and next frame information of the current frame for prediction using the Local Memory Attention (local time attention mechanism) module, and train, test and adjust the training parameters on the above-mentioned data set of the main components of the flying target to obtain better recognition effect; the video semantic segmentation method of the present invention improves the recognition effect and speed on the data set of the main components of the flying target, and the present invention is mainly used to solve the problem of poor recognition effect of the main components of the flying target in the existing video and insufficient utilization of timing information.

[0040] The specific implementation of the method of the present invention is further described in detail below in conjunction with embodiments.

[0041] As a possible implementation, Figure 1 As shown, the semantic segmentation method of the main components of a flying target in a video stream based on deep learning disclosed in an embodiment of the present invention includes the following steps:

[0042] Step a: Collect data: Build a flight target simulation data acquisition system, use a physical model to simulate the actual flight target, and use the video recording unit to shoot the flight target model supported by the robotic arm to collect video data containing the flight target.

[0043] a1: Build a flight target simulation data acquisition system, which is mainly composed of a flight target model, a recording unit (camera), a support unit (mechanical arm / stabilizer / slide rail), a green screen, a fill light and other equipment. Its purpose is to collect video image data of the flight target simulated by the model. The model is fixed on the mechanical arm or stabilizer and connected to the slide rail. The support unit will drive the model to move and simulate the posture of the flight target, and the recording unit will collect the model image to form the flight target video data source.

[0044] Step b: Construct a flight target video dataset: Use the flight target data collected by the flight target simulation data acquisition system to perform image processing operations such as background replacement to simulate the background and posture of the flight target in the real environment. Then annotate the data and divide the dataset into a training set and a test set for model training and testing.

[0045] b1: Perform background replacement processing on the data of the acquisition system. Use the open source software Natron to replace the green screen background in the video with a background similar to the actual scene of the real flight target to form a flight target video dataset;

[0046] b2: Image annotation: Use the Python-based data annotation software Labelme for data annotation. Referring to the formats of the two public video semantic segmentation datasets Cityscape and Camvid, capture an image from the video every 30 frames, read it into Labelme for annotation, mark the boundaries of each major component, and generate a json text containing the annotation information through the software. Then, use the Python script that comes with Labelme to generate annotated images. The script will generate an annotated image semantic map in png format with a bit depth of 8 based on the original image and annotation information. These semantic maps describe the semantic category information of each major component of the flying target in the image. The overall process of dataset construction is as follows: Figure 2 As shown in the figure, firstly, the background of the collected video data is replaced, and then the video frame images are deframed, the video frame images are normalized according to the specified image size, and the image format is processed (such as grayscale), and then each video frame image is named and labeled. In order to enrich the data samples, the original image and the labeled image are subjected to image enhancement processing such as flipping, displacement, rotation, etc., and the obtained data set is named FTMP data set.

[0047] Step c: Build and train a flight target video semantic segmentation network (i.e., AERFNet-LMANet video semantic segmentation network): Input the video training data into the video semantic segmentation model for training, so that the model can learn the features of the main components of the flight target. The semantic segmentation model takes video as input, contains multiple convolutional layers, and uses the Local MemoryAttention module to memorize temporal information to assist model judgment, thereby obtaining better test results.

[0048] c1: Construct a semantic segmentation model based on the attention mechanism. In the encoder part, the input image first passes through 3 downsampling layers (i.e., Downsampler block) and 5 residual layers (DR-bottleneck-1d) to reduce the number of channels, and then enters the CBAM (Convolutional Block Attention Module) module to introduce the attention mechanism; then the network will obtain the image features through a feature extraction layer composed of 8 DR-bottleneck-1d (1D convolutional layers with bottleneck residual blocks, the specific structure can be referred to the paper "Efficient ConvNet for Real-time Semantic Segmentation") residual layers; in the decoder part, the semantic map output by the encoder will be restored through two layers of upsampling networks composed of 1 upsampling layer (Upsampler block) and 2 DR-bottleneck-1d convolutional layers to generate the final output result. The network structure is as follows: Figure 3 As shown, Figure 3 "Out-F" in the figure represents the number of channels of the output feature map of the current module, "Out-Res" represents the spatial size of the output feature map of the current module, and "C" represents the number of flight target components to be identified, that is, the number of main components.

[0049] c2: Based on the above semantic segmentation network model, a Local MemoryAttention module based on the information of previous and next frames is introduced. The module uses the reading window obtained by the network through delayed output to read the information of the previous and next frames of the current frame in the video, and uses the Non-local method (using the feature similarity between image pixels to establish a long connection relationship between pixels to enhance the feature expression ability of each pixel in the image. This relationship does not have the local limitations of convolution in neural networks. In other words, no matter how far the pixels are apart, the Non-local method can explore long connection relationships. In other words, the non-local method has field capabilities that convolution cannot achieve) to integrate the features of the previous and next frames into the current frame, thereby assisting the model in judging the current frame.

[0050] c3: Train the above network model on the constructed simulated data set, calculate the intersection-over-union ratio between the prediction results given by the model for different parts of the flight target and the actual results, so as to adjust the parameters of each layer of the network, and iterate repeatedly until the optimal prediction result is obtained and the corresponding optimal network model is obtained, which is used for the semantic segmentation of the main components of the flight target.

[0051] This embodiment compares the parameter quantities corresponding to different residual block designs (taking 256-dimensional input as an example), as shown in Table 1, and compares the network model backbone proposed in the present invention with several other commonly used backbone networks to measure their differences in parameter quantities. In addition, this embodiment also compares the impact of different frame selection strategies (number of adjacent frame readings) on the recognition effect, where F is used to indicate the total number of previous and next frame readings except the current frame, as shown in Table 2.

[0052] Table 1 Comparison of parameters of different residual blocks

[0053]

[0054] Table 2 Comparison of results (IoU) of different adjacent frame reading numbers

[0055]

[0056]

[0057] According to Table 1, the semantic segmentation method of the present invention has a lower parameter quantity and has improved the speed of component recognition. When the speed is improved, the accuracy of the method of the present invention on the flight target data set is also improved. The present invention reads the information of the previous and next frames of the current frame in the video stream through a delay window, and integrates the features of the previous and next frames into the current frame through the Non-local method, making fuller use of the time sequence information of the past and future frames in the video, and achieving better recognition effect, such as Figure 4 In summary, the video semantic segmentation and recognition algorithm provided by the present invention realizes the automatic recognition of the main components of the flying target, eliminating the tedious and time-consuming manual means. By using the video semantic segmentation framework, the lightweight backbone reduces the amount of calculation; at the same time, by using the flow alignment module of optical flow transmission, the recognition accuracy of edge information is improved, and the recognition accuracy rate is greatly improved.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0059] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A semantic segmentation method for main components of flying targets in video streams based on deep learning, characterized in that: The steps include: Step 1, collect data: Build a flight target simulation data acquisition system, use a physical model to simulate the actual flight target, and use a video recording device to capture video of the flight target model to collect video data containing the flight target; Step 2: Build a flight target video dataset: Perform image processing operations on the video data collected in step 1 to simulate the background and posture of the flying target in a real environment; Labeling each to-be-identified component of the flying target, including the segmented component image area and the category label of the current component image area; Extracting part of the data from the constructed flight target video dataset as video training data; Step 3: Build and train a flight target video semantic segmentation network based on a deep neural network: The flight target video semantic segmentation network takes the video as input and is used to output the component segmentation results and component categories of each to-be-identified component of the flight target; The video training data is input into the flight target video semantic segmentation network for training, and the loss function of the flight target video semantic segmentation network is set based on the component segmentation results output by the flight target video semantic segmentation network and the real results to adjust the parameters of each layer of the network, and iterates repeatedly until the preset training end condition is met, so as to obtain the optimal video semantic segmentation network for the specified component of the flight target; The network structure of the flight target video semantic segmentation network is specifically as follows: The flying target video semantic segmentation network includes an encoder, a decoder and a local time attention mechanism module; In the encoder part, the input image first passes through two downsampling layers, then enters five 1D convolutional layers with bottleneck residual blocks, then passes through one downsampling layer and enters the convolutional attention module, and then the output of the convolutional attention module is passed through a feature extraction network composed of eight 1D convolutional layers with bottleneck residual blocks to extract the image features and obtain the image coding features; in the decoder part, the image coding features are restored through two upsampling networks composed of one upsampling layer and two 1D convolutional layers with bottleneck residual blocks to generate the final component segmentation results; The local time attention mechanism module is located between the encoder and the decoder. The local time attention mechanism module is used to memorize timing information to assist model judgment. The local time attention mechanism module obtains a reading window by delayed output, and extracts information about the previous and next frames of the current frame in the video of the current reading window, and integrates the features of the previous and next frames into the current frame through a Non-local method. The Non-local method uses the feature similarity between image pixels to establish a long connection relationship between pixels to enhance the feature expression capability of each pixel in the image.

2. The method according to claim 1, characterized in that In step 1, the constructed flight target simulation data acquisition system includes: a flight target model, a recording device, a support unit, a green screen and a fill light; The flying target model is used to support and drive the flying target model to move through the supporting unit, and simulate the posture of the flying target; the video recording device is used to collect the video image of the flying target model to obtain the video data containing the flying target.

3. The method according to claim 1, characterized in that In step 2, image processing operations are performed on the video data collected in step 1 to simulate the background and posture of the flying target in the real environment: The collected video data is processed by background replacement, and the green screen background in the collected video data is replaced with a background close to the actual scene of the real flight target.

4. The method according to claim 1, characterized in that In step 2, the components to be identified of the flight target include: fuselage, wings and mounts, and the mount objects are divided into components to obtain the categories of components to be identified.

5. The method according to claim 1, characterized in that In step 2, the component image region is an annotated image containing component annotation information generated by an image annotation tool.

6. The method according to claim 5, characterized in that Step 2 also includes performing data enhancement processing on the original image and the annotated image of the video data, wherein the data enhancement processing method includes: flipping, rotation and displacement.

7. The method according to any one of claims 1 to 6, characterized in that: In step 3, the loss function of the flight target video semantic segmentation network is: the intersection-over-union ratio between the component segmentation result output by the flight target video semantic segmentation network and the true result.

Images