A small-scale target component segmentation method based on deep learning

Abstract

The present invention discloses a small-scale target component segmentation method based on deep learning, which belongs to the field of image target detection and segmentation technology. The present invention first constructs a small-scale target data set, which includes a segmentation data set, a super-resolution reconstruction data set, and a target detection data set; for the small-scale target part segmentation task, a corresponding small-scale target component segmentation network model is set, and the model is trained based on the segmentation data set to obtain the optimal model; a small-scale target enhancement method is combined with super-resolution reconstruction and target detection related technology settings, which improves the segmentation accuracy of the segmentation algorithm for small-scale targets in a targeted manner without affecting the segmentation effect of ordinary targets.

Description

Technical Field

[0001] The present invention belongs to the technical field of image target detection and segmentation, and in particular relates to a small-scale target component segmentation method based on deep learning. Background Art

[0002] The component segmentation task developed on the basis of the semantic segmentation task is to further divide the semantic category information of the object with the help of semantic segmentation technology, and represent the different components of the object with different pixel values, so as to form a component segmentation map containing the information of each component of the object. For example, in the military field, the drone is divided into three parts: the nose, the fuselage and the payload. With the help of the results of component segmentation, ground laser weapons can achieve automatic and precise positioning and destruction of a certain part of the drone (such as the nose) in a targeted manner; in the field of life, human images are divided into parts such as the head, hair, tops, limbs, etc. With the help of virtual reality technology, the clothing parts can be replaced to achieve virtual dressing, reduce the cost of changing clothes for customers, and enhance the shopping experience of customers.

[0003] The current mainstream semantic segmentation algorithm has become mature and can achieve good segmentation results for targets of normal size. However, for the problem of component segmentation of small-scale targets, the segmentation effect of the current mainstream algorithm is not ideal. However, small-scale targets (the width and height of the target are less than 40 pixels) are widely present in the actual application of component segmentation. For example, in the field of remote sensing, the images sent back by drones or satellites have a wide coverage and are far away from the target. The number of pixel points constituting the target in the acquired image is small (for example, a dozen pixels can represent a forest or a piece of arable land), and the resolution is low. Therefore, it is necessary to accurately classify all small targets in the image to facilitate the monitoring of water areas, vegetation, arable land and other resources. In the military field, all military targets will be presented as small-scale targets in the image when they are far away from the image recording equipment. The high-precision segmentation of small-scale targets can greatly improve the long-range strike capability and strike accuracy of weapons, and thus gain an advantage in the actual battlefield against the enemy. Therefore, improving the segmentation ability of segmentation algorithms for small-scale targets has broad application prospects and practical significance. Summary of the invention

[0004] The present invention provides a small-scale target component segmentation method based on deep learning, which solves the technical problem that the existing neural network model has the problem that small targets are fuzzy and small in size, making it difficult to segment targets and having poor segmentation effect.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A small-scale target component segmentation method based on deep learning includes the following steps:

[0007] Step 1: Construct a small-scale target (target size is less than or equal to the specified size) dataset:

[0008] The dataset consists of three parts: segmentation dataset, super-resolution reconstruction dataset and target detection dataset. Each part of the dataset is annotated and verified, and each part of the dataset is expanded through data enhancement.

[0009] Step 2: Construct a small-scale target component segmentation network model, and use the back-propagation algorithm and gradient descent method to train the segmentation network model. The weight parameters of the segmentation network model are adjusted by feedback of the cost function between the model prediction value and the actual value. Repeat the iteration until the cost function is less than the set threshold to obtain a trained small-scale target component segmentation network model.

[0010] Step 3: Train the selected super-resolution reconstruction network based on the super-resolution reconstruction data set. After the training is completed, use it as a super-resolution reconstruction module. Train the selected target detection network based on the segmentation data set to obtain a target detection model. Based on the target detection model, build a small-scale target cropping module. The input of the small-scale target cropping module is the target detection result obtained by the target detection model. The target detection result is cropped according to the specified pruning strategy to obtain a cropped area.

[0011] Step 4, performing image preprocessing on the original image to be segmented for the small-scale target component segmentation, so that the size of the preprocessed image matches the input of the super-resolution reconstruction module;

[0012] The original image to be segmented after image preprocessing is input into the super-resolution reconstruction module for super-resolution reconstruction, and the super-resolution reconstruction result to be segmented is obtained and input into the target detection model and the small-scale target cropping module to obtain the cropped area of ​​the image to be segmented;

[0013] Input the cropped area of ​​the image to be segmented into the trained small-scale target component segmentation network model to obtain a segmentation map of the cropped area of ​​the image to be segmented;

[0014] The cropped area segmentation map is restored to the segmentation map of the original image to be segmented to obtain the small object segmentation result.

[0015] Furthermore, the step 1 comprises the following steps:

[0016] Step 1.1: Collect small-scale target images based on actual business scenarios, and classify the collected images by target size into small-scale target images and normal-scale target images, and mix the two in a 1:1 ratio to construct the original images of the small-scale target dataset;

[0017] Step 1.2: Use bicubic interpolation to downsample the small-scale target images to 1 / 2, 1 / 3, and 1 / 4 respectively to construct a super-resolution reconstruction dataset;

[0018] Step 1.3: Annotate the target boxes in each image and set category labels for each target box;

[0019] Step 1.4: The contours of different key components of the target to be segmented are marked with different color points. After conversion, a picture containing the component marking results is formed, a segmentation data set is constructed, and data enhancement processing is performed on the pictures in the segmentation data set;

[0020] Furthermore, in step 2, the small-scale target component segmentation network model includes two parts, an encoder and a decoder, wherein the encoder is used to extract image semantic feature information, and the encoder includes two sub-networks: a backbone network and a global context head network, wherein the input of the backbone network is the input image of the small-scale target component segmentation network model, and the backbone network is used to output a series of feature maps, denoted as feature maps Fi, wherein i∈[1,5], and the feature maps Fi are respectively input into the decoder of the i-th layer after passing through a convolution layer with a 1×1 convolution kernel, as the input feature map of the decoder of this layer, wherein the decoder includes 6 layers, and the input feature map of the 0th layer decoder is the feature map obtained by directly passing the input image of the small-scale target component segmentation network model through a 1×1 convolution kernel. , and use the feature map F5 as the input of the global context head network. The weights of each layer of decoders are generated by the global context head network (i.e., the network embedding of the input feature map of the decoder) and sent to the corresponding decoder. For the 5th layer decoder, the weight of this layer is calculated and merged with the input feature map of this layer to obtain the decoding result of the 5th layer. Starting from the 4th layer, the decoding structure of the j+1th layer is upsampled (bilinear interpolation) and used as the patch weight of the jth layer dynamic patch convolution. The dynamic patch convolution operation structure is calculated and merged with the weight of the jth layer decoder to obtain the decoding result of the jth layer, where j=4,3,2,1,0. The final segmentation map is obtained based on the decoding result of the 0th layer.

[0021] The technical solution provided by the present invention brings at least the following beneficial effects:

[0022] In the present invention, a small-scale target dataset is first constructed, which includes three parts: a segmentation dataset, a super-resolution reconstruction dataset and a target detection dataset; for the small-scale target part segmentation task, a corresponding small-scale target component segmentation network model is set, and the model is trained based on the segmentation dataset to obtain the optimal model; a small-scale target enhancement method is set in combination with super-resolution reconstruction and target detection related technologies, which improves the segmentation accuracy of the segmentation algorithm for small-scale targets in a targeted manner without affecting the segmentation effect of ordinary targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A schematic diagram of a processing process of a small-scale target component segmentation method based on deep learning provided by an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of a Copy-Pasting data enhancement strategy in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of a small-scale target component segmentation network model used in an embodiment of the present invention;

[0027] Figure 4 The figure is a schematic diagram for comparing recognition results in an embodiment of the present invention, wherein (a) is the original image to be segmented; (b) is a visualization result diagram of manually annotated labels; (c) is the segmentation result of HyperSeg-B6 without the SOEM method; and (d) is a segmentation result diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, a small-scale target component segmentation method based on deep learning provided by an embodiment of the present invention includes the following steps:

[0030] Step A: Construct a small-scale target dataset.

[0031] The dataset consists of three parts: segmentation dataset, super-resolution reconstruction dataset, and target detection dataset. Each part of the dataset has been annotated and verified, and the corresponding data enhancement method has been introduced to expand each part of the dataset. Each part consists of a training set and a test set. The training set is used to train the network model, and the test set is used to verify the segmentation performance of the network model.

[0032] A1: Collect small-scale target images based on actual business scenarios, and classify the collected images by target size into small-scale target images and ordinary-scale target images (in this embodiment, targets with a pixel composition greater than 40 are defined as ordinary targets, and targets with a pixel composition less than or equal to 40 are defined as small-scale targets), and mix the two in a 1:1 ratio to construct the original images of the small-scale target dataset.

[0033] A2: Perform bicubic interpolation and downsample the input image to 1 / 2, 1 / 3, and 1 / 4 to construct a super-resolution reconstruction dataset.

[0034] A3: Use the marking box in LabelImage software (an image labeling tool software) to select the target and label it with a category. When selecting the target, make sure that the entire target is in the box and all targets are correctly selected. After the selection is completed, the software will automatically convert the manually labeled information into an xml file, which contains information such as the number, location, and category of the target. The acquired data set is enhanced by flipping, rotating, adjusting brightness, contrast, etc. to construct a target detection data set.

[0035] A4: With the help of LabelMe software (an image labeling tool software), the contours of different key components of the target to be segmented are marked with different color points to form a corresponding json file, and then the labelme2voc script can be used to convert the json file into a png format image containing the component labeling results, thereby constructing a segmentation dataset. With the help of traditional data enhancement methods such as flipping, rotating, adjusting brightness and contrast, and Copy-Pasting data enhancement strategy, the dataset is enhanced to construct a small-scale target segmentation dataset.

[0036] A5: Implement the Copy-Pasting data augmentation strategy, extract the small-scale target and its corresponding mask map with the help of the mask map generated by annotation, and then place the extracted target on the new background with the help of the prior knowledge of the data set to generate a new small-scale target training set. The whole process is divided into three steps: target extraction, background selection, and target placement. Figure 2As shown. Specifically, the role of target extraction is to extract small targets from the original image and mask image. In the extraction process, the position of the small target in the original image is located by means of the mask image generated by annotation. Then, with the help of the numpy library (a Python library, mainly used to perform calculations on multidimensional arrays), the small target and the background are classified to obtain the small target image without background and its corresponding mask image. Background selection should rely on certain prior knowledge, and the scene picture or video where the target may actually appear should be selected as the background. Taking the Phantom 4 drone as an example, the scene where it works should be concentrated in the air in most cases. Therefore, it is possible to consider selecting a picture of the sky as the background for data expansion, rather than the deep sea or other scenes where the Phantom 4 drone rarely appears as the background. Target placement, after the target background selection is completed, it is possible to consider placing the target in a suitable position of the background in some specified postures for synthesis to generate a new small-scale target training set picture. Its placement process should also follow the corresponding prior knowledge, so that the generated picture conforms to objective physical principles and social common sense. Taking the Phantom 4 drone as an example, it is necessary to ensure that the placed drone is large in the distance and small in and out, and the drone cannot fly upside down, etc.

[0037] Step B: Build a small-scale object component segmentation network model based on HyperSeg (a block-by-block hypernetwork for real-time semantic segmentation).

[0038] B1: Construct a network model including input layer, first convolution layer - Nth convolution layer, fully connected layer and Softmax function. Among them, the backbone network (Backbone) can choose EfficientNetB6 to replace the original EfficientNetB2 in the network. Compared with the original EfficientNetB2 network, the EfficientNetB6 network structure is more complex, and the depth, width and resolution of the network are improved. The depth is 1.8 times the original depth of EfficientNetB1, and the width is 2.4 times the original. The image resolution has also changed from the original 240×240 to the current 528×528.

[0039] B2: Modify the minimum feature map size. Adjust the output of the Backbone subnetwork of the encoder so that the feature maps it outputs are adjusted from the original five to six. In this way, the number of feature maps directly involved in decoding that were originally input to the decoder increases from the original four to five, that is, the minimum feature map size will change from the original H / 16×W / 16 to H / 32×W / 32. Then, for the 512×512 resolution images in the dataset, the minimum feature map size extracted in the feature extraction stage will change from the original 32×32 to 16×16. In this way, the new encoder can not only extract the encoding features of the original encoder, but also the features of smaller targets in advance, greatly improving the network's feature extraction capability for small-scale targets.

[0040] The structure of the network after adjustment is as follows Figure 3 As shown. That is, in the embodiment of the present invention, the small-scale target component segmentation network model includes two parts, a super network encoder and a super network decoder, wherein the super network encoder is used to extract image semantic feature information, and the encoder includes two sub-networks: a Backbone network and a global context head network (Contexthead), wherein the input of the Backbone network is the original image information of dimension 3×H×W, and the output is a series of feature maps Fi (i∈[1,5]), wherein the feature map Fi directly passes through a convolution layer of a 1×1 convolution kernel as the encoding result Bi and is input to the corresponding decoder; the decoder generates the weight wi of each layer with the input generated by the global context head network, and then the decoder obtains the encoding input of each layer and the weight value of the corresponding layer to calculate the decoding result of the current layer, and the decoding result of the high-level layer is subjected to a bilinear interpolation and then dynamically convolved with the encoding result of the previous layer (Dynamic Patch-wise Convolution), and this convolution can ensure that different spatial positions of the network input have different convolution kernels by means of the dynamic change of spatial parameters at different positions. That is, the decoder starts from the 5th layer, calculates and merges the weight value of this layer with the feature map input (Bi) of this layer, and after upsampling, it is used as the patch weight of the dynamic patch convolution of the previous layer to participate in the calculation, and so on, until it is merged to the 0th layer to form the output result of the tracking. Figure 3 In the figure, the feature map F0 refers to the feature map obtained by directly passing the input image through a 1×1 convolution kernel; that is, the features extracted by the Backbone network are respectively passed through the convolution layer of the 1×1 convolution kernel to obtain the feature map Fi(i∈[1,5]), and the feature map Fi and the parameters at the corresponding position are embedded and output to the decoder. The decoder consists of 6 layers, and the weight value of each layer is generated by the global context head network. The input feature map of the 0th layer is F0.

[0041] B3: Use the back-propagation algorithm and gradient descent method to train the above network model, and adjust the network weight value through the cost function feedback between the model prediction value and the actual value. Repeat the iteration until the cost function is less than the set threshold. After obtaining the network model with the optimal weight value, input the test set into the optimal network model to obtain the recognition result.

[0042] C: A small-scale target enhancement network is set up in combination with super-resolution reconstruction and target detection. This improves the segmentation accuracy of the segmentation algorithm for small-scale targets without affecting the segmentation effect of ordinary targets.

[0043] C1: Based on the super-resolution reconstruction dataset, the selected super-resolution reconstruction network (such as the EDSR network, which is optimized and adjusted based on the SRResNet network architecture) is trained. After the training is completed, it is used as a super-resolution reconstruction module.

[0044] C2: Based on the component detection data set, the selected target detection network (such as SSD network) is trained to obtain the target detection model, and a small-scale target cropping module is constructed based on the model, that is, the small-scale target cropping module crops the small-scale targets obtained by the target detection model based on the specified cropping strategy. That is, when cropping an image, it is necessary to use the target detection model to simply locate the location of the small-scale target, and then crop the located target with the specified width and height to obtain a cropping result image containing the small-scale target.

[0045] C3: Based on the super-resolution reconstruction module and the small-scale target cropping module, the input image is super-resolved and reconstructed and small-scale targets are cropped to obtain a small-scale target with richer details, and the target is input into the small-scale target component segmentation network model for segmentation to obtain the segmentation result. The obtained segmentation result is input into the segmentation map restoration module (used to restore the cropped area segmentation map to the original image segmentation map) to restore the segmentation map to obtain the segmentation result of the tracked small-scale target. The segmentation result diagram is shown in Figure 4 As shown. That is, firstly, the input image is super-resolved and reconstructed through the super-resolution reconstruction module, and then the target detection is performed on the reconstructed super-resolution image through the target detection network to obtain the target detection result (locate the target and calculate the target size), and then the small target is cropped according to the specified cropping strategy to obtain the cropping area, and then the cropping area is segmented based on the small-scale target component segmentation network model to obtain the cropping area segmentation map, and finally, the cropping area segmentation map is restored to the segmentation map of the original image to obtain the small target segmentation result of the image to be segmented.

[0046] That is, the segmentation model constructed in B is trained with the help of the constructed data set to obtain the optimal segmentation model, and the small-scale target enhancement method constructed in C is applied based on the segmentation model to segment the input small-scale target image to obtain the final component segmentation result.

[0047] This embodiment compares the segmentation effects of the component segmentation algorithm on small-scale targets and ordinary targets of the Elf 4 before and after optimization, as shown in Table 1; and compares the changes in the segmentation effect of the model on the target before and after the introduction of the small-scale target data enhancement method, as shown in Table 2.

[0048] Table 1 Segmentation effect of component segmentation algorithm on small-scale targets and ordinary targets of Elf 4

[0049] Model Target Category Rotor IoU Subject IoU Load IoU QUR HyperSeg Small-scale targets 0.401 0.726 0.335 0.488 Improved Algorithm Small-scale targets 0.474 0.737 0.396 0.536 HyperSeg Common Target 0.828 0.918 0.841 0.862 Improved Algorithm Common Target 0.829 0.920 0.851 0.867

[0050] Table 2 compares the changes in the segmentation effect of the model on the target before and after the introduction of the small-scale target data enhancement method

[0051] Model Target Category Rotor IoU Subject IoU Load IoU QUR Enable SOEM Small-scale targets 0.501 0.736 0.416 0.551 Do not enable SOEM Small-scale targets 0.474 0.737 0.396 0.536 Enable SOEM Common Target 0.829 0.920 0.851 0.867 Do not enable SOEM Common Target 0.829 0.920 0.851 0.867

[0052] According to Table 1, the segmentation method of the present application can effectively improve the segmentation of small-scale targets and ordinary targets without losing a certain degree of real-time performance. According to Table 2, after the introduction of the SOEM method, the model's small-scale segmentation ability is greatly improved, among which the rotor and load are significantly improved by 0.02 and 0.015 respectively, and the overall improvement is 0.011. In other words, the SOEM method has improved the small-scale target segmentation ability of various segmentation models, proving the effectiveness of this module in improving the model's segmentation ability.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0054] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A small-scale target component segmentation method based on deep learning, characterized in that: The following steps are involved: Step 1: Construct a small-scale target dataset: The dataset consists of three parts: segmentation dataset, super-resolution reconstruction dataset and target detection dataset. Each part of the dataset is annotated and verified, and each part of the dataset is expanded through data enhancement. The step 1 comprises: Step 1.1: Collect small-scale target images based on actual business scenarios, and classify the collected images by target size into small-scale target images and normal-scale target images, and mix the two in a 1:1 ratio to construct the original images of the small-scale target dataset; Step 1.2: Use bicubic interpolation to downsample the small-scale target images to 1 / 2, 1 / 3, and 1 / 4 respectively to construct a super-resolution reconstruction dataset; Step 1.3: Annotate the target boxes in each image and set category labels for each target box to construct a target detection dataset; Step 1.4: The contours of different key components of the target to be segmented are marked with different color points. After conversion, a picture containing the component marking results is formed, a segmentation data set is constructed, and data enhancement processing is performed on the pictures in the segmentation data set; Step 2: Construct a small-scale target component segmentation network model, and use the back-propagation algorithm and gradient descent method to train the segmentation network model. The weight parameters of the segmentation network model are adjusted by feedback of the cost function between the model prediction value and the actual value. Repeat the iteration until the cost function is less than the set threshold to obtain a trained small-scale target component segmentation network model. Step 3: Train the selected super-resolution reconstruction network based on the super-resolution reconstruction data set. After the training is completed, use it as a super-resolution reconstruction module. Train the selected target detection network based on the target detection data set to obtain a target detection model. Based on the target detection model, build a small-scale target cropping module. The input of the small-scale target cropping module is the target detection result obtained by the target detection model. The target detection result is cropped according to the specified pruning strategy to obtain a cropped area. Step 4, performing image preprocessing on the original image to be segmented for the small-scale target component segmentation, so that the size of the preprocessed image matches the input of the super-resolution reconstruction module; The original image to be segmented after image preprocessing is input into the super-resolution reconstruction module for super-resolution reconstruction, and the super-resolution reconstruction result to be segmented is obtained and input into the target detection model and the small-scale target cropping module to obtain the cropped area of ​​the image to be segmented; Input the cropped area of ​​the image to be segmented into the trained small-scale target component segmentation network model to obtain a segmentation map of the cropped area of ​​the image to be segmented; The cropped area segmentation map is restored to the segmentation map of the original image to be segmented to obtain the small-scale target segmentation result.

2. The method according to claim 1, characterized in that In step 2, the small-scale target component segmentation network model includes two parts, an encoder and a decoder, wherein the encoder is used to extract image semantic feature information, and the encoder includes two sub-networks: a backbone network and a global context head network, wherein the input of the backbone network is the input image of the small-scale target component segmentation network model, and the backbone network is used to output a series of feature maps, denoted as feature maps Fi, wherein i∈[1,5], and the feature maps Fi are respectively input into the decoder of the i-th layer after a convolution layer with a 1×1 convolution kernel, as the input feature map of the decoder of this layer, wherein the decoder includes 6 layers, and the input feature map of the 0th layer decoder is the input map of the small-scale target component segmentation network model The feature map obtained by directly passing through the 1×1 convolution kernel is used, and the feature map F5 is used as the input of the global context head network. The weights of each layer of the decoder are generated through the global context head network and sent to the corresponding decoder. For the 5th layer decoder, the weight of this layer is calculated and merged with the input feature map of this layer to obtain the decoding result of the 5th layer. Starting from the 4th layer, the decoding structure of the j+1th layer is upsampled and used as the patch weight of the jth layer dynamic patch convolution. The dynamic patch convolution operation structure is calculated and merged with the weight of the jth layer decoder to obtain the decoding result of the jth layer, where j=4,3,2,1,0; The final segmentation map is obtained based on the decoding result of layer 0.

Images