Port Panoramic Perception System Based on Ship Instance Segmentation

By using a port panoramic perception system based on ship instance segmentation, and combining instance segmentation with a multi-scale cross-fusion feature extraction network, the problems of limited computing resources and false alarms for civilian vessels in the port panoramic perception system are solved, achieving efficient ship detection and water area extraction.

CN114565764BActive Publication Date: 2024-11-15BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210194201.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-01
Publication Date
2024-11-15
Estimated Expiration
2042-03-01

AI Technical Summary

Technical Problem

Existing technologies in port panoramic perception systems suffer from limited computing resources, high latency, and false alarms from civilian vessels when processing ship detection, segmentation, and water area extraction.

Method used

A port panoramic perception system based on ship instance segmentation is adopted. Combining instance segmentation and multi-scale cross-fusion feature extraction network, the system processes ship targets through pixel-level supervision information to distinguish between warships and civilian vessels and perform end-to-end port water area extraction.

Benefits of technology

It effectively reduced processing time, improved the accuracy of warship detection, reduced the false alarm rate of civilian vessels, and achieved efficient water area extraction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114565764B_ABST
    Figure CN114565764B_ABST
Patent Text Reader

Abstract

The present invention discloses a port panoramic perception system based on ship instance segmentation, including an image acquisition module, a neural network model and a panoramic perception result output module; the image acquisition module is used to acquire port images; the neural network model is used to perform warship detection, warship segmentation and water area extraction processing on the acquired port images, and obtain the warship target positioning frame, the specific shape contour of the warship target and the port water area distribution; the panoramic perception result output module is used to output the panoramic perception result according to the warship target positioning frame, the specific shape contour of the warship target and the port water area distribution. The present invention can process warship detection, warship segmentation and water area extraction at the same time through the constructed neural network model, and improve the accuracy of warship detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of port traffic management, and more particularly to a port panoramic perception system based on ship instance segmentation. Background Art

[0002] At present, extensive research on remote sensing technology in the military field has revealed the importance of panoramic perception systems in performing related tasks in some important military areas such as ports and airports. It plays an important role in the intelligent interpretation of remote sensing images because it can extract rich visual information from remote sensing images and assist the decision-making system in analyzing scene elements. Taking military ports as an example, in order to fully obtain information about key areas in remote sensing images, the visual perception system should be able to understand the scene and then provide guidance information to the decision-making system, including: the location of warship targets, the size and shape of warships that are different from civilian ships, and the distribution of port water areas. Port panoramic perception systems usually involve target detection to determine the location of ships. In addition, it is necessary to segment ships to determine their basic types, and segment and extract port water areas to determine the basic situation of the port. There are many methods to handle port-related tasks separately. For example, ship detection is handled by target detection algorithms; semantic segmentation methods are used to perform ship and water segmentation.

[0003] However, these methods have two problems: First, processing these tasks one by one takes longer than processing all the tasks at once. When deploying panoramic perception systems on equipment commonly used for remote sensing information processing, limited computing resources and latency should be considered. Second, both warships and civilian ships are considered as ship targets. If they cannot be marked separately, it is impossible to detect only warship targets, and civilian ships will cause a large number of false alarms during the detection process.

[0004] Therefore, how to provide a port panoramic perception system that can simultaneously process warship detection, warship segmentation and water area extraction in the input image is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a port panoramic perception system based on ship instance segmentation. On the one hand, instance segmentation adds pixel-level supervision information to the target detection task, which has obvious effects on targets with obvious size and shape characteristics such as warships, and can effectively eliminate the interference caused by small civilian ships to the detection; on the other hand, the instance segmentation task is based on target detection, thereby ensuring that the segmentation of warships will not be interfered by other similar contents on the water surface or on the shore. In addition, the present invention also adopts a multi-scale cross-fusion feature extraction network, which gets rid of the semantic segmentation network structure of encoding and decoding while sharing information between tasks, and directly extracts water areas efficiently on the output feature map.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a port panoramic perception system based on ship instance segmentation, including an image acquisition module, a neural network model and a panoramic perception result output module; the image acquisition module is used to acquire port images; the neural network model is used to perform warship detection, warship segmentation and water area extraction processing on the acquired port images, and obtain the warship target positioning frame, the specific shape outline of the warship target and the port water area distribution; the panoramic perception result output module is used to output the panoramic perception result according to the warship target positioning frame, the specific shape outline of the warship target and the port water area distribution.

[0007] Preferably, the neural network model includes a convolution layer, a multi-scale feature encoding network layer, a water area extraction module and an instance segmentation module; the convolution layer is used to downsample the collected port image to obtain a feature map; the multi-scale feature encoding network layer is used to classify the feature map to obtain feature maps of high, medium and low scales; the water area extraction module performs extraction processing based on the feature maps of high, medium and low scales to obtain a port water area distribution image; the instance segmentation module performs detection and segmentation processing based on the feature maps of high, medium and low scales to obtain a warship target positioning frame and a specific shape contour image of the warship target.

[0008] Preferably, the multi-scale feature encoding network layer includes a first deconvolution layer, and the first deconvolution layer is used to perform interpolation upsampling processing on the feature map to obtain the interpolation upsampled feature map.

[0009] Preferably, the water area extraction module includes a concatenate layer, a second deconvolution layer and an extraction layer; the concatenate layer is connected to the second deconvolution layer and the extraction layer in sequence; the concatenate layer is used to fuse the feature map after interpolation upsampling; the second deconvolution layer is used to upsample the fused feature map; the extraction layer is used to extract the upsampled feature map to obtain a port water area distribution image.

[0010] Preferably, the instance segmentation module includes a feature pyramid extraction module, a candidate box extraction module, a label box extraction module, a target detection module and a semantic segmentation module; the feature pyramid extraction module is connected to the candidate box extraction module, the label box extraction module, the target detection module and the semantic segmentation module in sequence; the feature pyramid extraction module is used to form a feature pyramid image from the feature maps of three different scales, high, medium and low; the candidate box extraction module is used to extract a candidate box image according to the feature pyramid image; the label box extraction module is used to extract a label box image according to the feature pyramid image; the target detection module is used to select the feature pyramid image according to the intersection and union of the candidate box image and the label box image to obtain the warship target positioning box; the semantic segmentation module is used to perform pixel-by-pixel marking within the warship target positioning box to obtain a specific shape contour image of the warship target.

[0011] Preferably, the method for determining the warship target positioning frame is: setting an intersection-and-union ratio threshold, if the intersection-and-union ratio of the candidate frame and the labeled frame is within the threshold range, it is determined to be a positive sample; otherwise, it is a negative sample.

[0012] Preferably, the determination formula of the warship target positioning frame is as follows:

[0013]

[0014] Among them, GT is the annotation box, box is the candidate box, and T now Indicates the current IoU threshold, 1 is a positive sample and 0 is a negative sample.

[0015] Preferably, the neural network model also includes a loss function, which is used to optimize the warship target positioning frame, the port water distribution image and the warship target specific shape contour image; the loss function includes the warship detection loss, the water extraction loss and the warship pixel segmentation loss; the loss function calculation formula is as follows:

[0016] L all =γ 1 L det +γ 2 L mask +γ 3 L water-seg

[0017] Among them, L all is the loss function, L det is the warship detection loss, γ 1 is the coefficient of warship detection loss, and the warship pixel segmentation loss L mask To output the corresponding loss according to the category, γ 2 is the coefficient of the warship pixel segmentation loss; Lwater-seg Extract the loss for the water area, γ 3 Extract the coefficient of loss for the water area.

[0018] Preferably, the warship detection loss includes a weighted sum of a classification loss component, a target loss component and a labeling box loss component, and the warship detection loss calculation formula is as follows:

[0019] L det =α 1 L olass +α 2 L obj +α 3 L box

[0020] The water area extraction loss includes a cross entropy loss component and an intersection-over-union loss component. The water area extraction loss calculation formula is as follows:

[0021] L water-seg =L ce +L IoU

[0022] Among them, L class is the classification loss component, α 1 is the weighted coefficient of the classification loss component, L obj is the target loss component, α 2 is the weighted coefficient of the target loss component, L box is the candidate box loss component, α 3 is the weighted coefficient of the candidate box loss component, L ce is the cross entropy loss component, L IoU is the intersection-and-combination loss component.

[0023] It can be known from the above technical solutions that, compared with the prior art, the present invention discloses a port panoramic perception system based on ship instance segmentation. The neural network model as a whole adopts a multi-task learning strategy. Through the ship instance segmentation task, the pixel-level labeled supervision information of the target is introduced into the training of target detection. The military ships and civilian ships are distinguished through scale features, and the false alarms of civilian ships are effectively eliminated in the warship detection. In addition, while performing the instance segmentation task, the neural network model can also realize end-to-end port water extraction, thereby completing the multi-task panoramic perception of the port area. In terms of implementation details, the present invention adopts a port water extraction network based on multi-scale feature fusion coding to meet the needs of the instance segmentation task in the encoding stage, while avoiding the redesign of a complex decoding network in the water extraction task. In addition, a dynamic sample allocation training strategy is also adopted to enable the neural network model to continuously generate high-quality training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0025] Figure 1 The accompanying drawing is a schematic diagram of the process provided by the present invention.

[0026] Figure 2 The accompanying drawing is a schematic diagram of the port water area extraction process based on multi-scale feature fusion coding provided by the present invention.

[0027] Figure 3 The accompanying drawing is a schematic diagram of the network flow of warship instance segmentation based on dynamic sample allocation provided by the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The embodiment of the present invention discloses a port panoramic perception system based on ship instance segmentation, characterized in that it includes an image acquisition module, a neural network model and a panoramic perception result output module; the image acquisition module is used to acquire port images; the neural network model is used to perform warship detection, warship segmentation and water area extraction processing on the acquired port images, and obtain the warship target positioning frame, the specific shape outline of the warship target and the port water area distribution; the panoramic perception result output module is used to output the panoramic perception result according to the warship target positioning frame, the specific shape outline of the warship target and the port water area distribution. The present invention can simultaneously process warship detection, warship segmentation and water area extraction in the input image, saving more processing time, while the traditional method of processing these tasks one by one takes longer time than processing all tasks at once.

[0030] In this embodiment, the neural network model includes a convolution layer, a multi-scale feature encoding network layer, a water area extraction module and an instance segmentation module; the convolution layer is used to downsample the collected port image to obtain a feature map; the multi-scale feature encoding network layer is used to classify the feature map to obtain high, medium and low scale feature maps; the water area extraction module extracts and processes according to the high, medium and low scale feature maps to obtain the port water area distribution image; the instance segmentation module detects and segments according to the high, medium and low scale feature maps to obtain the warship target positioning frame and the specific shape contour image of the warship target. Therefore, an image of the port area is input to the neural network model, and the output interpretation result includes three contents: the warship target position positioning frame, the specific shape contour of the warship target, and the water area distribution of the port area. The specific implementation method and architecture flow are as follows: the neural network model performs two convolution downsamplings on the input image, thereby reducing the amount of parameters and complexity of the calculation, and the obtained feature map is sent to a multi-scale feature fusion encoding network layer. In the multi-scale feature fusion encoding network, the high-resolution feature map will always maintain the resolution for convolution operations, and will be continuously fused with the low-resolution feature map generated by downsampling, thereby obtaining feature maps of three different scales. The feature maps of the three resolutions will undergo two types of processing: on the one hand, the three feature maps are interpolated and upsampled to the maximum resolution and then directly concatenated and merged. After the obtained fused feature map is upsampled twice by deconvolution, the prediction of the water area extraction semantic segmentation task can be realized. On the other hand, the three feature maps will form a feature pyramid (FPN), and then according to the architecture of the two-stage instance segmentation network, an RPN network is used to extract the candidate area of ​​the warship target and then perform classification regression and pixel-by-pixel instance segmentation.

[0031] In this embodiment, the port water area extraction network based on multi-scale feature fusion coding is as follows: Figure 2As shown, the input and output sizes of the multi-scale feature fusion coding network are feature maps with a quarter of the original image resolution. The downsampling of the input image and the upsampling of the output result are completed by convolution and deconvolution layers respectively, because in the multi-scale feature fusion coding network, the high-resolution feature map will always maintain the resolution for convolution operation, on the one hand to control the parameter amount, on the other hand to match the feature map size required by the instance segmentation task. The present invention adopts a high, medium and low multi-scale representation in parallel, and the representation of each resolution always runs through the entire coding part to reduce the loss of position information. The coding part is divided into four stages (stage), three multi-scale feature cross-fusion processes. The network starts with a high-resolution feature map as the first stage, gradually increases the sub-network from high resolution to low resolution, forms more stages, and connects the multi-resolution sub-networks in parallel. In the whole process, the multi-scale repeated fusion is performed by repeatedly exchanging information on the parallel multi-resolution sub-networks (the upward arrow is the convolution downsampling, and the downward arrow is the interpolation upsampling). In each stage, multiple series-connected residual structures are used to learn features. Compared with the traditional U-shaped semantic segmentation network structure, the multi-scale feature fusion encoding network of the present invention has two advantages: on the one hand, the present invention connects the high-resolution to low-resolution sub-networks in parallel, which can maintain high resolution and make the heatmap of the predicted segmentation result more accurate in the spatial dimension; instead of serial connection like most existing encoding networks, the resolution is restored through a low-to-high decoding process. On the other hand, most existing fusion schemes combine low-level and high-level feature representations in parallel when outputting results, while the present invention uses repeated multi-scale fusion to enhance high-resolution feature representations using low-resolution feature representations of the same depth and similar level, and vice versa, so that the former better integrates high-level semantic features and the latter better integrates low-level detail features. Therefore, the present invention can have a more adequate representation for highly connected areas such as water areas.

[0032] In this embodiment, the instance segmentation module includes a feature pyramid extraction module, a candidate box extraction module, a label box extraction module, a target detection module and a semantic segmentation module; the feature pyramid extraction module is connected to the candidate box extraction module, the label box extraction module, the target detection module and the semantic segmentation module in sequence; the feature pyramid extraction module is used to form a feature pyramid image from the feature maps of three different scales, high, medium and low; the candidate box extraction module is used to extract a candidate box image according to the feature pyramid image; the label box extraction module is used to extract a label box image according to the feature pyramid image; the target detection module is used to select the feature pyramid image according to the intersection and union of the candidate box image and the label box image to obtain the warship target positioning box; the semantic segmentation module is used to perform pixel-by-pixel marking within the warship target positioning box to obtain a specific shape contour image of the warship target.

[0033] In this embodiment, the classification in target detection is performed based on the annotation box, that is, the method for determining the warship target positioning box is: if the intersection over union (IoU) ratio between the candidate box and the annotation box is within a certain threshold range, it is identified as a positive sample, otherwise it is a negative sample.

[0034]

[0035] Among them, GT is the annotation box, box is the candidate box, and T now Indicates the current IoU threshold, 1 is a positive sample and 0 is a negative sample.

[0036] However, setting a fixed threshold during training often does not yield the best results. As the number of iterations increases, the model's ability to classify samples increases, and the number of positive samples increases dramatically. For example, in step 4, at the beginning, the blue and yellow boxes are considered positive samples according to a threshold, but as higher-quality pink boxes appear, if the threshold is raised, the blue boxes with lower quality will become negative samples, thereby ensuring that high-quality positive samples can be continuously generated during training, giving full play to the advantages of the two-stage model and improving the accuracy of warship instance segmentation. The dynamic change process of the intersection-over-union (IoU) threshold is as follows: First, the intersection-over-union (IoU) of the candidate box and its matching annotation box is calculated, and then the Kth positive box is selected. I The larger value is used as the current IoU threshold T now ,With the process of training iteration, more and more positive samples are added, T now It will increase with the increase of I. In the specific operation, first calculate the Kth IFor large IoU values, since one iteration will generate many batches, the IoU threshold is updated using the average value of the previous ones every C iterations. The dynamic sample allocation of the present invention avoids the cascade detection process, and thus avoids introducing a large number of redundant calculations. The instance segmentation part of the panoramic perception network of the present invention dynamically changes the threshold of the allocated samples, and adopts this dynamic setting strategy to improve the model performance.

[0037] In this embodiment, a loss function is also included. Since the multi-task training framework of the present invention is composed of an instance segmentation task and a water area extraction task, and the instance segmentation task includes two parts: target detection of warships and pixel-level segmentation of contours, the loss function of the panoramic perception network of the present invention includes warship detection, warship pixel-level segmentation loss, and water area extraction loss. The loss function calculation formula is as follows:

[0038] L all =γ 1 L det +γ 2 L mask +γ 3 L water-seg

[0039] For the pixel-level segmentation of warships, assuming there are K categories in total, the mask segmentation branch will output K binary masks. When calculating the loss, the sigmoid output of the corresponding category to which the pixel belongs is used to calculate the loss. This can avoid inter-class competition and completely decouple mask prediction and classification prediction. The common FCN method for traditional semantic segmentation is to perform multi-category softmax classification on each pixel and then calculate the cross entropy loss. This approach will cause inter-class competition;

[0040] Among them, L all is the loss function, L det is the warship detection loss, γ 1 is the coefficient of warship detection loss, and the warship pixel segmentation loss L mask To output the corresponding loss according to the category, γ 2 is the coefficient of the warship pixel segmentation loss; L water-seg Extract the loss for the water area, γ 3 Extract the coefficient of loss for the water area;

[0041] The warship target detection amount includes the weighted sum of the classification loss component, the target loss component and the positioning frame loss component; the warship detection loss calculation formula is as follows:

[0042] L det =α 1 L class +α 2 L obj +α3 L box

[0043] The water area extraction loss includes a cross entropy loss component and an intersection-over-union loss component. The water area extraction loss calculation formula is as follows:

[0044] L water-seg =L ce +L IoU

[0045] Among them, L class is the classification loss component, α 1 is the weighted coefficient of the classification loss component, L obj is the target loss component, α 2 is the weighted coefficient of the target loss component, L box is the candidate box loss component, α 3 is the weighted coefficient of the candidate box loss component, L ce is the cross entropy loss component, L IoU is the intersection-and-combination loss component.

[0046] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0047] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A port panoramic perception system based on ship instance segmentation, characterized by: It includes image acquisition module, neural network model and panoramic perception result output module; The image acquisition module is used to acquire port images; The neural network model is used to perform warship detection, warship segmentation and water area extraction processing on the collected port images, and obtain the warship target positioning frame, the specific shape and outline of the warship target and the port water area distribution; The panoramic perception result output module is used to output the panoramic perception result according to the warship target positioning frame, the specific shape outline of the warship target and the port water distribution; The neural network model includes a convolutional layer, a multi-scale feature encoding network layer, a water area extraction module and an instance segmentation module; The convolution layer is used to downsample the collected port image to obtain a feature map; The multi-scale feature encoding network layer is used to classify the feature map to obtain feature maps of three scales: high, medium and low; The water area extraction module performs extraction processing based on the feature maps of high, medium and low scales to obtain the port water area distribution image; The instance segmentation module performs detection and segmentation processing according to feature maps of high, medium and low scales to obtain a warship target positioning frame and a specific shape contour image of the warship target.

2. The port panoramic perception system based on ship instance segmentation according to claim 1 is characterized in that: The multi-scale feature encoding network layer includes a first deconvolution layer, and the first deconvolution layer is used to perform interpolation upsampling processing on the feature map to obtain the interpolation upsampled feature map.

3. The port panoramic perception system based on ship instance segmentation according to claim 2 is characterized in that: The water area extraction module includes a concatenate layer, a second deconvolution layer and an extraction layer; the concatenate layer is sequentially connected to the second deconvolution layer and the extraction layer; The concatenate layer is used to fuse the feature maps after interpolation and upsampling; The second deconvolution layer is used to upsample the fused feature map; The extraction layer is used to extract the upsampled feature map to obtain a port water area distribution image.

4. The port panoramic perception system based on ship instance segmentation according to claim 1 is characterized in that: The instance segmentation module includes a feature pyramid extraction module, a candidate box extraction module, a label box extraction module, a target detection module and a semantic segmentation module; the feature pyramid extraction module is sequentially connected to the candidate box extraction module, the label box extraction module, the target detection module and the semantic segmentation module; The feature pyramid extraction module is used to form a feature pyramid image from the feature maps of three different scales: high, medium and low; The candidate frame extraction module is used to extract a candidate frame image according to the feature pyramid image; The annotation box extraction module is used to extract the annotation box image according to the feature pyramid image; The target detection module is used to select the feature pyramid image according to the intersection and union of the candidate frame image and the labeled frame image to obtain the warship target positioning frame; The semantic segmentation module is used to perform pixel-by-pixel marking within the warship target positioning frame to obtain a specific shape contour image of the warship target.

5. The port panoramic perception system based on ship instance segmentation according to claim 4 is characterized in that: The method for determining the warship target positioning frame is as follows: setting an intersection-and-union ratio threshold, if the intersection-and-union ratio of the candidate frame and the labeled frame is within the threshold range, it is identified as a positive sample; otherwise, it is a negative sample.

6. The port panoramic perception system based on ship instance segmentation according to claim 5 is characterized in that: The determination formula of the warship target positioning frame is as follows: Among them, GT is the annotation box, box is the candidate box, and T now Indicates the current IoU threshold, 1 is a positive sample and 0 is a negative sample.

7. The port panoramic perception system based on ship instance segmentation according to claim 5 is characterized in that: The neural network model also includes a loss function, which is used to optimize the warship target positioning frame, the port water distribution image, and the warship target specific shape contour image; the loss function includes the warship detection loss, the water area extraction loss, and the warship pixel segmentation loss; the loss function calculation formula is as follows: L all =γ1L det +γ2L mask +γ3L water-seg Among them, L all is the loss function, L det is the warship detection loss, γ1 is the coefficient of warship detection loss, and the warship pixel segmentation loss L mask is the corresponding loss amount according to the category output, γ2 is the coefficient of the loss amount of warship pixel segmentation; L water-seg is the water area extraction loss, and γ3 is the coefficient of water area extraction loss.

8. The port panoramic perception system based on ship instance segmentation according to claim 7 is characterized in that: The warship detection loss includes the weighted sum of the classification loss component, the target loss component and the annotation box loss component. The warship detection loss calculation formula is as follows: L det =α1L class +α2L obj +α3L box The water area extraction loss includes a cross entropy loss component and an intersection-over-union loss component. The water area extraction loss calculation formula is as follows: L water-seg =L ce +L IoU Among them, L class is the classification loss component, α1 is the weighting coefficient of the classification loss component, L obj is the target loss component, α2 is the weighting coefficient of the target loss component, L box is the candidate box loss component, α3 is the weighted coefficient of the candidate box loss component, L ce is the cross entropy loss component, L IoU is the intersection-and-combination loss component.