A multi-modal maritime target detection method based on multi-scale capsules and Bi-FPN

Abstract

The application discloses a kind of multi-modal offshore target detection methods based on multi-scale capsule and Bi-FPN, comprising: S1, the RGB-D multi-modal image data of offshore scene is carried out multi-scale feature fusion by bidirectional feature pyramid;S2, the fusion effect of shallow feature is optimized by iterative fusion method;S3, the multi-scale feature obtained after feature fusion is input into classification network and regression network based on multi-scale capsule network respectively by hierarchical level;S4, the anchor frame class prediction matrix and the four-tuple vector of target object prediction frame coordinates required finally are calculated respectively by depth separable convolution and dynamic routing algorithm technology.According to the application, a variety of deep neural network technologies are fused, the accuracy of offshore scene target detection is improved, objects of different sizes and different degrees of shielding can also be identified, and the method has wide applicability and strong robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision algorithms, and in particular to a multi-modal offshore target detection method based on multi-scale capsules and Bi-FPN. BACKGROUND

[0002] In recent years, computer vision algorithms based on deep learning have achieved great success in medical, transportation, security and other fields. As a part of computer vision, target detection usually uses RGB images as input, and its task is to determine whether there is a target object in the image and to mark its category and location. At present, target detection has been extensively studied in the academic field and widely applied in real life, such as video fire detection, self-driving, and face recognition.

[0003] With the success of Tesla in the field of autonomous driving in recent years, depth estimation and depth completion technology has made great progress, and the corresponding depth image generation technology has also made a major breakthrough. Various sensors on vehicles provide more and more multi-modal information. These multi-modal data form a complementary relationship to some extent. For example, depth images make up for the spatial position information of missing objects in RGB images, and form a complement with the texture and color information of RGB images. How to effectively utilize the complementarity between multi-modal data to improve the accuracy of target detection has become the focus of researchers in recent years. However, unlike some target detection applications that have achieved certain results in the field of autonomous driving, target detection in the marine scene usually has many complex and variable interference factors such as sea waves and weather changes, which can easily cause noise. Existing multi-modal target detection methods are often limited by the fact that convolutional neural network architecture itself cannot understand the relationship between entities and is not good at handling viewpoint transformation. In the marine target detection application scene with a large amount of complex texture background, such as the detection of small target ships, reefs, buoys, etc. at a long distance, and due to the interference of sea waves on the sea surface, complex weather and other factors, the detection of images with a large amount of noise and frequent changes in observation angle and background, it is difficult to achieve real-time and accurate target detection. SUMMARY

[0004] In view of the deficiencies in the prior art, the present application aims to provide a multi-modal offshore target detection method based on multi-scale capsules and Bi-FPN, which fuses multiple deep neural network technologies, improves the accuracy of offshore scene target detection, and can also identify objects of different sizes and different degrees of shielding, has wide applicability and strong robustness. In order to achieve the above-mentioned purposes and other advantages according to the present application, a multi-modal offshore target detection method based on multi-scale capsules and Bi-FPN is provided, comprising:

[0005] S1, multi-scale feature fusion is performed on the RGB-D multi-modal image data of the sea scene by a bidirectional feature pyramid;

[0006] S2, the fusion effect of the shallow features is optimized by an iterative fusion method;

[0007] S3, the multi-scale features obtained after feature fusion are input into a classification network and a regression network based on a multi-scale capsule network in a hierarchical manner;

[0008] S4, a final required anchor class prediction matrix and a four-tuple vector of target object prediction box coordinates are calculated by using a depth separable convolution and a dynamic routing algorithm.

[0009] Preferably, the step S1 includes pre-processing the image to be detected, and extracting seven multi-scale features P1 to P7 from the input RGB image and depth image as the input of the bidirectional feature pyramid.

[0010] Preferably, the features at the P3 to P7 levels are used to perform weighted summation on the features of the two different modalities at the same channel and position, to obtain superimposed features Pisum after superimposition and fusion of the two modalities, and the superimposed features are used as the input features of the bidirectional feature pyramid Bi-FPN.

[0011] Preferably, in the Bi-FPN, the input at different levels is added with a separate weight wi, and a ReLU activation function is used after each weight wi to make the value non-negative, the Bi-FPN performs depth separable convolution on the processed data, and batch normalization and Swish activation function are used after each convolution to realize the fusion of the features at different levels.

[0012] Preferably, the structure of the multi-scale capsule classification network includes three layers: the first layer is a convolution layer, which is used to expand the channel number and output three tensors of different sizes;

[0013] The second layer is a multi-scale main capsule layer, which is used to encode the extracted feature information at different levels, each main capsule layer has 8D capsules with different convolution parameters, receives the primary features from the previous layer, and outputs the combination of the features;

[0014] The third layer is a digital capsule layer, which is connected with the multi-scale main capsule layer, and calculates the predicted vector through multiple weight matrices.

[0015] Compared with the prior art, the present application has the beneficial effects that: by utilizing the characteristics of the capsule network architecture that is not dependent on the viewing angle and has strong adaptability to data fluctuations, the background texture complexity caused by the complex and changeable environmental noise such as sea waves and weather in the marine scene, and the problems that the target objects such as ships and reefs to be detected are often far away and easy to be partially occluded, etc. are effectively improved by the multiple technologies such as bidirectional feature pyramid, multi-scale capsule network, depth separable convolution, batch normalization, Squash activation function and dynamic routing algorithm, thereby effectively improving the accuracy of marine target detection. The method can fuse multiple deep neural network technologies, improve the accuracy of marine scene target detection, and can also identify objects of different sizes and different degrees of occlusion, so it has wide applicability and strong robustness. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 FIG. 1 is a schematic diagram of the overall network architecture of the multi-modal marine target detection method based on multi-scale capsules and Bi-FPN according to the present application;

[0017] Figure 2 FIG. 3 is a schematic diagram of the multi-scale capsule classification network structure of the multi-modal marine target detection method based on multi-scale capsules and Bi-FPN according to the present application;

[0018] Figure 3 FIG. 4 is a schematic diagram of the multi-scale capsule regression network structure of the multi-modal marine target detection method based on multi-scale capsules and Bi-FPN according to the present application;

[0019] Figure 4 FIG. 5 is a schematic diagram of the iterative fusion strategy optimized bidirectional feature pyramid network structure of the multi-modal marine target detection method based on multi-scale capsules and Bi-FPN according to the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0021] REFERENCE Figures 1-4 A multi-modal marine target detection method based on multi-scale capsules and Bi-FPN, comprising: S1, performing multi-scale feature fusion on the RGB-D multi-modal image data of the marine scene by a bidirectional feature pyramid;

[0022] S2, optimizing the fusion effect of shallow features by an iterative fusion method;

[0023] S3. Input the multi-scale features obtained after feature fusion into the classification network and regression network based on multi-scale capsule network at different levels.

[0024] S4. Using depthwise separable convolution and dynamic routing algorithms, calculate the final required anchor box category prediction matrix and target object prediction box coordinate quadruples vector respectively.

[0025] Furthermore, step S1 includes preprocessing the image to be detected by extracting multi-scale features from the input RGB image and depth image at seven levels, from P1 to P7, as input to the bidirectional feature pyramid.

[0026] like Figure 1 As shown, features from levels P3 to P7 are used to perform weighted summation on features of two different modalities at the same channel and position, resulting in a superimposed feature Pisum after the two modalities are superimposed and fused. This superimposed feature is used as the input feature of the Bi-FPN bidirectional feature pyramid. In Bi-FPN, a separate weight wi is added to the input of each level, and a ReLU activation function is used after each weight wi to make its value non-negative. Bi-FPN performs depthwise separable convolution on the processed data and uses batch normalization and Swish activation function after each convolution to achieve the fusion of features at each level. For the RGB-D multimodal image of the marine scene to be detected, preprocessing, feature extraction, classification, and regression are performed. First, the image to be detected is preprocessed. The EfficientNet backbone network is used to extract multi-scale features from the input RGB image and depth image, which are seven levels from P1 to P7, as the input of the bidirectional feature pyramid. Then, using the features from layers P3 to P7, the features of two different modalities at the same channel and position are weighted and summed to obtain the superimposed feature Pisum after the two modalities are superimposed and fused. This superimposed feature is used as the input feature of the Bi-FPN (Bidirectional Feature Pyramid). In Bi-FPN, a separate weight wi is added to the input of each layer, and a ReLU activation function is used after each weight wi to make its value non-negative. Afterwards, Bi-FPN performs depthwise separable convolution on the above-processed data and uses batch normalization and the Swish activation function after each convolution to achieve the fusion of features at each layer. At the same time, in order to ensure that the features in the shallow feature layer P2 are fully fused with other deep features, this invention adopts an iterative fusion strategy in Bi-FPN to optimize the fusion of shallow features, so as to obtain a fully fused feature list as the input of the classification and regression network. Specifically, for the shallow features of layer P2, they are first downsampled to an appropriate size, and then added to Bi-FPN at each iteration time, and fused with the deep feature layers from bottom to top. The improved Bi-FPN network structure is as follows:Figure 4 As shown in the figure.

[0027] As shown in the figure. Figure 2 As shown in the figure.

[0028] The second layer is a multi-scale main capsule layer, which is used for encoding the extracted feature information of different levels. Each main capsule layer has 8D capsules with different convolution parameters, receives the primary features of the previous layer, and outputs the combination of features. Each main capsule layer includes 32 sub-capsules, which can be regarded as 32 parallel 2D convolution layers. Each capsule includes 8 convolution kernels. The final output after concatenation and splicing is batch X 6 X 6 X 8 X 32, and the channel number is 256. The last dimension of the output is cut, and the middle dimension is flattened to obtain 1152 output vectors. Each 8 X 1 vector in the output tensor is an initialization vector.

[0029] The third layer is a digital capsule layer. The digital capsule layer is connected to the multi-scale main capsule layer. The digital capsule layer calculates the predicted vector through multiple weight matrices. The digital capsule layer of the classification network has 80 digital capsules, representing 80 categories. Since the last channel number of the classification network is 80 X N (N represents the number of anchor boxes), the shape of the weight matrix required by the classification network is 80 X 1152 X NX 8, that is, each 8 X 1 initialization vector is multiplied by an 8 X N matrix to convert it into an N-dimensional vector. Then, the initialization vectors are converted to the same dimension through matrix multiplication by using the respective weight matrices, and are spliced by using the Concat function. The vectors are activated by using the Squash activation function. Finally, the activated vectors are input into a separate fully connected layer for decoding to output the probability of the category contained in each anchor box.

[0030] As shown in the figure. Figure 3 As shown in the figure, the structure of the multi-scale capsule regression network of the application is basically the same as that of the classification network. The main difference between the two is that in the third layer of the digital capsule layer, the digital capsule layer of the regression network has 4 digital capsules, representing the offset of the anchor box center point relative to the true box center point in the x and y directions, as well as the offset of the frame. In addition, since the last channel number of the regression network is 4 X N, the shape of the weight matrix required by the regression network is 4 X 1152 X NX 8. Each 8 X 1 initialization vector is multiplied by an 8 X N matrix to convert it into an N-dimensional vector and spliced by using the Concat function. The vectors are activated by using the dynamic routing algorithm. Finally, the activated vectors are input into a separate fully connected layer for decoding to output a four-tuple vector representing the offset.

[0031] The number of devices and the scale of processing illustrated herein are for simplicity and clarity and have no intent of limitation. Any size, scale, or number of devices, distributions, or processing steps is possible and within the scope of the application.

[0032] While the embodiments of the application have been disclosed as above, it should be apparent that they are susceptible to modifications and alternative forms. Specifically contemplated are, without limitation, embodiments of methods and apparatuses covering all steps as previously described or its equivalents. Modifications can be made by one skilled in the relevant art, and it is therefore contemplated to fall within the scope of the claims and their equivalents.

Claims

1. A multi-modal maritime target detection method based on multi-scale capsule and Bi-FPN, characterized in that, The method comprises the following steps: S1, performing multi-scale feature fusion on the RGB-D multi-modal image data of the sea scene through a bidirectional feature pyramid; in step S1, the to-be-detected image is preprocessed, and seven levels of P1 to P7 of extracted multi-scale features are extracted from the input RGB image and depth image as the input of the bidirectional feature pyramid; The features at the P3 to P7 levels are used to perform weighted summation on the features of two different modalities at the same channel and position, to obtain superimposed features Pisum after superimposition and fusion of the two modalities, and the superimposed features are used as the input features of the bidirectional feature pyramid Bi-FPN; For the shallow features at the P2 level, the shallow features are first down-sampled to a proper size, and then added to the Bi-FPN at each iteration time, and fused with the deep features from bottom to top; S2, optimizing the fusion effect of the shallow features through an iterative fusion method; S3, inputting the multi-scale features obtained after the feature fusion into a classification network and a regression network based on a multi-scale capsule network in a hierarchical manner; S4, calculating an anchor class prediction matrix and a four-tuple vector of a target object prediction box coordinate through a depth separable convolution and a dynamic routing algorithm.

2. The multi-modal maritime target detection method based on multi-scale capsule and Bi-FPN of claim 1, wherein, In the Bi-FPN, different levels of input are added with a single weight wi, and the value is non-negative after using the ReLU activation function after each weight wi, the Bi-FPN performs a depth separable convolution on the processed data, and uses batch normalization and a Swish activation function after each convolution to realize the fusion of features at different levels.

3. The multi-modal maritime target detection method based on multi-scale capsule and Bi-FPN of claim 1, wherein, The structure of the multi-scale capsule classification network comprises three layers: the first layer is a convolution layer, which is used to expand the channel number and outputs three tensors of different sizes; The second layer is a multi-scale main capsule layer, which is used to encode the extracted features at different levels, each main capsule layer has 8D capsules with different convolution parameters, receives the primary features from the previous layer, and outputs the combination of the features; The third layer is a digital capsule layer, which is connected with the multi-scale main capsule layer, and calculates the predicted vector through a plurality of weight matrices.

Images