Civil ship classification system based on transfer learning

Abstract

The invention discloses a civil ship classification system based on transfer learning, and belongs to the field of computer vision. Comprising a head module, a CNN module, a conversion module, a Transform module and a post-processing module. The head module and the CNN module jointly form a CNN layer and are used for identifying shallow layer information containing the color and the shape of an image; the conversion module is used for converting the feature map output by the CNN layer to form a Transform feature map; the Transform module is used for carrying out advanced / global feature extraction on the feature map which passes through the conversion module; the post-processing module comprises a global mean pooling layer and a full connection layer and is used for feature compression and category probability calculation. According to the method, the ship type is comprehensively judged and the ship position is determined by combining the advantages of the Transform and the CNN, that is, the advantage that the CNN is good at extracting image shallow layer features and the advantage that the Transform is good at understanding image deep layer information are utilized, and the high-precision detection effect is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a civilian ship classification system based on transfer learning. BACKGROUND

[0002] Computer vision is a discipline that studies how machines "see". Through the combination of cameras and computers, computer vision aims to achieve automatic identification of regions of interest in images, or understanding of the meaning expressed by the whole image, so that computers can better adapt to human observation or transmit to instruments for detection. The technologies designed in this field are very extensive, including image processing, pattern recognition, machine learning and other fields. The application scenarios of computer vision are also very extensive, such as industrial automation, intelligent identification and other fields. In the rapid development of artificial intelligence today, computer vision has become an indispensable important branch, promoting the development of human society.

[0003] Currently, artificial neural networks have been widely used in computer vision. Artificial neural networks in the field of computer vision are mainly divided into two categories, namely convolution neural network (CNN) based on convolution mechanism and Transformer network based on attention mechanism. Convolution neural network has excellent performance in processing image details and extracting image shallow features, while Transformer network has excellent effect in understanding deep information of images. The present application combines the advantages of CNN and Transformer, utilizes the characteristics of CNN good at perception and Transformer good at understanding, and proposes a neural network for identifying types of civilian ships. SUMMARY

[0004] The present application provides a civilian ship classification system based on transfer learning, in order to solve the above problems in the prior art.

[0005] In order to solve the above technical problems, the present application adopts the following technical solutions:

[0006] The present application provides a civilian ship classification system based on transfer learning, comprising: a head module, a CNN module, a conversion module, a Transformer module and a post-processing module.

[0007] The head module and the CNN module together form a CNN layer, which is used to identify shallow information containing color and shape of images.

[0008] The conversion module is used to convert the feature map output by the CNN layer to form a Transformer feature map.

[0009] The transformer module is used for high-level / global feature extraction on the feature map after the conversion module;

[0010] The post-processing module includes a global mean pooling layer and a fully connected layer, which are used for feature compression and class probability calculation.

[0011] Specifically, the conversion module forms a transformed feature map by dividing the feature map into blocks and splicing along the channel direction.

[0012] Specifically, the transformer module includes a convolution kernel, a layer normalization layer, a Gaussian error linear unit, and a Softmax function.

[0013] The convolution kernel is the core component of the neural network, responsible for extracting image features.

[0014] The layer normalization layer is used to stabilize the training process of the neural network.

[0015] The Gaussian error linear unit is an activation function.

[0016] The Softmax function is an activation function.

[0017] Specifically, the standard convolution kernel name is defined as Convk×k, and the standard convolution expression is:

[0018] f Convk×k (x)=wx+b;

[0019] where f Convkxk () is the convolution kernel operation function, k is the size parameter of the convolution kernel, w is the convolution kernel coefficient matrix, b is the convolution kernel bias coefficient, and x is the input, i.e. the feature map.

[0020] Specifically, the LN expression is:

[0021]

[0022] where f LN () is the LN operation function, γ is the weight coefficient of LN, β is the bias coefficient of LN, x is the input, μ is the mean of x in the channel direction, σ is the standard deviation of x in the channel direction, and ∈ is a constant (∈=10 -5 ).

[0023] Specifically, the Gaussian error linear unit expression is:

[0024]

[0025] where f GELU () is the GELU operation function, μ is the mean of x, σ is the standard deviation of x, and x is the input.

[0026] Specifically, the Softmax function expression is:

[0027]

[0028] Wherein, f Softmax () is a Softmax operation function, x i is the i-th feature subgraph of the input feature map, and N is the total number of input x feature subgraphs, that is, the number of channels.

[0029] Specifically, the input feature map x will undergo the following process in the Transformer module:

[0030] x is normalized by the LN layer to obtain x0;

[0031] x0 passes through three Conv 3x3 to obtain three groups of feature vectors x1, x2 and x3;

[0032] x1 and x2 are multiplied and then passed through a Softmax activation function to obtain a transposed attention mechanism map x4;

[0033] x4 is multiplied by x3 and then passed through a Conv 1x1 for feature fusion to obtain x5;

[0034] x5 is subjected to LN normalization, Conv 3x3, GELU, and Conv 1x1 to obtain x8;

[0035] x8 and x7 are added to obtain the output y.

[0036] The beneficial effects provided by the present application are:

[0037] The present application provides a civil ship classification system based on transfer learning, which aims to combine the advantages of Transformer and CNN, that is, to use the advantages of CNN in extracting image shallow features and the advantages of Transformer in understanding image deep information to comprehensively determine the ship type and the ship position, and to realize high-precision detection effect.

[0038] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is a network overall structure schematic diagram of the embodiment of the present application.

[0040] Figure 2 It is a CNN layer transfer schematic diagram of the embodiment of the present application.

[0041] Figure 3 This is a schematic diagram of the conversion module structure according to an embodiment of the present invention.

[0042] Figure 4 This is a schematic diagram of the Transformer module structure according to an embodiment of the present invention.

[0043] Figure 5 This is a schematic diagram of the post-processing module structure according to an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0045] This invention provides a civilian vessel identification system based on transfer learning. The network employs shallow modules of a large CNN network pre-trained on a large dataset (such as ImageNet) to extract shallow information from images; it then utilizes the proposed Transformer module to understand deeper information within the images. On one hand, this network can leverage the CNN layers to transfer prior weights from the large dataset, improving its detection performance and achieving the goal of transfer learning; on the other hand, it can utilize the Transformer layer to deepen its comprehensive understanding of image information, further enhancing its detection accuracy.

[0046] (1) Module section

[0047] like Figure 1 The diagram shown is a schematic representation of the overall network structure according to an embodiment of the present invention. The present invention mainly relates to the following four parts:

[0048] 1) Header Module

[0049] 2) CNN module

[0050] 3) Conversion Module

[0051] 4) Transformer module

[0052] 5) Post-processing module

[0053] The overall workflow of this invention is as follows:

[0054] 1) Input image;

[0055] 2) The head module of the CNN layer performs downsampling and dimensionality augmentation on the input image, performs preliminary sampling, and outputs a feature map;

[0056] 3) CNN module of CNN layer extracts features of the feature map of 2), and learns shallow information of the image:

[0057] 4) The conversion module blocks the feature map of 3) and splices along the channel direction to form a transformed feature map (i.e. the smallest semantic unit in natural language processing, named Token);

[0058] 5) The Transformer layer extracts the output feature map of 4) by using the Transformer module one by one;

[0059] 6) The post-processing layer calculates the feature map of 5) to obtain a classification result (category information and category probability);

[0060] 7) The category result is output.

[0061] (2) Module part

[0062] 1) Head module and CNN module

[0063] As shown in Figure 1 , the head module and the CNN module together constitute the CNN layer. The CNN layer can be directly taken from the shallow module of the existing pre-trained large model. Because for the CNN network, its shallow module is mainly responsible for identifying the color, shape and other shallow information of the image. Whether it is a professional field dataset or a general field dataset, the color and shape of the image and other shallow information are common. Therefore, the network directly migrates this part of the network to itself, which can use the excellent weights pre-trained by big data to "guide" itself to learn more accurate information, thereby improving its performance.

[0064] As shown in Figure 2 , it is a CNN layer migration schematic diagram of the embodiment of the application. This diagram takes ResNet50 as an example. Generally speaking, all CNN networks follow a 5-layer structure. In Figure 2 , the part of the network migrated is the first layer and the second layer of ResNet. The migration method is to directly copy the first layer and the second layer (including weights and network structure) of ResNet to the CNN layer of the network.

[0065] 2) Conversion module

[0066] The conversion module is used to convert the feature map output by the CNN layer to form a transformed feature map. It will block the feature map and splice along the channel direction to form a transformed feature map (i.e. the smallest semantic unit in natural language processing, named Token). The specific process is shown in Figure 3 , it is a conversion module structure schematic diagram of the embodiment of the application.

[0067] 3) Transformer module

[0068] like Figure 4 The diagram shown is a schematic of the Transformer module structure according to an embodiment of the present invention. The feature map after transformation by the transformation module will be subjected to high-level / global feature extraction by the Transformer module proposed in this invention.

[0069] The convolutional kernel is a core component of a neural network, responsible for extracting image features. Its standard convolutional kernel name is defined as Convk×k, and the standard convolutional expression can be written as:

[0070] f Convk×k (x)=wx+b

[0071] Among them, f Convk×k () is the convolution kernel operation function, k is the size parameter of the convolution kernel, w is the convolution kernel coefficient matrix, b is the convolution kernel bias coefficient, and x is the input.

[0072] Layer Normalization (LN) is a technique that normalizes feature maps along the channel directions, which can stabilize the training process of neural networks. The LN expression can be written as:

[0073]

[0074] Among them, f LN () represents the LN operation function, γ is the LN weight coefficient, β is the LN bias coefficient, x is the input, μ is the mean of x in the channel direction, σ is the standard deviation of x in the channel direction, and ∈ is a constant (∈ = 10). -5 ).

[0075] The Gaussian Error Linear Unit (GELU) is the activation function in this invention.

[0076] Its expression is:

[0077]

[0078] Among them, f GELU () is the GELU operation function, μ is the mean of x, σ is the standard deviation of x, and x is the input.

[0079] The Softmax function is also the activation function in this invention. Its expression is:

[0080]

[0081] Among them, f Softmax() is a Softmax operation function, x i is the i-th feature subgraph of the input feature graph, and N is the total number of feature subgraphs (channel number) of the input x.

[0082] The input feature graph x will undergo the following process in the Transformer module:

[0083] a) x is normalized by the LN layer to obtain x0;

[0084] b) x0 passes through three Conv 3x3 to obtain three groups of feature vectors x1, x2 and x3;

[0085] c) x1 and x2 are multiplied and then pass through the Softmax activation function to obtain the transposed attention mechanism graph x4 (essentially a probability graph);

[0086] d) x4 is multiplied by x3 and then passes through a Conv 1x1 for feature fusion to obtain x5;

[0087] e) x5 passes through LN normalization, Conv 3x3, GELU, and Conv 1x1 to obtain x8;

[0088] f) x8 and x7 are added to obtain the output y.

[0089] The output y of the Transformer module is a feature graph with global attention mechanism, which can well express the global information and deep information of the image.

[0090] 4) Post-processing module

[0091] As shown in Figure 5 , it is a post-processing module structure schematic diagram of the embodiment of the application. It is composed of a global average pooling layer (Global Average Pooling, GAP) and a fully connected layer (Fully Connected, FC), which is used for feature compression and class probability calculation.

[0092] The scheme provided by the application has the following effects:

[0093] The application provides a civil ship classification system based on transfer learning, which aims to combine the advantages of Transformer and CNN, that is, to use the advantages of CNN in extracting image shallow features and the advantages of Transformer in understanding image deep information to comprehensively determine the ship type and the ship position, and to realize high-precision detection effect.

[0094] Optionally, in some possible implementations, all or part of the above embodiments can be included.

[0095] It is to be understood that the terminology "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" used in the description

[0096] Of course, various corresponding changes and modifications can be made to the present application by those skilled in the art without departing from the spirit and scope of the present application, and such changes and modifications shall be included in the scope of protection of the present application.

Claims

1. A civil vessel classification system based on transfer learning, characterized in that, include: Head module, CNN module, transformation module, Transformer module, post-processing module; The head module and the CNN module together form the CNN layer, which is used to identify shallow information containing the color and shape of an image; The transformation module is used to transform the feature maps output by the CNN layers into Transformer-style feature maps; The Transformer module is used to perform high-level / global feature extraction on the feature map after the transformation module; The post-processing module includes a global mean pooling layer and a fully connected layer for feature compression and class probability calculation.

2. The civil ship classification system based on transfer learning according to claim 1, characterized in that, The transformation module divides the feature map into blocks and splices them along the channel direction to form a Transformer-styled feature map.

3. The civil ship classification system based on transfer learning according to claim 2, characterized in that, The Transformer module includes convolutional kernels, layer processing normalization layers, Gaussian error linear units, and the Softmax function; Convolutional kernels are the core component of neural networks, responsible for extracting image features; The normalization layer is used to stabilize the training process of the neural network; The Gaussian error linear unit is the activation function; The Softmax function is the activation function.

4. The civil vessel classification system based on transfer learning according to claim 3, characterized in that, The standard convolution kernel is defined as Convk×k, and the standard convolution expression is: f Convk×k (x)=wx+b; Among them, f Convk×k () is the convolution kernel operation function, k is the size parameter of the convolution kernel, w is the convolution kernel coefficient matrix, b is the convolution kernel bias coefficient, and x is the input, i.e., the feature map.

5. The civil ship classification system based on transfer learning according to claim 3, characterized in that, The LN expression is: Among them, f LN () represents the LN operation function, γ is the LN weight coefficient, β is the LN bias coefficient, x is the input, μ is the mean of x in the channel direction, σ is the standard deviation of x in the channel direction, and ∈ is a constant (∈ = 10). -5 ).

6. The civil ship classification system based on transfer learning according to claim 3, characterized in that, The expression for the Gaussian error linear unit is: Among them, f GELU () is the GELU operation function, μ is the mean of x, σ is the standard deviation of x, and x is the input.

7. The civil ship classification system based on transfer learning according to claim 3, characterized in that, The expression for the Softmax function is: Among them, f Softmax () is the Softmax operation function, x i Let be the i-th feature sub-map of the input feature map, and N be the total number of feature sub-maps of the input x, i.e., the number of channels.

8. The civil ship classification system based on transfer learning according to any one of claims 1 to 7, characterized in that, The input feature map x will undergo the following process in the Transformer module: x is normalized after passing through an LN layer to obtain x0; x0 is processed through three Conv 3×3 processes to obtain three sets of feature vectors x1, x2 and x3; After multiplying x1 and x2, the transposed attention mechanism diagram x4 is obtained by applying the Softmax activation function; After multiplying x4 and x3, feature fusion is performed through a Conv 1×1 to obtain x5; After LN normalization, Conv 3×3, GELU, and Conv 1×1, x5 is obtained as x8; Adding x8 and x7 together gives the output y.

Images