Method for Identifying Marine Disaster-Bearing Bodies Based on Siamese Neural Network under Small-Sample Conditions

Abstract

A method for identifying marine disaster-bearing bodies based on a siamese neural network under small-sample conditions, which includes establishing an image sample library; expanding the sample library through data augmentation methods; annotating the samples; introducing the attention mechanism SKNet network and combining it with the ResNet101 network to construct a backbone feature extraction network; based on the backbone feature extraction network, constructing a siamese neural network with the same dual-path structure and weight sharing; using the siamese neural network to extract features from the input data; calculating the distance between the dual-path feature vectors through a loss function; and outputting the category information of the marine disaster-bearing body to which it belongs. In view of the characteristics of multi-scale, diversity, and small samples of marine disaster-bearing bodies, the present invention combines a convolutional network with an improved three-channel SKNet network, enhances the feature extraction ability and feature effectiveness of the algorithm, improves the adaptive ability of the algorithm to small-sample and multi-scale targets, and is more suitable for the identification and classification of marine disaster-bearing bodies under small-sample conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of marine emergency disaster prevention, and relates to a method for identifying marine disaster-bearing bodies. Background Art

[0002] Typhoon disasters are one of the most destructive natural disasters in the world. With the continuous enhancement of China's comprehensive national strength, the level of marine disaster early warning and forecasting has also been continuously improved. Provinces and regions have built comprehensive marine observation and forecasting service platforms to respond to numerical forecasting of marine environment and disasters. This platform uses observation means such as shore stations, buoys, satellites, and radars to provide services for early warning of marine disasters such as storm surges, sea waves, and wind fields in the early stage. The marine disaster early warning and forecasting services provided by the platform mainly play a role in large areas and as warnings, and there are deficiencies in small-area and refined early warning information services. Especially during major marine disasters, differential early warning services for different regions cannot be realized, and the basis for disaster prevention and mitigation deployment is mainly qualitative and less quantitative. Therefore, it is of great significance to conduct research on the disaster-causing mechanism of refined vulnerable disaster-bearing bodies, construct an accurate disaster loss assessment model, realize the evaluation of vulnerable disaster-bearing bodies and disaster-causing assessment, provide a basis for refined management of disaster prevention and mitigation during strong typhoons, assist the government in commanding and making decisions on marine disaster prevention and mitigation work, and minimize disaster losses to the greatest extent.

[0003] Due to the complexity of typhoon disasters (including different types of disasters such as strong wind disasters, strong wave disasters, and seawater intrusion) and the uncertainty of the vulnerability assessment of marine disaster-bearing bodies (rigid, non-rigid, and different types of disaster-bearing bodies such as point-like, linear, and planar, as well as the uncertainty of disaster-causing and disaster losses), conventional meteorological simulation means can only roughly predict the time, path, etc. of the formation of meteorological disasters, and cannot estimate the losses and social impacts that disasters may bring, making it difficult to meet the current social development's demand for precise and intelligent response. Therefore, how to scientifically evaluate the vulnerability of marine disaster-bearing bodies, find out the disaster-causing mechanism of marine disaster-bearing bodies, and establish a disaster loss assessment model, realize the whole-process disaster loss assessment and early warning before, during, and after typhoon storm surge disasters, and improve the emergency management level of marine disaster prevention and mitigation is an important issue related to public safety and the stable and rapid development of the national economy.

[0004] Most traditional methods for evaluating the vulnerability of disaster-bearing bodies construct an evaluation index system and a physical model for calculating weights for the vulnerability of disaster-bearing bodies based on the common elements of the entity and relationship of the vulnerability of disaster-bearing bodies, so as to realize the representation, integration, and storage of the knowledge of the vulnerability of disaster-bearing bodies. This method largely depends on the experience of researchers and the selection of weight parameters, and the disaster-causing factors and disaster-causing responses of disasters are a complex non-linear relationship, with strong uncertainty and difficulty in control. Therefore, it is difficult to accurately realize the evaluation of the vulnerability of disaster-bearing bodies and the assessment of disaster losses only relying on traditional physical models. Summary of the Invention

[0005] The present invention takes typical vulnerable marine disaster-bearing bodies as the research object, and proposes a method for identifying marine disaster-bearing bodies based on Siamese neural network under small sample conditions. For various typical marine disaster-bearing bodies in the form of points, lines, and surfaces, a small sample library is constructed; a Siamese neural network structure is designed to train the sample library, and a recognition model of marine disaster-bearing bodies under small sample conditions is obtained for disaster-bearing body evaluation and disaster warning, thereby constructing a disaster response model for vulnerable disaster-bearing bodies.

[0006] To achieve the above object, the present invention adopts the following technical solutions.

[0007] The method for identifying marine disaster-bearing bodies based on Siamese neural network under small sample conditions described in the present invention mainly includes three parts: (1) Data augmentation: The small sample data set is augmented by methods such as rotation angle, random cropping, and adjustment of picture brightness; (2) Design of the backbone feature extraction network: Its function is to extract features, and various neural networks can be applied; (3) Construction of the Siamese neural network: This network has two sub-networks with the same structure and shared weights, which are trained through a supervised metric learning, and then the features extracted by that network are reused for small sample learning. The specific steps are as follows:

[0008] Step S1: Use drones to collect refined data of typical marine disaster-bearing bodies, and combine satellite remote sensing images to establish its image data set, including target images and annotation information, etc.

[0009] Step S2: Augment the sample library by methods such as random angle rotation, random cropping, contrast adjustment, and gray level equalization.

[0010] Step S3: Introduce an attention mechanism, design a backbone feature extraction network based on a convolutional neural network to extract image features. Through multiple combinations of SKNet and convolutional neural networks, image features are extracted to obtain a feature map. Among them, convolutional neural network algorithms such as ResNet101, VGG16, ResNet, and IncRes V2 are used, and the ResNet101 network is adopted in the present invention.

[0011] Furthermore, the implementation steps of the backbone feature extraction network with the introduced attention mechanism are as follows:

[0012] (1) Construct a feature extraction network with an improved three-channel SKNet attention mechanism to extract features from the input image and generate a preliminary feature map;

[0013] (2) Take the feature map generated in (1) as the input image, perform convolutional processing with the ResNet101 network to further extract features, and finally output the result through a connection layer.

[0014] Furthermore, the feature extraction network of the improved three-channel SKNet attention mechanism is specifically implemented as follows:

[0015] 1) Perform convolution operations on the input data using convolution kernels of 3*3, 5*5, and 7*7 respectively to obtain outputs U1, U2, and U3;

[0016] 2) Use element-wise summation to fuse the results of the three branches: U = U1 + U2 + U3. U is a feature map of size C*H*W (where C represents the number of channels, H represents the height, and W represents the width) that integrates information from multiple receptive fields. Then, by taking the average in the H and W dimensions, a vector of size C*1*1 is obtained, representing the importance of each channel.

[0017] Let F gp represent the global average pooling operation, and the channel-wise statistical information is represented by s (s ∈ R C ). s c represents the c-th element of s, and its calculation formula is as shown in Formula 1:

[0018]

[0019] 3) Apply a fully connected layer to the C*1*1 vector for a linear transformation to obtain an information z of size Z*1*1 (as shown in Equation (2)). Then, use three linear transformations respectively to restore from the Z dimension to a C-dimensional vector and extract the information of each channel dimension.

[0020] z = F fc (s) = δ(B(W s )) (2)

[0021] where: z ∈ R d×1 ; δ is the ReLU function; B is batch normalization, W ∈ R d×C , where d = max(C / r, L), r represents the reduction ratio, and L is the minimum value of d.

[0022] 4) Use softmax for normalization to obtain the corresponding scores representing the importance of each channel. Then, multiply them by the corresponding U1, U2, and U3 respectively to obtain A1, A2, and A3. Then, add the three modules together for fusion to obtain Y. Y has undergone information refinement compared to U and integrates information from multiple receptive fields.

[0023] Let a, b, c be the three weight matrices of Select, A, B ∈ R C×d , A i represents the i-th row of A, a i is the i-th element of a, Bi and b i and A i Similarly, and a i + b i + c i = 1, and the final feature map Y is as shown in Eqs. (3) and (4) (where e is a natural number).

[0024] Yi = a i × A1 + b i × A2 + c i × A3 (3)

[0025]

[0026] Step S4: Based on the backbone feature extraction network, construct a siamese neural network structure. This network has two inputs, and uses a neural network to map the inputs to a new space, forming the representations of the inputs in the new space.

[0027] Furthermore, the specific implementation steps of the siamese neural network are as follows:

[0028] (1) Based on the backbone feature network constructed in step S3, construct two networks with the same structure and shared weights;

[0029] (2) After the two inputs pass through the backbone feature extraction network, a multi-dimensional feature is obtained, which is flattened into one dimension to obtain the one-dimensional vectors of the two inputs;

[0030] (3) During training, different paired samples are constructed in a combined manner, input into the network for training, and the loss is calculated through a distance cross-entropy at the top layer. According to the distance between the sample pairs, it is judged whether they belong to the same class, and the corresponding probability distribution is generated; in the prediction stage, the siamese network processes each sample pair between the test sample and the support set, and the final prediction result is the class with the highest probability on the support set. This method restricts the structure of the input and automatically discovers features that can be generalized from new samples. It is trained through a supervised metric learning based on the siamese network, and then the features extracted by that network are reused for single / few-shot learning.

[0031] Furthermore, the loss calculation principle is as Figure 2 shown:

[0032] 1) There are two identical sub-neural networks with shared weights W;

[0033] 2) It belongs to weakly supervised learning, and the samples are sample pairs: ((X1, X2), Y). Among them, X1 and X2 are similar samples, Y = 1, otherwise Y = 0;

[0034] 3) The sub-network accepts two inputs, which are transformed into vectors by the neural network, and then the distance between the two vectors is calculated (select the distance metric according to requirements). The calculation formula is as follows:

[0035]

[0036]

[0037] Among them, N represents the number of samples, that is, the number of sample pairs; Y represents the label, that is, Y = 0 or Y = 1; E ω represents the Euclidean distance, that is, E ω = |X1 - X2|2; m represents the distance threshold of dissimilar samples, that is, the distance between two dissimilar samples is in [0, m]. When it exceeds m, the loss of two dissimilar samples can be regarded as 0.

[0038] 4) Perform two fully connected operations on this distance. The second fully connected operation is to a single neuron, and the result of this neuron is taken as sigmoid to make its value between 0 and 1, representing the similarity degree of the two input pictures.

[0039] Step S5: Use the sample library to train the network parameters to obtain the ocean disaster-bearing body recognition model.

[0040] Step S6: Input the data to be recognized and the known category data in pairs using the model, calculate the similarity, and thus obtain the classification result with the highest similarity.

[0041] The present invention takes a series of related scientific issues such as the vulnerability assessment, risk assessment, and disaster response of ocean disaster-bearing bodies as the research main line, aiming to establish an analysis and evaluation model combining traditional physical models and artificial intelligence models. In the physical model for calculating weights, the fuzzy theory and the analytic hierarchy process are combined to complete the process of determining the index weights from qualitative to quantitative according to expert opinions. In the deep learning evaluation model, an ocean disaster-bearing body disaster loss sample library is established, and the influence weights of each disaster-causing factor on the disaster loss situation are automatically analyzed through deep learning methods, and the damaged degree of the disaster-bearing body and the risk level of the disaster are automatically judged, providing a universal vulnerability analysis and disaster loss assessment framework, reducing the limitations of the vulnerability knowledge of different types of disaster-bearing bodies, and improving the scientificity and reliability of the evaluation results. Finally, the accuracy of the two models is verified through comparative experiments, and a more reliable ocean disaster-bearing body disaster loss assessment method combining the two models is established to achieve more accurate, scientific, and intelligent ocean disaster emergency management. This method has important significance for improving the emergency management discipline in theory; in practice, the vulnerability research can provide a scientific basis for the formulation of emergency measures, that is, it is beneficial to guiding the practice of emergency management, minimizing the harm of disaster events to the greatest extent, and improving the effectiveness and scientificity of emergency management.

[0042] Compared with the prior art, the present invention has the following beneficial technical effects:

[0043] 1) By introducing the SKNet network model, the present invention improves the network structure of the original convolutional neural network. Compared with the original ResNet101 network, the improved network enhances the ability of the network to extract image features for complex scenes and is more suitable for the extraction and recognition of complex marine targets.

[0044] 2) The present invention uses convolutional kernels of multiple sizes in the network, which has scale adaptability and can better adapt to the detection of multi-scale marine targets, enabling accurate detection of multiple targets with different scales (a large scale span) by one model.

[0045] 3) By using a dual-path Siamese neural network structure, the present invention enhances the network's learning ability for small samples. Compared with a single-path feature extraction network, under small sample conditions, the present invention has higher recognition accuracy and stability and can be used for rapid modeling and recognition of typical marine disaster-bearing bodies. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is the network structure diagram of the present invention.

[0047] Figure 2 is the backbone feature extraction network structure diagram with an attention mechanism introduced.

[0048] Figure 3 is the dual-path Siamese neural network structure diagram.

[0049] Figure 4 is the schematic diagram of the loss calculation principle based on cross-entropy of the present invention.

[0050] Figure 5 is the test effect diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] The following further illustrates the present invention with reference to the accompanying drawings through specific embodiments.

[0052] A method for identifying marine disaster-bearing bodies based on a Siamese neural network under small sample conditions includes the following steps:

[0053] S1. Collect high-resolution satellite and UAV remote sensing image data, and establish an image sample library for typical marine disaster-bearing bodies (such as houses, shore dikes, oyster rafts, ships, etc.).

[0054] S2. Use satellite remote sensing images and UAV high-definition images to capture screenshots to obtain marine target samples, with each image having a resolution of 600*600 pixels. Expand the sample library by rotating at any angle, randomly cropping, adding noise, etc. A total of 160 training sample images are established, including 40 images of various targets (houses, shore dikes, oyster rafts, ships), and 80 test sample images. The data distribution is shown in Table 1.

[0055] Table 1 Statistical data of different types of marine disaster-bearing bodies

[0056]

[0057] S3. Use the LabelImg tool to annotate the samples and establish data labels corresponding to the sample types. The content of the label file includes: sample path, labeled target coordinate range, and target category.

[0058] S4. Feature extraction of the first path of images in the Siamese network:

[0059] (1) For the image data input in the first path of the sample pair, first normalize it to a size of 256*256, and then preprocess the image using the three-channel SKNet attention module.

[0060] a. Perform convolution operations on the input data using convolution kernels of 3*3, 5*5, and 7*7 respectively to obtain three feature maps;

[0061] b. Use element-wise summation to fuse the three feature maps to obtain a feature map of size C*H*W (C represents the number of channels, H represents the height, and W represents the width); then, by taking the average in the H and W dimension directions, obtain a vector of size C*1*1, which represents the importance degree of each channel. As shown in Equation 1.

[0062] c. Perform a linear transformation on the C*1*1 vector using a fully connected layer to obtain an information z of size Z*1*1 (as shown in Equation 2), and then use three linear transformations respectively to restore from the Z dimension to a C-dimensional vector and extract the information of each channel dimension.

[0063] d. Use softmax for normalization processing to obtain the corresponding scores representing the importance degree of each channel, and then multiply each of the three feature maps obtained in a by the scores respectively to obtain 3 new modules; then add the 3 modules together for fusion to obtain a feature map that integrates information from multiple receptive fields, which is used as the output after the three-channel SKNet preprocesses the image.

[0064] (2) Use the ResNet101 network to further extract features from the feature map with attention information output in (1). The calculation process of the ResNet101 network is as follows:

[0065] a. First, it passes through a convolutional layer of 7×7×64.

[0066] b. Then it passes through 3 + 4 + 23 + 3 = 33 residual blocks; each residual block contains 3 convolutional layers, namely: 1×1, 3×3, 1×1. Among them, the two 1×1 convolutional layers are respectively used to reduce and increase the feature dimension. The main purpose is to reduce the number of parameters, thereby reducing the computational amount, and after dimensionality reduction, data training and feature extraction can be carried out more effectively and intuitively. Therefore, a total of 33×3 = 99 convolutional operations are performed in this part.

[0067] Among them, the calculation principle of the residual block is as follows:

[0068] The original network input x can be fitted to output F(x), and the expected output is H(x). Now we let H(x) = F(x) + x, then our network only needs to learn to output a residual F(x) = H(x) - x. The mapping learned by the two fully connected layers is H(x), that is to say, these two layers can gradually fit H(x). Assuming that H(x) has the same dimension as x, then fitting H(x) is equivalent to fitting the residual function H(x) - x. Let the residual function F(x) = H(x) - x, then the original function becomes F(x) + x. So directly add a cross-layer connection on the basis of the original network. Here, the cross-layer connection is also very simple, which is to pass the identity mapping of x. Instead of directly letting F(x) learn the potential mapping, it is better to learn the residual H(x) - x, that is, F(x): = H(x) - x. In this way, the original forward path becomes F(x) + x, and use F(x) + x to fit H(x).

[0069] c. The last fully connected layer is used for classification.

[0070] A total of 1 + 99 + 1 = 101 layers of convolution and fully connected calculations are performed (as shown in Table 1), and finally the feature vector v1 of the first input image is obtained.

[0071] Table 1 ResNet101 network structure

[0072]

[0073] S5. Feature extraction of the second image of the siamese network: For the data of the second input in the sample pair, use the same steps as in S4 for feature extraction processing to obtain the feature vector v2 of the second input image. Among them, the second network has the same structure as the first network and shares weights.

[0074] S6. Loss calculation: Flatten the two obtained multi-dimensional vectors into two one-dimensional vectors, and calculate the Euclidean distance E between the two vectors wAs shown in Equation (7), determine whether they belong to the same class based on the distance of the sample pairs, and generate the corresponding probability distribution.

[0075]

[0076] After a batch of samples are trained, calculate the overall loss according to Formulas (5) and (6).

[0077] S7. Randomly test the recognition model with the samples not participating in the training.

[0078] It can be preliminarily seen from the training effect that under the condition of small samples, the overall stability of this algorithm is good. Due to the small number of sample data, if only traditional convolutional neural networks are used for training, such as Faster R-CNN, YOLOv5, etc., the model is difficult to converge, and the recognition accuracy of the test samples is relatively low. The overall test results are shown in Tables 2 and 3.

[0079] Table 2 Comparative Test Results

[0080]

[0081] Table 3 Classification Test Results of the Algorithm of the Present Invention

[0082]

[0083] It can be seen from the test results that under the same training and test conditions, the recognition accuracy of the improved algorithm of the present invention for small sample targets has been significantly improved.

[0084] The above are only the preferred embodiments of the present invention, and do not limit the implementation manner and protection scope of the present invention. For those skilled in the art, it should be able to realize that all the equivalent replacements and obvious changes made by using the content of the specification of the present invention should be included in the protection scope of the present invention.

Claims

1. An ocean disaster-bearing body recognition method based on a siamese neural network under small sample conditions, Its features include the following steps: Step S1: Use drones to conduct refined data collection on typical marine disaster-bearing bodies, and combine satellite remote sensing images to establish its image dataset, including target images and annotation information; Step S2: Perform data augmentation on the sample library through methods such as random angle rotation, random cropping, contrast adjustment, and gray level equalization; Step S3: Introduce an attention mechanism, design a backbone feature extraction network based on a convolutional neural network to extract image features; through multiple combinations of SKNet and a convolutional neural network, perform image feature extraction to obtain a feature map, and the convolutional neural network algorithm is ResNet101; Step S4: Based on the backbone feature extraction network, construct a siamese neural network structure. This network has two inputs, and uses a neural network to map the inputs to a new space to form representations of the inputs in the new space; Step S5: Use the sample library to train the network parameters to obtain a marine disaster-bearing body recognition model; Step S6: Input the data to be recognized and the known category data in pairs into the model, calculate the similarity, and thus obtain the classification result with the highest similarity; For the backbone feature extraction network with an attention mechanism described in Step S3, the steps are as follows: (1) Construct an improved three-channel SKNet attention mechanism feature extraction network to extract features from the input image and generate a preliminary feature map. The steps are as follows: 1) Perform convolution operations on the input data using convolution kernels of 3*3, 5*5, and 7*7 respectively to obtain the output U 1、 U 2、 U 3; 2) Use element-wise summation to fuse the results of the three branches: U = U 1 +U 2 +U 3, U is a feature map of size C*H*W that fuses information from multiple receptive fields, where C represents the number of channels, H represents the height, and W represents the width; then, by taking the average in the H and W dimensions, a vector of size C*1*1 is obtained, representing the importance of each channel; Let F gp denote the global average pooling operation, and the channel-wise statistical information is represented by s ( s ∈R C ), and s c denotes s the c-th element of, and the calculation formula is as shown in Equation (1): （1） 3) Apply a linear transformation to the vector of C*1*1 using a fully connected layer to obtain information of Z*1*1 z , as shown in Equation (2); then use three linear transformations respectively to restore from the Z dimension to the C-dimensional vector and extract the information of each channel dimension; （2） Wherein: z ∈R d×1 ; δ is the ReLU function; B is batch normalization, W ∈R d×C , where d =max( C / r, L ), r represents the reduction ratio, L is d the minimum value; 4) Use softmax for normalization to obtain the corresponding scores representing the importance of each channel, and then multiply them by the corresponding U 1, U 2, U 3 to obtain A 1, A 2, A 3; then add the three modules together for fusion to obtain Y , Y relative to U has undergone information refinement and integrated information from multiple receptive fields; Let a, b, c be the three weight matrices of Select, A, B ∈R C×d , A i denotes A the i th row of, a i is a the i th element of, B i 、 b i and A i ，a i Similarly, and a i +b i +c i = 1, the final feature mapping Y is as shown in Eqs. (3) and (4); where e is a natural number; （3） （4） (2) Use the feature map generated in (1) as the input image, perform convolutional processing with the ResNet101 network to further extract features, and finally output the result through a connection layer; The steps of the siamese neural network described in Step S4 are as follows: (1) Based on the backbone feature network constructed in Step S3, construct two networks with the same structure and shared weights; (2) After the two inputs pass through the backbone feature extraction network, a multi-dimensional feature is obtained, which is flattened into a one-dimensional vector to obtain the one-dimensional vectors of the two inputs; (3) During training, different paired samples are constructed in a combined manner and input into the network for training. At the top layer, the loss is calculated through a distance cross-entropy. According to the distance between the sample pairs, it is judged whether they belong to the same class, and the corresponding probability distribution is generated; in the prediction stage, the siamese network processes each sample pair between the test sample and the support set, and the final prediction result is the class with the highest probability on the support set; this method restricts the input structure and automatically discovers features that can be generalized from new samples, and is trained through a supervised metric learning based on the siamese network, and then the features extracted by the network are reused for single / small sample learning; The steps of the loss calculation are as follows: 1) There are two identical sub-neural networks with shared weights WT; 2) Belongs to weakly supervised learning, and the samples are sample pairs: (( X 1, X 2), V ); among them, X 1, X 2 are similar samples, V represents the label, V = 1, otherwise V = 0; 3) The sub-network accepts two inputs, and is transformed into vectors by the neural network, and then the distance between the two vectors is calculated. The calculation formula is as follows: （5） （6） Among them, N represents the number of samples, that is, the number of sample pairs; V represents the label, that is V = 0 or V = 1; E WT represents the Euclidean distance, that is E WT = | X 1 - X 2|2; m represents the distance threshold of dissimilar samples, that is, the distance between two dissimilar samples is in [0, m]. When it exceeds m, the loss of the two dissimilar samples can be regarded as 0; 4) Perform two fully connected operations on this distance, and the second fully connected operation is to a neuron. Take the sigmoid of the result of this neuron to make its value between 0 and 1, representing the similarity degree of the two input pictures.

Images