Zero sample learning-based sonar image recognition method and system

Abstract

The invention provides a zero sample learning-based sonar image recognition method and system, and the method comprises the steps: collecting a sonar image of a target, carrying out the data enhancement processing, constructing a sonar image zero sample recognition data set, and dividing the data set into a training set and a test set; extracting quantitative features of the sonar image in the training set; constructing an attribute matrix on the training set and the test set; obtaining a mapping relation model of the quantitative characteristics and the attribute matrix on the training set through an encoder; extracting quantitative characteristics of the test set through the mapping relation model, calculating similarity, obtaining a prediction result, and evaluating the accuracy. According to the method, unknown classes can be identified, and the problem of class label missing is effectively solved; through direct extraction of the pre-trained deep convolutional neural network, deep and essential features of the sonar image can be extracted, the method can be suitable for various different targets, and generalization and reliability are high. The problem of domain offset of the same attribute of the training set and the test set in different categories is effectively solved by adopting an auto-encoder structure.

Description

Technical Field

[0001] The present invention belongs to the field of sonar image recognition, and in particular, relates to a sonar image recognition method and system based on zero-sample learning. Background Art

[0002] Image sonar is a widely used sensing device in the field of underwater detection and target search. Commonly used image sonars include side scan sonar, forward-looking sonar, synthetic aperture sonar, etc. Sonar image recognition plays an important role in underwater exploration and detection. In the study of sonar image recognition, current research work can be divided into traditional learning methods and small sample learning methods according to the sample situation of the data set required for classification training.

[0003] Traditional learning studies classification and recognition problems in closed-set scenarios. The categories of the training set and the test set are consistent, and the model is trained through a large amount of labeled data. Traditional learning methods can be divided into two categories based on image processing and deep learning methods according to whether manual feature extraction is required. The image processing method requires special feature extraction and then uses traditional classifiers for feature recognition; the deep learning method directly uses the automatic learning ability of image features of deep neural networks to achieve sonar image classification through network design and data set training. Among them, the image processing method needs to be carried out according to the steps of image preprocessing, manual feature extraction, classifier classification, etc., and end-to-end automatic image recognition cannot be achieved. At the same time, the number of images processed is small, and the algorithm recognition rate and robustness are not high. The recognition effect of deep learning methods is significantly higher than that of image processing methods, but it also faces great challenges. Sonar images are difficult to reach the level of optical images in terms of the quantity and quality of data sets, which makes deep learning technology that relies on big data subject to great restrictions in sonar image recognition.

[0004] In order to solve the problem of small sample sizes in sonar image datasets, some studies have adopted small sample learning methods. Small sample learning aims to enable the model to effectively learn and predict when faced with only a small number of samples. The small sample learning method uses a large-scale dataset with irrelevant categories but sufficient quantity to build a model, and then transfers it to the small sample learning task of a new category. The small sample learning method has alleviated the problem of deep learning's dependence on the size of the dataset to a certain extent, but the processing idea has not broken through the conventional deep learning method flow. The existing sonar image recognition method uses a direct mapping learning method between data and labels, which cannot solve the problem of missing category labels and does not have the ability to recognize new and unknown categories that did not appear in the training process.

[0005] Patent document "Zero-sample image recognition method based on attribute feature vector and reversible generative model" (CN111612047A) discloses a new reversible generative model to solve the zero-sample image recognition problem, which directly maps the real data distribution to the multivariate Gaussian distribution with the help of multiple affine coupling transformations. The essence is to transform the zero-sample recognition problem into traditional supervised learning, and use the SVM model to classify and recognize the generated pseudo images. A large number of pseudo images need to be generated, so the amount of calculation is large, and it does not have a fully reversible bidirectional mapping capability, and the stability of the generated data is unreliable.

[0006] Therefore, to address the problem that existing sonar image recognition methods cannot identify new categories of targets that have not appeared in the training process, a sonar image recognition method based on zero-shot learning is proposed. Summary of the invention

[0007] In view of the defects in the prior art, the object of the present invention is to provide a sonar image recognition method and system based on zero-sample learning.

[0008] A sonar image recognition method based on zero-sample learning provided by the present invention includes:

[0009] Step S1: collect sonar images of the target and perform data enhancement processing, construct a sonar image zero-sample recognition dataset and divide it into a training set and a test set;

[0010] Step S2: extracting quantitative features of sonar images in the training set, and constructing attribute matrices on the training set and the test set;

[0011] Step S3: Obtaining a mapping relationship model between quantitative features and attribute matrix through an encoder on the training set;

[0012] Step S4: extract the quantitative features of the test set through the mapping relationship model to calculate the similarity, obtain the prediction results and evaluate the accuracy.

[0013] Preferably, the step S1 comprises:

[0014] Step S1.1: Collect public sonar images covering various targets, perform image enhancement through image filtering and histogram equalization, and unify the pixel size of each image to 224*224*3.

[0015] Step S1.2: Data expansion is performed by random rotation transformation, flipping transformation, scaling transformation, translation transformation and random addition of noise.

[0016] Step S1.3: Divide the data set, taking P of the sonar images of L-type targets as the training set and the remaining Q as the test set, L = P + Q.

[0017] The construction attribute matrix includes:

[0018] Step S2.1.1: Based on the prior knowledge of the sonar image target, determine the attribute theme of the sonar image from the appearance, shape and material of the target.

[0019] Step S2.1.2: Construct an attribute vector based on the determined attribute theme, set all attributes to binary values, and the attribute vector is represented as

[0020] in, Indicates the descriptive feature value of the k-th attribute of the i-th category. When it is 0, it means there is no such attribute, and when it is 1, it means there is such attribute;

[0021] M represents the total number of attributes.

[0022] Step S2.1.3: Construct attribute vectors for the training set and the test set respectively, and merge the attribute vectors to form the attribute matrix S = [S p ,S q ];

[0023] Among them, S p Represents the training set attribute matrix;

[0024] S q Represents the test set attribute matrix.

[0025] Preferably, the step of extracting quantitative features of the sonar image comprises:

[0026] Step S2.2.1: Load the pre-trained deep learning network trained on the large-scale image dataset ImageNet, which consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer.

[0027] Step S2.2.2: Remove the fully connected layer and output layer of the pre-trained deep learning network backend, and retain the input layer and convolutional pooling layer structure as the sonar image feature extractor.

[0028] Step S2.2.3: Input the image of the training set into the feature extractor, obtain the convolution calculation plus the offset through forward propagation calculation, and the activation function f(x) is nonlinearly mapped to obtain the feature X of the i-th layer i ,

[0029] The feature map size is 7*7*512, and is transposed and expanded to 1*25088 vector;

[0030] Among them, Z i Represents the convolution kernel weight of the i-th layer;

[0031] b i Indicates the offset of the i-th layer;

[0032] Represents the convolution kernel and i-1 layer feature X i-1 Convolution calculation.

[0033] The step S4 comprises:

[0034] Step S4.1: Input the image of the test set into the image feature extractor to obtain the quantitative features x of the test set i .

[0035] Step S4.2: Quantize the feature x i Obtain the corresponding attribute vector Wx through the encoder i .

[0036] Step S4.3: Normalize the cross-correlation attribute vector Wx using zero mean i And the test set attribute matrix S q Perform similarity matching calculation and select the most similar label as the prediction result φ(x i )=argminD(Wx i ,S q );

[0037] Where D() represents the similarity value calculated using zero-mean normalized cross-correlation;

[0038] W represents weight;

[0039] S q Represents the test set attribute matrix.

[0040] Step S4.4: Count the prediction results of each test set image, and evaluate the accuracy by the recognition accuracy Acc, Acc = (TP + TN) / (TP + TN + FP + FN);

[0041] Among them, TP represents the number of positive samples correctly identified;

[0042] FP represents the number of false positive negative samples;

[0043] TN represents the number of negative samples correctly identified;

[0044] FN represents the number of positive samples that are missed.

[0045] Preferably, step S3 comprises:

[0046] Step S3.1: Set up an autoencoder structure, where the encoder module in the autoencoder compresses the input features into a latent space representation to achieve S = h(X), and the decoder module in the autoencoder reconstructs the latent space representation to obtain an output R = g(S), and the complete autoencoder structure achieves R = g(h(X));

[0047] Wherein, h() represents the encoding function;

[0048] g() represents the decoding function;

[0049] S represents the latent space representation;

[0050] R represents the decoded output;

[0051] X represents the features of the input.

[0052] Step S3.2: Set the autoencoder bidirectional mapping learning training objective function to stWX = S;

[0053] Wherein, the subscript F represents the Euclidean norm;

[0054] W represents weight;

[0055] S represents the attribute matrix.

[0056] Step S3.3: Relax the strict constraint condition of WX=S to Use the training set to solve the mapping matrix W and obtain the mapping relationship model;

[0057] in, Represent the losses of decoder and encoder respectively;

[0058] λ represents a hyperparameter that adjusts the weight of the two losses.

[0059] Preferably, the targets include eight types: a solid sphere, a round cage, a solid cube, a cylinder, an airplane model, a metal barrel, a square cage and a tire.

[0060] The training set takes 5 targets, and the test set takes the remaining 3 targets.

[0061] In the step S2, vector features are extracted through a deep learning network including a VGG16 deep learning model, VGG, ResNet and GooleNet;

[0062] Independent labels are represented as mutually related label semantic vectors, and the attribute matrix is ​​the intermediate semantic layer shared by the space of vector features and labels.

[0063] The properties include rigid, grid, square, regular, round, cylindrical, straight, solid, and hollow.

[0064] The encoder is a bidirectionally mapped semantic autoencoder structure with a hyperparameter λ=0.5.

[0065] A sonar image recognition system based on zero-sample learning provided by the present invention includes:

[0066] Module M1: Collect sonar images of the target and perform data enhancement processing, build a sonar image zero-shot recognition dataset and divide it into a training set and a test set;

[0067] Module M2: Extract the quantitative features of sonar images in the training set and construct the attribute matrix on the training set and the test set;

[0068] Module M3: Obtain the mapping relationship model between quantitative features and attribute matrix through the encoder on the training set;

[0069] Module M4: Calculate the similarity by extracting the quantitative features of the test set through the mapping relationship model, obtain the prediction results and evaluate the accuracy.

[0070] Preferably, the module M1 comprises:

[0071] Module M1.1: Collect public sonar images covering a variety of targets, enhance the images through image filtering and histogram equalization, and unify the pixel size of each image to 224*224*3.

[0072] Module M1.2: Data augmentation through random rotation, flipping, scaling, translation and random noise addition.

[0073] Module M1.3: Divide the data set, taking P of the sonar images of L-type targets as the training set and the remaining Q as the test set, L = P + Q.

[0074] The construction attribute matrix includes:

[0075] Module M2.2.1: Based on prior knowledge of sonar image targets, determine the attribute themes of sonar images from the appearance, shape and material of the target.

[0076] Module M2.2.2: Construct an attribute vector based on the determined attribute theme, set all attributes to binary values, and the attribute vector is represented as

[0077] in, Indicates the descriptive feature value of the k-th attribute of the i-th category. When it is 0, it means there is no such attribute, and when it is 1, it means there is such attribute;

[0078] M represents the total number of attributes.

[0079] Module M2.2.3: Construct attribute vectors for the training set and the test set respectively, and merge the attribute vectors to form the attribute matrix S = [S p ,S q ];

[0080] Among them, S p Represents the training set attribute matrix;

[0081] S q Represents the test set attribute matrix.

[0082] Preferably, the step of extracting quantitative features of the sonar image comprises:

[0083] Module M2.1.1: Load the pre-trained deep learning network trained on the large-scale image dataset ImageNet, which consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer.

[0084] Module M2.1.2: Remove the fully connected layer and output layer of the pre-trained deep learning network backend, and retain the input layer and convolutional pooling layer structure as the sonar image feature extractor.

[0085] Module M2.1.3: Input the training set image into the feature extractor, obtain the convolution calculation plus the offset through forward propagation calculation, and the activation function f(x) is nonlinearly mapped to obtain the feature X of the i-th layer i ,

[0086] The feature map size is 7*7*512, and is transposed and expanded to 1*25088 vector;

[0087] Among them, Z i Represents the convolution kernel weight of the i-th layer;

[0088] b i Indicates the offset of the i-th layer;

[0089] Represents the convolution kernel and i-1 layer feature X i-1 Convolution calculation.

[0090] The module M4 comprises:

[0091] Module M4.1: Input the images of the test set into the image feature extractor to obtain the quantitative features x of the test set i .

[0092] Module M4.2: Quantify the features x i Obtain the corresponding attribute vector Wx through the encoder i .

[0093] Module M4.3: Normalizing the cross-correlation attribute vector Wx using zero mean i And the test set attribute matrix S q Perform similarity matching calculation and select the most similar label as the prediction result φ(x i )=argminD(Wxi,S q );

[0094] Where D() represents the similarity value calculated using zero-mean normalized cross-correlation;

[0095] W represents weight;

[0096] S q Represents the test set attribute matrix.

[0097] Module M4.4: Count the prediction results of each test set image and evaluate the accuracy by the recognition accuracy Acc, Acc = (TP + TN) / (TP + TN + FP + FN);

[0098] Among them, TP represents the number of positive samples correctly identified;

[0099] FP represents the number of false positive negative samples;

[0100] TN represents the number of negative samples correctly identified;

[0101] FN represents the number of positive samples that are missed.

[0102] Preferably, the module M3 comprises:

[0103] Module M3.1: Set up the autoencoder structure. The encoder module in the autoencoder compresses the input features into a latent space representation to achieve S = h(X). The decoder module in the autoencoder reconstructs the latent space representation to obtain the output R = g(S). The complete autoencoder structure achieves R = g(h(X)).

[0104] Wherein, h() represents the encoding function;

[0105] g() represents the decoding function;

[0106] S represents the latent space representation;

[0107] R represents the decoded output;

[0108] X represents the features of the input.

[0109] Module M3.2: Set the autoencoder bidirectional mapping learning training objective function to stWX = S;

[0110] Wherein, the subscript F represents the Euclidean norm;

[0111] W represents weight;

[0112] S represents the attribute matrix.

[0113] Module M3.3: Relax the strict constraint of WX=S to Use the training set to solve the mapping matrix W and obtain the mapping relationship model;

[0114] in, Represent the losses of decoder and encoder respectively;

[0115] λ represents a hyperparameter that adjusts the weight of the two losses.

[0116] Preferably, the targets include eight types: a solid sphere, a round cage, a solid cube, a cylinder, an airplane model, a metal barrel, a square cage and a tire.

[0117] The training set takes 5 targets, and the test set takes the remaining 3 targets.

[0118] The module M2 extracts vector features through a deep learning network including a VGG16 deep learning model, VGG, ResNet and GooleNet;

[0119] Independent labels are represented as mutually related label semantic vectors, and the attribute matrix is ​​the intermediate semantic layer shared by the space of vector features and labels.

[0120] The properties include rigid, grid, square, regular, round, cylindrical, straight, solid, and hollow.

[0121] The encoder is a bidirectionally mapped semantic autoencoder structure with a hyperparameter λ=0.5.

[0122] Compared with the prior art, the present invention has the following beneficial effects:

[0123] 1. The present invention can realize the recognition of unknown classes and effectively solve the problem of missing category labels by training known class data, extracting corresponding features, and then embedding auxiliary knowledge.

[0124] 2. The present invention utilizes the powerful feature extraction capability of the deep learning model in sonar image feature extraction, and directly performs image quantitative feature extraction through a pre-trained deep convolutional neural network, without the need for cumbersome image feature processing engineering, and can extract the deep and essential features of the sonar image, and can be applied to a variety of different targets, with high generalization and reliability.

[0125] 3. In the mapping process between the feature space and the semantic embedding space of zero-shot learning of sonar images, the present invention adopts an autoencoder structure to integrate the advantages of forward mapping and reverse mapping, so that the semantic embedding space shared by known classes and unknown classes retains the original feature information as much as possible, effectively solving the domain shift problem of the same attribute in the training set and the test set in different categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0127] Figure 1A flowchart for implementing a sonar image recognition method based on zero-shot learning;

[0128] Figure 2 Schematic diagram of the number of samples in each category of the data set;

[0129] Figure 3 Example diagram for dataset augmentation;

[0130] Figure 4 Schematic diagram of the division of training set and test set;

[0131] Figure 5 This is a schematic diagram of the structure of the adopted VGG16 deep learning model;

[0132] Figure 6 It is a schematic diagram of the mapping relationship between different spatial data;

[0133] Figure 7 Schematic diagram of the attribute matrix constructed for the dataset;

[0134] Figure 8 Schematic diagram of the autoencoder structure. DETAILED DESCRIPTION

[0135] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0136] According to the sonar image recognition method based on zero-sample learning provided by the present invention, by training known class data, extracting corresponding features, and then embedding auxiliary knowledge, the intelligent recognition of unknown classes is realized by using prior knowledge, which is an effective means to solve the problem of missing class labels. Figure 1 For example, the specific steps include:

[0137] Step S1: Collect sonar images covering multiple categories of targets and perform data enhancement processing, build a sonar image zero-shot recognition dataset by category, use known class data as a training set, and unknown class data as a test set;

[0138] Specifically, the step S1 includes:

[0139] Step S1.1: Collect public sonar images covering various targets, and enhance the images by image filtering and histogram equalization, unifying the pixel size of each image to 224*224*3;

[0140] In more preferred embodiments, the types L of the targets are selected to be 8, Figure 2For example, they include solid spheres, round cages, solid cubes, cylinders, airplane models, metal barrels, square cages, and tires.

[0141] Step S1.2: Figure 3 Taking the preprocessed data set as an example, data expansion is performed by random rotation transformation, flipping transformation, scaling transformation, translation transformation and random noise addition.

[0142] Step S1.3: Divide the data set after the expansion in step S1.2, and use P types of sonar images of L types of targets as training sets, and the remaining Q types as test sets, L = P + Q.

[0143] In more preferred embodiments, the data set is divided into Figure 4 For example, L=8, P=5, that is, 5 types of sonar images of 8 types of targets are used as training sets, and the remaining 3 types are used as test sets.

[0144] Step S2: extracting sonar image features based on a pre-trained deep learning network, inputting the training set sonar images into the pre-trained deep learning network to calculate and obtain image quantitative features;

[0145] Specifically, the step S2 includes:

[0146] Step S2.1: Load the deep learning network trained on the large-scale image dataset ImageNet, such as VGG, ResNet, GooleNet, etc. The pre-trained network consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer;

[0147] In more preferred embodiments, the deep learning network is a VGG16 deep learning model. Figure 5 For example, the network consists of convolution, pooling, and fully connected modules.

[0148] Step S2.2: Remove the network backend fully connected layer and output layer structure, and retain the input layer and convolutional pooling layer structure as the sonar image feature extractor;

[0149] Step S2.3: Input the training set image into the feature extractor in step S2.2. The convolution network obtains the convolution calculation output activation value as the image quantization feature through forward propagation calculation. The forward propagation calculation can be expressed as:

[0150]

[0151] Among them, Z i represents the weight of the convolution kernel of the i-th layer, Represents the convolution kernel and i-1 layer feature X i-1 Convolution calculation of ;

[0152] The result is then added to the offset b of the i-th layeri Add, and finally obtain the feature X of the i-th layer through nonlinear mapping of the activation function f(x) i .

[0153] The feature map size is 7*7*512, and its transpose is expanded to 1*25088 vector.

[0154] Step S3: Based on the prior knowledge of sonar target images, an attribute matrix of the semantic embedding space is constructed from the perspectives of target appearance, shape, material, etc., and independent labels are represented as interrelated label semantic vectors. The attribute matrix is ​​the intermediate semantic layer shared by the feature space and the label;

[0155] The relationship between data space, feature space, semantic embedding space and label space is Figure 6 For example.

[0156] Specifically, step S3 includes:

[0157] Step S3.1: Determine the sonar image attribute theme based on the prior knowledge of the sonar target image, and the theme is developed from the perspectives of the sonar image target appearance, shape, material, etc.;

[0158] In more preferred examples, the sonar image attribute themes are defined from the perspectives of sonar image target appearance, shape, material, etc., and the 10 optimal attributes of the data set are determined, including hard, grid, square, regular, circular, cylindrical, straight line, solid, hollow, etc.

[0159] Step S3.2: Construct an attribute vector based on the attribute theme determined in step S3.1, and set all attributes to binary values, that is, 0 / 1 indicates the presence or absence of the attribute. The attribute vector of any category can be expressed as

[0160] in, Represents the descriptive feature value of the kth attribute of the ith category, and M is the total number of attributes.

[0161] In more preferred embodiments, M=10.

[0162] Step S3.3: Construct attribute vectors for the known class training set and the unknown class test set according to step S3.2 respectively, and merge the attribute vectors to form the attribute matrix S = [S p ,S q ];

[0163] Among them, S p is the attribute matrix of the known class training set, S q is the attribute matrix of the unknown class test set.

[0164] by Figure 7For example, in the attribute matrix visualization display, the horizontal axis is the attribute and the vertical axis is the category. The black square indicates that it belongs to the attribute, and the white square indicates that it does not belong to the attribute.

[0165] Step S4: setting a bidirectional mapping semantic autoencoder structure, the encoder module in the autoencoder compresses the input features into a potential representation space, and then uses the decoder module in the autoencoder to reconstruct the potential space representation, and obtains a mapping relationship model between the feature space and the embedding space through learning of a training set of known classes;

[0166] In the process, there is no need to generate unknown pseudo images, use SVM classifier for reclassification, or achieve image-to-attribute mapping through affine coupling transformation. Instead, the autoencoder structure based on bidirectional mapping uses known data to train the model. The model can input unknown images and output corresponding attribute vectors, and then calculate the similarity between the attribute vector and the unknown target attribute matrix to complete the recognition, thus realizing the complete bidirectional mapping capability from image to attribute and enabling the model to learn more general essential information.

[0167] Specifically, step S4 includes:

[0168] Step S4.1: Set up the autoencoder structure to learn the mapping between feature space and embedding space. Figure 8 For example, the encoder module in the autoencoder compresses the input feature X into a latent space representation to achieve S = h(X), and the decoder module in the autoencoder reconstructs the latent space representation to obtain the output R = g(S). Then the complete autoencoder structure achieves R = g(h(X)), so that the feature input feature X and the decoded output R are as close as possible, and the encoded data can be restored to the original data as much as possible;

[0169] h() represents the encoding function, g() represents the decoding function, S represents the latent space representation, and R represents the decoded output.

[0170] Step S4.2: Set the autoencoder bidirectional mapping learning training objective function to stWX = S;

[0171] The subscript F represents the Euclidean norm and W represents the weight.

[0172] Step S4.3: Relax the strict constraint condition of WX=S in step S4.2 to The two terms in the formula correspond to the losses of the decoder and encoder respectively. The hyperparameter λ is used to adjust the proportion of the two losses. Finally, the mapping matrix W is solved using the known class training set.

[0173] In more preferred embodiments, λ=0.5 is set.

[0174] Step S5: Send the test sample features into the mapping relationship model, that is, the mapping matrix W, to obtain the corresponding attribute vector, and then use the zero-mean normalized cross-correlation to calculate the similarity between the attribute vector and the test set category attribute matrix, take the most similar label as the predicted value, and use the unknown class recognition accuracy as the evaluation indicator.

[0175] In the process of unknown target recognition, a large amount of data is not required. Instead, the test sample is sent to the mapping relationship model to obtain the corresponding attribute vector, and then the zero-mean normalized cross-correlation is used to calculate the similarity between the attribute vector and the test set category attribute matrix, and the recognition result is directly given.

[0176] Specifically, step S5 includes:

[0177] Step S5.1: Input the test set sample into the image feature extractor of step S2.2 to obtain the quantitative feature x of the test sample i ;

[0178] Step S5.2: The quantitative feature x obtained in step S5.1 i Send it to the encoder model to get the attribute vector Wx corresponding to the test sample i ;

[0179] Step S5.3: Normalize the cross-correlation attribute vector Wx using zero mean i And the test set category attribute matrix S q Perform similarity matching calculation and select the most similar label as the prediction result φ(x i )=argminD(Wx i ,S q );

[0180] Wherein, D() indicates that the similarity value is calculated using zero-mean normalized cross-correlation.

[0181] Step S5.4: Calculate the recognition result of each test set sample according to steps S5.1-S5.3, that is, the prediction result of S5.3, and use the recognition accuracy Acc as the model result evaluation indicator:

[0182] Acc=(TP+TN) / (TP+TN+FP+FN);

[0183] Among them, TP represents the number of positive samples correctly identified, FP represents the number of false positive negative samples, TN represents the number of negative samples correctly identified, and FN represents the number of false positive positive samples.

[0184] In more preferred examples, the final statistical results show that Figure 4 The average recognition accuracy of the three unknown class test sets is Acc = 81.7%, and the recognition accuracy of the three types of targets is Acc respectively. bucket=71.4%,Acc cage =83.6%,Acc tyre =82.3%, which is able to better identify new categories that have not appeared in the training process.

[0185] The present invention also provides a sonar image recognition system based on zero-sample learning. The sonar image recognition system based on zero-sample learning can be implemented by executing the process steps of the sonar image recognition method based on zero-sample learning, that is, those skilled in the art can understand the sonar image recognition method based on zero-sample learning as a preferred implementation of the sonar image recognition system based on zero-sample learning.

[0186] A sonar image recognition system based on zero-sample learning provided by the present invention includes:

[0187] Module M1: Collect sonar images of the target and perform data enhancement processing, build a sonar image zero-shot recognition dataset and divide it into a training set and a test set;

[0188] Module M2: Extract the quantitative features of sonar images in the training set and construct the attribute matrix on the training set and the test set;

[0189] Module M3: Obtain the mapping relationship model between quantitative features and attribute matrix through the encoder on the training set;

[0190] Module M4: Calculate the similarity by extracting the quantitative features of the test set through the mapping relationship model, obtain the prediction results and evaluate the accuracy.

[0191] In more preferred embodiments, the module M1 includes:

[0192] Module M1.1: Collect public sonar images covering a variety of targets, enhance the images through image filtering and histogram equalization, and unify the pixel size of each image to 224*224*3.

[0193] Module M1.2: Data augmentation through random rotation, flipping, scaling, translation and random noise addition.

[0194] Module M1.3: Divide the data set, taking P of the sonar images of L-type targets as the training set and the remaining Q as the test set, L = P + Q.

[0195] The construction attribute matrix includes:

[0196] Module M2.2.1: Based on prior knowledge of sonar image targets, determine the attribute themes of sonar images from the appearance, shape and material of the target.

[0197] Module M2.2.2: Construct an attribute vector based on the determined attribute theme, set all attributes to binary values, and the attribute vector is represented as

[0198] in, Indicates the descriptive feature value of the k-th attribute of the i-th category. When it is 0, it means there is no such attribute, and when it is 1, it means there is such attribute;

[0199] M represents the total number of attributes.

[0200] Module M2.2.3: Construct attribute vectors for the training set and the test set respectively, and merge the attribute vectors to form the attribute matrix S = [S p ,S q ];

[0201] Among them, S p Represents the training set attribute matrix;

[0202] S q Represents the test set attribute matrix.

[0203] In more preferred embodiments, the extracting of quantitative features of sonar images includes:

[0204] Module M2.1.1: Load the pre-trained deep learning network trained on the large-scale image dataset ImageNet, which consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer.

[0205] Module M2.1.2: Remove the fully connected layer and output layer of the pre-trained deep learning network backend, and retain the input layer and convolutional pooling layer structure as the sonar image feature extractor.

[0206] Module M2.1.3: Input the training set image into the feature extractor, obtain the convolution calculation plus the offset through forward propagation calculation, and the activation function f(x) is nonlinearly mapped to obtain the feature X of the i-th layer i ,

[0207] The feature map size is 7*7*512, and is transposed and expanded to 1*25088 vector;

[0208] Among them, Z i Represents the convolution kernel weight of the i-th layer;

[0209] b i Indicates the offset of the i-th layer;

[0210] Represents the convolution kernel and i-1 layer feature X i-1 Convolution calculation.

[0211] The module M4 comprises:

[0212] Module M4.1: Input the images of the test set into the image feature extractor to obtain the quantitative features x of the test set i .

[0213] Module M4.2: Quantify the features x i Obtain the corresponding attribute vector Wx through the encoder i .

[0214] Module M4.3: Normalizing the cross-correlation attribute vector Wx using zero mean i And the test set attribute matrix S q Perform similarity matching calculation and select the most similar label as the prediction result φ(x i )=argminD(Wx i ,S q );

[0215] Where D() represents the similarity value calculated using zero-mean normalized cross-correlation;

[0216] W represents weight;

[0217] S q Represents the test set attribute matrix.

[0218] Module M4.4: Count the prediction results of each test set image and evaluate the accuracy by the recognition accuracy Acc, Acc = (TP + TN) / (TP + TN + FP + FN);

[0219] Among them, TP represents the number of positive samples correctly identified;

[0220] FP represents the number of false positive negative samples;

[0221] TN represents the number of negative samples correctly identified;

[0222] FN represents the number of positive samples that are missed.

[0223] In more preferred embodiments, the module M3 includes:

[0224] Module M3.1: Set up the autoencoder structure. The encoder module in the autoencoder compresses the input features into a latent space representation to achieve S = h(X). The decoder module in the autoencoder reconstructs the latent space representation to obtain the output R = g(S). The complete autoencoder structure achieves R = g(h(X)).

[0225] Wherein, h() represents the encoding function;

[0226] g() represents the decoding function;

[0227] S represents the latent space representation;

[0228] R represents the decoded output;

[0229] X represents the features of the input.

[0230] Module M3.2: Set the autoencoder bidirectional mapping learning training objective function to stWX = S;

[0231] Wherein, the subscript F represents the Euclidean norm;

[0232] W represents weight;

[0233] S represents the attribute matrix.

[0234] Module M3.3: Relax the strict constraint of WX=S to Use the training set to solve the mapping matrix W and obtain the mapping relationship model;

[0235] in, Represent the losses of decoder and encoder respectively;

[0236] λ represents a hyperparameter that adjusts the weight of the two losses.

[0237] In more preferred embodiments, the targets include eight types: solid sphere, round cage, solid cube, cylinder, airplane model, metal barrel, square cage and tire.

[0238] The training set takes 5 targets, and the test set takes the remaining 3 targets.

[0239] The module M2 extracts vector features through a deep learning network including a VGG16 deep learning model, VGG, ResNet and GooleNet;

[0240] Independent labels are represented as mutually related label semantic vectors, and the attribute matrix is ​​the intermediate semantic layer shared by the space of vector features and labels.

[0241] The properties include rigid, grid, square, regular, round, cylindrical, straight, solid, and hollow.

[0242] The encoder is a bidirectionally mapped semantic autoencoder structure with a hyperparameter λ=0.5.

[0243] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0244] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A sonar image recognition method based on zero-sample learning, characterized in that: include: Step S1: collect sonar images of the target and perform data enhancement processing, construct a sonar image zero-sample recognition dataset and divide it into a training set and a test set; Step S2: extracting quantitative features of sonar images in the training set, and constructing attribute matrices on the training set and the test set; Step S3: Obtaining a mapping relationship model between quantitative features and attribute matrix through an encoder on the training set; Step S4: extract the quantitative features of the test set through the mapping relationship model to calculate the similarity, obtain the prediction results and evaluate the accuracy.

2. The sonar image recognition method based on zero-sample learning according to claim 1, characterized in that: The step S1 comprises: Step S1.1: Collect public sonar images covering various targets, perform image enhancement by image filtering and histogram equalization, and unify the pixel size of each image to 224*224*3; Step S1.2: Data augmentation is performed by random rotation transformation, flip transformation, scaling transformation, translation transformation and random noise addition; Step S1.3: divide the data set, and use P of the sonar images of L types of targets as the training set, and the remaining Q types as the test set, L = P + Q; The construction attribute matrix includes: Step S2.1.1: Determine the attribute themes of the sonar image from the appearance, shape and material of the target based on the prior knowledge of the sonar image target; Step S2.1.2: Construct an attribute vector based on the determined attribute theme, set all attributes to binary values, and the attribute vector is represented as in, Indicates the descriptive feature value of the k-th attribute of the i-th category. When it is 0, it means there is no such attribute, and when it is 1, it means there is such attribute; M represents the total number of attributes; Step S2.1.3: Construct attribute vectors for the training set and the test set respectively, and merge the attribute vectors to form the attribute matrix S = [S p ,S q ]; in, S p Represents the training set attribute matrix; S q Represents the test set attribute matrix.

3. The sonar image recognition method based on zero-sample learning according to claim 1, characterized in that: The step of extracting the quantitative features of the sonar image comprises: Step S2.2.1: Load the pre-trained deep learning network trained on the large-scale image dataset ImageNet, which consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer; Step S2.2.2: Remove the fully connected layer and output layer of the pre-trained deep learning network, and keep the input layer and convolutional pooling layer structure as the sonar image feature extractor; Step S2.2.3: Input the image of the training set into the feature extractor, obtain the convolution calculation plus the offset through forward propagation calculation, and the activation function f(x) is nonlinearly mapped to obtain the feature X of the i-th layer i , The feature map size is 7*7*512, and is transposed and expanded to 1*25088 vector; Among them, Z i Represents the weight of the convolution kernel of the i-th layer; b i Indicates the offset of the i-th layer; Represents the convolution kernel and i-1 layer feature X i-1 Convolution calculation of ; The step S4 comprises: Step S4.1: Input the image of the test set into the image feature extractor to obtain the quantitative features x of the test set i ; Step S4.2: Quantize the feature x i Obtain the corresponding attribute vector Wx through the encoder i ; Step S4.3: Normalize the cross-correlation attribute vector Wx using zero mean i And the test set attribute matrix S q Perform similarity matching calculation and select the most similar label as the prediction result φ(x i )=argminD(Wx i ,S q ); Where D() represents the similarity value calculated using zero-mean normalized cross-correlation; W represents weight; S q represents the test set attribute matrix; Step S4.4: Count the prediction results of each test set image, and evaluate the accuracy by the recognition accuracy Acc, Acc = (TP + TN) / (TP + TN + FP + FN); Among them, TP represents the number of positive samples correctly identified; FP represents the number of false positive negative samples; TN represents the number of negative samples correctly identified; FN represents the number of positive samples that are missed.

4. The sonar image recognition method based on zero-sample learning according to claim 1, characterized in that: The step S3 comprises: Step S3.1: Set up an autoencoder structure, where the encoder module in the autoencoder compresses the input features into a latent space representation to achieve S = h(X), and the decoder module in the autoencoder reconstructs the latent space representation to obtain an output R = g(S), and the complete autoencoder structure achieves R = g(h(X)); Wherein, h() represents the encoding function; g() represents the decoding function; S represents the latent space representation; R represents the decoded output; X represents the input features; Step S3.2: Set the autoencoder bidirectional mapping learning training objective function to Wherein, the subscript F represents the Euclidean norm; W represents weight; S represents the attribute matrix; Step S3.3: Relax the strict constraint condition of WX=S to Use the training set to solve the mapping matrix W and obtain the mapping relationship model; in, Represent the losses of the decoder and encoder respectively; λ represents a hyperparameter that adjusts the weight of the two losses.

5. The sonar image recognition method based on zero-sample learning according to claim 1, characterized in that: The targets include 8 types: solid sphere, round cage, solid cube, cylinder, airplane model, metal barrel, square cage and tire; The training set takes 5 targets, and the test set takes the remaining 3 targets; In the step S2, vector features are extracted through a deep learning network including a VGG16 deep learning model, VGG, ResNet and GooleNet; Independent labels are represented as mutually related label semantic vectors, and the attribute matrix is ​​the intermediate semantic layer shared by the vector feature space and the label; Said properties include hard, grid, square, regular, round, cylindrical, straight, solid, and hollow; The encoder is a bidirectionally mapped semantic autoencoder structure with a hyperparameter λ=0.

5.

6. A sonar image recognition system based on zero-sample learning, characterized in that: include: Module M1: Collect sonar images of the target and perform data enhancement processing, build a sonar image zero-shot recognition dataset and divide it into a training set and a test set; Module M2: Extract the quantitative features of sonar images in the training set and construct the attribute matrix on the training set and the test set; Module M3: Obtain the mapping relationship model between quantitative features and attribute matrix through the encoder on the training set; Module M4: Calculate the similarity by extracting the quantitative features of the test set through the mapping relationship model, obtain the prediction results and evaluate the accuracy.

7. The sonar image recognition system based on zero-sample learning according to claim 6, characterized in that: The module M1 comprises: Module M1.1: Collect public sonar images covering a variety of targets, enhance the images through image filtering and histogram equalization, and unify the pixel size of each image to 224*224*3; Module M1.2: Data augmentation through random rotation, flipping, scaling, translation and random noise addition; Module M1.3: Divide the data set, taking P of the sonar images of L-type targets as the training set and the remaining Q as the test set, L = P + Q; The construction attribute matrix includes: Module M2.2.1: Based on the prior knowledge of sonar image targets, determine the attribute themes of sonar images from the appearance, shape and material of the target; Module M2.2.2: Construct an attribute vector based on the determined attribute theme, set all attributes to binary values, and the attribute vector is represented as in, Indicates the descriptive feature value of the k-th attribute of the i-th category. When it is 0, it means there is no such attribute, and when it is 1, it means there is such attribute; M represents the total number of attributes; Module M2.2.3: Construct attribute vectors for the training set and the test set respectively, and merge the attribute vectors to form the attribute matrix S = [S p ,S q ]; in, S p Represents the training set attribute matrix; S q Represents the test set attribute matrix.

8. The sonar image recognition system based on zero-sample learning according to claim 6, characterized in that: The step of extracting the quantitative features of the sonar image comprises: Module M2.1.1: Load the pre-trained deep learning network trained on the large-scale image dataset ImageNet, which consists of input layer, convolution layer, pooling layer, fully connected layer and output layer; Module M2.1.2: Remove the fully connected layer and output layer of the pre-trained deep learning network, and keep the input layer and convolutional pooling layer structure as the sonar image feature extractor; Module M2.1.3: Input the training set image into the feature extractor, obtain the convolution calculation plus the offset through forward propagation calculation, and the activation function f(x) is nonlinearly mapped to obtain the feature X of the i-th layer i , The feature map size is 7*7*512, and is transposed and expanded to 1*25088 vector; Among them, Z i Represents the convolution kernel weight of the i-th layer; b i Indicates the offset of the i-th layer; Represents the convolution kernel and i-1 layer feature X i-1 Convolution calculation of ; The module M4 comprises: Module M4.1: Input the images of the test set into the image feature extractor to obtain the quantitative features x of the test set i ; Module M4.2: Quantify the features x i Obtain the corresponding attribute vector Wx through the encoder i ; Module M4.3: Normalizing the cross-correlation attribute vector Wx using zero mean i And the test set attribute matrix S q Perform similarity matching calculation and select the most similar label as the prediction result φ(x i )=argminD(Wx i ,S q ); Where D() represents the similarity value calculated using zero-mean normalized cross-correlation; W represents weight; S q represents the test set attribute matrix; Module M4.4: Count the prediction results of each test set image and evaluate the accuracy by the recognition accuracy Acc, Acc = (TP + TN) / (TP + TN + FP + FN); Among them, TP represents the number of positive samples correctly identified; FP represents the number of false positive negative samples; TN represents the number of negative samples correctly identified; FN represents the number of positive samples that are missed.

9. The sonar image recognition system based on zero-sample learning according to claim 6, characterized in that: The module M3 comprises: Module M3.1: Set up the autoencoder structure. The encoder module in the autoencoder compresses the input features into a latent space representation to achieve S = h(X). The decoder module in the autoencoder reconstructs the latent space representation to obtain the output R = g(S). The complete autoencoder structure achieves R = g(h(X)). Wherein, h() represents the encoding function; g() represents the decoding function; S represents the latent space representation; R represents the decoded output; X represents the features of the input; Module M3.2: Set the autoencoder bidirectional mapping learning training objective function to Wherein, the subscript F represents the Euclidean norm; W represents weight; S represents the attribute matrix; Module M3.3: Relax the strict constraint of WX=S to Use the training set to solve the mapping matrix W and obtain the mapping relationship model; in, Represent the losses of the decoder and encoder respectively; λ represents a hyperparameter that adjusts the weight of the two losses.

10. The sonar image recognition system based on zero-sample learning according to claim 6, characterized in that: The targets include 8 types: solid sphere, round cage, solid cube, cylinder, airplane model, metal barrel, square cage and tire; The training set takes 5 targets, and the test set takes the remaining 3 targets; The module M2 extracts vector features through a deep learning network including a VGG16 deep learning model, VGG, ResNet and GooleNet; Independent labels are represented as mutually related label semantic vectors, and the attribute matrix is ​​the intermediate semantic layer shared by the vector feature space and the label; Said properties include hard, grid, square, regular, round, cylindrical, straight, solid, and hollow; The encoder is a bidirectionally mapped semantic autoencoder structure with a hyperparameter λ=0.5.

Images