Underwater target classification method based on acousto-optic image fusion transformer and related device

Abstract

The application belongs to the technical field of image classification in computer vision, and discloses a kind of underwater target classification method and related device based on acousto-optic image fusion Transformer;Among them, the underwater target image classification method includes the following steps: obtaining optical and its corresponding acoustic image to be classified;Based on the acoustic and optical image to be classified, a pre-trained deep image classification network is used for underwater target classification to obtain a classification result.The application proposes an acousto-optic fusion module for acousto-optic image fusion, a feature extraction module for feature extraction and feature mixing, enhances the diversity and semantic richness of features, and further improves the accuracy of underwater target image classification, solving the technical problems of high misclassification of underwater target images caused by insufficient single modal image feature extraction and insufficient feature richness capture in the prior art.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image classification in computer vision, and particularly relates to an underwater target classification method based on an acousto-optic image fusion Transformer and a related device. BACKGROUND

[0002] With the development of intelligentization of underwater unmanned equipment, the role of underwater target recognition technology is increasingly prominent. Underwater target image classification refers to a technology of classifying targets in images obtained by underwater imaging equipment, which can be applied to underwater target search and rescue, underwater environment detection, ship navigation and obstacle avoidance, and is a key step for intelligentization of underwater unmanned equipment and ocean environment monitoring. Explanatorily, the marine water environment is complex and has many environmental disturbances (for example, affected by pixel size, focusing distance, shooting angle, environmental dirt, external light reflection factors, etc.), and the target in the obtained image is not easy to distinguish, which will affect the accuracy of the underwater target image classification task. Further explanatorily, underwater targets have inter-class similarity and intra-class difference in morphology, which determines that the class of underwater targets is more difficult to judge. In summary, underwater target image classification is a very challenging task.

[0003] Existing deep learning-based image classification methods are mainly based on convolutional neural networks (CNN) and vision Transformers (ViT). Convolutional neural networks use the translational invariance prior knowledge of images to learn underwater target features through convolution operations to aggregate pixel information layer by layer, thereby realizing recognition on a single data source image, such as an optical image collected by an underwater optical sensor (such as an underwater camera) or an acoustic image obtained by an acoustic sensor (such as a sonar). However, the CNN-based image classification method is limited by the local receptive field and cannot effectively capture long-distance context information, resulting in low classification accuracy. In contrast, vision Transformers perform well in modeling long-distance dependencies of features and can usually achieve higher accuracy (Top-1 Accuracy) than traditional convolutional neural networks in image classification tasks. However, the classification method of a single data type and a single feature mode Transformer is difficult to fully describe the rich texture details and feature diversity of underwater targets due to the lack of richness of feature expression. Therefore, it is crucial to propose a multi-source data fusion enhancement recognition method to improve the recognition accuracy of underwater targets. In addition, traditional vision Transformer image classification networks often lack feature enhancement modules, which makes it difficult to effectively distinguish subtle differences or difficult-to-distinguish features in underwater targets, thereby affecting the classification accuracy of acoustic or optical images.

[0004] Multi-source information fusion technology aims to integrate different types of data, make full use of the advantages of each data source, and extract more rich and representative information. However, the existing multi-source information fusion method usually faces the problem that the flow of multi-modal information between feature channels is limited, and the traditional point-by-point convolution fusion method is computationally expensive. In addition, the existing image recognition method based on Transformer often uses ordinary self-attention mechanism, and fails to fully consider how to enhance the saliency information capture of the fused feature map. Therefore, it is of important theoretical and practical significance to construct an efficient visual Transformer cognitive computing architecture based on acoustic image and optical image fusion, which can enhance the saliency feature extraction capability in complex underwater environment and improve the underwater target classification accuracy. SUMMARY

[0005] The purpose of the present application is to provide an underwater target classification method based on acoustic-optical image fusion Transformer and related device, aiming to solve the problem of insufficient saliency feature capture of underwater target image in the prior art, which leads to high misclassification. In the technical solution disclosed by the present application, an acoustic-optical fusion module is proposed to obtain a multi-modal fusion image tensor. At the same time, in order to increase the information flow between the data channels of multiple modalities, the present application designs an acoustic-optical image channel mixing module to enhance the multi-modal information fusion effect. In terms of feature extraction, the present application uses a Transformer architecture based on self-attention to extract the features of the fused image tensor. In order to further improve the global dependency modeling capability of the self-attention mechanism, the present application designs a pooling calibration attention mechanism and proposes a feature calibration module based on style pooling. The method of the present application improves the richness of underwater target features through acoustic-optical image fusion, and improves the semantic saliency of underwater target features through feature enhancement self-attention module, thereby improving the classification accuracy of underwater target image.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect of the present application, an underwater target classification method based on acoustic-optical image fusion Transformer is provided, comprising the following steps:

[0008] Acoustic and optical images of the underwater target to be classified are obtained;

[0009] Based on the acoustic and optical images of the underwater target to be classified, a pre-trained deep image classification network is used for underwater target classification to obtain a classification result;

[0010] Wherein,

[0011] The deep image classification network comprises an acoustic-optical fusion module, a feature extraction module and a linear classification head connected in sequence.

[0012] The acoustic-optical fusion module first performs image preprocessing on the acoustic image and the optical image, and converts them into an acoustic image tensor and an optical image tensor respectively; then, channel splicing and shuffling are performed on the acoustic image tensor and the optical image tensor to obtain an acoustic-optical fusion image tensor; the feature extraction module adopts a multi-stage encoder structure, and each stage of the encoder includes a patch embedding and a plurality of encoder modules based on a pooling calibration attention mechanism connected in series;

[0013] The encoder module includes a convolutional position encoding module, a first layer normalization operation, a pooling calibration attention module, a second layer normalization operation and a feedforward neural network layer connected in series, and the input of the first layer normalization operation and the output of the pooling calibration attention module are added through a jump connection layer, and the input of the second layer normalization operation and the output of the feedforward neural network are added;

[0014] The pooling calibration attention module includes a feature calibration module based on style pooling, a multi-head transpose self-attention module and a convolutional relative position encoding module; wherein the feature calibration module based on style pooling is used to input the query Q, key K and value V matrices formed by linear mapping of the output feature map of the first layer normalization operation, and output the query Q * , key K * and value V * matrices after feature calibration; the multi-head transpose self-attention module is used to input the query Q * , key K * and value V * matrices after feature calibration, and output a multi-head transpose self-attention module output feature map; the convolutional relative position encoding module is used to input the query Q * and value V * matrices after feature calibration output by the feature calibration module based on style pooling, and output a convolutional relative position encoding module output matrix; the sum of the multi-head transpose self-attention module output feature map and the convolutional relative position encoding module output matrix forms a pooling calibration attention module output feature map.

[0015] In a second aspect, the application provides an underwater target classification system based on acoustic-optical image fusion Transformer, which is used to implement the underwater target classification method based on acoustic-optical image fusion Transformer as described in any one of the first aspect.

[0016] In a third aspect, the application provides an electronic device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the underwater target classification method based on acoustic-optical image fusion Transformer as described in any one of the first aspect when executing the program.

[0017] In a fourth aspect, the application provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the underwater target classification method based on acoustic-optical image fusion Transformer according to any one of the first aspect of the application.

[0018] Compared with the prior art, the application has the following beneficial effects:

[0019] The application provides an underwater target classification method based on acoustic-optical image fusion Transformer, which classifies through a pre-trained deep image classification network, wherein the deep image classification network comprises an acoustic-optical fusion module, a feature extraction module and a linear classification head; the acoustic-optical fusion module performs channel splicing and channel shuffling on an acoustic image tensor and an optical image tensor; the feature extraction module extracts features from the acoustic-optical fusion image tensor, and inputs the output acoustic-optical fusion feature map into the linear classification module to obtain a classification result. In summary, in view of the problem of single feature extraction of a single modal underwater target image in the prior art, the application fully utilizes the multi-modal features of acoustic images and optical images, enhances the diversity of features, can effectively improve the classification accuracy of underwater targets, and reduces misclassification. The application adopts an image pre-fusion method, uses a single branch for feature extraction, has lower computational burden and higher inference speed compared with a double-branch image post-fusion method, and introduces a channel shuffling and feature calibration module based on style pooling to further enhance feature flow and feature extraction. The method of the application is based on a visual Transformer framework, introduces multi-modal visual cognitive computing into the automatic classification field, can effectively improve the classification accuracy of underwater targets, and each module is simple to implement and does not rely too much, and has strong applicability. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description; obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 is a flowchart of an underwater target classification method based on acoustic-optical image fusion Transformer provided by an embodiment of the application.

[0022] Figure 2 is a structural diagram of a deep image classification network in an embodiment of the application.

[0023] Figure 3is a structural schematic diagram of an encoder module in an embodiment of the present application.

[0024] Figure 4 is a structural schematic diagram of a pooling calibration attention module in an embodiment of the present application.

[0025] Figure 5 is a schematic diagram of an underwater target image classification result in a specific embodiment of the present application; wherein, Figure 5 (a) in the figure is a schematic diagram of the classification result of the reefs category image sample, Figure 5 (b) in the figure is a schematic diagram of the classification result of the shipwrecks category image sample, Figure 5 (c) in the figure is a schematic diagram of the classification result of the fishes category image sample, Figure 5 (d) in the figure is a schematic diagram of the classification result of the (Autonomous Underwater Vehicles) AUVs category image sample, Figure 5 (e) in the figure is a schematic diagram of the classification result of the frogmen category image sample.

[0026] Figure 6 is a schematic diagram of an underwater target classification system based on sound-light image fusion Transformer provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] Please refer to Figure 1 and Figure 2 In an embodiment of the present application, a method for classifying underwater targets based on sound-light image fusion Transformer is provided, comprising the following steps:

[0029] Step 1: Obtain acoustic and optical images of the underwater target to be classified; for example, the optical image to be classified is an underwater RGB image (i.e., a color image) containing the underwater target to be classified; the acoustic image to be classified is generated from the optical image through a style transfer network, a process that is conventional in the art and will not be described in detail here.

[0030] Step 2: Based on the acoustic and optical images of the underwater target to be classified, a pre-trained depth image classification network is used to classify the underwater target and obtain the classification result; interpretably, the classification result is output in the form of category and probability;

[0031] Among them, such as Figure 2 As shown, the deep image classification network includes an acoustic-optical fusion module, a feature extraction module, and a linear classification head connected in sequence.

[0032] The acoustic-optical fusion module first performs image preprocessing on the paired input acoustic and optical images, including two operations: First, the length and width of the acoustic and optical images are scaled to a query value of 224×224 pixels, respectively. Then, each pixel in the acoustic and optical images is normalized from 0-255 to the 0-1 range, thereby forming acoustic and optical image tensors. Next, the acoustic and optical image tensors are concatenated by channel, and then the acoustic and optical image tensors are ordered and shuffled along the channel dimension to form alternating acoustic and optical image tensor layers, thus obtaining the acoustic-optical fusion image tensor.

[0033] The feature extraction module includes four sequentially connected stages: Stage 1, Stage 2, Stage 3, and Stage 4. Stage 1 includes a 4x patch embedding and several encoder modules connected in series. Stages 2, 3, and 4 each include a 2x patch embedding and several encoder modules connected in series. Explained, the 4x patch embedding is used as input to the acoustic-optical fusion image tensor and outputs a 4x downsampled feature map. The 2x patch embedding is used as input to the output feature map of the previous stage encoder module and outputs a 2x downsampled feature map. The encoder module is used to input the 4x patch embedding. Alternatively, the feature map is downsampled by 2x, and the output feature map is output by the encoder module; to further illustrate, the 4x patch embedding performs 4x downsampling on the input acousto-optic fusion image tensor. The downsampling operation employs two convolution operations with a filter size of 3×3 and a stride of 2, and one depthwise separable convolution operation with a filter size of 3×3 and a stride of 1; the 2x patch embedding performs 2x downsampling on the output feature map of the encoder module in the previous stage. The downsampling operation employs one depthwise separable convolution operation with a filter size of 3×3 and a stride of 2.

[0034] like Figure 3As shown, the encoder module includes a convolutional position encoding, a first layer normalization operation, a pooling calibration attention module, a second layer normalization operation, and a feedforward neural network layer, and adds the input of the first layer normalization operation and the output of the pooling calibration attention module through a skip connection layer, and adds the input of the second layer normalization operation and the output of the feedforward neural network; specifically, the convolutional position encoding is used to input the input feature map of the encoder module, and outputs a convolutional position encoding output feature map; the first layer normalization operation is used to input the convolutional position encoding output feature map, and outputs a first layer normalization operation output feature map; the pooling calibration attention module is used to input the first layer normalization operation output feature map, and outputs a pooling calibration attention module output feature map; the second layer normalization operation is used to input the sum of the pooling calibration attention module output feature map and the convolutional position encoding output feature map, and outputs a second layer normalization operation output feature map; the feedforward neural network layer is used to input the second layer normalization operation output feature map, and outputs a feedforward neural network output feature map; and the encoder module output feature map is the sum of the feedforward neural network output feature map and the second layer normalization operation input feature map.

[0035] As shown in Figure 4 , the pooling calibration attention module includes a style-pooling-based feature calibration module, a multi-head transpose self-attention module, and a convolutional relative position encoding module; the application adopts the style-pooling-based feature calibration module to enhance the multi-head transpose self-attention module; the style-pooling-based feature calibration module adopts three feature calibration networks to perform style-pooling-based feature calibration operations on the input query Q, key K, and value V matrices, respectively, where the query Q, key K, and value V matrices are obtained by linear mapping of the first layer normalization operation output feature map; each feature calibration network includes a trunk network and a branch network, and the output of the feature calibration network is a broadcast-based Hadamard product of the output of the trunk network and the output of the branch network; wherein the operation of the trunk network is an identity mapping, and the operation of the branch network includes a global average pooling, a global maximum pooling, a one-dimensional convolution, and a sigmoid activation function operation connected in turn; through average pooling and maximum pooling, the branch network can effectively extract the channel correlation of the query Q, key K, and value V matrices formed by the acoustic-optical fusion image feature map, and mix the adjacent channel correlation information through one-dimensional convolution, and then obtain the weight of each acoustic image channel and optical image channel of the underwater target through the Sigmoid function and act on the query Q * , key K * , and value V * matrices to form feature-calibrated query Q * , key K * , and value V *The matrix enhances the saliency of the characteristics by suppressing or enhancing the characteristics of the multi-source heterogeneous characteristic information channel, and forms subsequent pooling calibration attention to capture the enhanced long-distance dependency relationship, thereby improving the acoustic-optical image fusion and classification accuracy. The multi-head transpose self-attention module inputs the query Q * , the key K * , and the value V * matrix, and outputs the feature map of the multi-head transpose self-attention module; the convolution relative position encoding module is used for inputting the query Q * , the value V * matrix, and outputs the matrix output by the convolution relative position encoding module; and the feature map output by the pooling calibration attention module is the sum of the feature map output by the multi-head self-attention module and the matrix output by the convolution relative position encoding module.

[0036] In the embodiment of the application, the visual Transformer-based cognitive computing technology is applied to the field of automatic recognition of underwater targets, the acoustic image tensor and the optical image tensor are spliced and shuffled in channels through an acoustic-optical fusion module, and an improved feature extraction module is proposed for acoustic-optical image feature extraction; the acoustic-optical fusion module is used for acoustic image feature and optical image feature fusion, which enhances the diversity and richness of the features, and accordingly an image classification network is constructed to solve the underwater target image classification task. Further explanation, the feature extraction module and the feature fusion module disclosed in the embodiment of the application are based on basic convolution and matrix operations, the input and output data forms are consistent, do not need to rely too much, and are convenient to apply to various underwater target depth image classification models, and have wide application prospects. In summary, the underwater target classification method based on acoustic-optical image fusion Transformer provided in the embodiment of the application uses the method of acoustic image and optical image fusion and feature extraction to improve the diversity of underwater target features, can enhance the diversity and semantic richness of the extracted features, and improve the underwater target image classification accuracy. Therefore, the technical scheme disclosed in the embodiment of the application can solve the technical problems of underwater target image misclassification caused by insufficient single-modal image feature extraction and insufficient consideration of feature richness in the prior art.

[0037] In an embodiment of the application, the acoustic-optical fusion module splices and shuffles the acoustic image tensor and the optical image tensor in channels; inputs the acoustic image and the optical image, and outputs the acoustic-optical fusion image tensor. The acoustic-optical fusion module can be:

[0038]

[0039] In the formula, X a is the acoustic image tensor, X o is the optical image tensor, and X aoX is a concatenated tensor of acoustic image tensor and optical image tensor, X is a sound-light fusion image tensor, Cat is a channel concatenation; Shuffle is a channel shuffle, X a1 a2 a3 X1, X2, X3 represent channel 1, channel 2, channel 3 of the acoustic image tensor respectively. o1 o2 o3 X1, X2, X3 represent channel 1, channel 2, channel 3 of the optical image tensor respectively.

[0040] In an embodiment of the present application, the training process of the depth image classification network comprises:

[0041] 1) Data preprocessing: scale the acoustic image and the optical image in the training set and the acoustic image and the optical image in the validation set to 224x224, and convert them into the form of a tensor, the pictures in the training set and the validation set are paired acoustic images and optical images; after channel concatenation and channel shuffle, a sound-light fusion image tensor is formed;

[0042] 2) Data loading and batch processing: according to the set batch size 64, load the data of the training set and the validation set in batches;

[0043] 3) Model initialization: initialize the defined sound-light image fusion Transformer-based optimizer and cross-entropy loss function;

[0044] 4) Model training: for each training cycle (epoch), input the sound-light fusion image tensor into the network through forward propagation, obtain the prediction result through the feature extraction module and the linear classification head; use the cross-entropy loss function to calculate the difference between the prediction result and the actual label; through the error back propagation algorithm, calculate the gradient and update the network parameters to minimize the loss;

[0045] 5) Model verification: after the end of each training cycle, evaluate the model performance by calculating the cross-entropy loss and the accuracy on the validation set;

[0046] 6) Model saving: if the accuracy on the validation set is improved, save the current optimal model.

[0047] In a specific embodiment of the present application, the flow of the underwater target classification method comprising the network training process specifically comprises the following steps:

[0048] ​​​​Step 1: constructing an optical image dataset of underwater targets to be identified; wherein the image satisfies three color channels of RGB (i.e. a color image), and contains corresponding artificial classification results; the pixel size of the dataset ranges from 480 to 1920 in length and width; a corresponding acoustic image dataset is generated using a style transfer network; the dataset is randomly divided into a training set and a validation set; the collected pictures are transmitted in pairs to a computer executing an algorithm.

[0049] Step 2: constructing an image classification network, including a sound-light fusion module, a feature extraction module and a linear classification head connected in sequence;

[0050] In the embodiment of the application, the sound-light fusion module performs channel splicing on the acoustic image tensor and the optical image tensor, inputs the acoustic image and the optical image, and outputs a sound-light fusion image tensor. The sound-light fusion module is:

[0051]

[0052] In the formula, X a is an acoustic image tensor, X o is an optical image tensor, X ao is a sound-light fusion image tensor, Cat is channel splicing; Shuffle is channel shuffling, X a1 , X a2 , X a3 respectively represent channel 1, channel 2 and channel 3 of the acoustic image tensor, X o1 , X o2 , X o3 respectively represent channel 1, channel 2 and channel 3 of the optical image tensor.

[0053] The feature extraction module includes stage 1, stage 2, stage 3 and stage 4 connected in sequence; the stage 1 includes a 4-fold patch embedding and two encoder modules connected in sequence; the stage 2, the stage 3 and the stage 4 each include a 2-fold patch embedding, and include 2, 6 and 2 encoder modules respectively; the 4-fold patch embedding module includes two convolution operations with a step size of 2 and a filter size of 3x3, and a depth separable convolution operation with a step size of 1 and a filter size of 3x3:

[0054]

[0055] In the formula, X1 is the sound-light fusion image tensor input to the feature extraction module, is a feature map output by the convolution operation, is a feature map output by the depth separable convolution, Conv 3×3 is a convolution operation with a filter size of 3x3, DWC 3×3is a depth separable convolution operation with a filter size of 3x3; after each convolution operation, batch normalization and Hardswish activation function processing are performed once; two convolution operations perform 4 times down sampling on the resolution of the input image, and expand the channel number of the input image to 40; the depth separable convolution operation does not change the image resolution and channel number; the 2 times patch embedding module of stages 2, 3 and 4 comprises a depth separable convolution operation with a filter size of 3x3 and a step size of 2, which performs 2 times down sampling on the resolution of the input feature map, and expands the channel number of the input feature map to 60, 80 and 120 respectively.

[0056] In the embodiment of the application, the encoder module comprises, in sequence, a convolution position encoding, a first layer normalization operation, a pooling calibration attention module, a second layer normalization operation and a feedforward neural network layer, and the input of the first layer normalization operation and the output of the pooling calibration attention module are added through a jump connection layer, and the input of the second layer normalization operation and the output of the feedforward neural network are added.

[0057] The convolution position encoding operation is as follows:

[0058]

[0059] In the formula, X2 is the output feature map of the 4 times patch embedding or the 2 times patch embedding in the feature extraction module or the encoder module, is the convolution position encoding output feature map, and Conv 3×3 is a convolution operation with a filter size of 3x3.

[0060] In the embodiment of the application, the pooling calibration attention module comprises a style-pooling-based feature calibration module, a multi-head transpose self-attention module and a convolution relative position encoding module; the style-pooling-based feature calibration module is used to input the query Q, key K and value V matrices formed by linear mapping of the output feature map of the first layer normalization operation, and output the feature-calibrated query Q * , key K * and value V * matrix; the multi-head transpose self-attention module is used to input the feature-calibrated query Q * , key K * and value V * matrix, and output the multi-head transpose self-attention module output feature map; the convolution relative position encoding module is used to input the feature-calibrated query Q * , key K *The matrix output by the convolution relative position encoding module is input into the multi-head transpose self-attention module, and the matrix output by the multi-head transpose self-attention module is input into the pooling calibration attention module.

[0061] The feature calibration module based on style pooling is,

[0062]

[0063] In the formula, Q * , K * , and V * are the query, key, and value matrices after feature calibration; Q, K, and V are query, key, and value matrices formed by linear mapping of the feature map output by the first layer normalization operation; σ(·) is a Sigmoid activation function; Conv(·) is a one-dimensional convolution operation with a convolution kernel size; [Ave, Max] # is a style pooling operation in the channel dimension, including global average pooling Ave and global maximum pooling Max; # indicates different operation matrices; is a Hadamard product.

[0064] The multi-head transpose self-attention module is,

[0065]

[0066] In the formula, Att is the feature map output by the multi-head transpose self-attention module; Q * , K * , and V * are the query, key, and value matrices after feature calibration; d is a scaling factor; Softmax(·) is an exponential normalization function; (·) T represents matrix transposition;

[0067] The convolution relative position encoding module is,

[0068]

[0069] In the formula, ReP is the matrix output by the convolution relative position encoding module; DConv 3，5 is a deep convolution operation with a filter size of 3x3 and 5x5 performed on each channel; represents a Hadamard product.

[0070] The feature map output by the pooling calibration attention module composed of the feature map output by the multi-head transpose self-attention module and the matrix output by the convolution relative position encoding module is,

[0071]

[0072] In the formula, ConvAtt represents the output feature map of the pooling calibration attention module.

[0073] In the embodiment of the present application, the classification module adopts a linear classification head for generating the target category; the linear classification head is a commonly used module in the field of image classification.

[0074] Step 3: data preprocessing, before training the network with pairs of acoustic images and optical images, each picture is first scaled to 224 pixels in width and height, so that the picture size is 224x224 pixels, and finally the picture pixels are normalized from 0-255 to the interval 0-1 to form an acoustic image tensor and an optical image tensor, respectively.

[0075] Step 4: after the acoustic image tensor and the optical image tensor are paired, channel splicing and channel shuffling are performed to form a sound-light fusion image tensor, and then a sound-light fusion feature map is obtained through a feature extraction module, and then an underwater target image classification result is output through a linear classification head module;

[0076] In addition, in each training process, backpropagation is performed starting from the loss function value; an AdamW optimizer is used to optimize the network parameters according to the gradient information obtained by backpropagation, thereby guiding the neural network to achieve accurate image classification results according to the input pictures.

[0077] In the following specific embodiments of the present application: the optical image dataset used contains a total of 3745 optical images of underwater environment shots of 5 categories, the acoustic image dataset used contains a total of 3745 acoustic images of the optical images generated by a style transfer network, and the width and height of the images are 480-1920 pixels; the style transfer network used is a convolutional neural network-based style transfer network; the acoustic dataset and the optical dataset are randomly divided into a training set consisting of 2452 images and a validation set consisting of 1293 images, respectively; the images will be first scaled to 224x224 pixels in the preprocessing stage; the initialization method of the image classification network in the embodiment of the present application is random initialization; the running environment is a computer with frameworks such as PyTorch, which can read given pictures and complete the construction and training of the model of the present method. The training time of the embodiment of the present application on a CPU of Gold6626R@2.90GHz, 8G memory and a GPU of NVIDIA GeForce RTX3090 is about 1 hour.

[0078] In the embodiment of the present application, the specific implementation steps include: first, setting the relevant training parameters, setting the optimizer used for network updating in the present application to AdamW optimizer, setting the momentum value to 0.05, setting the initial learning rate to 0.001, and setting the weight decay coefficient to 0.00001. The learning rate adjustment strategy is set to linear warm-up and cosine simulated annealing, and the warm-up period is 5. The deep image classification network of the embodiment of the present application is composed of an acoustic-optical fusion module, a feature extraction module and a linear classification head. Each feature extraction network includes 2, 2, 6 and 2 encoder modules in the four stages, respectively. The input of the network is an RGB image (i.e. a color image), which is processed into an acoustic-optical fusion image tensor in the acoustic-optical fusion module. The texture and abstract semantic information of the acoustic-optical fusion image tensor are extracted by the feature extraction module, and the corresponding output feature map is obtained after multiple stages of feature extraction to obtain an acoustic-optical fusion feature map. Then it is passed to the linear classification head module to obtain a classification result image containing target categories and probabilities. Next, when training the network using the divided data set, 64 pictures are randomly selected from the training set each time to input into the network, and the selected AdamW optimizer is used for parameter updating. After 30 iterations on the data set, the training is completed. Finally, the pictures in the validation set are input into the trained network for classification to obtain the results of the embodiment of the present application, as shown in Figure 5 Figure 5 Fig. (a) is a classification result schematic diagram of reefs category image samples, Figure 5 Fig. (b) is a classification result schematic diagram of shipwrecks category image samples, Figure 5 Fig. (c) is a classification result schematic diagram of fishes category image samples, Figure 5 Fig. (d) is a classification result schematic diagram of (Autonomous Underwater Vehicles) AUVs category image samples, Figure 5 Fig. (e) is a classification result schematic diagram of frogmen category image samples. From the categories and their probabilities of underwater targets given in the embodiment, the method proposed in the present application achieves good underwater target image classification results. Among the five underwater target image samples given, the underwater categories are correctly given with high probability, which reflects the effectiveness of the present application in the task of accurate classification of underwater images.

[0079] ​In summary, the embodiment of the present application discloses an underwater target classification method based on acoustic-optical image fusion Transformer, the acoustic-optical fusion module of the method is used for multi-modal image fusion, the feature extraction module is used for multi-modal image tensor feature extraction and feature mixing; by introducing the fusion of acoustic features and optical features, the diversity of the features is enhanced. The method of the present application introduces visual cognitive computing into the field of automatic classification, which can effectively improve the accuracy of underwater target image classification, and each module is simple to realize and does not rely too much, and has strong applicability. The underwater target classification method based on acoustic-optical image fusion Transformer provided by the present application has the implementation result of the calibration of the target type in the image, and can solve the technical problems of underwater target image misclassification caused by insufficient single modal image feature extraction and insufficient consideration of feature richness in the prior art.

[0080] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment, please refer to the method embodiment of the present application.

[0081] Please refer to Figure 6 In the embodiment of the present application, a kind of underwater target classification system based on acoustic-optical image fusion Transformer is provided, comprising:

[0082] Data acquisition module, it is used to obtain underwater target acoustic image and optical image to be classified;

[0083] Deep image classification network underwater target classification module, it is used to underwater target classification based on the underwater target acoustic image and optical image to be classified, using pre-trained deep image classification network, obtains classification result;

[0084] Wherein,

[0085] The deep image classification network includes acoustic-optical fusion module, feature extraction module and linear classification head connected in turn;

[0086] The acoustic-optical fusion module first carries out image preprocessing to acoustic image and optical image, respectively converts into acoustic image tensor and optical image tensor;Then, a pair of acoustic image tensor and optical image tensor are channel splicing and mixed to obtain acoustic-optical fusion image tensor;The feature extraction module adopts multi-stage encoder structure, and the encoder of each stage includes patch embedding and a plurality of encoder modules based on pooling calibration attention mechanism connected in series;

[0087] The encoder module includes a convolutional position encoding module, a first-layer normalization operation, a pooling calibration attention module, a second-layer normalization operation, and a feedforward neural network layer connected in series. The input of the first-layer normalization operation is added to the output of the pooling calibration attention module, and the input of the second-layer normalization operation is added to the output of the feedforward neural network, respectively, through skip connection layers.

[0088] The pooling-based calibration attention module includes a style-pooling-based feature calibration module, a multi-head transpose self-attention module, and a convolutional relative position encoding module. The style-pooling-based feature calibration module takes as input a query Q, key K, and value V matrix formed by linear mapping of the feature map output from the first-layer normalization operation, and outputs a feature-calibrated query Q. * Key K * Value V * Matrix; the multi-head transposed self-attention module is used as input for the feature-calibrated query Q. * Key K * Value V * The matrix outputs a multi-head transpose self-attention module output feature map; the convolutional relative position encoding module is used as input to the feature-calibrated query Q output by the style pooling-based feature calibration module. * Value V * The output matrix of the convolutional relative position encoding module is used to form the output feature map of the pooling calibration attention module.

[0089] In an embodiment of the present application, a computer device is provided, which comprises a processor and a memory for storing a computer program comprising program instructions, the processor being configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method process or a corresponding function. The processor in the embodiments of the present application can be used to execute the operation of the underwater target classification method based on the acoustic-optical image fusion Transformer.

[0090] In an embodiment of the present application, a storage medium is provided, specifically a computer readable storage medium (Memory), which is a memory device in a computer device, used for storing programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the computer device, and of course can also include an expansion storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the underwater target classification method based on the acoustic-optical image fusion Transformer in the above embodiments.

[0091] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0092] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0093] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0095] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing and illustrating, not limiting the technical solutions of the present application. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. An underwater target classification method based on acoustic-optical image fusion Transformer, characterized in that, Includes the following steps: Acquire acoustic and optical images of the underwater target to be classified; Based on the acoustic and optical images of the underwater target to be classified, a pre-trained depth image classification network is used to classify the underwater target and obtain the classification result. in, The deep image classification network includes an acoustic-optical fusion module, a feature extraction module, and a linear classification head connected in sequence. The acoustic-optical fusion module first preprocesses the acoustic and optical images, converting them into acoustic image tensors and optical image tensors, respectively. Then, it performs channel splicing and shuffling on the paired acoustic and optical image tensors to obtain the acoustic-optical fusion image tensor. The feature extraction module adopts a multi-stage encoder structure, with each stage of the encoder including cascaded patch embedding and several encoder modules based on a pooling calibration attention mechanism. The encoder module includes a convolutional position encoding module, a first-layer normalization operation, a pooling calibration attention module, a second-layer normalization operation, and a feedforward neural network layer connected in series. The input of the first-layer normalization operation is added to the output of the pooling calibration attention module, and the input of the second-layer normalization operation is added to the output of the feedforward neural network, respectively, through skip connection layers. The pooling calibration attention module includes a style pooling-based feature calibration module, a multi-head transpose self-attention module, and a convolutional relative position encoding module; wherein, the style pooling-based feature calibration module is used to input the query formed by linear mapping of the feature map output from the first layer normalization operation. ,key ,value Matrix, outputting a feature-calibrated query. ,key ,value Matrix; the multi-head transposed self-attention module is used as input for feature-calibrated queries. ,key ,value The matrix outputs a multi-head transpose self-attention module output feature map; the convolutional relative position encoding module is used as input to the feature calibration output of the style pooling-based feature calibration module after feature calibration. ,value The output matrix of the convolutional relative position encoding module is used to form the output feature map of the pooling calibration attention module. The calculation of the pooling calibration attention module is expressed as follows: ； in, ； ； ； In the formula, Use the Sigmoid activation function; This is an operation for one-dimensional convolution with a kernel size of 1. Each is for , , Style pooling operations along the matrix channel dimension include global average pooling (Ave) and global max pooling (Max). For Hadamah accumulation; For each channel, the filter size is... , Depth-wise convolution operations; This is the scaling factor; It is the exponential normalization function; This is the matrix transpose. To calibrate the output of the attention module using pooling.

2. The underwater target classification method based on acoustic-optical image fusion Transformer according to claim 1, characterized in that, The aforementioned acoustic-optical fusion module performs channel splicing and channel shuffling on acoustic image tensors and optical image tensors of the same size and number of channels to obtain an acoustic-optical fusion image tensor. The channel shuffling involves orderly shuffling of the acoustic image tensor and the optical image tensor according to their channels, so that the channels of the acoustic-optical fusion image tensor are alternating layers of acoustic image tensor and optical image tensor.

3. The underwater target classification method based on acoustic-optical image fusion Transformer according to claim 2, characterized in that, In the deep image classification network, the computation of the acoustic-optical fusion module is represented as follows: ； ； In the formula, For acoustic image tensors, For optical image tensors, It is a concatenated tensor of acoustic image tensor and optical image tensor. To splice by channel; For mixed washing in the channel; For audio-visual fusion image tensors; Channels 1, 2, and 3 represent the acoustic image tensor, respectively. These represent channels 1, 2, and 3 of the optical image tensor, respectively.

4. The underwater target classification method based on acoustic-optical image fusion Transformer according to claim 1, characterized in that, The style pooling-based feature calibration module includes three modules for query calibration. ,key ,value A matrix is ​​used to perform a style pooling-based feature calibration network. Each feature calibration network includes a backbone network and branch networks. The backbone network operates on an identity mapping, while the branch networks operate on global average pooling, global max pooling, one-dimensional convolution, and sigmoid activation function operations in sequence. The Hadamard product of the backbone network output and the branch network outputs, based on broadcasting, serves as the output of the feature calibration network. The outputs of the three feature calibration networks are respectively the feature-calibrated queries. ,key ,value matrix.

5. The underwater target classification method based on acoustic-optical image fusion Transformer according to claim 1, characterized in that, The feature extraction module includes stages 1, 2, 3, and 4 connected in sequence; stage 1 includes a 4x patch embedding and several encoder modules connected in series; stage 2, stage 3, and stage 4 each include a 2x patch embedding and several encoder modules connected in series. The 4x patch embedding is used to downsample the input acousto-optic fusion image tensor by 4x and output a 4x downsampled feature map; The 2x patch embedding is used to downsample the output feature map of the encoder module in the previous stage by 2x, and output a 2x downsampled feature map. The encoder module is used to encode the 4x or 2x downsampled feature map, and outputs the feature map output by the encoder module.

6. The underwater target classification method based on acoustic-optical image fusion Transformer according to claim 1 or 5, characterized in that, In the encoder module: The convolutional position encoding module is used to input the input feature map of the encoder module and output the convolutional position encoded output feature map; The first-layer normalization operation is used to input the convolution position encoding output feature map and output the first-layer normalization operation output feature map. The pooling calibration attention module is used to input the feature map output by the first layer normalization operation and output the feature map output by the pooling calibration attention module. The second-layer normalization operation is used as input to the sum of the output feature map of the pooling calibration attention module and the output feature map of the convolution position encoding, and outputs the feature map of the second-layer normalization operation. The feedforward neural network layer is used to input the output feature map of the second layer normalization operation and output the feedforward neural network output feature map. The encoder module output feature map is the sum of the feedforward neural network output feature map and the input feature map of the second-layer normalization operation.

7. An underwater target classification system based on acoustic-optical-image fusion Transformer, characterized in that, include: The data acquisition module is used to acquire acoustic and optical images of the underwater targets to be classified. The underwater target classification module of the depth image classification network is used to classify underwater targets based on acoustic and optical images of the underwater targets to be classified, using a pre-trained depth image classification network to obtain classification results. in, The deep image classification network includes an acoustic-optical fusion module, a feature extraction module, and a linear classification head connected in sequence. The acoustic-optical fusion module first preprocesses the acoustic and optical images, converting them into acoustic image tensors and optical image tensors, respectively. Then, it performs channel splicing and shuffling on the paired acoustic and optical image tensors to obtain the acoustic-optical fusion image tensor. The feature extraction module adopts a multi-stage encoder structure, with each stage of the encoder including cascaded patch embedding and several encoder modules based on a pooling calibration attention mechanism. The encoder module includes a convolutional position encoding module, a first-layer normalization operation, a pooling calibration attention module, a second-layer normalization operation, and a feedforward neural network layer connected in series. The input of the first-layer normalization operation is added to the output of the pooling calibration attention module, and the input of the second-layer normalization operation is added to the output of the feedforward neural network, respectively, through skip connection layers. The pooling calibration attention module includes a style pooling-based feature calibration module, a multi-head transpose self-attention module, and a convolutional relative position encoding module; wherein, the style pooling-based feature calibration module is used to input the query formed by linear mapping of the feature map output from the first layer normalization operation. ,key ,value Matrix, outputting a feature-calibrated query. ,key ,value Matrix; the multi-head transposed self-attention module is used as input for feature-calibrated queries. ,key ,value The matrix outputs a multi-head transpose self-attention module output feature map; the convolutional relative position encoding module is used as input to the feature calibration output of the style pooling-based feature calibration module after feature calibration. ,value The output matrix of the convolutional relative position encoding module is used to form the output feature map of the pooling calibration attention module. The calculation of the pooling calibration attention module is expressed as follows: ； in, ； ； ； In the formula, Use the Sigmoid activation function; This is an operation for one-dimensional convolution with a kernel size of 1. Each is for , , Style pooling operations along the matrix channel dimension include global average pooling (Ave) and global max pooling (Max). For Hadamah accumulation; For each channel, the filter size is... , Depth-wise convolution operations; This is the scaling factor; It is the exponential normalization function; This is the matrix transpose. To calibrate the output of the attention module using pooling.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the underwater target classification method based on the Transformer for acoustic-optical image fusion as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the underwater target classification method based on the Transformer for audio-visual image fusion as described in any one of claims 1 to 6.

Images