Underwater sound target recognition system and method based on multi-mode joint representation

Abstract

The invention provides an underwater acoustic target recognition system and method based on multi-modal joint representation. The system comprises a multi-modal feature extractor, a feature fusion module, a fusion feature extractor and a full connection layer. The processing process of the system comprises the steps that multi-modal features based on waveforms and spectrograms are extracted through a waveform network and a spectrogram network of a multi-modal feature extractor respectively, the two-way structural design allows the network to capture features of an underwater acoustic target from different angles, and understanding of signal features is enhanced; the two kinds of feature information of a spectrogram and a waveform are fused through a feature fusion module, and the two kinds of features are fused through a multi-head attention mechanism; and finally, constructing a fusion feature extractor to extract global features. The system disclosed by the invention obtains a very competitive result on two widely used ship radiation noise data sets. Experimental results show that the performance of the system exceeds that of a traditional waveform-based and spectrogram-based model on the premise of keeping a relatively low parameter quantity.

Description

Technical Field

[0001] The present application belongs to the field of underwater acoustic target recognition, and specifically relates to an underwater acoustic target recognition system and method based on multi-modal joint characterization. Background Art

[0002] Hydroacoustic target recognition determines the type of underwater targets (such as ships, marine life, etc.) by analyzing and processing underwater sound waves received by sonar equipment. Hydroacoustic target recognition technology has a wide range of applications in the ocean field, covering areas such as ocean observation, marine resource exploration, and marine military operations, and can provide an important basis for human marine military activities and marine economy. Hydroacoustic target recognition technology is extremely critical for quickly and accurately obtaining underwater target information. It has become an important research direction in the field of hydroacoustic signal processing, and is also a hot topic and recognized problem for researchers from various countries.

[0003] Traditional underwater acoustic target recognition methods usually require the extraction of manually designed features to train classifiers. Feature extraction methods are mainly divided into three categories: time domain, frequency domain, and time-frequency domain. Among these methods, time-frequency domain features have been widely used because they combine rich information in the time domain and frequency domain. Time-frequency domain analysis converts the original one-dimensional underwater acoustic waveform into a two-dimensional time-frequency spectrum. This conversion not only reduces the size of the input data, but also helps to reduce the impact of noise, thereby achieving good results in underwater acoustic target recognition tasks. Common time-frequency feature spectrograms include CQT (Constant-QTransform), GFCC (Gammatone Frequency Cepstral Coefficient), MFCC (Mel Frequency Cepstral Coefficients), Mel spectrum, etc. Extracting these spectral features requires manually adjusting the filter parameters based on the recognition results. In practical applications, researchers usually need to consider the different characteristics of underwater acoustic signals and the specific requirements of the task to set the filter parameters. However, when faced with complex and changeable marine environments, fixed parameter settings are often difficult to adapt to the changes of various underwater scenes and targets. In addition, studies have shown that low-dimensional acoustic features limit the generalization ability of the model, while high-dimensional representations can more effectively capture the inherent characteristics of target signals. These factors have caused limitations in the performance of traditional underwater acoustic target recognition methods, especially in terms of adaptability and generalization ability.

[0004] Thanks to the powerful ability of deep learning in feature learning, as well as the accumulation and public sharing of underwater acoustic datasets, underwater acoustic target recognition methods based on deep learning have received widespread attention in recent years. According to the modal differences of the input data, these methods can be divided into two categories: one is based on spectrograms, and the other is based on raw waveforms. In spectrogram-based methods, time-frequency spectrograms are regarded as inputs, and deep neural networks (DNNs) are used to extract deep features. However, in spectrogram-based methods, the original underwater acoustic signals often lose temporal detail information after time-frequency processing. At the same time, as mentioned earlier, setting the parameters for extracting spectrograms also depends on prior knowledge. Waveform-based methods directly use raw waveforms as inputs and obtain discriminative feature representations from these raw waveforms through deep neural networks. Although waveform-based methods can accurately perceive time domain waveforms, due to the superposition characteristics of time domain signals, such methods are easily interfered by unstable noise, resulting in weak robustness to noise.

[0005] Comprehensively considering the spectrogram-based and waveform-based underwater acoustic target recognition methods, it can be found that the two types of methods characterize the characteristics of underwater acoustic target signals from different perspectives, and each has its own advantages and limitations. For example, spectrogram-based methods can express information effectively and efficiently, but such methods require preprocessing steps and are more vulnerable to adversarial attacks; while waveform-based methods utilize the subtle temporal structure of the original signal and do not require preprocessing, so they are suitable for mobile or embedded devices, but they have more noise, and the computational cost and data requirements are usually high. However, there are still few existing studies on the problem of underwater acoustic target recognition with multiple input data modalities. How to complement the advantages of spectrogram and waveform representations to improve the performance and efficiency of underwater acoustic target recognition is still a major difficulty. Summary of the invention

[0006] The purpose of the present invention is to complement the advantages of the two representation methods of spectrum and original waveform, and to propose an underwater acoustic target recognition system and method based on multimodal joint representation, so as to improve the target recognition performance with a small number of parameters.

[0007] In order to achieve the above objectives, this application proposes an underwater acoustic target recognition system based on multi-modal joint representation, including:

[0008] A multimodal feature extractor for extracting the spectrogram and waveform features of underwater acoustic signals;

[0009] Feature fusion module, used to fuse the output of spectrum features and waveform features;

[0010] A fusion feature extractor is used to extract the global features of the underwater acoustic signal;

[0011] The fully connected layer is used to output the final underwater acoustic target category.

[0012] As an improvement of the above system, the multimodal feature extractor includes a waveform network and a spectrogram network, wherein:

[0013] The waveform network is used to extract the waveform features of the underwater acoustic waveform, including 7 Conv1D-BN-ReLU blocks;

[0014] The Conv1D-BN-ReLU block includes a one-dimensional convolution layer, a batch normalization, and a first activation function;

[0015] The one-dimensional convolution layer of the first Conv1D-BN-ReLU block uses a convolution downsampling operation to reduce the size of the input underwater acoustic waveform; the one-dimensional convolution layers of the second to fifth Conv1D-BN-ReLU blocks use one-dimensional maximum pooling downsampling;

[0016] The spectrogram network is used to take the Mel spectrum of the water acoustic waveform as input and extract the spectrogram features of the water acoustic waveform, including 5 Conv2D-BN-ReLU blocks and 1 frequency average pooling module;

[0017] The Conv2D-BN-ReLU block includes a two-dimensional convolutional layer, a batch normalization, and a first activation function;

[0018] The first Conv2D-BN-ReLU block performs downsampling operations in frequency and time dimensions;

[0019] The 2nd to 5th Conv2D-BN-ReLU blocks only downsample in the frequency dimension.

[0020] As an improvement of the above system, the convolution kernel size of the one-dimensional convolution layer of the Conv1D-BN-ReLU block is 9; the step size of the one-dimensional convolution layer of the first Conv1D-BN-ReLU block is 4; and the first activation function is ReLU.

[0021] As an improvement of the above system, in the Mel spectrum of the underwater acoustic waveform, the number of Mel filters is 64, the number of STFT sampling points is 1024, and the Hamming window frame shift interval is 512.

[0022] As an improvement of the above system, the feature fusion module adopts a Transformer-based fusion strategy;

[0023] The feature fusion module uses the waveform features and the spectrogram features as the key vector and the query vector in the attention mechanism respectively; obtains the similarity matrix by the dot product of the transposed waveform features and the spectrogram features, and further obtains the attention matrix by Softmax;

[0024] The waveform features and the spectrogram features are concatenated in the channel dimension, and the mixed features are obtained as the value vector by projecting the multi-layer perceptron;

[0025] The channel weighted result is obtained by calculating the dot product of the value vector and the attention matrix, and then global average pooling is used to fuse all channel information.

[0026] As an improvement of the above system, the fusion feature extractor includes two multi-layer perceptron modules and a third multi-layer perceptron;

[0027] The multi-layer perceptron module includes layer normalization, a first multi-layer perceptron, a second activation function, and a second multi-layer perceptron;

[0028] After the input data is layer normalized, the first multi-layer perceptron performs global feature extraction, and after being processed by the second activation function, the feature extraction of the second multi-layer perceptron is added to the input data in a residual connection manner.

[0029] As an improvement to the above system, the second activation function is a nonlinear activation function.

[0030] As an improvement of the above system, the training process of the system includes:

[0031] Step 1: Preprocess the original underwater acoustic signal: resample the original underwater acoustic data set, slice the data set, and divide it into training set and test set;

[0032] Step 2: Set training hyperparameters, loss function, and optimizer;

[0033] Step 3: Extract the Mel spectrum features of each original underwater acoustic signal waveform in the data set;

[0034] Step 4: Load the training set and send the underwater acoustic waveforms and their corresponding Mel spectra in the training set into the underwater acoustic target recognition system in batches for training until convergence.

[0035] As an improvement of the above system, the method of slicing and dividing the training set and the test set includes: slicing first and then dividing the data set and first dividing the data set and then slicing;

[0036] During training, the AdamW optimizer was used, and the initial training learning rate was 0.001. Batch training was adopted, and the underwater acoustic waveforms and their corresponding Mel spectra in the training set were input into the underwater acoustic target recognition system in batches. The training batch size was 64, and the number of training rounds was 100. The cross entropy loss was used as the optimization objective function.

[0037] The present application also provides a method for underwater acoustic target recognition based on multi-modal joint characterization, which is implemented based on the above system and includes:

[0038] The preprocessed underwater acoustic signal is input into the underwater acoustic target recognition system, and the recognition result is output.

[0039] Compared with the prior art, the advantages of this application are:

[0040] The present invention provides an underwater acoustic target recognition system and method based on multimodal joint characterization, which utilizes the advantages of spectrogram and waveform to extract the joint features of the two and improve the accuracy of target recognition. The system of the present invention achieves comparable or even better performance than other methods when the network parameters only account for 20%. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The following is the architecture diagram of the underwater acoustic target recognition system based on multi-modal joint representation;

[0042] Figure 2 Shown is a schematic diagram of the feature fusion module;

[0043] Figure 3 The figure shows the ROC curve comparison of different underwater acoustic target recognition methods on the ShipsEar dataset;

[0044] Figure 4 The figure shows the ROC curve comparison of different underwater acoustic target recognition methods on the DeepShip dataset;

[0045] Figure 5 Shown is a confusion matrix diagram of the method of the present invention on the ShipsEar dataset;

[0046] Figure 6 Shown is a visualization t-SNE graph of some underwater acoustic target recognition methods on the ShipsEar dataset;

[0047] Figure 7 Shown is a visualization t-SNE graph of some underwater acoustic target recognition methods on the DeepShip dataset. DETAILED DESCRIPTION

[0048] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, the present application provides an underwater acoustic target recognition system based on multi-modal joint representation, comprising:

[0050] Multimodal Feature Extractor (MFE), feature fusion module, fusion feature extractor (FFE) and fully connected layer. The multimodal feature extractor consists of two sub-networks: Spectrogram-Net and Wave-Net, which are used to extract spectrogram features respectively. and waveform characteristics .

[0051] The waveform network converts the original hydroacoustic waveform As input, the network contains seven Conv1D-BN-ReLU blocks. Among them, Conv1D-BN-ReLU represents a one-dimensional convolutional layer, accompanied by batch normalization (Batch Normalization, BN) and ReLU (Rectifier Linearity Unit) processes. The first block uses a convolution downsampling operation with a step size of 4 (common step sizes include 1, 2, 3 or larger) to quickly reduce the input size. The following four Conv1D-BN-ReLU blocks use one-dimensional maximum pooling downsampling. The last two blocks are used to further refine the deep features and produce the final output waveform features. In the waveform network, all convolution kernels use a large convolution kernel with a kernel size of 9 (common kernel sizes include 1, 3, 5, 7, and 9) to expand the receptive field as quickly as possible.

[0052] The spectrogram network extracts the Mel spectrum of the original underwater acoustic waveform As input (the number of Mel filters is set to 64, the number of STFT sampling points is set to 1024, and the Hamming window frame shift interval is set to 512), the network contains five Conv2D-BN-ReLU blocks for time-frequency feature extraction. Among them, Conv2D-BN-ReLU represents a two-dimensional convolution layer, accompanied by batch normalization and ReLU processes. After each block, a maximum pooling downsampling module is set. Among them, the first downsampling module performs downsampling operations in both frequency and time dimensions, while the subsequent downsampling modules only downsample in the frequency dimension. Then, the frequency average pooling module FAP is used to remove the frequency dimension, integrate all frequency information in each time frame, and ensure that the extracted features Features extracted with waveform networks Consistent in time dimension.

[0053] The feature fusion module fuses the outputs of the spectrogram network and the waveform network. In this module, the spectrogram features and waveform features are combined to make full use of these two different feature information. The feature fusion module adopts a Transformer-based fusion strategy to measure the similarity of channel dimensions. Waveform features and spectral features are aligned in the time dimension, where Represents the number of channels, Represents the time dimension. and They are regarded as the key vector and query vector in the attention mechanism respectively. and The transposed dot product obtains the similarity matrix, and the attention matrix is ​​further obtained through Softmax. On the other hand, in order to obtain and Mixed features of , concatenating them in the channel dimension, get Then, the mixed features are obtained by projection multi-layer perceptron MLP (Projection Multi-Layer Perceptron) , and use it as the value vector. The dot product calculation with the attention matrix obtains the final channel weighted result This process can be expressed by the formula:

[0054]

[0055]

[0056] in, is the Softmax function. The traditional multi-head attention mechanism is used to divide the query vector, key vector, and value vector into "heads" with the same number of channels, that is, ,in Represents the number of heads. After refining the channel features using the attention matrix, global average pooling (GAP) is used to fuse all channel information in the network.

[0057] The global fusion feature extractor contains two multi-layer perceptron modules (MLPBlock) for extracting global features. The features extracted by the feature fusion module are subjected to temporal global average pooling (GAP) to remove the temporal dimension and reduce the feature dimension, thus making feature extraction based on full connection feasible.

[0058] For each MLPBlock, enter First, it is sent to layer normalization for channel normalization to speed up the training process.

[0059]

[0060] in, and Respectively represent input The mean and variance of . Set to , to prevent division by zero. and Represents the weights and biases in the affine transformation used to scale the output values.

[0061] Afterwards, The first layer MLP uses the GeLU nonlinear activation function, and the output of the second layer MLP is connected to the input in a residual connection. Add together to get the final output result. The complete process of MLBlock can be expressed as follows:

[0062]

[0063]

[0064] in, is a Gaussian random variable with zero mean and unit variance, and Represents probability.

[0065] Finally, a fully connected layer is used as the classification head to produce the final category output.

[0066] The underwater acoustic target recognition system based on multimodal joint characterization proposed in this application combines the advantages of both spectrogram and waveform data modalities, and can achieve better recognition accuracy with a small number of parameters.

[0067] The present application also provides a method for underwater acoustic target recognition based on multi-modal joint characterization, which is implemented based on the above system and includes:

[0068] (1) Preprocess the original underwater acoustic signal, including resampling and data set division.

[0069] (2) Construct an underwater acoustic target recognition system based on multimodal joint representation.

[0070] (3) Set training hyperparameters, loss function, and optimizer.

[0071] (4) Extract the Mel spectrum features of each original underwater acoustic signal waveform.

[0072] (5) The underwater acoustic waveforms and their corresponding Mel spectra in the training set are sent into the model in batches for training until convergence.

[0073] (6) The underwater acoustic waveforms and their corresponding Mel spectra in the test set are sent to the trained model in batches and the recognition results are output.

[0074] The specific steps of the underwater acoustic target recognition method based on multi-modal joint characterization are described below in combination with the accompanying drawings and experimental results.

[0075] Step 001, the present invention uses two publicly available ship radiated noise datasets (ShipsEar and DeepShip) to verify the performance of the method. First, the two datasets are resampled with a sampling frequency of 16kHz. Then the datasets are sliced ​​and divided into training sets and test sets. The present invention adopts two dataset division methods, which are as follows:

[0076] • Random segmentation: All original signals are segmented into 5-second segments with an overlap of 2.5 seconds, and randomly divided into training and test sets in a ratio of 1:9. In this segmentation method, the segments in the training and test sets can come from the same signal.

[0077] • Unrelated segmentation: First, all original signals are randomly divided into a ratio of 8:2 to generate training sets and test sets. Under this division method, the audio clips in the generated training set and test set do not belong to the same signal, which ensures that there is no overlap between the training set and the test set, which is closer to the actual application scenario.

[0078] Step 002: construct an underwater acoustic target recognition model based on multimodal joint representation. The recognition model includes a multimodal feature extractor, a feature fusion module and a fusion feature extractor. The multimodal feature extractor consists of two sub-networks, a spectrogram network and a waveform network, which are used to extract spectrogram features respectively. and waveform characteristics The two sub-networks are designed with appropriate convolution kernel size and downsampling step size to align the temporal features in the subsequent feature fusion. Table 1 shows the detailed parameter settings of the spectrogram and waveform sub-networks. Then, the aligned two features are and Input into the feature fusion module for feature integration. Inspired by the attention mechanism, we select and As query vector and key vector, and use their concatenated projection vector as value vector. The fused features are input into the fusion feature extractor to further extract deep features. The fusion feature extractor is composed of two multi-layer perceptron modules. Finally, a fully connected layer is used as the classification head to produce the final category output. Figure 1 This is a flow chart of the underwater acoustic target recognition method based on multi-modal joint representation.

[0079] Table 1 Detailed parameter settings of the spectrogram and waveform subnetworks

[0080]

[0081] Step 003, set the training hyperparameters, optimizer and loss function, and use the training set to train the model and update the model parameters. Specifically, use the AdamW optimizer, the initial training learning rate is 0.001, and the default weight decay is used. The training batch size is set to 64, and the number of training rounds is set to 100. In order to improve the convergence of the network, a step-by-step decay strategy is adopted for the learning rate. The learning rate is reduced by 0.1 times after the 30th and 60th rounds. Use cross entropy loss as the optimization objective function.

[0082] Step 004: extract the Mel spectrum of each underwater acoustic signal sample in the data set, set the number of Mel filters to 64, the number of STFT sampling points to 1024, and the Hamming window frame shift interval to 512.

[0083] Step 005, sending the underwater acoustic waveforms and their corresponding Mel spectra in the training set into the model in batches for iterative training, and updating the parameters until all training rounds are completed;

[0084] Step 006: Send the underwater acoustic waveforms and their corresponding Mel spectra in the test set to the trained model in batches, output the recognition results and evaluate the performance. Use the trained model to predict the category of the test set, use the overall accuracy (Acc) to quantitatively evaluate the recognition performance, and use the network parameters to evaluate the model complexity. The Acc formula is as follows:

[0085]

[0086] in, is the number of correctly classified fragments, is the total number of fragments.

[0087] The recognition results are shown in Tables 2-4, where Table 2 is the recognition performance comparison results of the random segmentation of the ShipsEa data set, Tables 3 and 4 are the recognition performance comparison results of the unrelated segmentation of the ShipsEar and DeepShip data sets, and HANet is the method of the present invention. Table 5 is a comparison of the parameter amounts of different recognition methods. The values ​​in bold indicate the best results. As can be seen from Table 2, compared with the waveform-based and spectrogram-based methods, the method of the present invention achieved the best recognition accuracy of 98.60%, which shows that combining the two data modalities of spectrogram and waveform is effective. Compared with other comparison algorithms, the method of the present invention achieved the highest recognition accuracy in Class B, and achieved competitive performance in the remaining 4 categories. As can be seen from Table 5, the parameter amount of the method of the present invention is the smallest among all methods, and is more than 4 times smaller than DaiNet, which has the second smallest parameter amount, which shows that the method of the present invention has achieved the best balance between recognition performance and model complexity.

[0088] Table 2 Comparison of recognition performance of ShipsEa dataset randomly split

[0089]

[0090] Table 3 Comparison of recognition performance of unrelated segmentation of ShipsEar and DeepShip datasets

[0091]

[0092] Table 4 Comparison of recognition performance of unrelated segmentation of ShipsEar and DeepShip datasets

[0093]

[0094] Table 5 Comparison of parameters of different recognition methods

[0095]

[0096] As can be seen from Tables 3-4, on the ShipsEar dataset, the proposed method shows the best performance compared with the waveform-based and spectrogram-based methods. In addition, the proposed network shows excellent performance on both categories B and C, and it also shows competitive performance on category E. In addition to the relatively small sample size of the ShipsEar dataset, the difficulty of unrelated segmentation of the dataset itself also makes all methods in the experiment have low accuracy on certain specific categories such as category A. This situation may be due to the fact that the representation of these categories in the dataset is relatively vague or their characteristics are not obvious enough, making it difficult for the model to accurately distinguish these categories. From 4-6, it can be found that for the DeepShip dataset, the proposed method still achieves the highest recognition accuracy, significantly surpassing the waveform-based and spectrogram-based methods. The proposed network achieves the highest recognition performance on both tankers and passenger ships. In addition, it can be observed that for the ShipsEar dataset, the performance of waveform-based methods (EnvNet and DaiNet) is generally lower than that of spectrogram-based methods. This is mainly due to the limited data volume of the ShipsEar dataset, and the spectrogram-based preprocessing step can reduce the data volume and noise, thereby reducing the risk of overfitting. On the contrary, for the DeepShip dataset, due to its large number of samples, the waveform-based method can also obtain sufficient samples to learn effective features. Therefore, in this case, the waveform-based method shows similar performance to the spectrogram-based method. Combined with the comparison of network parameter quantities shown in Table 5, it can be concluded that the method of the present invention achieves the best balance between recognition performance and network parameter quantity, which further proves the effectiveness of the method.

[0097] also, Figure 3 —4 uses the ROC (Receiver Operator Characteristics) curve and the corresponding AUC (Area Under the Curve) indicator to conduct a more comprehensive quantitative evaluation of different recognition methods. Ideally, the model hopes to be as close to the (0, 1) coordinates in the upper left corner as possible on the ROC curve. The AUC value measures the area under the ROC curve and provides a quantitative performance metric. The larger the AUC value, the better the performance of the model. As can be seen from the figure, for the DeepShip dataset with a large amount of data, the ROC curve and AUC of the method of the present invention perform best, which shows that the method of the present invention has good recognition ability when processing large-scale datasets. On ShipsEar with a small amount of data, the AUC indicator of the method of the present invention is slightly lower than that of ConvNext, but considering that the number of parameters used by ConvNext is 73 times that of the method of the present invention, the method of the present invention is more attractive in practical applications.

[0098] Figure 5 The confusion matrix of the method of the present invention on the ShipsEar dataset. It can be observed from the figure that the confusion matrix of the method of the present invention presents a clear diagonal pattern, which means that the model has high recognition accuracy and less misclassification. In order to further demonstrate the effectiveness of the proposed method in this chapter, the last layer feature output of the four methods on the ShipsEar and DeepShip datasets (based on spectrograms: ResNet18 and RegNet; based on waveforms: DaiNet; based on hybrid features: the method of the present invention) is visualized using t-SNE, as shown in the figure. Figure 6 and Figure 7 In general, the method of the present invention exhibits better clustering performance, with a larger inter-class distance and a smaller intra-class distance, which proves the effectiveness of the method of the present invention in feature extraction.

[0099] The present application may also provide a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0100] The user interface may include a display, a keyboard or a pointing device, such as a mouse, a trackball, a touch pad or a touch screen.

[0101] It is understood that the memory in the disclosed embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0102] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.

[0103] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.

[0104] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:

[0105] Execute the steps of the above method.

[0106] The above method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The above-disclosed methods, steps and logic block diagrams can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the above-disclosed method can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0107] It is understood that the embodiments described in the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0108] For software implementation, the technology of the present application can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0109] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. An underwater acoustic target recognition system based on multimodal joint representation, characterized in that: include: A multimodal feature extractor for extracting the spectrogram and waveform features of underwater acoustic signals; Feature fusion module, used to fuse the output of spectrum features and waveform features; A fusion feature extractor is used to extract the global features of the underwater acoustic signal; and The fully connected layer is used to output the final underwater acoustic target category.

2. The underwater acoustic target recognition system based on multimodal joint representation according to claim 1 is characterized in that: The multimodal feature extractor includes a waveform network and a spectrogram network, wherein: The waveform network is used to extract the waveform features of the underwater acoustic waveform, including 7 Conv1D-BN-ReLU blocks; The Conv1D-BN-ReLU block includes a one-dimensional convolution layer, a batch normalization, and a first activation function; The one-dimensional convolution layer of the first Conv1D-BN-ReLU block uses a convolution downsampling operation to reduce the size of the input underwater acoustic waveform; the one-dimensional convolution layers of the second to fifth Conv1D-BN-ReLU blocks use one-dimensional maximum pooling downsampling; The spectrogram network is used to take the Mel spectrum of the water acoustic waveform as input and extract the spectrogram features of the water acoustic waveform, including 5 Conv2D-BN-ReLU blocks and 1 frequency average pooling module; The Conv2D-BN-ReLU block includes a two-dimensional convolutional layer, a batch normalization, and a first activation function; The first Conv2D-BN-ReLU block performs downsampling operations in frequency and time dimensions; The 2nd to 5th Conv2D-BN-ReLU blocks only downsample in the frequency dimension.

3. The underwater acoustic target recognition system based on multimodal joint representation according to claim 2 is characterized in that: The convolution kernel size of the one-dimensional convolution layer of the Conv1D-BN-ReLU block is 9; the step size of the one-dimensional convolution layer of the first Conv1D-BN-ReLU block is 4; and the first activation function is ReLU.

4. The underwater acoustic target recognition system based on multimodal joint representation according to claim 2 is characterized in that: In the Mel spectrum of the underwater acoustic waveform, the number of Mel filters is 64, the number of STFT sampling points is 1024, and the Hamming window frame shift interval is 512.

5. The underwater acoustic target recognition system based on multimodal joint representation according to claim 1 is characterized in that: The feature fusion module adopts a Transformer-based fusion strategy; The feature fusion module uses the waveform features and the spectrogram features as the key vector and the query vector in the attention mechanism respectively; obtains the similarity matrix by the dot product of the transposed waveform features and the spectrogram features, and further obtains the attention matrix by Softmax; The waveform features and the spectrogram features are concatenated in the channel dimension, and the mixed features are obtained as the value vector by projecting the multi-layer perceptron; The channel weighted result is obtained by calculating the dot product of the value vector and the attention matrix, and then global average pooling is used to fuse all channel information.

6. The underwater acoustic target recognition system based on multimodal joint representation according to claim 1 is characterized in that: The fusion feature extractor includes two multi-layer perceptron modules and a third multi-layer perceptron; The multi-layer perceptron module includes layer normalization, a first multi-layer perceptron, a second activation function, and a second multi-layer perceptron; After the input data is layer normalized, the first multi-layer perceptron performs global feature extraction, and after being processed by the second activation function, the feature extraction of the second multi-layer perceptron is added to the input data in a residual connection manner.

7. The underwater acoustic target recognition system based on multi-modal joint representation according to claim 6 is characterized in that: The second activation function is a nonlinear activation function.

8. The underwater acoustic target recognition system based on multi-modal joint representation according to claim 1 is characterized in that: The training process of the system includes: Step 1: Preprocess the original underwater acoustic signal: resample the original underwater acoustic data set, slice the data set, and divide it into training set and test set; Step 2: Set training hyperparameters, loss function, and optimizer; Step 3: Extract the Mel spectrum features of each original underwater acoustic signal waveform in the data set; Step 4: Load the training set and send the underwater acoustic waveforms and their corresponding Mel spectra in the training set into the underwater acoustic target recognition system in batches for training until convergence.

9. The underwater acoustic target recognition system based on multi-modal joint representation according to claim 8 is characterized in that: The method of slicing and dividing the training set and the test set includes: slicing first and then dividing the data set and first dividing the data set and then slicing; During training, the AdamW optimizer was used, and the initial training learning rate was 0.

001. Batch training was adopted, and the underwater acoustic waveforms and their corresponding Mel spectra in the training set were input into the underwater acoustic target recognition system in batches. The training batch size was 64, and the number of training rounds was 100. The cross entropy loss was used as the optimization objective function.

10. A method for underwater acoustic target recognition based on multi-modal joint characterization, implemented based on the system described in any one of claims 1 to 9, comprising: The preprocessed underwater acoustic signal is input into the underwater acoustic target recognition system, and the recognition result is output.

Images