Underwater sound intelligent spectrum sensing method fusing multi-modal data and application

Abstract

The invention discloses an underwater acoustic intelligent spectrum sensing method fusing multi-modal data and application, and belongs to the technical field of ocean Internet of Things. The method comprises the following steps: firstly, constructing an underwater wireless sensor network model, obtaining a multi-modal database of underwater acoustic signals, and obtaining picture data in a frequency domain through wavelet transform after a system obtains time sequence data of the underwater acoustic signals; secondly, extracting high-quality features of the time series data through an unsupervised learning denoising auto-encoder; and finally, a Mu-YOLO perceptron of a parallel double BackBone fusion multi-modal technology is constructed, and one BackBone takes DSC module depth separable convolution as a main part to capture key time characteristics. And the other BackBone captures key spatial features by taking CSPDarknet as a main part, and realizes high-precision spectrum sensing by fusing diversified and multi-layer features of multi-modal data.

Description

Technical Field

[0001] The present invention relates to the field of marine Internet of Things technology, and in particular to an underwater acoustic intelligent spectrum sensing method and application integrating multi-modal data. Background Art

[0002] The Ocean Internet of Things (Ocean Internet of Things, OoT) has been embedded in the key research agenda of the global ocean field and is one of the key means to occupy the commanding heights of the ocean in the future. The ocean is closely related to human production and life. The Ocean Internet of Things, with the Internet of Things (IoT) technology as the core and relying on the rapid development of new technologies, is an integrated marine information network that integrates real-time monitoring, data collection, resource exploration and other technologies, and ultimately realizes a large-scale interconnected system. The deployment scale of the Ocean Internet of Things is extremely large, requiring the roadbed and unmanned ships, ships, submarines, buoys and other marine communication equipment to cooperate in information exchange, and to make comprehensive use of various communication network resources as much as possible to complete complex marine tasks, such as marine exploration, marine aquaculture, marine transportation, marine biotechnology, and maritime military defense.

[0003] Various marine devices are deployed in underwater environments. The communication between marine devices constitutes an underwater wireless sensor network, which can be used to monitor, collect and transmit various data of the marine Internet of Things. Due to the particularity of the underwater environment, the available spectrum resources are relatively small, and the increase in the number of marine devices has generated massive data, and the competition for spectrum resources is becoming increasingly fierce. Accurate and reliable spectrum sensing can effectively respond to the growing demand for data transmission and achieve highly reliable communication and spectrum sharing.

[0004] Mastering diverse, comprehensive, and real-time marine information is a prerequisite for the development of the marine Internet of Things, and communication spectrum resources are the support for realizing marine informatization. In the marine Internet of Things environment, multiple networks coexist and generate massive amounts of data. The competition for underwater acoustic spectrum among marine communication equipment is fierce, and it is challenging to seize limited underwater acoustic spectrum resources. Moreover, in the face of complex and changeable marine environments, there is also the problem of low utilization of underwater acoustic spectrum. Summary of the invention

[0005] In view of the shortcomings and deficiencies in the prior art, the present invention provides a method and application of underwater acoustic intelligent spectrum sensing by fusing multimodal data with diversified and multi-level features to achieve high-precision spectrum sensing, which can detect spectrum holes with high precision and effectively avoid interference with users occupying channels, and realize low false alarm probability and high detection probability under low transmission power.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: The underwater acoustic intelligent spectrum sensing method integrating multimodal data provided by the present invention comprises the following steps:

[0007] First, an underwater wireless sensor network model is constructed to obtain a multi-modal database of underwater acoustic signals: time series data and image data. After the system obtains the time series data of the underwater acoustic signal, it uses wavelet transform to obtain image data in the frequency domain.

[0008] Secondly, the high-quality features of time series data are extracted through unsupervised learning denoising autoencoder;

[0009] Finally, the features of time series data and image data are fused, and then a series of effective feature extractions at different levels are performed to achieve interaction between the two modalities, build a Mul-YOLO sensor that integrates multimodal technology, and parallel backbone elastic capture of multimodal underwater acoustic features;

[0010] The Mul-YOLO sensor is a YOLO with two parallel backbones. One BackBone mainly uses the DSC module deep separable convolution to capture key temporal features; the other BackBone mainly uses CSPDarknet to capture key spatial features. The constructed Mul-YOLO sensor can gradually mine the underlying structural information of the underwater acoustic signal, integrate the diversified and multi-level features of multimodal data, and achieve high-precision spectrum perception.

[0011] Preferably, the time series data is efficiently represented by DAE and then input into the Mul-YOLO network that integrates multimodal technology together with the image data, so that the data of the two modalities have consistent feature representation and scale, and the features of the corresponding levels are fused for modal interaction, building a bridge for the interaction between time series data and image data, and continuously exploring the intrinsic relationship between the two modalities, so that different modal data are mapped to similar spaces, which is conducive to the interaction and information transfer between modalities, and mining more comprehensive and multi-level information to improve the overall perception performance.

[0012] Preferably, if H0 indicates that the current frequency band is in a silent state, and H1 indicates that the marine communication equipment occupies the current frequency band, the binary hypothesis of underwater acoustic spectrum perception is:

[0013]

[0014] Among them, x(t) is the transmitted signal, on(t) is the ocean environment noise including turbulence noise and wind noise, and * represents convolution;

[0015] Spectrum sensing is essentially a binary classification problem. The underwater acoustic data is input into the binary classification network, and the detection probability (P d ) and false alarm probability (FalseAlarmProbability, P f ) two parameters to measure spectrum perception, as shown below:

[0016] P d =P(H1|H1)

[0017] P f =P(H1|H0)

[0018] If P d If the value is too low, it will be impossible to reliably identify whether there is a user in the underwater acoustic channel; f If it is too high, it will cause frequent false warnings.

[0019] Preferably, an underwater wireless sensor network model is constructed to obtain a multi-modal database of underwater acoustic signals. The specific steps are as follows:

[0020] The underwater acoustic channel is expressed as

[0021]

[0022] τ is the arrival time of the sound ray, A is the normalized amplitude, j represents the intrinsic sound ray reaching the receiving point, δ(t-τ j ) represents the time τ j An impulse at

[0023] Finally, the underwater acoustic channel terminal outputs the array signal S n The expression is as follows:

[0024] S n ={s1, s2, ..., s n}

[0025] n represents the sequence length n∈(1, 2, ..., N), N is the total sampling value of the signal, and S is obtained. n The real part of is:

[0026] R_S n =real(S n )={r_s1, r_s2,..., r_S n};

[0027] For time series data, the length of each sample is m, the label is y0 when the underwater acoustic spectrum is idle, and the label is y1 when the marine communication equipment occupies the underwater acoustic spectrum. The x-th sample feature R_S(x) can be expressed as:

[0028] R_S(x)=[r_S m+1 , r_s m+2 ,......,r_s 2m ]

[0029] The time series data set constructed with 2k samples can be expressed as:

[0030] TD={[R_S(1);y0],[R_S(2);y0],...,[R_S(k);y0]

[0031] [R'_S(1);y1],[R'_S(2);y1],...,[R'_S(k);y1]};

[0032] For image data, the channel output signal is transformed using a one-dimensional continuous wavelet transform to calculate the square

[0033] WT f (a,b)= <f(t),ψ a，b (t)>

[0034]

[0035] The inner product of the integrable function f(t) and the wavelet function ψ(t), a is the scale factor, b is the time shift factor, as shown below: ;

[0037] Obtaining the data matrix, dividing the data samples, and constructing a 2k sample data set can be expressed as:

[0038] P={[R_C(1);y0],[R_C(2);y0],……,[R_C(k);y0]

[0039] [R'_C(1);y1],[R'_C(2);y1],...,[R'_C(k);y1]};

[0040] Convert the features of each sample into a grayscale image to obtain an image dataset;

[0041] Based on the above steps, a multimodal database constructed by time series data and image data is obtained.

[0042] Preferably, the high-quality features of time series data are extracted by combining unsupervised learning DAE. The specific steps are as follows:

[0043] For time series data, the signal feature matrix is ​​input into the DAE network to extract more efficient features;

[0044] The DAE network injects randomness by adding noise to the time series data. The constructed autoencoder performs denoising operations after training and learning the data rules, thereby obtaining the original time series data before noise pollution, identifying the potential patterns and essential features in the underwater acoustic time series data, and obtaining robust features.

[0045] The noise added to the original time series data is a Gaussian distributed random number array with a mean of 0, a standard deviation of 1, and the same shape as the input data;

[0046] The DAE structure consists of two parts: encoding and decoding. DAE maps the noisy time series data to the hidden layer through the encoder, and then maps it to the output layer through the decoder. 1θ With y 2θ′ is a nonlinear activation function, θ is the parameterization of the weight bias matrix {w, b}, and the encoding and decoding process is as follows:

[0047] c=y 1θ (a) = s(ω1a+b1)

[0048] d=y 2θ′ (c) = s(ω2c+b2)

[0049] In the encoder, convolution activation function layers and maximum pooling layers appear alternately to enhance the network’s expressiveness. Two convolution activation function layers introduce nonlinear operations to learn more complex and abstract features, and two maximum pooling layers are used to retain the key features of time series data.

[0050] In the decoder, convolution activation function layers and upsampling layers appear cyclically. Three convolution activation function layers are responsible for representation, and two upsampling layers are responsible for amplifying the feature matrix to restore the original data.

[0051] Preferably, the image data characterization network framework - underwater acoustic image data module, the specific operation steps are as follows:

[0052] For the underwater acoustic image data module, the Backbone module in YOLOv5 is used as the main network architecture, and three effective feature layers are output. CBS is constructed by convolution, BN and SiLU activation functions to capture local feature information. The C3 module increases the depth and receptive field of the network and improves the ability of feature extraction.

[0053] The size of the original underwater acoustic image data is 1*200*40. After Stage 1, the effective feature size of the shallow layer is 256*50*10, the effective feature size of the deep layer is 512*25*5 after Stage 2, and the effective feature size of the deep layer is 1024*13*3 after Stage 3.

[0054] In the Stage 3 module, the feature size before inputting SPP is 1024*13*13, and the output is 1024*13*13. The SPP module adopts a pooling window with multiple steps and splices it with the original feature layer to retain the underwater acoustic frequency domain features at different scales to the maximum extent, effectively avoiding the problem of repeated feature extraction and enriching the feature expression capability.

[0055] Preferably, the time series data characterization network framework - underwater acoustic time series data module, the specific operation steps are as follows:

[0056] By utilizing the lightweight characteristics of the DSC module network, the multi-level detail features of underwater acoustic time series data can be captured while maintaining high performance, thus achieving efficient underwater acoustic spectrum perception.

[0057] The temporal representation network goes through three stages to obtain three effective feature layers. The input temporal shape is 1*200*40, the size of feat4 is 256*50*10, the size of feat5 is 512*25*5, and the size of feat6 is 1024*13*3;

[0058] In the network architecture for characterizing time series data, s1 indicates a step size of 1, and s2 indicates a step size of 2. The network is mainly constructed by two modules: the CBR module and the DSC module.

[0059] The CBR module consists of a convolution layer with a convolution kernel size of 3, a BN layer, and a ReLU6 activation function; the DSC module consists of DW and PW, which can effectively reduce the number of parameters;

[0060] If dh represents the height of the output feature, dw represents the width of the output feature, fh represents the height of the convolution kernel, fw represents the width of the convolution kernel, ci represents the number of input channels, and co represents the number of output channels, the standard convolution calculation amount is

[0061] flops=d h ×d w ×f h ×f w ×c i ×c o

[0062] The computational complexity of the DSC module is 1 / c of ​​that of the standard convolution. o +1 / (f h ×f w ) times, the amount of calculation is:

[0063] flops=d h ×d w ×f h ×f w ×c i +c i ×c o ×d h ×d w

[0064] Different from the CBR module, DW uses depthwise convolution to implement separate convolution for each channel, while PW uses ordinary convolution with a convolution kernel of 1 to adjust the channel;

[0065] The features after the DAE network are input into Stage4, and the shallow effective feature layer feat4 is obtained through the series connection of CBR module and DSC module structure; feat4 is represented by the stacked DSC module to obtain the effective feature layer feat5; and then the deep effective feature layer feat6 is obtained through Stage6;

[0066] Stacking multiple convolutional layers to obtain effective features at different levels helps with subsequent modal interaction, understands underwater acoustic data from multiple perspectives, enriches feature diversity, and makes the network more expressive.

[0067] Preferably, the modal interaction module has the following specific operation steps:

[0068] First, the features of time series data and image data are fused, and then passed through a series of feature extraction layers to achieve interaction between the two modalities;

[0069] In the modal interaction module, the significant information of different modes, feat1 and feat4, feat2 and feat5, and feat3 and feat6, are fused and then sent to the modal interaction network for continuous feature fusion and feature extraction, so as to improve the processing capability of underwater acoustic multimodal data, adapt to the diversified underwater acoustic data source information, and improve the sensor accuracy;

[0070] The upsampling layer is designed to carefully expand the size of the underwater acoustic feature map to enhance the modal interaction model's perception of the time and frequency domain features in the underwater acoustic signal, effectively improving the understanding of complex acoustic scenes. The Coordinate Attention (CA) mechanism is added to the modal interaction network to enhance the network's ability to extract features from underwater acoustic time and frequency domain data, and to more flexibly focus on the correlation between modalities.

[0071] CA embeds aggregated features in both horizontal and vertical directions into channel information, capturing spatial information while retaining the precise information of the target plane position, thereby generating a coordinate-aware feature map that is sensitive to both the target direction and position.

[0072] f 1×1 represents a 1×1 convolutional layer, σ represents the activation function, and the underwater acoustic feature tensor input to CA is f i ∈F c×h×w , first f i Input to the horizontal and vertical pooling layers (xavg, yavg) and spliced, and then passed through f 1×1 With σ, we get the channel coded output F, as follows:

[0073] F = δ{f 1×1 [xavg(f i ),yavg(f i)]}

[0074] r represents the compression rate. This feature is further decomposed to obtain the two-directional tensor F c / r×h 、F c / r×w , and then pass through f in their respective directions 1×1 After σ, the feature f with dimensions c×h×w is expanded o , the calculation formula is as follows:

[0075]

[0076] After the multimodal interaction is completed, the input is flattened and sent to the fully connected layer. The underwater acoustic channel classification is then realized through the softmax function. The loss function uses the cross entropy loss function:

[0077]

[0078] c represents the category, y is the true distribution, Predict distribution for hydroacoustic channels.

[0079] Preferably, after the time series data is efficiently represented by DAE, it is input into the Mul-YOLO network that integrates multimodal technology together with the image data, and the features of the corresponding levels are fused for modal interaction to build a bridge for the interaction between time series data and image data. A combination of multiple CBS, C3, UpSample, and CA modules is used to continuously explore the intrinsic relationship between the two modalities, so that different modal data are mapped to similar spaces.

[0080] The present invention provides an underwater acoustic intelligent spectrum sensing method integrating multi-modal data. It has the following beneficial effects:

[0081] (1) The method for underwater acoustic intelligent spectrum perception integrating multimodal data of the present invention, firstly, the system obtains the time series data of the underwater acoustic signal, and obtains the image data by wavelet transform. The time series data and the image data constitute a multimodal database, which covers a more comprehensive time and space perspective, and improves the complementarity and diversity of features; secondly, the unsupervised learning denoising autoencoder is combined to extract high-quality features of the time series data; finally, in order to make up for the fact that the traditional YOLO cannot absorb the features of multimodal data, a parallel dual-backbone YOLO is constructed, one backbone mainly uses the depthwise separable convolution to capture the key time features, and the other backbone mainly uses the CSPDarknet to capture the key spatial features. The constructed Mul-YOLO sensor can gradually mine the underlying hierarchical structure information of the underwater acoustic signal, integrate the diversified and multi-level features of multimodal data, and realize high-precision spectrum perception.

[0082] Compared with underwater acoustic spectrum sensing implemented by ConvNeXt, MobileViT, and SwinTransformer methods, the detection probability of the proposed scheme is increased by about 5%-14%, the false alarm probability is optimized by about 0.5%-6%, and low false alarm probability and high detection probability are achieved under low transmission power. Experimental results show that the proposed scheme can detect spectrum holes with high accuracy and effectively avoid interfering with users occupying the channel.

[0083] (2) The underwater acoustic time series data is obtained through an autoencoder to obtain high-quality features, and the time series data is transformed through wavelet transform to obtain image data in the frequency domain. The multimodal data composed of the time series data and the image data effectively avoids the problem of only considering a single feature and lacking feature diversity.

[0084] (3) To break through the limitation of traditional YOLO that cannot absorb multi-modal features, a dual-BackBone Mul-YOLO architecture is designed. The CBR module and depthwise separable convolution are used to accumulate multi-level key temporal features. Convolutional layers, CBS, C3 and spatial pyramid pooling are used to realize feature extraction and fusion to mine key spatial features. The parallel backbone flexibly captures multi-modal underwater acoustic features.

[0085] (4) In order to comprehensively utilize the advantages of each modality and better express the essence of the data, the modal interaction module first integrates the spatiotemporal features of different levels, and then constructs a multi-branch network through the C3, CBS, and UpSample modules to learn the spatiotemporal multi-scale characteristics. Driven by CA, it flexibly pays attention to the correlation between the two modalities, realizes aggregation operations, and ultimately improves the perception efficiency.

[0086] The ocean Internet involves a variety of data types, and the application of multimodal technology in the communication field to provide reliable data transmission has important research value. The application of multimodal technology condenses the correlation between perceived user data, reduces conflicts between each other, and improves underwater acoustic spectrum perception performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 This is a flow chart of underwater acoustic intelligent spectrum sensing that integrates multimodal data according to the present invention;

[0088] Figure 2 for Figure 1 Schematic diagram of the structure of the encoder;

[0089] Figure 3 for Figure 1 Schematic diagram of the structure of the decoder;

[0090] Figure 4 for Figure 1 The structural diagram of the network architecture for representing image data;

[0091] Figure 5for Figure 1 A schematic diagram of the structure of the network architecture for characterizing time series data;

[0092] Figure 6 for Figure 1 Schematic diagram of the structure of the mesomodal interaction network architecture;

[0093] Figure 7 It is the trend chart of the model convergence index change;

[0094] Figure 8 ROC curve 1 for ablation experiment;

[0095] Fig. 9 This is the ROC curve 2 of the ablation experiment;

[0096] Fig.10 is the relationship between PD and emitted sound power;

[0097] Fig.11 It is the relationship diagram between PF and emitted sound power;

[0098] Fig.12 This is the ROC curve when the transmitted sound power is 158.17dB;

[0099] Fig.13 This is the ROC curve when the transmitted sound power is 160.17dB;

[0100] Fig.14 The influence of wind speed on the underwater acoustic spectrum perception performance. DETAILED DESCRIPTION

[0101] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0102] Example 1

[0103] like Figure 1As shown, the underwater acoustic intelligent spectrum perception method for fusing multimodal data provided by the present invention includes the following steps: first, the system obtains the time series data of the underwater acoustic signal, and obtains the image data by wavelet transform. The time series data and the image data constitute multimodal data (Multi-modaldata). The multimodal data covers a more comprehensive time and space perspective, and improves the complementarity and diversity of features; secondly, the unsupervised learning denoising autoencoder is combined to extract high-quality features of the time series data; finally, in order to make up for the fact that the traditional YOLO cannot absorb the features of multimodal data, a parallel dual-backbone YOLO is constructed, one backbone mainly uses deep separable convolution to capture key time features, and the other backbone mainly uses CSPDarknet to capture key spatial features. The constructed Mul-YOLO sensor can gradually mine the underlying hierarchical structure information of the underwater acoustic signal, fuse the diversified and multi-level features of multimodal data, and realize high-precision spectrum perception.

[0104] If H0 indicates that the current frequency band is in a silent state, and H1 indicates that marine communication equipment occupies the current frequency band, the binary hypothesis of underwater acoustic spectrum perception is:

[0105]

[0106] Among them, x(t) is the transmitted signal, on(t) is the ocean environment noise including turbulence noise and wind noise, and * represents convolution;

[0107] Spectrum sensing is essentially a binary classification problem. The underwater acoustic data is input into the binary classification network, and the detection probability (P d ) and false alarm probability (FalseAlarmProbability, P f ) two parameters to measure spectrum perception, as shown below:

[0108] P d =P(H1|H1)

[0109] P f =P(H1|H0)

[0110] If P d If the value is too low, it will be impossible to reliably identify whether there is a user in the underwater acoustic channel; f Too high will result in frequent false alarms.

[0111] Example 2

[0112] Construct an underwater wireless sensor network model, generate time series data under the underwater acoustic channel, and generate image data through wavelet transform. The specific steps are as follows:

[0113] The underwater acoustic channel is expressed as

[0114]

[0115] τ is the arrival time of the sound ray, A is the normalized amplitude, j represents the intrinsic sound ray reaching the receiving point, δ(t-τ j ) represents the time τ j An impulse at

[0116] Finally, the underwater acoustic channel terminal outputs the array signal:

[0117] S n ={s1, s2, ..., s n}

[0118] n represents the sequence length n∈(1, 2, ..., N), N is the total sampling value of the signal, and S is obtained. n The real part of is:

[0119] R_S n =real(S n )={r_s1, r_s2,..., r_s n};

[0120] For time series data, the length of each sample is m, the label is y0 when the underwater acoustic spectrum is idle, and the label is y1 when the marine communication equipment occupies the underwater acoustic spectrum. The x-th sample feature R_S(x) can be expressed as:

[0121] R_S(x)=[r_s m+1 , r_s m+2 ,......,r_s 2m ]

[0122] The time series data set constructed with 2k samples can be expressed as:

[0123] TD={[R_S(1);y0],[R_S(2);y0],...,[R_S(k);y0]

[0124] [R'_S(1);y1], [R'_S(2);y1], ..., [R'_S(k);y1]};

[0125] For image data, the channel output signal is transformed using a one-dimensional continuous wavelet transform to calculate the inner product of the square integrable function f(t) and the wavelet function ψ(t), where a is the scale factor and b is the time shift factor, as shown in the following formula:

[0126] WT f (a, b) = <f(t),ψ a，b (t)>

[0127]

[0128] Obtaining the data matrix, dividing the data samples, and constructing a 2k sample data set can be expressed as:

[0129] P={[R_C(1);y0],[R_C(2);y0],...,[R_C(k);y0]

[0130] [R'_C(1);y1], [R'_C(2);y1], ..., [R'_C(k);y1]};

[0131] The features of each sample are converted into a grayscale image to obtain an image dataset.

[0132] The multimodal database constructed by time series data and image data obtained based on the above steps covers rich and diverse data information.

[0133] Example 3

[0134] Combined with unsupervised learning DAE to extract high-quality features of time series data, the specific steps are as follows:

[0135] For time series data, the signal feature matrix is ​​input into the DAE network to extract more efficient features. The DAE network injects randomness into the time series data by adding noise. The constructed autoencoder performs denoising operations after training and learning the data rules, thereby obtaining the original time series data before noise pollution, identifying the potential patterns and essential features in the underwater acoustic time series data, and obtaining robust features.

[0136] The noise added to the original time series data is a Gaussian distributed random number array with a mean of 0, a standard deviation of 1, and the same shape as the input data;

[0137] The DAE structure consists of two parts: encoding and decoding. Figure 2 and Figure 3 As shown in Figure 1, DAE maps the noisy time series data to the hidden layer through the encoder, and then maps it to the output layer through the decoder; 1θ With y 2θ is a nonlinear activation function, θ is the parameterization of the weight bias matrix {w, b}, and the encoding and decoding process is as follows:

[0138] c=y 1θ (a) = s(ω1a+b1)

[0139] d=y 2θ′ (c) = s(ω2c+b2)

[0140] In the encoder, convolution activation function layers and maximum pooling layers appear alternately to enhance the network’s expressiveness. Two convolution activation function layers introduce nonlinear operations to learn more complex and abstract features, and two maximum pooling layers are used to retain the key features of time series data.

[0141] In the decoder, convolution activation function layers and upsampling layers appear cyclically. Three convolution activation function layers are responsible for representation, and two upsampling layers are responsible for amplifying the feature matrix to restore the original data.

[0142] Example 4

[0143] Image data representation network framework. For underwater acoustic image data, this module uses the Backbone module in YOLOv5 as the main framework and outputs three effective feature layers. The network architecture is as follows Figure 4 As shown in the figure, CBS is constructed by convolution, BN and SiLU activation functions to capture local feature information. The C3 module increases the depth and receptive field of the network and improves the ability of feature extraction.

[0144] The size of the original underwater acoustic image data is 1*200*40. After Stagel, the effective feature size of the shallow layer is 256*50*10. After Stage2, the effective feature size is 512*25*5. After Stage3, the effective feature size of the deep layer is 1024*13*3.

[0145] In the Stage 3 module, the feature size before inputting SPP is 1024*13*13, and the output is 1024*13*13. The SPP module adopts a pooling window with multiple steps and splices it with the original feature layer to retain the underwater acoustic frequency domain features at different scales to the maximum extent, effectively avoiding the problem of repeated feature extraction and enriching the feature expression capability.

[0146] Example 5

[0147] The time series data representation network framework uses the lightweight characteristics of the DSC module network to capture the multi-level detailed features of underwater acoustic time series data while maintaining high performance, thus achieving efficient underwater acoustic spectrum perception;

[0148] The temporal representation network goes through three stages to obtain three effective feature layers. The input temporal shape is 1*200*40, the size of feat4 is 256*50*10, the size of feat5 is 512*25*5, and the size of feat6 is 1024*13*3;

[0149] The network architecture of time series data representation is as follows Figure 5 As shown, s1 represents a step size of 1, and s2 represents a step size of 2. The network is mainly constructed by two modules: the CBR module and the DSC module;

[0150] The CBR module consists of a convolution layer with a convolution kernel size of 3, a BN layer, and a ReLU6 activation function; the DSC module consists of DW and PW, which can effectively reduce the number of parameters;

[0151] If dh represents the height of the output feature, dw represents the width of the output feature, fh represents the height of the convolution kernel, fw represents the width of the convolution kernel, ci represents the number of input channels, and co represents the number of output channels, the standard convolution calculation amount is

[0152] flops=dh×dw×fh×fw×ci×co

[0153] The computational complexity of the DSC module is 1 / co+1 / (fh×fw) times that of the standard convolution, and the computational complexity is:

[0154] flops=dh×dw×fh×fw×ci+ci×co×dh×dw

[0155] Different from the CBR module, DW uses depthwise convolution to implement separate convolution for each channel, while PW uses ordinary convolution with a convolution kernel of 1 to adjust the channel;

[0156] The features after the DAE network are input into Stage4, and the shallow effective feature layer feat4 is obtained through the series connection of CBR module and DSC module structure; feat4 is represented by the stacked DSC module to obtain the effective feature layer feat5; and then the deep effective feature layer feat6 is obtained through Stage6;

[0157] Stacking multiple convolutional layers to obtain effective features at different levels helps with subsequent modal interaction, understands underwater acoustic data from multiple perspectives, enriches feature diversity, and makes the network more expressive.

[0158] Example 6

[0159] Modal interaction network architecture, such as Figure 6 As shown in the figure, the module first fuses the features of time series data with the image data, and then goes through a series of feature extraction layers to achieve interaction between the two modalities;

[0160] In the modal interaction module, the significant information of different modes, feat1 and feat4, feat2 and feat5, and feat3 and feat6, are fused and then sent to the modal interaction network for continuous feature fusion and feature extraction, so as to improve the processing capability of underwater acoustic multimodal data, adapt to the diversified underwater acoustic data source information, and improve the sensor accuracy;

[0161] The upsampling layer is designed to carefully expand the size of the underwater acoustic feature map to enhance the modal interaction model's perception of the time domain and frequency domain features in the underwater acoustic signal, effectively improving the understanding of complex acoustic scenes; CA is added to the modal interaction network to enhance the network's feature extraction capabilities for underwater acoustic time domain data and frequency domain data, and to more flexibly focus on the correlation between modalities; unlike other attention mechanisms, CA embeds aggregated features in both horizontal and vertical directions into channel information, capturing spatial information while retaining the precise information of the target plane position, thereby generating a coordinate-aware feature map that is sensitive to both target direction and position.

[0162] f 1×1 represents a 1×1 convolutional layer, σ represents the activation function, and the underwater acoustic feature tensor input to CA is f i ∈F c×h×w , first f i Input to the horizontal and vertical pooling layers (xavg, yavg) and spliced, and then passed through f 1×1 With σ, the calculation formula of the output F after channel coding is as follows:

[0163] F = δ{f 1×1 [xavg(f i ),yavg(f i )]}

[0164] r represents the compression rate. This feature is further decomposed to obtain the two-directional tensor F c / r×h 、F c / r×w , and then pass through f in their respective directions 1×1 After σ, the feature f with dimensions c×h×w is expanded o , the calculation formula is as follows:

[0165]

[0166] After the multimodal interaction is completed, the input is flattened to the fully connected layer, and then the underwater acoustic channel classification is realized through the softmax function. The loss function adopts the cross entropy loss function.

[0167]

[0168] c represents the category, y is the true distribution, Predict distribution for hydroacoustic channels.

[0169] Experimental analysis

[0170] The equipment models used in this experiment are shown in Table 1.

[0171] Table 1 Equipment Model

[0172]

[0173]

[0174] The Adam optimizer was used to train the network, with a learning rate of 0.00001, a batch_size of 32, and a range of 156.17dB-168.17dB with a spacing of 2dB. 1966 time series data samples and 1966 image data samples were taken for each emission sound power, with a training set: validation set: test set = 8:1:1.

[0175] Figure 7 The effect of the number of trainings on the performance of underwater acoustic spectrum perception is shown. The convergence of the model gradually stabilizes with the increase of the number of trainings. Considering the computing resources, and the fact that the accuracy and loss values ​​of the training set and the validation set are stable when Epoch is 7, the model performance has not been greatly improved, and the number of trainings is determined to be 7.

[0176] (1) Ablation experiment

[0177] In order to verify the influence of multimodal technology on underwater acoustic spectrum perception, ablation experiments were conducted, MobileNetV1 for unimodal time series data features, and CSPDarknet for unimodal image data features. The accuracy of underwater acoustic spectrum perception under different transmission sound powers is shown in Table 2. The application of multimodal technology can improve the accuracy of underwater acoustic spectrum perception. The lower the transmission sound power, the more obvious the multimodal advantage. When the transmission sound power is 156.17dB, the underwater acoustic spectrum perception accuracy of multimodal data is 86.8%, which is 23.4% and 8.5% higher than that of unimodal time series and image data, respectively.

[0178] Table 2 Ablation experiment results

[0179]

[0180] When the transmitted sound power is 158.17dB and 160.17dB, the ROC curves are as follows: Figure 8 and Fig. 9 As shown in the figure, the spectrum sensing of underwater acoustic frequency domain signals is better than that of time domain signals. It can be seen intuitively that the proposed scheme has the advantage of applying multimodal technology. Taking into account the multimodal characteristics, the spectrum sensing performance is effectively improved compared with single-modal spectrum sensing.

[0181] (2) Comparative experiment

[0182] ConvNeXt, MobileViT and SwinTransformer are used to implement the underwater acoustic spectrum perception comparison test. Fig.10, the lower the transmission sound power, the more obvious the detection probability is, which is more conducive to finding idle channels, timely discovering and utilizing available spectrum resources, and improving spectrum utilization efficiency. Referring to the two ablation experiments, the detection probability of Mul-YOLO is better than the detection probability under single mode; referring to the three comparative experiments, the detection probability of Mul-YOLO has obvious advantages. When the transmission power is 156.17dB, the detection probability is 80%. Compared with other schemes, the detection probability of the proposed scheme is improved by about 5%-14%.

[0183] Low false alarm probability is of great significance in reducing channel preemption and minimizing interference to other users. Fig.11 , the increase in channel quality is accompanied by a decrease in the false alarm probability. The lower the transmitted sound power, the more significant the improvement effect of Mul-YOLO. When the transmit power is 160.17dB, the false alarm probability of Mul-YOLO and the ablation experiment CSPDarknet is 3%. From the perspective of AUC value, the Mul-YOLO scheme is 0.15 higher than CSPDarknet. When the transmit power is 156.17dB, the false alarm probability of Mul-YOLO is only 8%, the false alarm probability of ConvNeXt is 15%, the false alarm probability of MobileViT is 47%, and the false alarm probability of SwinTransformer is 38%. Compared with other algorithms, when the transmit sound power is between 156.17dB-166.17dB, the false alarm probability of Mul-YOLO is optimized by about 0.5%-6%. The multimodal technology and network architecture in the proposed scheme play a key role in realizing accurate underwater acoustic spectrum perception in complex environments.

[0184] Fig.12 and Fig.13 The ROC curves are for the transmitted acoustic power of 158dB and 160dB respectively. The AUC value is the area enclosed by the curve and the horizontal axis. The AUC value is proportional to the ability of the sensor to determine whether there is a user in the underwater acoustic channel. Compared with other algorithms, the ROC curve of the Mul-YOLO scheme is closer to the upper left corner, and the performance of the underwater acoustic spectrum sensor is better. When the transmitted acoustic power is 158.17dB and the false positive rate is 0.2, the true positive rate is Mul-YOLO>MobileViT>SwinTransformer>ConvNeXt from high to low. The true positive rate of Mul-YOLO is as high as 99%, and the true positive rate of MobileViT is about 93%, with a difference of about 6%. When the transmitted acoustic power is 160.17dB, the ROC curves of all schemes are closer to the upper left corner, and the AUC value is larger. In the ablation experiment, the application of multimodal technology improves the performance of the underwater acoustic spectrum sensor. In the comparative experiment, the constructed underwater acoustic spectrum sensor can capture multi-level features and has a higher detection accuracy. Simulation results show that the proposed scheme has significant advantages.

[0185] Fig.14 The influence of wind speed on the performance of underwater acoustic spectrum perception was explored, and the transmitted acoustic power was constant at 162.17dB. As the wind speed increases, the ROC curve of underwater acoustic spectrum perception gradually moves away from the upper left corner, the AUC value gradually decreases, and the performance of spectrum perception gradually deteriorates. Increased wind speed will increase the ocean environmental noise, which will in turn affect the signal-to-noise ratio. Therefore, under the same transmitted acoustic power, the greater the wind speed, the worse the underwater acoustic spectrum perception performance.

[0186] Compared with underwater acoustic spectrum sensing implemented by ConvNeXt, MobileViT, and SwinTransformer methods, the detection probability of the proposed scheme is increased by about 5%-14%, the false alarm probability is optimized by about 0.5%-6%, and low false alarm probability and high detection probability are achieved under low transmission power. Experimental results show that the proposed scheme can detect spectrum holes with high accuracy and effectively avoid interfering with users occupying the channel.

[0187] The marine Internet involves a variety of data types. The multimodal technology of the present invention is applied to the communication field to provide reliable data transmission, which has important research value. The application of multimodal technology condenses the correlation between perceived user data, reduces conflicts between each other, and improves the underwater acoustic spectrum perception performance.

[0188] In summary, the underwater acoustic intelligent spectrum sensing method integrating multimodal data of the present invention adopts multimodal technology to understand the instantaneous and periodic characteristics of complex underwater acoustic signals from a comprehensive perspective. The constructed Mul-YOLO sensor can learn the structural characteristics of underwater acoustic signals from multiple angles, integrate the diversified and multi-level characteristics of multimodal data, and realize high-precision spectrum sensing. It can detect spectrum holes with high precision and effectively avoid interfering with users occupying channels. Compared with other schemes in the prior art, the method of the present invention explores the nonlinear relationship of complex underwater acoustic signals, hierarchically learns the abstract characteristics of underwater acoustic signals, and achieves low false alarm probability and high detection probability under low transmission power.

[0189] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An underwater acoustic intelligent spectrum sensing method integrating multimodal data, characterized in that: The steps include: First, an underwater wireless sensor network model is constructed to obtain a multi-modal database of underwater acoustic signals: time series data and image data. After the system obtains the time series data of the underwater acoustic signal, it uses wavelet transform to obtain image data in the frequency domain. Secondly, the high-quality features of time series data are extracted through unsupervised learning denoising autoencoder; Finally, the features of time series data and image data are fused, and then a series of effective feature extractions at different levels are performed to achieve interaction between the two modes, build a Mul-YOLO sensor that integrates multimodal technology, and parallel dual-trunk elastic capture of multimodal underwater acoustic features; The Mul-YOLO sensor is a YOLO with two parallel backbones, one of which is based on the DSC module deep separable convolution to capture key temporal features; Another BackBone mainly uses CSPDarknet to capture key spatial features. The constructed Mul-YOLO sensor can gradually mine the underlying structural information of the underwater acoustic signal, integrate the diversified and multi-level features of multimodal data, and achieve high-precision spectrum perception.

2. According to claim 1, the method for underwater acoustic intelligent spectrum perception integrating multimodal data is characterized in that: After being efficiently represented by DAE, the time series data is input into the Mul-YOLO network that integrates multimodal technology together with the image data, so that the data of the two modalities have consistent feature representation and scale. The features of the corresponding levels are fused for modal interaction, building a bridge for the interaction between time series data and image data, and continuously exploring the intrinsic relationship between the two modalities. Different modal data are mapped to similar spaces, which is conducive to interaction and information transfer between modalities, and mining more comprehensive and multi-level information to improve the overall perception performance.

3. The underwater acoustic intelligent spectrum sensing method for fusing multimodal data according to claim 1 is characterized in that: If H0 means the current frequency band is silent, and H1 means the ocean communication The signal equipment occupies the current frequency band, and the binary assumption of underwater acoustic spectrum perception is: Among them, x(t) is the transmitted signal, on(t) is the ocean environment noise including turbulence noise and wind noise, and * represents convolution; Spectrum sensing is essentially a binary classification problem. The underwater acoustic data is input into the binary classification network. The detection probability (P d ) and false alarm probability (FalseAlarmProbability, P f ) two parameters to measure spectrum perception, as shown below: P d =P(H1|H1) P f =P(H1|H0) If P d If the value is too low, it will be impossible to reliably identify whether there is a user in the underwater acoustic channel; f If it is too high, it will cause frequent false warnings.

4. The underwater acoustic intelligent spectrum sensing method for fusing multimodal data according to claim 1 is characterized in that: Construct an underwater wireless sensor network model and obtain a multi-modal database of underwater acoustic signals. The specific steps are as follows: The underwater acoustic channel is expressed as τ is the arrival time of the sound ray, A is the normalized amplitude, j represents the intrinsic sound ray reaching the receiving point, δ(t-τ j ) represents the time τ j An impulse at The final expression of the underwater acoustic channel terminal output array signal Sn is as follows: S n ={s1,s2,……,s n } n represents the sequence length n∈(1,2,……,N), N is the total sampling value of the signal, and S is obtained. n The real part of is: R_S n =real(S n )={r_s1,r_s2,……,r_s n }; For time series data, the length of each sample is m, the label is y0 when the underwater acoustic spectrum is idle, and the label is y1 when the marine communication equipment occupies the underwater acoustic spectrum. The x-th sample feature R_S(x) can be expressed as: R_S(x)=[r_s m+1 ,r_s m+2 ,……,r_s 2m ] The time series data set constructed with 2k samples can be expressed as: TD={[R_S(1);y0],[R_S(2);y0],...,[R_S(k);y0] [R'_S(1);y1],[R'_S(2);y1],...,[R'_S(k);y1]}; For image data, the channel output signal is transformed using a one-dimensional continuous wavelet transform to calculate the inner product of the square integrable function f(t) and the wavelet function ψ(t), where a is the scale factor and b is the time shift factor, as shown in the following formula: WT f (a,b)=<f(t),ψ a , b (t)〉 Obtaining the data matrix, dividing the data samples, and constructing a 2k sample data set can be expressed as: P={[R_C(1);y0],[R_C(2);y0],……,[R_C(k);y0] [R'_C(1);y1],[R'_C(2);y1],...,[R'_C(k);y1]}; Convert the features of each sample into a grayscale image to obtain an image dataset; Based on the above steps, a multimodal database constructed by time series data and image data is obtained.

5. The underwater acoustic intelligent spectrum sensing method for fusing multimodal data according to claim 2 is characterized in that: Combined with unsupervised learning DAE to extract high-quality features of time series data, the specific steps are as follows: For time series data, the signal feature matrix is ​​input into the DAE network to extract more efficient features; The DAE network injects randomness by adding noise to the time series data. The constructed autoencoder performs denoising operations after training and learning the data rules, thereby obtaining the original time series data before noise pollution, identifying the potential patterns and essential features in the underwater acoustic time series data, and obtaining robust features. The noise added to the original time series data is a Gaussian distributed random number array with a mean of 0, a standard deviation of 1, and the same shape as the input data; The DAE structure consists of two parts: encoding and decoding. DAE maps the noisy time series data to the hidden layer through the encoder, and then maps it to the output layer through the decoder. 1θ With y 2θ′ is a nonlinear activation function, θ is the parameterization of the weight bias matrix {w, b}, and the encoding and decoding process is as follows: c=y 1θ (a)=s(ω1a+b1) d=y 2θ′ (c)=s(ω2c+b2) In the encoder, convolution activation function layers and maximum pooling layers appear alternately to enhance the network’s expressiveness. Two convolution activation function layers introduce nonlinear operations to learn more complex and abstract features, and two maximum pooling layers are used to retain the key features of time series data. In the decoder, convolution activation function layers and upsampling layers appear cyclically. Three convolution activation function layers are responsible for representation, and two upsampling layers are responsible for amplifying the feature matrix to restore the original data.

6. The underwater acoustic intelligent spectrum sensing method for fusing multimodal data according to claim 1 is characterized in that: The network framework for underwater acoustic image data representation is as follows: For the underwater acoustic image data module, the Backbone module in YOLOv5 is used as the main network architecture, and three effective feature layers are output. CBS is constructed by convolution, BN and SiLU activation functions to capture local feature information. The C3 module increases the depth and receptive field of the network and improves the ability of feature extraction. The size of the original underwater acoustic image data is 1*200*40. After Stage 1, the effective feature size of the shallow layer is 256*50*10, the effective feature size of the deep layer is 512*25*5 after Stage 2, and the effective feature size of the deep layer is 1024*13*3 after Stage 3. In the Stage 3 module, the feature size before inputting SPP is 1024*13*13, and the output is 1024*13*13. The SPP module adopts a pooling window with multiple steps and splices it with the original feature layer to retain the underwater acoustic frequency domain features at different scales to the maximum extent, effectively avoiding the problem of repeated feature extraction and enriching the feature expression capability.

7. The underwater acoustic intelligent spectrum sensing method for fusing multimodal data according to claim 6 is characterized in that: The network framework for underwater acoustic time series data characterization is as follows: By utilizing the lightweight characteristics of the DSC module network, the multi-level detail features of underwater acoustic time series data can be captured while maintaining high performance, thus achieving efficient underwater acoustic spectrum perception. The temporal representation network goes through three stages to obtain three effective feature layers. The input temporal shape is 1*200*40, the size of feat4 is 256*50*10, the size of feat5 is 512*25*5, and the size of feat6 is 1024*13*3; In the time series data representation network architecture, s1 indicates a step size of 1, and s2 indicates a step size of 2. The network is mainly constructed by two modules: the CBR module and the DSC module. The CBR module consists of a convolution layer with a convolution kernel size of 3, a BN layer, and a ReLU6 activation function; the DSC module consists of DW and PW, which can effectively reduce the number of parameters; If d h Represents the height of the output feature, d w Indicates the width of the output feature, f h The height of the convolution kernel, f w The width of the convolution kernel, c i Number of input channels, c o The number of output channels and the amount of standard convolution calculation are flops=d h ×d w ×f h ×f w ×c i ×c o The computational effort of the DSC module is 1 / c of ​​that of the standard convolution. o +1 / (f h ×f w ) times, the amount of calculation is: flops=d h ×d w ×f h ×f w ×c i +c i ×c o ×d h ×d w Different from the CBR module, DW uses depthwise convolution to implement separate convolution for each channel, while PW uses ordinary convolution with a convolution kernel of 1 to adjust the channel; The features after the DAE network are input into Stage4, and the shallow effective feature layer feat4 is obtained through the series connection of CBR module and DSC module structure; feat4 is represented by the stacked DSC module to obtain the effective feature layer feat5; and then the deep effective feature layer feat6 is obtained through Stage6; Stacking multiple convolutional layers to obtain effective features at different levels helps with subsequent modal interaction, understands underwater acoustic data from multiple perspectives, enriches feature diversity, and makes the network more expressive.

8. The underwater acoustic intelligent spectrum sensing method for fusing multimodal data according to claim 7 is characterized in that: Modal interaction module, the specific operation steps are as follows: First, the features of time series data and image data are fused, and then passed through a series of feature extraction layers to achieve interaction between the two modalities; In the modal interaction module, the significant information of different modes, feat1 and feat4, feat2 and feat5, and feat3 and feat6, are fused and then sent to the modal interaction network for continuous feature fusion and feature extraction, so as to improve the processing capability of underwater acoustic multimodal data, adapt to the diversified underwater acoustic data source information, and improve the sensor accuracy; The upsampling layer is designed to carefully expand the size of the underwater acoustic feature map to enhance the modal interaction model's perception of the time and frequency domain features in the underwater acoustic signal, effectively improving the understanding of complex acoustic scenes; The coordinate attention mechanism (CA) is added to the modal interaction network to enhance the network's ability to extract features from underwater acoustic time domain data and frequency domain data, and to more flexibly focus on the correlation between modalities. CA embeds aggregated features in both horizontal and vertical directions into channel information, capturing spatial information while retaining the precise information of the target plane position, thereby generating a coordinate-aware feature map that is sensitive to both the target direction and position. f 1×1 represents a 1×1 convolutional layer, σ represents the activation function, and the underwater acoustic feature tensor input to CA is f i ∈F c×h×w , first f i Input to the horizontal and vertical pooling layers (xavg, yavg) and spliced, and then passed through f 1×1 With σ, we get the channel coded output F, as follows: F=δ{f 1×1 [xavg(f i ),yavg(f i )]} r represents the compression rate. This feature is further decomposed to obtain the two-directional tensor F c / r×h 、F c / r×w , and then pass through f in their respective directions 1×1 After σ, the feature f with dimensions c×h×w is expanded o , the calculation formula is as follows: After the multimodal interaction is completed, the input is flattened and sent to the fully connected layer. The underwater acoustic channel classification is then realized through the softmax function. The loss function uses the cross entropy loss function: c represents the category, y is the true distribution, Predict distribution for the hydroacoustic channel.

9. The underwater acoustic intelligent spectrum sensing method for fusing multimodal data according to claim 2 is characterized in that: After the time series data is efficiently represented by DAE, it is input into the Mul-YOLO network that integrates multimodal technology together with the image data. The features of the corresponding levels are fused for modal interaction to build a bridge for the interaction between time series data and image data. A combination of multiple CBS, C3, UpSample, and CA modules is used to continuously explore the intrinsic relationship between the two modalities, so that different modal data are mapped to similar spaces.

10. Application of an underwater acoustic intelligent spectrum sensing method integrating multimodal data according to any one of claims 1 to 9, characterized in that: Used in the field of communications.

Images