Method for training a mathematical model to generate artificial sonar signals

Abstract

A method (20) for training a mathematical model (40, 40', 40") to generate artificial sonar signals (34') is disclosed, comprising the following steps: a) Providing the mathematical model (40, 40', 40") comprising a cascade - an audio encoder (42) designed to generate features from audio signals (34, 52), - an adaptive instance normalization module (50) configured to use the features of two audio signals (48, 52') to obtain features of a resulting audio signal (48`) that exhibits the characteristic features of both audio signals (34, 52); and - an audio decoder (44) configured to convert the features (48') of the resulting audio signal into the resulting audio signal (34'); b) Providing a database of training data pairs, each comprising an underwater audio signal (52) and either a target signal (34) generated based on physical properties of a target or a target reference signal; c) Training the model (40, 40', 40"), with the training data pairs from the database, to obtain artificial sonar signals as the resulting audio signal (34').

Description

[0001] The invention relates to improved sonar data generation, for example for the training of sonar operators or the automatic classification or identification of underwater targets.

[0002] Underwater acoustic target identification (UATR) aims to classify or identify unknown vessels based on their unique acoustic signatures. These signatures are primarily characterized by frequency components generated by the vessel's propulsion systems, hull features, and other ambient noise. Deep learning has demonstrated significant potential in UATR tasks, but its effectiveness depends on the availability of large and diverse datasets.

[0003] While existing datasets like ShipsEar and DeepShip provide extensive recordings of ship noise across numerous vessels, they primarily capture the variability between ships. However, they lack sufficient variability within a single ship, meaning that recordings of individual ships under different operating conditions (e.g., varying speeds, fuel levels, and environmental influences) are underrepresented. This data scarcity limits the ability of deep learning models to generalize effectively, thus reducing their robustness in real-world applications. A similar problem exists in the training of sonar operators, which also cannot train on a sufficient number of diverse recordings of the same target.

[0004] The object of the present invention is therefore to create an improved concept for sonar data generation.

[0005] The problem is solved by the subject matter of the independent patent claims. Further advantageous embodiments are the subject matter of the dependent patent claims.

[0006] Exemplary implementations show a method for training a mathematical model to generate artificial sonar signals with the following steps a) to c).

[0007] a) Provide the mathematical model with a cascade comprising an audio encoder, an Adaptive Instance Normalization (AdalN) module, and an audio decoder. That is, the model to be trained has this architecture. The audio encoder generates features from audio signals. The features can be (especially random) physical features, such as certain significant frequency patterns, but especially features intangible to humans that nevertheless represent the audio signal well. The Adaptive Instance Normalization module uses the features of two audio signals to obtain features of a resulting audio signal that exhibits the characteristic features of both audio signals. That is, the features of two audio signals form the input values, and features of a resulting audio signal form the output values. Advantageously, the number of input values ​​per audio signal is equal to the number of output values.The audio decoder can convert the characteristics of the resulting audio signal into the final audio signal. The resulting audio signal of the trained model advantageously combines the details, i.e., the dominant frequencies, of the first audio signal with the structure, i.e., the timbre, of the second audio signal. The first audio signal thus exhibits, in particular, a line spectrum. The second audio signal is, in particular, more broadband than the first audio signal. The first audio signal can also be referred to as the "content signal." The second audio signal can also be referred to as the "style signal."

[0008] Such a mathematical model can also be called a variational autoencoder (VAE) with adaptive instance normalization. An example of a variational autoencoder can be found in N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, "Soundstream: An end-to-end neural audio codec." An example of instance normalization can be found in X. Huang and S. Belongie, "Arbitrary style transfer in real-time with adaptive instance normalization." The codebook described below yields a vector-quantized variational autoencoder (VQ-VAE).

[0009] The audio encoder and decoder can have a cascade of residual layers, with each block capable of changing the output dimensions by a factor of, for example, two. This small, incremental dimensional change ensures smoother scaling up and down and reduces the impact of aliasing artifacts.

[0010] The target signals and the underwater audio signals can be fed to the encoder as one-dimensional raw waveforms, i.e., time series. Advantageously, a decoupled encoder is used, which maps the input separately to compressions of varying sizes. It has been found that nested encoder structures allow insufficient information flow to the deeper layers, as the early encoders tend to overpass quickly. In addition to the model level described so far, the model can have further hierarchical levels. These hierarchical levels are given the index i. This results in a hierarchical, optionally vector-quantized, variational autoencoder.

[0011] The mathematical model can thus have at least one further hierarchy level, arranged parallel to the audio encoder and the adaptive instance normalization module. This further hierarchy level comprises a cascade consisting of another audio encoder, another adaptive instance normalization module, and another audio decoder. The further audio encoder can generate features from the audio signals that differ in number from the features of the audio encoder itself. The further adaptive instance normalization module can use the features of the two audio signals to obtain features of a further resulting audio signal, which exhibits the characteristic features of both audio signals. The further audio decoder can scale the number of features of the further resulting audio signal at this further hierarchy level to the number of features in the first hierarchy level.The audio decoder can now combine the features of the next hierarchy level after the next decoder with the features of the first hierarchy level after the Adaptive Instance Normalization module to create the resulting audio signal. This results in a first and a second hierarchy level, each encoding the incoming audio signals with different compression. This preserves different features, thereby improving the quality of the resulting audio signal.

[0012] Both the target signals and the underwater audio signals can be individually encoded using a stack of residual-1D convolution layers with the encoder. After each i-th encoder output, the features of the target signal can be... i (c) through the AdaIN layer to the features of the underwater audio signal z i(s) are normalized and optionally scaled. Vector quantization and decoding begins with higher-level features, i.e., with large values ​​for i. The stylized feature vectors t i (c, s) can optionally be assigned to the next codewords from the codebook of the hierarchy level.

[0013] The quantized features can then be decoded into the dimensions of the next lower layer and added to the stylized features. This process is repeated for each hierarchy level, with the lowest decoder stage mapping the combined features back into the original input dimension.

[0014] b) Providing a database of training data pairs, each comprising an underwater audio signal and either a target signal generated based on the physical properties of a target or a reference signal of the target. That is, the first audio signal is a generated target signal. Optionally, generating the target signal is part of the procedure. It is already possible to simulate, i.e., artificially generate, the resulting frequency spectrum of an audio signal based on the target's noise sources, the drive's rotational speed. The advantage is that these generated target signals can be produced with a multitude of variations for the same target. For example, different speed settings of the target, or different ambient noises emitted from the target's interior, can be simulated to generate the variations.Therefore, the simulation can generate virtually an infinite number of target signals.

[0015] A real, recorded underwater signal is suitable as a second audio signal. This second audio signal can be any underwater sound signal. For example, the second audio signal can include background noise and animal sounds. Advantageously, the second audio signals are recorded under various environmental conditions. Such audio signals already exist in virtually unlimited numbers or can be recorded without difficulty. When using the recorded reference signal as the target signal, as well as when using the simulated target signal, the resulting audio signal can be varied by changing the underwater audio signal. Thus, it is possible to create a large database even when using recorded reference signals as the target signal, although the number of recorded reference signals is limited.

[0016] In step c), the model is trained using the training data pairs from the database to generate the artificial sonar signals as the resulting audio signal. A first and a second audio signal (i.e., a training data pair) are provided. The model is trained on a sequence of these training data pairs. Specifically, the training optimizes the quality of the characteristic features of both the underwater audio signal and the target signal in the resulting audio signal. One quality criterion for the underwater audio signal could be its timbre. One quality criterion for the target signal could be its line spectrum. That is, the resulting audio signal should reflect both the line spectrum of the target signal and the timbre of the underwater audio signal. The training data is advantageously available as time series.

[0017] The idea is to train a mathematical model that improves acoustic models—i.e., resulting artificial sonar signals—by incorporating ship-specific spectral properties from reference recordings or simulated target signals. The mathematical model can separate static timbre features from dynamic details generated by propulsion and engine noise. Adaptive instance normalization integrates ship-specific spectral properties with a parametrically controlled simulation of broadband underwater sound. This enables the generation of realistic and variably generated acoustic signals. By synthesizing different training samples that preserve ship-specific spectral attributes while introducing controlled variations, the limitations of existing datasets are reduced. With the artificial sonar signals (i.e., the artificial audio signals)

[0018] In exemplary implementations, the model training in step c) is carried out in two stages. In the first of these two stages, the generation of the target signal's features in the audio encoder is optimized. That is, the audio encoder, particularly in conjunction with the audio decoder, is initially trained to extract representative features from audio signals that enable the most accurate possible decoding of the audio signal. In the second of these two stages, the quality of the characteristic properties of the underwater audio signal and the target signal in the resulting audio signal is optimized. Advantageously, in the second stage, the model is trained to adapt the (particularly broadband) timbre of the underwater audio signal while simultaneously preserving the (particularly narrowband) line spectra of the target signal.

[0019] For example, in step c), the encoder and decoder are trained with the target signals in the first stage, thus reducing the deviation of the resulting audio signals from the original target signals. In the second stage, the trained encoder weights can remain unchanged, and the trained decoder weights can be discarded to obtain an untrained decoder. After training the audio encoder or discarding the audio decoder weights, the decoder can be trained in step c) such that the deviation between the feature vectors of the target signal extracted by the encoder and the corresponding feature vectors of the resulting audio signal is reduced. This means that the result of the mathematical model, the resulting audio signal, is processed again with another, preferably identical, encoder to calculate the features from the re-encoded artificial sonar signal.Advantageously, the encoder is the same one used in the mathematical model. Specifically, both encoders have the same weights and architecture. This training criterion is also known as content loss. Additionally, the decoder is trained to reduce the deviation of statistical properties between the feature vector of the underwater audio signal and the feature vector of the target signal. This training criterion is also known as style loss. It has been shown that the model performs better with two-stage training than with one-stage training.

[0020] In exemplary implementations, the Adaptive-Instance-Normalization module is configured to perform normalization and, optionally, scaling of the features (i.e., the feature vector) of the target signal based on a statistical measure of the features (i.e., the feature vector) of the underwater audio signal. The normalization and, optionally, the scaling can be performed per feature, i.e., element-wise within the feature vector.

[0021] In further embodiments, the cascade of the mathematical model includes a codebook configured to map a result of the Adaptive Instance Normalization module to a predetermined code in the codebook according to a mathematical operation, with the audio decoder being configured to decode the predetermined code. That is, a code space with regions is created. Each region is represented by a code vector from the codebook. The Adaptive Instance Normalization module then outputs a feature vector located in one of the regions of the code space. However, instead of the original feature vector, the code vector belonging to that region is used for decoding. This process can also be called vector quantization. Advantageously, in the two-stage training, the codebook is trained with the encoder in the first stage. Advantageously, the codewords of the codebook are no longer modified in the second stage.

[0022] In exemplary embodiments, generating the target signal is part of the method. For example, generating the target signal can be achieved by superimposing synthesized, known partial noises of the target, or this step can be included. In particular, the known partial noises are any selection from an engine noise, a transmission noise, a drive shaft noise, a propeller noise, and ambient noises from inside the target.

[0023] Furthermore, a signal processing unit for generating artificial sonar signals with a mathematical model is disclosed. The signal processing unit performs the following steps a) to c): a) Provision of the mathematical model with a cascade comprising the audio encoder, the adaptive instance normalization module, and the audio decoder. The mathematical model is the same for which an exemplary training procedure is described above; however, it is also possible to train the model using a different training procedure. That is, in particular, the audio encoder is trained to generate features from audio signals; the adaptive instance normalization module is trained to use the features of two audio signals to obtain features of a resulting audio signal that exhibits the characteristic features of both audio signals; and the audio decoder is trained to convert the features of the resulting audio signal into the resulting audio signal. b) Accessing a database of payload data pairs, each comprising an underwater audio signal and a target signal generated based on the physical properties of a target. The payload data pairs have the same properties as the training data pairs in the previously described training procedure. However, unlike in classical classification tasks, it is also possible to use the same data for training and for generating the data, since the model is not intended to classify into specific groups but rather to generate signals. In this case, overfitting the mathematical model will also lead to sonar signals and not to a false detection. However, it is important to ensure that there is sufficient variability in the generated data sets, i.e., that not only learned data sets are output. In other words, it is possible for the payload data pairs and the training data pairs to be identical. c) Using the payload data pairs to generate an artificial sonar signal for each payload data pair. That is, the payload data pairs are presented to the model as input data, and the model, in particular the audio decoder, outputs the resulting sonar signal.

[0024] Analogous to the signal processing unit, a method for generating artificial sonar signals with a mathematical model comprising the following steps a) to c) is disclosed: a) Providing the mathematical model with a cascade comprising - an audio encoder trained to generate features from audio signals, - an adaptive instance normalization module trained to use the features of two audio signals to obtain features of a resulting audio signal that exhibits the characteristic features of both audio signals; and - an audio decoder designed to convert the characteristics of the resulting audio signal into the resulting audio signal; b) Accessing a database of payload pairs comprising each an underwater audio signal and a target signal generated based on the physical properties of a target; c) Using the payload data pairs to generate an artificial sonar signal for each payload data pair.

[0025] A "vehicle from a maritime environment" refers, for example, to watercraft (e.g., ships, submarines, whether manned or unmanned) or aircraft over water or on the coast.

[0026] Additionally, the generated target signals can also be created using this method or the analog signal processing unit.

[0027] Furthermore, a (further) signal processing unit for classifying or identifying vehicles in the maritime environment is disclosed with a (further) mathematical model, wherein the further mathematical model is trained with artificial sonar data generated with the signal processing unit described above.

[0028] Vehicle identification or classification can be performed using a further model, i.e., an artificial intelligence, within the sonar data (i.e., the payload data). The model can optionally comprise multiple, particularly different, sub-models. A (sub-)model describes the architecture of one or more algorithms. The model can be optimized through training by adjusting weights, i.e., parameters. Most common, well-known algorithms are suitable, especially artificial neural networks, Bayesian classifiers, support vector machines (SVMs), and linear regression. Deep learning approaches, such as transformer-based artificial neural networks and / or convolutional artificial neural networks, can also be used. If the model comprises several sub-models for vehicle identification or classification, similar algorithms (e.g.,various artificial neural networks), but also different algorithms (e.g. an artificial neural network and a linear regression) can be combined.

[0029] In particular, it has been shown that no special algorithms or algorithm architectures are needed to create the model for vehicle identification or classification. It is sufficient to train commercially available algorithms with appropriate data. Such algorithms are also known as COTS (Components Off-The-Shelf). It has been found that the structure—for example, the number of neurons or layers in artificial neural networks—is irrelevant for training the model, as long as a minimum level of complexity is achieved. Models that perform well in public benchmarks are particularly suitable.Furthermore, it has been shown that even simple feedforward neural networks, especially those with few layers, are already suitable for the classification and / or identification of vehicles using the features from the deep model.

[0030] Preferably, at least one (partial) model is implemented as an artificial neural network. The artificial neural network can, for example, be based on Transformer technology and / or be or include a convolutional artificial neural network. However, a classic feedforward neural network, such as a multilayer perceptron (MLP), is preferred.

[0031] If a single model without sub-models is used, it will preferentially identify all vehicles. If multiple sub-models are used, each sub-model can, for example, identify one vehicle.

[0032] Artificially generated sonar signals, especially in combination with real, recorded sonar data, can be used to train the model or sub-models.

[0033] Artificial (i.e., synthetic) training data is suitable. Artificial data can be used particularly when the amount of real training data is insufficient. Preferably, a combination of real and artificial data can be used for training. That is, artificial data can be supplemented with real sonar data. Real sonar data can be acquired during training runs, training maneuvers, or in the field. Furthermore, there are publicly accessible databases with large amounts of sonar data. Preferably, the real data is pre-labeled, for example, by a trained sonar operator. That is, the actual vehicles in the sonar data are identified beforehand. This is called labeling. This enables supervised learning of the model. Manual labeling is not necessary with artificial sonar data, since it is known where (i.e.,The system identifies when an object was simulated (at what bearing and time) and where only noise is present. Labeling can therefore be automated. The idea is to train a model, i.e., an artificial intelligence, using machine learning. This model, using suitable features, is capable of detecting, classifying, and preferably identifying vehicles. Such models are more reliable and accurate than the current classification and identification methods used by sonar operators, i.e., humans.

[0034] Analogous to the (further) signal processing unit, a method for classifying or identifying vehicles in the maritime environment is disclosed with a further mathematical model, wherein the further mathematical model is trained with artificial sonar data generated with the previously described signal processing unit or the corresponding method.

[0035] Furthermore, the use of the aforementioned signal processing unit or the corresponding method for generating artificial sonar data for a training simulator for training sonar operators is disclosed. This makes it possible to use more realistic and non-recurring sonar signals for training sonar operators, thereby improving the training.

[0036] A computer program is now disclosed, comprising instructions which, when the program is executed by a computer, cause it to execute the mathematical model using a database of training data pairs or user data pairs according to one of the described methods.

[0037] In other words, a timbre transfer approach is revealed that enhances acoustic models by incorporating ship-specific spectral properties from reference recordings. The approach can employ a hierarchical vector-quantized variational autoencoder (VQ-VAE) to separate static timbre features from dynamic content representations generated by propulsion and engine noise. Using adaptive instance normalization, ship-specific spectral properties can be integrated into a parametrically controlled simulation of broadband ship noise. This enables the generation of realistic and variably conditioned acoustic signals. By synthesizing various training samples that preserve ship-specific spectral attributes while introducing controlled variations, the limitations of existing datasets can be minimized.

[0038] Preferred embodiments of the present invention are explained below with reference to the accompanying drawings. These show: Fig. 1: a schematic representation of an acoustic model for generating artificially generated, i.e. synthetic, target signals; Fig. 2: a schematic representation of a mathematical model for generating artificial sonar data, wherein Fig. 2a the model for a first stage of training and Fig. 2b shows a second stage of training; Fig. 3: a schematic representation of variants of the mathematical model from Fig. 2, wherein Fig. 3a the model with two layers and Fig. 3b shows the model with three layers as an example; Fig. 4a: a schematic representation of a signal processing unit for generating artificial sonar signals; Fig. 4b: a schematic representation of a signal processing unit for the classification of vehicles in a maritime environment.

[0039] Before exemplary embodiments of the present invention are explained in detail below with reference to the drawings, it should be noted that identical, functionally equivalent or equivalent elements, objects and / or structures in the different figures are provided with the same reference numerals, so that the description of these elements shown in different exemplary embodiments is interchangeable or can be applied to one another.

[0040] Fig. Figure 1 shows a schematic representation of a parametric acoustic model 20 that simulates ship sound x(t) by combining deterministic harmonic and non-harmonic components. The generated signals serve as the “content” domain for the in Fig. 2 and Fig. The mathematical model shown in Figure 3 allows control over narrowband frequency bands. Random parameters, frequency attenuation, and phase shift are used to generate diverse datasets. While the acoustic model 20 takes essential spectral features into account, it lacks the complexity of the real world.

[0041] The acoustic model 20 takes into account three main sources of target noise: Machine noises: Generated by the main motor 22 (x CFR (t)), the gearbox 24 (x PSR (t)) and wave 26 ((x CSR (t)). These noises are typically harmonic, as they originate from rotating mechanical components. Propulsion noise: Generated by the ship's propeller 28 (x BR(t)) and cavitation effects 30 (n(t)). In contrast to the machine noise, the drive noise contains both narrowband and broadband components. The propeller blade speed 28 defines important spectral peaks, while cavitation 30 generates a continuous noise spectrum. Background noise 32 (x AUX (t)): Generated by onboard generators, pumps, and ventilation systems. Unlike propulsion and engine noise, auxiliary noise is independent of the ship's speed and remains relatively stable over time.

[0042] The model can calculate the acoustic signals for each individual component using a corresponding fundamental frequency, phase shift, and amplitude. For machine and drive signals, the amplitude and phase shift of additional, especially any, harmonic frequencies can also be taken into account.

[0043] The engine speed, known as the crankshaft rate f, can be used as a control variable for this simulation. CSR (t) is denoted. The rotating crankshaft generates a subthreshold, time-dependent harmonic signal x. CSR (t), which is mainly influenced by the significant harmonics generated by the cylinder ignitions of the engine.

[0044] The frequency of the cylinder ignition frequency f CFR is determined using the number of cylinders n cylinders and the number of strokes n strokes defined as follows: fCFR=2fCSRncylindresnstrokes

[0045] Since the engine is typically mechanically connected to the propeller via a gearbox, the propeller shaft speed F PSR using the gear ratio r GRR calculated as follows: fPSR=rGRR⋅fCSR

[0046] However, it is important to note that many ships, mainly larger ones, have a direct connection between the crankshaft and the propeller shaft, i.e., GRR is equal to 1.

[0047] The rotating propeller generates a broadband noise. This broadband noise is modulated by the rotating blades themselves, depending on the blade speed f. BR , which in turn depend on the number of leaves n blade about the connection f BR = f PSR · n blade depends.

[0048] Using the calculated fundamental frequencies, the individual waveforms can be calculated. The following equation shows the calculation of a single harmonic set, using the cylinder ignition frequency as an example: xCFR(t)=∑i=1NCFRαCFR,i⋅sin(i⋅2π⋅fCFR⋅t+ΦCFR) where x is the time-dependent acoustic signal, N CFR the number of harmonic frequencies, a CFR,ithe amplitude of the i-th harmonic, f CFR the fundamental frequency and Φ CFR represents the associated phase shift.

[0049] The complete acoustic signal, i.e. the generated sonar signal 34, can then be calculated by simply adding the individual waveforms as follows: x(t)=xCFR(t)+xCSR(t)+xPSR(t)+xAUX(t)+n(t)⋅xBR(t) where n(t) describes the cavitation noise. This can be modeled, for example, with white noise.

[0050] This simple simulation allows for the control of the narrowband frequency spectrum. Random parameters, frequency attenuation, and phase shifts are used to create a diverse dataset.

[0051] Fig. Figure 2 shows a mathematical model 40 for generating artificial sonar signals 42. The model 40 can be trained using a two-stage process. Fig. Figure 2a shows the configuration of the Model 40 for the first stage of training. Fig. Figure 2b shows the final architecture of the model 40, which is required for the second stage of training. In the first stage, the model 40 first learns to reconstruct simulated target signals 34, i.e., to form a reconstructed target signal 34'. Subsequently, in the second stage, the model 40 is trained to transfer the spectral properties of real ship sound recordings to these synthetic signals. In other words, in the first stage, an encoder 42, a decoder 44, and optionally a codebook 46 are trained to reconstruct 34' of simulated ship sounds 34 in order to create a structured space that encodes the main acoustic properties of ship sounds 34.

[0052] In the first stage, depicted in Fig. 2a The encoder 42 maps the generated sonar signal 34 [s] into a (feature) space by extracting features 48. The decoder 44 outputs the resulting audio signal 34' as a reconstruction of the original signal 34 from the features 48, ensuring that important narrowband frequency components, such as those introduced by propeller harmonics and engine noise, are preserved. The training objective includes, for example, an L1 loss in the time domain, which ensures high accuracy and phase preservation, while an additional spectral convergence loss ensures the reconstruction of the correct frequency bands. However, other loss functions can also be selected. To compare the feature quality, the artificially generated sonar signal is encoded again by another encoder 42'. This second encoder has the same parameters as encoder 42.Thus, the similarity of features 48 after the first encoder 42 can be compared with the similarity of features 49 after the second encoder 42' to determine the quality of the encoder / decoder path and therefore the quality of features 48 in representing the original (synthetic) target signal 34. Features 49 can also be referred to as control features, as they are only needed for training. Through the initial learning to reconstruct deterministic synthetic signals, the model 40 establishes a structured space that creates the necessary foundation for the subsequent second stage, style transfer.

[0053] Optionally, a codebook 46 can be connected between encoder 42 and decoder 44. Using codebook 46, the quantized feature space can be divided into regions, each assigned a single code vector. This means that the virtually infinite number of possible value combinations in the feature vector that encoder 42 can output is quantized to a number N of code vectors. In other words, each actual feature vector is mapped to a code vector. Using codebook 46, decoder 44 then decodes the code vector instead of the feature vector. This reduces the decoding complexity for decoder 44, resulting in overall improved results for the reconstructed target signal 34'. In particular, the variance in the deviation of the reconstructed target signals 34' from the original target signal 34 is reduced. This process is also referred to as stabilizing the training process.

[0054] In the second stage, depicted in Fig. In step 2b, the final model 50 is trained. Unlike the first stage, model 50 is now trained on both input data, i.e., the target signals 34 [s] and underwater audio signals 52 [c]. Advantageously, the same encoder, i.e., in particular the same architecture and the same weights, is used to extract the feature vectors of the target signals and the underwater audio signals. Model 42 now generates feature vectors 48 and 52' corresponding to the target signals 34 and the underwater audio signals 52, respectively. The two feature vectors 48, 52' form the input values ​​of an Adaptive-Instance-Normalization module 50. The module 50 transfers the timbre of the underwater audio signals 52, which are encoded in the corresponding feature vector 52', to the target signals 34, which are encoded in the corresponding feature vector 48.

[0055] The Adaptive Instance Normalization Module 50 can achieve this transfer by matching the statistics of the feature vectors of the underwater audio signal 52 z(s) to the corresponding feature vectors of the target signal 34 z(c), where z represents the vector. The output t 48' of the Adaptive Instance Normalization Module 50 can be mapped as follows: t=AdaIN(z(c),z(s))=σ(z(s))⋅(z(c)−μ(z(c))σ(z(c)))+μ(z(s))

[0056] The features of the target signal are normalized and scaled by the spatial mean µ and the standard deviation σ of the features of the underwater audio signal, where t corresponds to the adapted properties of the target signal.

[0057] The second-stage training involves freezing the encoder 42 and, optionally, the codebook 46. If the codebook is used, it is also possible that it will be marginally trained along with the second-stage training. However, the weights of the decoder 44 are discarded, and the decoder is retrained. The decoder training can be performed using a weighted loss function that combines both the timbre and the mapping of the significant frequency components of the target signal. The weighted loss can be defined by L = L c + λL s be defined, L represents the total loss, L c the loss of frequency components as well as L s represents the loss of timbre, which is weighted by the factor λ.

[0058] The loss of frequency components can be expressed as the Euclidean distance between the coded content features. j (c) and the coded features of the output signal zj (ŷ) can be described. The distances can be described for each i-th hierarchical level (i=1 in Fig. 2, i=2 in Fig. 3a and i=3 in Fig. 3b) are added, where L represents the total number of levels. The loss of the frequency components can be calculated as follows: Lc=∑i=1L‖zi(y^)−zi(c)‖2

[0059] The tone color adjusts the statistics of the target signal's features to those of the underwater audio signal. Therefore, the mean µ and standard deviation σ can be calculated at different intermediate coding levels ϕ, where the index j represents the level index of the K intermediate coding levels used. The intermediate coding levels are the encoder layers. The tone color loss can be calculated as follows: LS=∑i=1L∑j=1K‖μ(Φi,j(y^))−μ(Φi,j(s))‖2+∑i=1L∑j=1K‖σ(Φi,j(y^))−σ(Φi,j(s))‖2

[0060] Fig. Figure 3 shows two adapted models 50' with an additional hierarchy level 54a ( Fig. 3a) or two further hierarchy levels 54a, 54b ( Fig. 3b) while the model 50 from Fig. 2a and Fig. 2b has only one hierarchy level. The hierarchy levels are denoted by the index i. The training described above already describes the training of the model with i hierarchy levels. The notation of the reference symbols continues for the further levels.

[0061] Advantageously, the hierarchy levels have different compression levels. That is, the encoders encode the input signals with feature vectors of varying sizes. The decoders (44', 44") of the higher layers each have the task of mapping the number of features of the current layer i to the number of features of the layer i-1 below it. The features can then be summed and optionally added to the codebook of the layer below. This process is carried out for each level until all features in the lowest layer have been summed and fed to decoder 44 to generate the artificial sonar signal.

[0062] Fig. Figure 4a shows a schematic representation of a signal processing unit 60 for generating artificial sonar signals 34'. The signal processing unit includes, purely as an example, the mathematical model 40. However, it is also possible, for example, to use models 40' or 40". Optionally, the model is trained using the training procedure described above. By accessing a database 62, the signal processing unit receives the data pairs 34, 52 to generate the artificial sonar signals 34'.

[0063] Fig. Figure 4b discloses a signal processing unit 64 for classifying or identifying vessels in a maritime environment. The signal processing unit 64 comprises a further mathematical model 66, which is shown here by way of example as a multilayer perceptron (MLP). For training purposes, the further model 66 has access to the artificial sonar signals 34' generated by the signal processing unit 60. Fig.4a have been generated. Optionally, the signal processing unit 64 can be additionally trained with real sonar data. For classification or identification, the model 66 receives real sonar data 68 into the input layer. The number of output nodes in the output layer 70 can correspond to the number of different vehicles to be classified or identified.

[0064] Although some aspects have been described in connection with a device, it is understood that these aspects also constitute a description of the corresponding process, so that a block or component of a device can also be understood as a corresponding process step or as a feature of a process step. Similarly, aspects described in connection with or as a process step also constitute a description of a corresponding block, detail, or feature of a corresponding device.

[0065] Depending on the implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example, a magnetic or optical storage medium, on which electronically readable control signals, e.g., a computer program, are stored. These signals can interact with, or interact with, a programmable computer system (CPU and / or GPU) in such a way that the respective method is carried out. For this reason, the digital storage medium should be computer-readable. Embodiments can therefore include a data carrier that has electronically readable control signals capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.

[0066] In some embodiments, a programmable logic device (for example, a field-programmable gate array, an FPGA) can be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array can interact with a microprocessor to perform one of the methods described herein. Generally, in some embodiments, the methods are performed by any hardware device. This can be general-purpose hardware such as a computer processor (CPU) or a graphics processing unit (GPU), or hardware specific to the method, such as an ASIC. Distributed execution across the CPU and GPU is also possible.

[0067] The embodiments described above merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to other people skilled in the art. Therefore, it is intended that the invention be limited only by the scope of protection set forth in the following claims and not by the specific details presented herein by way of description and explanation of the embodiments. Reference symbol list: 20 acoustic model 22 Main engine 24 gearboxes 26 wave 28 propellers 30 Cavitation 32 background noises 34 generated sonar signal / target signal 34' resulting artificial sonar signal 40 mathematical model 42 encoders 42' further encoder 44 decoders 46 Codebook 48 characteristics of the generated sonar signal 49 control features of the additional encoder 50 Adaptive Instance Normalization Module 52 underwater audio signals 52' Characteristics of the underwater audio signal 54 hierarchical levels 60 Signal processing unit 62 database 64 additional signal processing units 66 further mathematical model 68 sonar data 70 Starting level

Claims

[1] Method (20) for training a mathematical model (40, 40', 40") to generate artificial sonar signals (34') comprising the following steps: a) Providing the mathematical model (40, 40', 40") comprising a cascade - an audio encoder (42) designed to generate features from audio signals (34, 52), - an adaptive instance normalization module (50) configured to use the features of two audio signals (48, 52') to obtain features of a resulting audio signal (48') that exhibits the characteristic features of both audio signals (34, 52); and - an audio decoder (44) configured to convert the features (48') of the resulting audio signal into the resulting audio signal (34'); b) Providing a database of training data pairs, each comprising an underwater audio signal (52) and either a target signal (34) generated based on physical properties of a target or a target reference signal; c) Training the model (40, 40', 40"), with the training data pairs from the database, to obtain artificial sonar signals as the resulting audio signal (34'). [2] Method according to claim 1, wherein the training of the model in step c) is carried out in two stages, wherein in a first stage of the two stages the generation of the features (48) of the target signal (34) in the audio encoder (42) is optimized and in a second stage of the two stages the quality of the characteristic properties of the underwater audio signal (52) and the target signal (34) in the resulting audio signal (34') is optimized. [3] Method according to claim 2, wherein in step c) in the first stage the encoder (42) and the decoder (44) are trained with the target signals (34) so ​​that a deviation of the resulting audio signals (34') from the original target signals (34) is reduced. [4] Method according to claim 3, wherein in step c) in the second stage the trained weights of the encoder (42) remain unchanged and the trained weights of the decoder (44) are discarded to obtain an untrained decoder (44). [5] Method according to one of claims 3 or 4, wherein in step c) the decoder (44) is trained in the second stage such that a deviation of feature vectors (48) of the target signal extracted by the encoder (42) to corresponding feature vectors (49) of the resulting audio signal is reduced and that a deviation of statistical properties of the feature vector (52') of the underwater audio signal to the feature vector (48) of the target signal is reduced. [6] Method according to any one of claims 3 to 5, wherein the Adaptive-Instance-Normalization module (50) is configured to perform a normalization of the features (48), i.e., the feature vector, of the target signal based on a statistical measure of the features (52'), i.e., the feature vector, of the underwater audio signal. [7] Method according to one of the preceding claims, wherein the cascade of the mathematical model (40, 40', 40") comprises a codebook (46) configured to assign a result of the Adaptive-Instance-Normalization module (50) according to a mathematical operation to a predetermined code of the codebook (46), wherein the audio decoder (44) is configured to decode the predetermined code. [8] Method according to claim 7 as referring back to any one of claims 3 to 6, wherein the codebook (46) is trained in the first stage and the codewords remain unchanged for the training of the second stage. [9] Method according to one of the preceding claims, wherein step b) comprises generating the target signal (34). [10] Method according to claim 9, wherein generating the target signal (34) comprises superimposing synthesized, known partial noises of the target, in particular wherein the known partial noises comprise any selection from an engine noise (22), a transmission noise (24), a drive shaft noise (26), a propeller noise (28) and incident noises (32) from inside the target. [11] Method according to any of the preceding claims, - wherein the mathematical model has a further hierarchy level (54a), wherein the further hierarchy level (54a) is arranged parallel to the audio encoder (42) and the adaptive instance normalization module (50), - wherein the next hierarchy level comprises a cascade including another audio encoder (42a, 42b), another adaptive instance normalization module (50') and another audio decoder (44'), - wherein the further audio encoder (42a) is configured to generate features (48a, 52a') from the audio signals (34, 52) which differ in number from the features (48, 52') of the audio encoder (42); - wherein the further adaptive instance normalization module (50`) is configured to use the features (48a, 52a') of the two audio signals to obtain features (48a') of a further resulting audio signal which has the characteristic features of both audio signals (48a, 52a'); and - wherein the further audio decoder (44`) is configured to scale the number of features (48a') of the further resulting audio signal of the further hierarchy level to the number of features (48') in the first hierarchy level; - wherein the audio decoder is configured to convert an addition of the features of the further hierarchy level after the further decoder (44') and the features of the first hierarchy level after the Adaptive-Instance-Normalization module (50) into the resulting audio signal (34'). [12] Signal processing unit for generating artificial sonar signals with a mathematical model, wherein the signal processing unit is configured to perform the following steps: a) Providing the mathematical model with a cascade comprising - an audio encoder trained to generate features from audio signals, - an adaptive instance normalization module trained to use the features of two audio signals to obtain features of a resulting audio signal that exhibits the characteristic features of both audio signals; and - an audio decoder designed to convert the characteristics of the resulting audio signal into the resulting audio signal; b) Accessing a database of payload pairs comprising each an underwater audio signal and a target signal generated based on the physical properties of a target; c) Using the payload data pairs to generate an artificial sonar signal for each payload data pair. [13] Signal processing unit according to claim 12, wherein the mathematical model is trained by machine learning. [14] Signal processing unit according to one of claims 12 or 13, wherein the mathematical model is trained using the method according to one of claims 1 to 11. [15] Method for generating artificial sonar signals using a mathematical model comprising the following steps: a) Providing the mathematical model with a cascade comprising - an audio encoder trained to generate features from audio signals, - an adaptive instance normalization module trained to use the features of two audio signals to obtain features of a resulting audio signal that exhibits the characteristic features of both audio signals; and - an audio decoder designed to convert the characteristics of the resulting audio signal into the resulting audio signal; b) Accessing a database of payload pairs comprising each an underwater audio signal and a target signal generated based on the physical properties of a target; c) Using the payload data pairs to generate an artificial sonar signal for each payload data pair. [16] Signal processing unit for classifying or identifying vehicles in a maritime environment with a further mathematical model, wherein the further mathematical model is trained with artificial sonar data generated with the signal processing unit according to any one of claims 12 to 14. [17] Method for classifying or identifying vehicles in a maritime environment using a further mathematical model, wherein the further mathematical model is trained with artificial sonar data generated by the signal processing unit according to one of claims 12 to 14 or the method according to claim 15. [18] Use of the signal processing unit according to any one of claims 12 to 14 or the method according to claim 15 for generating artificial sonar data for a training simulator for training sonar operators. [19] Computer program comprising instructions which, when the program is executed by a computer, cause it to train a mathematical model using a database of training data pairs according to the method of any one of claims 1 to 11 or to execute the method according to claim 15 or claim 17 using a database of user data pairs.

Images