Method for detecting events in sonar data

Abstract

The invention relates to a method for detecting events (22) in passive sonar data (24), comprising the following step: applying a mathematical model to a time series of the sonar data (24) in order to detect, by means of the model, an event (22) or a plurality of events in the sonar data (24) in the time-frequency range.

Description

[0001] 2024,114

[0002] Methods for detecting events in sonar data

[0003] Description

[0004] The invention relates to the detection of events in sonar data. Events in sonar data can be natural sounds such as whale songs or technically generated sounds.

[0005] Currently, events are detected, for example, using energy detection based on clustering in the frequency domain. However, this type of event detection is prone to false positives or the detection of uninteresting events, as factors other than energy are generally not taken into account.

[0006] The object of the present invention is therefore to create an improved concept for the detection of events in sonar data.

[0007] The problem is solved by the subject matter of the independent patent claims. Further advantageous embodiments are the subject matter of the dependent patent claims.

[0008] Exemplary embodiments show methods for detecting events in passive sonar data (hereinafter referred to as sonar data) with the following step: a) Applying a mathematical model to a time series of sonar data in order to detect one or more events in the sonar data in the time-frequency domain using the model. The mathematical model is advantageously trained using machine learning, i.e., it is a model generated by machine learning. In particular, a frequency analysis, e.g., section by section in the sequence of sonar data, can be applied to the sonar data to obtain a representation of the sonar data against frequency and time. The model is then optionally applied to the time-frequency representation of the sonar data. Thus, the events can be detected, for example, via their characteristic frequencies. The representation of the detection 2024.114 is done, for example, by encircling or forming a box, in particular a rectangle, around the time-frequency cells in which the detection takes place.

[0009] This means, for example, that a continuous, and especially overlapping, time-frequency transformation of the raw data from the underwater transducers used to record the sonar data is performed. Any known time-frequency transformation, such as a Short-Time Fourtier Transform (SFTS) or a wavelet transformation, is suitable. In other words, the sonar data is provided to the model in a temporal sequence, in a time-frequency representation, to perform event detection. This visual time-frequency representation forms the input data for the model. For example, time intervals of a predetermined duration, e.g., 60 seconds, are used as input values ​​for the time-frequency transformation. Furthermore, a new time-frequency transformation is generated, e.g., every 10 seconds, and provided to the model for detection, resulting in overlapping time-frequency representations.

[0010] Such a visual representation can be called a waterfall display. Furthermore, using this model has the advantage that events in the time-frequency domain can be directly categorized, for example, into continuous wave (CW) or frequency-modulated (FW) events, or interesting and uninteresting events. Uninteresting events can then be suppressed directly, i.e., not displayed or output. Due to the overlapping information in the time-frequency representations, different event detections can occur. For unambiguous detection, detections with high confidence can be preferred, for example, by checking overlapping areas from different transformation images for plausibility.

[0011] Alternatively, events, i.e., detections, can be displayed in color. In particular, interesting events can be represented in, for example, signal colors, and uninteresting events in, for example, pastel colors. Furthermore, the event detection approach using the model allows for monitoring the acoustic quality of the sonar data. The acoustic quality of the sonar data can be determined based on the different characteristics of radio interference compared to underwater sound signals. Radio interference, in particular, is characterized by a comparatively higher signal energy and rapid changes. It has also been shown that events below the classical detection threshold can be reliably detected using the model.

[0012] The idea is to perform event detection in sonar data using artificial intelligence. The artificial intelligence comprises a model that can optionally include multiple, and in particular, different, sub-models. A (sub-)model describes a trained algorithm. Most common, well-known algorithms are suitable, especially artificial neural networks, Bayesian classifiers, support vector machines (SVMs), and linear regression. However, deep learning approaches such as transformer-based artificial neural networks and / or convolutional artificial neural networks (CNNs) are preferred. If the model includes several sub-models for event detection, similar algorithms (e.g., different artificial neural networks) as well as different algorithms (e.g., an artificial neural network and linear regression) can be combined.

[0013] The sonar data consists of underwater sound signals, i.e., the output data of a passive sonar system. Active sonar signals and underwater communication signals, for example, can be detected within this passive sonar data. Based on these detections, the associated ships or submarines can also be located. In some implementations, the mathematical model can differentiate detections by type. This means that, in the visual representation, biosignals (e.g., animal sounds), frequency-modulated active sonar signals (FM signal), continuous active sonar signals (CW signals), underwater communication signals, etc., are distinguished from one another. This distinction can be made, for example, by using different colors to highlight the detection.

[0014] In particular, it has been shown that no special algorithms are needed to perform event detection. It is sufficient to train commercially available algorithms with 2024,114 corresponding data points. Such algorithms are also known as COTS (Components-Off-The-Shelf). It has been found that the structure—for example, the number of neurons or the number of layers in artificial neural networks—is irrelevant for event detection, as long as a minimum level of complexity is achieved. Models that perform well in public benchmarks are particularly suitable. Examples of such benchmarks include the "COCO dev-test" and the "COCO val2017." Models based on the DINO (DETR with Improved deNoising anchOr box) algorithm or the YOLO (You Only Look Once) algorithm are also examples of suitable models.

[0015] In "Tianhe Ren, Jianwei Yang, Shilong Liu, Ailing Zeng, Feng Li, Hao Zhang, Hongyang Li, Zhaoyang Zeng, Lei Zhang. A Strong and Reproducible Object Detector with Only Public Datasets. arXiv:2304.13027 [cs.CV], (https: / / doi.org / 10.48550 / arXiv.2304.13027)" and "Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, Heung-Yeung Shum. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. arXiv:2203.03605 [cs.CV], (https: / / doi.org / 10.48550 / arXiv.2203.03605)” examples of the models are revealed.

[0016] Preferably, however, the model, or at least a (partial) model, is implemented as an artificial neural network. The artificial neural network can, for example, be based on Transformer technology and / or be or comprise a convolutional artificial neural network.

[0017] If a single model without sub-models is used, it will preferentially detect all potentially occurring events. If multiple sub-models are used, the individual sub-models can, for example, detect different event categories. Event categories can be natural noises, harmonic noises, or short-term events. However, any other event categories can also be found.

[0018] Suitable training data includes real sonar data acquired during training runs, training maneuvers, or in operational use. Preferably, the data should be pre-classified, for example, by a trained sonar operator (e.g., 2024.114). This means that the actual events in the sonar data are identified beforehand. This process is called labeling. This enables supervised learning of the model. Synthetic data, i.e., data generated using technical means, is also suitable for training the model. Synthetic data can be used particularly when the amount of real training data is insufficient. In this case, a combination of real and synthetic data is preferably used for training.

[0019] In exemplary implementations, the method comprises the following further steps:

[0020] - Continuous recording of sonar data and continuous monitoring of this data by the model for events. This means the amount of sonar data continuously increases with new measurements. The model can then also continuously search for events. For example, adding a measurement to the beginning of the sonar data time series might cause the most recent measurement in the series to be excluded. In other words, the model can consider sonar data within a predefined time period in the past. This results in real-time monitoring of the sonar data for events using the model.

[0021] In further embodiments, the method includes the step of scaling the recorded sonar data. It has been found that event detection functions more reliably when the sonar data is pre-scaled. The scaling is performed, for example, to a variance of the value of the sonar data, i.e., in the case of an image of brightness displayed on a monitor, from one across all considered sonar data points, i.e., for example, a displayed sonar image.

[0022] Optionally, the method includes applying direction-finding to the sonar data to obtain a bearing-time representation of the sonar data, and applying the model to the bearing-time representation of the sonar data to detect the event or multiple events per bearing, or applying the model to individual time series of the sonar data per bearing to detect the event or multiple events per bearing. Thus, it is possible to detect events in a spatial representation of 2024.114. The model can directly obtain all data, i.e., the bearing-time representation, and detect the event or multiple events from the data and determine their bearings. Alternatively, the time series can also be fed into the model per bearing. However, the events will then typically be detected at different adjacent bearings. A model-based or deterministic (e.g.,Energy-based analysis can then separate the events and determine the correct bearing for each event.

[0023] In implementation examples, signal processing, specifically normalization, is applied to the time series of sonar data to obtain preprocessed sonar data. One color channel of an image file can then be used to provide the model with the time series of the sonar data, and another color channel can simultaneously provide the model with the preprocessed time series of the sonar data. That is, the input data for the model is compiled in the form of an image file. An image file typically has three color channels, e.g., RGB (Red, Green, Blue), or four color channels, e.g., CMYK (Cyan, Yellow, Magenta, Black). However, the sonar data can be represented in grayscale, so only one color channel is needed. The other color channels can then be used to provide the model with representations of the sonar data obtained through signal processing.This makes it easier for the model to extract features relevant for detection.

[0024] In particular, it has been shown that normalization is necessary both along the time axis, since the detections sought are time-limited, and along the frequency axis, since the background, which is mostly frequency-dependent, would otherwise limit the contrast if only normalized data were provided to the model.

[0025] In a further embodiment, the model is also used to classify a detected event. That is, not only is the event detected (i.e., recognized), but it is also assigned to an event category or specifically named (e.g., whale song, ship propeller, etc.). 2024.114

[0026] Further embodiments show, analogously, a data processing unit for detecting events in sonar data, which can perform the described process steps.

[0027] Furthermore, a system comprising a data processing unit and a passive sonar for recording the sonar data is disclosed. The passive sonar thus provides the sonar data, while the data processing unit performs the data processing, in particular the application of the model to the sonar data. The passive sonar can be a low-frequency passive sonar. The passive sonar can be used to create situational awareness images, i.e., in particular the detection of events, especially objects, in the vicinity of the passive sonar.

[0028] Analogously, a computer program comprising instructions is disclosed which, when the program is executed by a computer, cause it to execute the method according to one of the preceding claims using sonar data.

[0029] Preferred embodiments of the present invention are explained below with reference to the accompanying drawing. It shows:

[0030] Fig. 1 : a schematic block diagram of a system for detecting events in sonar data.

[0031] Before exemplary embodiments of the present invention are explained in detail below with reference to the drawings, it should be noted that identical, functionally equivalent or equivalent elements, objects and / or structures in the different figures are provided with the same reference numerals, so that the description of these elements shown in different exemplary embodiments is interchangeable or can be applied to one another.

[0032] Fig. 1 shows a schematic block diagram of a system 20 for detecting events 22 in sonar data 24. The sonar data 24 can be processed by means of 2024.114

[0033] Underwater sound transducers 26, i.e., a sonar, are generated from received underwater sound 28. A data processing unit 30, i.e., for example, a computer, processes the sonar data 24 and applies a machine learning-trained model for detecting the events 22 to the sonar data. The result can then be a representation 32 of the sonar data with the events contained therein.

[0034] The disclosed (underwater) sound transducers are designed for use underwater, particularly in the sea. The transducers can convert underwater sound into an electrical signal (e.g., voltage or current) corresponding to the sound pressure, the (received) underwater sound signal. Furthermore, it is possible for the transducers to convert an applied electrical voltage into underwater sound. The electrical voltage can follow a predefined pattern and then be referred to as the (transmitted) sonar signal, while the underwater sound resulting from the sonar signal to be transmitted is referred to as the (transmitted) sonar signal. Examples of sonar signals are a chirp (frequency-modulated signal) or, as a special case of the chirp, a sweep (linearly frequency-modulated signal). The transducers can therefore be used as underwater sound receivers and / or underwater sound transmitters.The transducers can be made of a piezoelectric material, such as a piezoceramic, to serve as the sensor material. A plurality of underwater transducers, or one or more underwater transducers in conjunction with a signal processing unit, can be referred to as a sonar system. The transducers can be used for (active and / or passive) sonar (sound navigation and ranging). The transducers are preferably not suitable for, or are not used for, medical applications. Likewise, the transducers are preferably not used for, or are not suitable for, the ultrasonic testing of materials.

[0035] Although some aspects have been described in connection with a device, it is understood that these aspects also represent a description of the corresponding process, so that a block or component of a device is also to be understood as a corresponding process step or as a feature of a process step. Similarly, aspects described in 2024.114

[0036] In connection with or as a process step, also represents a description of a corresponding block or detail or feature of a corresponding device.

[0037] Depending on the implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example, a magnetic or optical storage medium, on which electronically readable control signals, e.g., a computer program, are stored. These signals can interact with a programmable computer system (CPU and / or GPU) in such a way that the respective method is carried out. For this reason, the digital storage medium should be computer-readable. Embodiments can therefore include a data carrier that has electronically readable control signals capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.

[0038] In some embodiments, a programmable logic device (for example, a field-programmable gate array, an FPGA) can be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array can interact with a microprocessor to perform one of the methods described herein. Generally, in some embodiments, the methods are performed by any hardware device. This can be general-purpose hardware such as a computer processor (CPU) or a graphics processing unit (GPU), or hardware specific to the method, such as an ASIC. Distributed execution across the CPU and GPU is also possible.

[0039] The embodiments described above merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to other people skilled in the art. Therefore, it is intended that the invention be protected only by the scope of protection set forth below in 2024.114.

[0040] The patent claims are not limited by the specific details presented herein by way of description and explanation of the exemplary embodiments.

[0041] 2024,114

[0042] Reference symbol list:

[0043] 20 System

[0044] 22 Event 24 Sonar data

[0045] 26 Water-based sound transducers / passive sonar

[0046] 28 Water sound

[0047] 30 Data processing unit

[0048] 32 sonar images with detected events

Claims

2024,114 Patent claims 1. Method for detecting events (22) in passive sonar data (24) comprising the following step: - Applying a mathematical model to a time series of sonar data (24) to detect an event (22) or a plurality of events in the sonar data (24) in the time-frequency domain using the model.

2. The method according to claim 1, comprising the following further steps: - continuous recording of sonar data (24); - Continuously checking the sonar data with the model for events (22).

3. Method according to claim 2 with the following further step: - Scaling the recorded sonar data (24).

4. A method according to any of the preceding claims, comprising the following further steps: - Applying direction calculation to the sonar data (24) to obtain a representation of the sonar data versus bearing and time; - Applying the model to the bearing-time representation of the sonar data (24) to detect the event or plurality of events per bearing or applying the model to individual time series of the sonar data per bearing to detect the event or plurality of events per bearing.

5. A method according to any of the preceding claims, comprising the following further steps: - Applying a frequency analysis to the sonar data (24) to obtain a representation of the sonar data (24) against frequency and time; - Applying the model to the time-frequency representation of the sonar data (24).

6. A method according to any of the preceding claims, comprising the following further steps: 2024,114 - Applying preprocessing, in particular normalization, to the time series of sonar data to obtain preprocessed sonar data, - Using one color channel of an image file to provide the model with the time series of the sonar data and another color channel to simultaneously provide the model with the pre-processed time series of the sonar data.

7. A method according to any of the preceding claims, comprising the following further step: - Classifying a detected event (22) using the model.

8. Method according to one of the preceding claims, wherein the mathematical model is trained using machine learning.

9. Data processing unit (30) for detecting events (22) in sonar data (24), wherein the data processing unit (30) is configured to perform the method according to one of the preceding claims.

10. System (20) comprising the data processing unit (30) according to claim 9 and a passive sonar (26) for recording the sonar data (24).

11. Computer program comprising instructions which, when the program is executed by a computer, cause it to execute the method according to one of the preceding claims using sonar data (24).

Images