Methods for detecting events in sonar data

Abstract

A method for detecting events (22) in sonar data (24) is disclosed comprising the following step: - Applying a machine learning-generated model to a time series of sonar data (24) to detect an event (22) or a plurality of events in the sonar data (24) using the model.

Description

[0001] The invention relates to the detection of events in sonar data. Events in sonar data can be natural sounds such as whale songs or technically generated sounds.

[0002] Currently, events are detected, for example, using energy detection based on clustering in the frequency domain. However, this type of event detection is prone to false positives or the detection of uninteresting events, as factors other than energy are generally not taken into account.

[0003] The object of the present invention is therefore to create an improved concept for the detection of events in sonar data.

[0004] The problem is solved by the subject matter of the independent patent claims. Further advantageous embodiments are the subject matter of the dependent patent claims.

[0005] Exemplary implementations show methods for detecting events in sonar data with the following step: a) Applying a machine learning model to a time series of sonar data to detect one or more events within the sonar data. That is, the sonar data is provided to the model in a temporal sequence to perform event detection. A visual representation of such a time series is, for example, a waterfall display. Furthermore, using the model has the advantage that the events can be directly categorized, for example, as continuous wave (CW) or frequency-modulated (FW) events, or as interesting and uninteresting events. Uninteresting events can then be directly suppressed, i.e., not displayed or output. Alternatively, the events can be color-coded, with interesting events highlighted, for example, in a specific color.Signal colors and uninteresting events, for example, can be displayed in pastel colors. Furthermore, the event detection approach using the model allows for monitoring the acoustic quality of the sonar data. The acoustic quality of the sonar data can be determined based on the different characteristics of radio interference compared to underwater sound signals. In particular, radio interference is characterized by a comparatively higher signal energy and rapid changes. It has also been shown that events below the conventional detection threshold can be reliably detected using the model.

[0006] The idea is to perform event detection in sonar data using artificial intelligence. The artificial intelligence comprises a model that can optionally include multiple, and in particular, different, sub-models. A (sub-)model describes a trained algorithm. Most common, well-known algorithms are suitable, especially artificial neural networks, Bayesian classifiers, support vector machines (SVMs), and linear regression. However, deep learning approaches such as transformer-based artificial neural networks and / or convolutional artificial neural networks (CNNs) are preferred. If the model includes several sub-models for event detection, similar algorithms (e.g., different artificial neural networks) as well as different algorithms (e.g., an artificial neural network and linear regression) can be combined.

[0007] In particular, it has been shown that no special algorithms are needed to perform event detection. It is sufficient to train commercially available algorithms with appropriate data. Such algorithms are also known as COTS (Components-Off-The-Shelf). It has been found that the structure—for example, the number of neurons or the number of layers in artificial neural networks—is irrelevant for event detection, as long as a minimum level of complexity is achieved. Models that perform well in public benchmarks are particularly suitable. Examples of such benchmarks include the "COCO dev-test" and the "COCO val2017." Models based on the DINO (DETR with Improved deNoising anchOr box) algorithm or the YOLO (You Only Look Once) algorithm are also examples of suitable models.

[0008] In "Tianhe Ren, Jianwei Yang, Shilong Liu, Ailing Zeng, Feng Li, Hao Zhang, Hongyang Li, Zhaoyang Zeng, Lei Zhang. A Strong and Reproducible Object Detector with Only Public Datasets. arXiv:2304.13027 [cs.CV]. (https: / / doi.org / 10.48550 / arXiv.2304.13027)" and "Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, Heung-Yeung Shum. DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection the models revealed.

[0009] Preferably, however, the model, or at least a (partial) model, is implemented as an artificial neural network. The artificial neural network can, for example, be based on Transformer technology and / or be or comprise a convolutional artificial neural network.

[0010] If a single model without sub-models is used, it preferentially detects all potentially occurring events. If multiple sub-models are used, the individual sub-models can, for example, detect different event categories. Event categories can be natural noises, harmonic noises, or short-term events. However, any other event categories can also be found.

[0011] Suitable training data includes real sonar data acquired during training runs, training maneuvers, or in operational use. Ideally, the data should be pre-classified, for example, by a trained sonar operator. This means that the actual events in the sonar data are identified beforehand. This process is called labeling. This enables supervised learning of the model. Synthetic data, i.e., data generated using technical means, is also suitable for training the model. Synthetic data can be used particularly when the amount of real training data is insufficient. In this case, a combination of real and synthetic data is preferably used for training.

[0012] In exemplary implementations, the method comprises the following further steps: - Continuous recording of sonar data and continuous monitoring of this data by the model for events. This means the amount of sonar data continuously increases with new measurements. The model can then also continuously search for events. For example, adding a measurement to the beginning of the sonar data time series might cause the most recent measurement in the series to be excluded. In other words, the model can consider sonar data within a predefined time period in the past. This results in real-time monitoring of the sonar data for events using the model.

[0013] In further embodiments, the method includes the step of scaling the recorded sonar data. It has been found that event detection functions more reliably when the sonar data is pre-scaled. The scaling is performed, for example, to a variance of the value of the sonar data, i.e., in the case of an image of brightness displayed on a monitor, from one across all considered sonar data points, i.e., for example, a displayed sonar image.

[0014] Optionally, the method includes applying direction-finding to the sonar data to obtain a bearing-time representation of the sonar data, and applying the model to the bearing-time representation of the sonar data to detect the event or multiple events per bearing, or applying the model to individual time series of the sonar data per bearing to detect the event or multiple events per bearing. Thus, it is possible to detect events in a spatial representation. The model can directly receive all the data, i.e., the bearing-time representation, and detect the event or multiple events from the data and determine their bearings. Alternatively, the time series can also be fed into the model per bearing. However, the events will then typically be detected at different adjacent bearings. A model-based or deterministic (e.g.,Energy-based analysis can then separate the events and determine the correct bearing for each event.

[0015] In addition to or as an alternative to direction finding, frequency analysis can be applied to the sonar data to obtain a representation of the sonar data against frequency and time, and the model can then be applied to this time-frequency representation. This allows events to be detected, for example, via their characteristic frequencies.

[0016] In a further embodiment, the model is also used to classify a detected event. That is, not only is the event detected (i.e., recognized), but it is also assigned to an event category or specifically named (e.g., whale song, ship propeller, etc.).

[0017] Further embodiments show, analogously, a data processing unit for detecting events in sonar data, which can perform the described process steps.

[0018] Furthermore, a system comprising a data processing unit and a passive sonar for recording the sonar data is disclosed. The passive sonar thus provides the sonar data, while the data processing unit performs the data processing, in particular the application of the model to the sonar data.

[0019] Analogously, a computer program comprising instructions is disclosed which, when the program is executed by a computer, cause it to execute the method according to one of the preceding claims using sonar data.

[0020] Preferred embodiments of the present invention are explained below with reference to the accompanying drawing. It shows: Fig. 1: A schematic block diagram of a system for detecting events in sonar data.

[0021] Before exemplary embodiments of the present invention are explained in detail below with reference to the drawings, it should be noted that identical, functionally equivalent or equivalent elements, objects and / or structures in the different figures are provided with the same reference numerals, so that the description of these elements shown in different exemplary embodiments is interchangeable or can be applied to one another.

[0022] Fig.Figure 1 shows a schematic block diagram of a system 20 for detecting events 22 in sonar data 24. The sonar data 24 can be generated from received underwater sound 28 using underwater sound transducers 26, i.e., a sonar. A data processing unit 30, i.e., for example, a computer, processes the sonar data 24 and applies a machine learning-trained model for detecting the events 22 to the sonar data. The result can then be a representation 32 of the sonar data with the events contained therein.

[0023] The disclosed (underwater) sound transducers are designed for use underwater, particularly in the sea. The transducers can convert underwater sound into an electrical signal (e.g., voltage or current) corresponding to the sound pressure, the (received) underwater sound signal. Furthermore, it is possible for the transducers to convert an applied electrical voltage into underwater sound. The electrical voltage can follow a predefined pattern and then be referred to as the (transmitted) sonar signal, while the underwater sound resulting from the sonar signal to be transmitted is referred to as the (transmitted) sonar signal. Examples of sonar signals are a chirp (frequency-modulated signal) or, as a special case of the chirp, a sweep (linearly frequency-modulated signal). The transducers can therefore be used as underwater sound receivers and / or underwater sound transmitters.The transducers can utilize a piezoelectric material, such as a piezoceramic, as the sensor material. Multiple underwater transducers, or one or more underwater transducers combined with a signal processing unit, can be referred to as a sonar system. The transducers can be used for (active and / or passive) sonar (sound navigation and ranging). The transducers are preferably not suitable for, or are not used in, medical applications. Likewise, the transducers are preferably not used for, or are not suitable for, the ultrasonic testing of materials.

[0024] Although some aspects have been described in connection with a device, it is understood that these aspects also constitute a description of the corresponding process, so that a block or component of a device can also be understood as a corresponding process step or as a feature of a process step. Similarly, aspects described in connection with or as a process step also constitute a description of a corresponding block, detail, or feature of a corresponding device.

[0025] Depending on the implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example, a magnetic or optical storage medium, on which electronically readable control signals, e.g., a computer program, are stored. These signals can interact with, or interact with, a programmable computer system (CPU and / or GPU) in such a way that the respective method is carried out. For this reason, the digital storage medium should be computer-readable. Embodiments can therefore include a data carrier that has electronically readable control signals capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.

[0026] In some embodiments, a programmable logic device (for example, a field-programmable gate array, an FPGA) can be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array can interact with a microprocessor to perform one of the methods described herein. Generally, in some embodiments, the methods are performed by any hardware device. This can be general-purpose hardware such as a computer processor (CPU) or a graphics processing unit (GPU), or hardware specific to the method, such as an ASIC. Distributed execution across the CPU and GPU is also possible.

[0027] The embodiments described above merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to other people skilled in the art. Therefore, it is intended that the invention be limited only by the scope of protection set forth in the following claims and not by the specific details presented herein by way of description and explanation of the embodiments. Reference symbol list: 20 System 22nd event 24 sonar data 26 Water-based sound transducers / passive sonar 28 Water sound 30 Data processing unit 32 sonar images with detected events QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Tianhe Ren, Jianwei Yang, Shilong Liu, Ailing Zeng, Feng Li, Hao Zhang, Hongyang Li, Zhaoyang Zeng, Lei Zhang. A Strong and Reproducible Object Detector with Only Public Datasets. arXiv:2304.13027 [cs.CV]. (https: / / doi.org / 10.48550 / arXiv.2304.13027

[0008] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, Heung-Yeung Shum. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. arXiv:2203.03605 [cs.CV]. (https: / / doi.org / 10.48550 / arXiv.2203.03605

[0008]

Claims

[1] Method for detecting events (22) in sonar data (24) comprising the following step: - Applying a machine learning-generated model to a time series of sonar data (24) to detect an event (22) or a plurality of events in the sonar data (24) using the model. [2] Method according to claim 1, comprising the following further steps: - continuous recording of sonar data (24); - Continuously checking the sonar data with the model for events (22). [3] Method according to claim 2 with the following further step: - Scaling the recorded sonar data (24). [4] A method according to any of the preceding claims, comprising the following further steps: - Applying direction calculation to the sonar data (24) to obtain a representation of the sonar data versus bearing and time; - Applying the model to the bearing-time representation of the sonar data (24) to detect the event or plurality of events per bearing or applying the model to individual time series of the sonar data per bearing to detect the event or plurality of events per bearing. [5] A method according to any of the preceding claims, comprising the following further steps: - Applying a frequency analysis to the sonar data (24) to obtain a representation of the sonar data (24) against frequency and time; - Applying the model to the time-frequency representation of the sonar data (24). [6] Method according to any of the preceding claims, comprising the following further step: - Classifying a detected event (22) using the model. [7] Data processing unit (30) for detecting events (22) in sonar data (24), wherein the data processing unit (30) is configured to perform the method according to one of the preceding claims. [8] System (20) comprising the data processing unit (30) according to claim 7 and a passive sonar (26) for recording the sonar data (24). [9] Computer program comprising instructions which, when the program is executed by a computer, cause it to perform the method according to any of the preceding claims using sonar data (24).