Method for generating a clarified positional image

Abstract

A method for generating a clarified positional image (20) is disclosed, wherein the clarified positional image (20) is generated on the basis of sensor data (22) from sensors (24) located in water using a model (28) generated by means of machine learning (26).

Description

[0001] METHOD FOR CREATING A RECOVERED SITUATIONAL PICTURE

[0002] Description

[0003] The invention relates to the creation of a detailed situational picture based on data from sensors located in water, using a computer-implemented model trained by machine learning. Such a model can also be referred to as artificial intelligence. In particular, sensor data from different types of sensors are used.

[0004] A situational image is the spatial representation of information, especially objects, either absolutely or relative to a reference point (e.g., a platform, particularly the platform on which the sensors are located). Besides any objects such as mines, unmanned or manned underwater vehicles, shipwrecks, reefs, marine animals (e.g., whales or schools of fish), manned or unmanned (e.g., floating) surface vehicles (e.g., ships), other information such as boundary layers in the water, the actual speed of sound in water (e.g., for different directions of propagation), etc., can also be displayed.

[0005] A comprehensive situational picture is understood to be a situational picture that depicts at least categorized objects, preferably classified objects if they are technical objects. Object categories include, for example, "surface craft," "submersible craft," "whale," "obstacle lying on the seabed," "aircraft," etc. For technical objects such as ships, it is possible to further classify them. In this case, the technical object is assigned an additional attribute, such as its type (e.g., manned submarine, specifically submarine 212cd; torpedo, specifically heavyweight torpedo DM2A4; manned ship, e.g., corvette K130; etc.).

[0006] Currently, for each sensor or sensor array, especially for multiple similar sensors, a separate display of the sensor data is used. Often, the sensor data is also processed with different algorithms to visualize different information. For sonar systems, as an example of a sensor array, a waterfall display and a DEMON (Detection of Envelope Modulation on Noise) analysis can be performed and displayed. Based on this information, a trained sonar operator can create a situational awareness picture. However, the fusion of the information is usually performed by the sonar operator.

[0007] The object of the present invention is therefore to create an improved concept for generating a situational image.

[0008] The problem is solved by the subject matter of the independent patent claims. Further advantageous embodiments are the subject matter of the dependent patent claims.

[0009] Exemplary embodiments show a method for generating a detailed situational picture, wherein, based on sensor data from sensors located in the water, at least a part of the detailed situational picture is generated using a machine learning model. The situational picture is preferably an underwater situational picture. However, it can also be a combined underwater-surface situational picture. That is, the combined underwater-surface situational picture can also depict surface contacts, which are categorized or classified, for example, using non-acoustic sensors. The detailed situational picture is updated, preferably continuously or event-driven. It is therefore a dynamic situational picture. Furthermore, the model advantageously allows for a prediction, i.e., a forecast, of the individual object movements.

[0010] The idea is to analyze sensor data using artificial intelligence and create a comprehensive situational picture. The artificial intelligence comprises a model that can optionally include multiple, and in particular, different sub-models. A (sub-)model describes a trained algorithm. Most common, well-known algorithms are suitable, especially artificial neural networks, Bayesian classifiers, support vector machines (SVMs), linear regression, and especially deep learning approaches such as artificial neural networks based on transformer technology and / or convolutional artificial neural networks (CNNs), etc. If the model comprises several sub-models for creating the comprehensive situational picture, similar algorithms (e.g., different artificial neural networks) as well as different algorithms (e.g., an artificial neural network and linear regression) can be combined.In particular, it has been shown that no special algorithms are needed to create the situational awareness picture. It is sufficient to train commercially available algorithms with appropriate data. Such algorithms are also known as COTS (Components-Off-The-Shelf). It has been found that the structure—for example, the number of neurons or the number of layers in artificial neural networks—is irrelevant for creating the situational awareness picture, as long as a minimum level of complexity is achieved. Models that perform well in public benchmarks are particularly suitable. Examples of such benchmarks include the "COCO dev-test" and the "COCO val2017." Models based on the DINO (DETR with Improved deNoising anchOr box) algorithm or the YOLO (You Only Look Once) algorithm are also examples of suitable models.

[0011] In “Tianhe Ren, Jianwei Yang, Shilong Liu, Ailing Zeng, Feng Li, Hao Zhang, Hongyang Li, Zhaoyang Zeng, Lei Zhang. A Strong and Reproducible Object Detector with Only Public Datasets. arXiv:2304.13027 [cs.CV], (https: / / doi.org / 10.48550 / arXiv.2304.13027)“ and „Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, Heung-Yeung Shum. DINO: DETR with Improved DeNoising for Anchor Box End Detection. arXiv:2203.03605 [cs.CV], (https: / / doi.org / 10.48550 / arXiv.2203.03605)“ are Examples for the Models available.

[0012] Preferably, at least one (sub-)model is implemented as an artificial neural network. This artificial neural network can, for example, be based on Transformer technology and / or be a convolutional artificial neural network. If a single model without sub-models is used, it preferably generates the entire situational picture. If multiple sub-models are used, the individual sub-models can generate parts of the situational picture. These parts of the situational picture can relate to different pieces of information (e.g., the location of an object (detection) and the type of object (categorization or classification)). The parts of the situational picture can also represent spatially distinct areas of the situational picture, for example, different sectors arranged at different angles.

[0013] Suitable training data includes real sensor data acquired during training runs, training maneuvers, or in operational use. Preferably, the data should be pre-classified, for example, by a trained sonar operator. This means that the actual situational awareness picture is created for the training data. This enables supervised learning of the model. Synthetic data, i.e., data generated using technical means, is also suitable for training the model. Synthetic data can be used particularly when the amount of real training data is insufficient. In this case, a combination of real and synthetic data is preferably used for training.

[0014] Various types of sensors are suitable, including acoustic sensors such as underwater sonar transducers, which can be operated as active or passive sonar, as well as non-acoustic sensors. Examples of non-acoustic sensors include accelerometers, AIS (Automatic Identification System) sensors, optical sensors, quantum sensors, and magnetic sensors. The sensors can be used individually or in an array with other, particularly similar, sensors. A sonar array is one example of a sensor array. Sensor arrays offer the advantage of providing additional information by analyzing signal differences between individual sensors. In the case of a sonar array, this includes beamforming, for example, by summing the received sonar signals with a predetermined delay.In a magnetic sensor array, the direction of a magnetic field can be determined via differential measurement. The sensor data can therefore preferably include data from different sensor types. In addition to sensor data from sensors located in the water, sensor data from sensors located above the water can also be used. However, it is also possible to use only sensor data from sensors located in the water.

[0015] In exemplary implementations, the method includes preprocessing the sensor data so that the model generates the resolved situational picture based on the preprocessed sensor data. Ideally, the model should be able to process the data without preprocessing. This is becoming increasingly likely with advancements in artificial intelligence technology. However, it has currently been shown that preprocessing the sensor data can be advantageous. This data preprocessing can include, for example, one or any combination of the following steps: direction determination, feature extraction, normalization, smoothing, and signal analysis.

[0016] That is, if the preprocessing includes normalization, the model can create the resolved situational picture based on the normalized sensor data. If the preprocessing includes feature extraction, the model can create the resolved situational picture based on the extracted features. If the preprocessing includes direction determination, the model can create the resolved situational picture based on sensor data from different directions of incidence.

[0017] Feature extraction involves performing various operations, particularly mathematical ones, on the sensor data to describe its properties. For example, the (e.g., moving) mean or the variance of the sensor signal can be suitable as a feature, to name just two possibilities. Signal analysis can be performed, for example, using a DEMON analysis. The data generated in this way can be used as input for the model. If the model is divided into sub-models, these sub-models can process different data. One sub-model, for example, can receive the DEMON data along with object positions to categorize or classify the objects. The object positions can be obtained from the sensor data, such as normalized and / or smoothed sensor data, using another sub-model.Additionally or alternatively, the extracted features can also enable or support the categorization or classification of objects.

[0018] Additionally or alternatively, submodels can receive the sensor data in parallel after direction determination, so that the resolved situational image is generated in parallel by, in particular, identical submodels. The sensor data for one direction can be used as input data for each submodel. Furthermore, additionally or alternatively, submodels can be trained to process sensor data from different types of sensors.

[0019] In further embodiments, the model features feedback, enabling it to generate a subsequent resolved situational picture based on at least a portion of the current resolved situational picture and the sensor data. This means that the current resolved situational picture can also serve as an input parameter for the model. In particular, the feedback allows the system to reflect on and optionally correct itself. For example, a plausibility check can be performed for the objects. An object can only move a finite distance between two consecutive observation points, nor can its classification or category change simultaneously. Nevertheless, a change in categorization or classification should be possible if warranted based on a larger available dataset.The plausibility check can be performed as part of post-processing or using the model.

[0020] The procedure can further include adapting the model using current sensor data. In principle, if the procedure is carried out on a watercraft, whether surface or underwater (manned or unmanned, i.e., autonomous or remotely controlled), it is possible to adapt the model during a maneuver. However, since it is unpredictable how this will change the model, it will usually only be adapted between two maneuvers. The sensor data can be stored in a database for this purpose and used at a base station to train the model. If a test confirms an improvement in the model, the model can be distributed to all platforms that use it. This allows the model to be trained on new watercraft, which are often monitored and "measured" by other platforms during one of their first voyages.The platforms determine the sensor data of the new watercraft. This enables a subsequent classification of the new watercraft.

[0021] The method can be used for various purposes. Examples include anti-submarine warfare (ASW), mine hunting, use in a combat management system (CMS), in a torpedo (e.g., for target detection), for reconnaissance (ACINT - Acoustic Intelligence), or in a submarine.

[0022] Analogously, a computer program is disclosed, comprising instructions which, when the program is executed by a computer, cause it to perform the procedure described above.

[0023] Furthermore, a data processing unit for creating an informed situational picture is disclosed, wherein the data processing unit is configured to execute the aforementioned procedure based on the sensor data.

[0024] Likewise, a system comprising the data processing unit and a plurality of sensors located in the water is disclosed, wherein the data processing unit is configured to use the sensor data from the sensors located in the water to execute the procedure described above.

[0025] The system's data processing unit can be located on a first platform, and the underwater sensors can be located on a second platform. The second platform is configured to transmit the sensor data from the underwater sensors to the first platform via a data transmission link, in particular a radio link or an underwater communication link. The data processing unit can also execute the aforementioned procedure based on the received sensor data. Notwithstanding the above, it is also possible for additional sensors to be located on the first platform, and for the data processing unit to use the sensor data from both the sensors on the first platform and the sensors on the second platform to generate the resulting situational awareness picture.The decentralized arrangement of the sensors makes it possible to detect objects from different viewpoints, although under certain conditions, object detection may only be possible from a specific viewpoint.

[0026] Furthermore, the system can include a screen layout, displaying both the reconstructed situational picture and a classic representation of the sensor data. This classic representation of the sensor data could, for example, include a DEMON analysis or a waterfall display.

[0027] Preferred embodiments of the present invention are explained below with reference to the accompanying drawings. These show:

[0028] Fig. 1 : a schematic block diagram of a method for creating a detailed situational picture;

[0029] Fig. 2: a schematic block diagram of a method for creating a clarified situational picture according to exemplary embodiments;

[0030] Fig. 3: a schematic block diagram of a method for creating a detailed situational picture according to exemplary embodiments;

[0031] Fig. 4: a schematic block diagram of a method for creating a detailed situational picture according to exemplary embodiments.

[0032] Before exemplary embodiments of the present invention are explained in detail below with reference to the drawings, it should be noted that identical, functionally equivalent or equivalent elements, objects and / or structures in the different figures are provided with the same reference numerals, so that the description of these elements shown in different exemplary embodiments is interchangeable or can be applied to one another.

[0033] Fig. 1 shows a schematic block diagram of a method for generating a detailed situational awareness picture 20, wherein the detailed situational awareness picture 20 is generated based on sensor data 22 from sensors 24 located in the water, using a model 28 generated by machine learning 26. The initial training 26 takes place before the use of the model 28. However, it is also possible to update the model 28 during operation, i.e., to train it further.

[0034] As training data 30 for machine learning, i.e. the training 26, either real sensor data, synthetic sensor data or a combination of real and synthetic sensor data 22 can be used.

[0035] To execute the procedure using the trained model 28, the model 28 can access the sensor data 22, particularly continuously. The sensor data 22 is typically available in digitized, i.e., sampled, form as a time-value pair. Using the sensor data 22, the model 28 outputs situational awareness data 32. The situational awareness data 32 can represent a complete situational awareness picture 20 or a difference from the (temporally) previous situational awareness picture 20.

[0036] Fig. 2 shows a schematic block diagram of an embodiment of the method from Fig. 1. In this embodiment, the model 28 comprises a plurality of parallel-operating sub-models 28', 28", 28'", 28"". The models 28', 28", 28'", 28"" can each contribute positional image data 32 to the resolved positional image 20. For example, the sub-models 28', 28", 28'", 28"" can process different sensor data 22.

[0037] Fig. 3 shows a schematic block diagram of another embodiment of the method from Fig. 1. In contrast to the embodiment in Fig. 2, here two sub-models 28', 28" are arranged sequentially. That is, the first sub-model 28' processes the sensor data 22 and outputs preliminary situational data 32'. The second sub-model 28" processes the preliminary situational data 32' and outputs the situational data 32. It should be noted that one and / or both sub-models 28', 28" can also include further parallel sub-models, analogous to the embodiment in Fig. 2.

[0038] 2. For example, the first sub-model 28' can contain a plurality of sub-models. The second sub-model 28" can evaluate the respective preliminary situational awareness data 32' of the parallel sub-models and output the situational awareness data 32. The sub-models can be trained with different training data 30', 30". This also applies to the sub-models of the execution example from Fig. 2.

[0039] Fig. 4 shows a schematic block diagram of an embodiment of the block diagram from Fig. 1. The method additionally includes preprocessing 34 of the sensor data 22 to obtain preprocessed sensor data 36. The model 28 processes the preprocessed sensor data 36. Based on the preprocessed sensor data 36, ​​the model 28 outputs the situational awareness data 32. In addition to or as an alternative to the preprocessing, the method according to this embodiment includes postprocessing 38 to obtain postprocessed situational awareness data 32. The postprocessing can, for example, involve preparing the results or, optionally, fusing the data.

[0040] The disclosed (underwater) sound transducers are designed for use underwater, particularly in the sea. The transducers can convert underwater sound into an electrical signal (e.g., voltage or current) corresponding to the sound pressure, the (received) underwater sound signal. Furthermore, it is possible for the transducers to convert an applied electrical voltage into underwater sound. The electrical voltage can follow a predefined pattern and then be referred to as the (transmitted) sonar signal, while the underwater sound resulting from the sonar signal to be transmitted is referred to as the (transmitted) sonar signal. Examples of sonar signals are a chirp (frequency-modulated signal) or, as a special case of the chirp, a sweep (linearly frequency-modulated signal). The transducers can therefore be used as underwater sound receivers and / or underwater sound transmitters.The transducers can utilize a piezoelectric material, such as a piezoceramic, as the sensor material. Multiple underwater transducers, or one or more underwater transducers combined with a signal processing unit, can be referred to as a sonar system. The transducers can be used for (active and / or passive) sonar (sound navigation and ranging). The transducers are preferably not suitable for, or are not used in, medical applications. Likewise, the transducers are preferably not used for, or are not suitable for, the ultrasonic testing of materials.

[0041] Although some aspects have been described in connection with a device, it is understood that these aspects also constitute a description of the corresponding process, so that a block or component of a device can also be understood as a corresponding process step or as a feature of a process step. Similarly, aspects described in connection with or as a process step also constitute a description of a corresponding block, detail, or feature of a corresponding device.

[0042] Depending on the implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example, a magnetic or optical storage medium, on which electronically readable control signals, e.g., a computer program, are stored. These signals can interact with a programmable computer system (CPU and / or GPU) in such a way that the respective method is carried out. For this reason, the digital storage medium should be computer-readable. Embodiments can therefore include a data carrier that has electronically readable control signals capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.

[0043] In some embodiments, a programmable logic device (for example, a field-programmable gate array, an FPGA) can be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array can interact with a microprocessor to perform one of the methods described herein. Generally, in some embodiments, the methods are performed by any hardware device. This can be general-purpose hardware such as a computer processor (CPU) or a graphics processing unit (GPU), or hardware specific to the method, such as an ASIC. Distributed execution across the CPU and GPU is also possible.

[0044] The embodiments described above merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to other people skilled in the art. Therefore, it is intended that the invention be limited only by the scope of protection set forth in the following claims and not by the specific details presented herein by way of description and explanation of the embodiments.

[0045] Reference symbol list:

[0046] 20 clarified situation picture

[0047] 22 sensor data points, 24 sensors

[0048] 26 Training

[0049] 28 Model

[0050] 30 training data

[0051] 32 Situational awareness data 32' Preliminary situational awareness data

[0052] 32" post-processed situational awareness data

[0053] 34 Preprocessing

[0054] 36 pre-processed sensor signals

[0055] 38 Post-processing

Claims

Patent claims 1. Method for creating a clear situational picture (20), wherein the clear situational picture (20) is created based on sensor data (22) from sensors (24) located in the water, using a model (28) generated with machine learning (26).

2. Method according to claim 1, wherein a preprocessing (34) of the sensor data (22) is performed, such that the model (28) creates the clarified situational picture (20) based on the preprocessed sensor data (22).

3. Method according to claim 2, wherein the preprocessing (34) comprises normalization such that the model (28) creates the clarified situational picture (20) based on the normalized sensor data (22).

4. Method according to one of claims 2 or 3, wherein the preprocessing (34) comprises feature extraction such that the model (28) creates the clarified situational image (20) based on the extracted features.

5. Method according to one of claims 2 to 4, wherein the preprocessing (34) comprises direction formation, such that the model (28) creates the clarified situational image (20) based on sensor data (22) from different directions of incidence.

6. Method according to one of the preceding claims, wherein the model (28) is formed with an artificial neural network as the algorithm.

7. Method according to one of the preceding claims, wherein the model (28) has feedback such that the model (28) creates a subsequent clarified situation picture (20) based on the current clarified situation picture (20) and the sensor data (22).

8. Method according to one of the preceding claims, wherein the model (28) enables a prediction of the movements of the objects represented in the clarified situational image.

9. Computer program comprising instructions which, when the program is executed by a computer, cause it to execute the method according to one of the preceding claims.

10. Data processing unit for creating an informed situational picture (20), wherein the data processing unit is configured to execute the method according to one of claims 1 to 8.

11. System comprising the data processing unit according to claim 10 and a plurality of sensors located in the water, wherein the data processing unit is configured to use the sensor data (22) of the sensors (24) located in the water to carry out the method according to any one of claims 1 to 8.

12. System according to claim 11, wherein the data processing unit is arranged on a first platform and wherein the sensors (24) located in the water are arranged on a second platform, wherein the second platform is configured to send the sensor data (22) of the sensors (24) located in the water to the first platform via a data transmission link, in particular a radio link, wherein the data processing unit is configured to execute the method according to any one of claims 1 to 8 based on the received sensor data (22).

13. System according to one of claims 11 or 12, comprising a screen arrangement, wherein the clarified situational image (20) is displayed on the screen arrangement as well as a classic representation of the sensor data.

Images