Procedure for creating an informed situation report

Abstract

A method for creating a reconnaissance situation picture (20) is disclosed, wherein the reconnaissance situation picture (20) is created based on sensor data (22) from sensors (24) located in the water, by means of a model (28) generated by machine learning (26).

Description

[0001] The invention relates to the creation of a reconnaissance situation report based on data from in-water sensors using a computer-implemented model trained using machine learning. Such a model can also be referred to as artificial intelligence. In particular, sensor data from different sensor types is used.

[0002] A situation picture is the spatial representation of information, particularly objects, either absolute or relative to a reference point (e.g., a platform, particularly the platform on which the sensors are located). In addition to any objects such as mines, unmanned or manned underwater vehicles, wrecks, reefs, marine animals (e.g., whales or schools of fish), manned or unmanned (e.g., floating) surface vehicles (e.g., ships), other information such as boundary layers in the water, actual waterborne sound speed (e.g., for different propagation directions), etc. can also be displayed.

[0003] A reconnaissance situation picture is understood to be one that depicts at least categorized objects, preferably classified objects when it comes to technical objects. Object categories include, for example, "surface vehicle," "underwater vehicle," "whale," "obstacle lying on the waterbed," "flying object," etc. Technical objects such as ships can be classified additionally. In this case, the technical object is assigned a further characteristic, for example, its type (e.g., manned submarine, in particular, the U-boat 212cd; torpedo, in particular, the DM2A4 heavy torpedo; manned ship, e.g., the K130 corvette; etc.).

[0004] Currently, a separate display of the sensor data is provided for each sensor or for each sensor arrangement of a plurality of, particularly similar, sensors. The sensor data is often processed using different algorithms to visualize different information. For sonar systems, as an example of a sensor arrangement, a waterfall display and a DEMON (Detection of Envelope Modulation on Noise) analysis can be performed and displayed. Based on this information, a trained sonar operator can create a situation report. However, the fusion of the information is usually carried out by the sonar operator.

[0005] CN 1 16 577 789 A discloses a real-time target detection system for an autonomous underwater vehicle (AUV).

[0006] DE 10 2021 208 506 A1 discloses an underwater vehicle and a method for improving a situational picture of an underwater vehicle.

[0007] The object of the present invention is therefore to create an improved concept for the creation of a situation report.

[0008] This object is achieved by the subject matter of the independent patent claims. Further advantageous embodiments are the subject matter of the dependent patent claims.

[0009] Embodiments show a method for creating a reconnaissance situation report, wherein at least part of the reconnaissance situation report is created based on sensor data from sensors located in the water using a model generated using machine learning. The situation report is preferably an underwater situation report. However, it can also be a combined underwater-surface situation report. This means that the combined underwater-surface situation report can also show surface contacts that are categorized or classified, for example, using non-acoustic sensors. The reconnaissance situation report is updated, preferably continuously or event-based. It is therefore a dynamic situation report. Furthermore, the model advantageously allows a forecast, i.e. a preview, of the individual object movements.

[0010] The idea is to evaluate the sensor data using artificial intelligence and to create a reconnaissance picture. The artificial intelligence comprises a model, which can optionally include a plurality of, in particular different, sub-models. A (sub-)model describes a trained algorithm. Most common algorithms are suitable, in particular artificial neural networks, Bayes classifiers, support vector machines (SVMs), linear regression, and in particular deep learning approaches such as artificial neural networks based on transformer technology and / or a convolutional artificial neural network (CNN), etc. If the model comprises several sub-models for creating the reconnaissance picture, similar algorithms (e.g. different artificial neural networks) or different algorithms (e.g. an artificial neural network and a linear regression) can be combined.In particular, it has been shown that no special algorithms are needed to create the reconnaissance situational picture. It is sufficient to train algorithms offered for commercial use with appropriate data. Such algorithms are also referred to as COTS (Components-Off-The-Shelf). It has been shown that the structure—for example, the number of neurons or the number of layers in artificial neural networks—is irrelevant for creating the reconnaissance situational picture as long as a minimum level of complexity is achieved. Models that also perform well in public benchmarks are particularly suitable. Examples of benchmarks include the "COCO dev-test" and the "COCO val2017." Examples include models based on the DINO (DETR with Improved deNoising anchOr box) algorithm or the YOLO (You Only Look Once) algorithm.

[0011] In "Tianhe Ren, Jianwei Yang, Shilong Liu, Ailing Zeng, Feng Li, Hao Zhang, Hongyang Li, Zhaoyang Zeng, Lei Zhang. A Strong and Reproducible Object Detector with Only Public Datasets. arXiv:2304.13027 [cs.CV]. (https: / / doi.org / 10.48550 / arXiv.2304.13027)" and "Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, Heung-Yeung Shum. DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection the models revealed.

[0012] Preferably, however, at least one (partial) model is implemented as an artificial neural network. The artificial neural network can, for example, be based on transformer technology and / or be or include a convolutional artificial neural network.

[0013] If a single model is used without submodels, this model preferably generates the entire reconnaissance situation picture. If multiple submodels are used, the individual submodels can generate parts of the situation picture. The parts of the situation picture can refer to different information (e.g., location of an object (detection) and type of object (categorization or classification)). However, the parts of the situation picture can also be spatially different areas of the situation picture, for example, different sectors arranged at different angles.

[0014] Real sensor data obtained during training drives, training maneuvers, or in operations is suitable as training data. The data should preferably be classified in advance, for example by a trained sonar operator. This means that the actual situational picture is created based on the training data. This enables supervised learning of the model. Synthetic data, i.e. data generated using technical means, is also suitable for training the model. Synthetic data can be used in particular when the volume of real training data is insufficient for training. In this case, a combination of real and synthetic data can preferably be used for training.

[0015] Various types of sensors are suitable, for example acoustic sensors, such as water sound transducers operated as active sonar or passive sonar, as well as non-acoustic sensors. Suitable non-acoustic sensors include acceleration sensors, AIS sensors (Automatic Identification System), optical sensors, quantum sensors or magnetic sensors. The sensors can be used alone or in an arrangement with other, particularly similar, sensors. One example of a sensor arrangement is a sonar array. Sensor arrangements have the advantage that further information can be obtained from signal differences between the individual sensors. In the case of a sonar array, direction formation (beamforming) is worth mentioning here, for example by adding the received sonar signals together with a predetermined delay.In a magnetic sensor arrangement, the direction of a magnetic field can be determined using a differential measurement. The sensor data can therefore preferably include data from different sensor types. In addition to the sensor data from sensors located in the water, sensor data from sensors located above the water can also be used. However, it is also possible to use only sensor data from sensors located in the water.

[0016] According to the invention, the method comprises preprocessing the sensor data so that the model creates the reconnaissance situation report based on the preprocessed sensor data. In principle, it is desirable for the model to be able to process the data without preprocessing. This is becoming increasingly likely with advances in artificial intelligence technology. However, it has currently been shown that preprocessing the sensor data can be advantageous. According to the invention, the preprocessing comprises feature extraction so that the model creates the reconnaissance situation report based on the extracted features. The preprocessing of the data can further comprise, for example, one or any combination of the following steps: direction generation, normalization, smoothing, signal analysis.During feature extraction, various operations, particularly mathematical ones, are performed on the sensor data in order to describe its properties. For example, the (e.g. moving) mean or the variance of the sensor signal can be suitable as a feature, to name just two examples. A DEMON analysis can be used as signal analysis. The data generated in this way can be used as input data for the model. If the model is divided into sub-models, the sub-models can process different data. A sub-model can, for example, receive the DEMON data together with object positions in order to categorize or classify the objects. The object positions can, for example, be obtained from the sensor data using another sub-model, e.g. the normalized and / or smoothed sensor data.Additionally or alternatively, the extracted features can also enable or support a categorization or classification of objects.

[0017] Additionally or alternatively, submodels can receive the sensor data in parallel after the direction has been determined, so that the reconnaissance picture is created in parallel by, especially identical, submodels. Each submodel can use the sensor data for one direction as input data. Furthermore, or alternatively, submodels can be trained to process sensor data from different types of sensors.

[0018] In further embodiments, the model has feedback, so that the model creates a reconnaissance situation picture that follows the current reconnaissance situation picture based on at least part of the current reconnaissance situation picture and the sensor data. This means that the current reconnaissance situation picture can also represent an input parameter for the model. In particular, the feedback results in the possibility for the system to reflect itself and optionally correct itself. For example, a plausibility check can be carried out for the objects. For example, an object can only move a finite distance within two consecutive observation points in time and cannot change its classification or category all at once. Nevertheless, a change in the categorization or classification should be possible if this is appropriate on the basis of a larger existing data set.The plausibility check can be carried out as part of post-processing or using the model.

[0019] The method may further comprise adapting the model using current sensor data. In principle, if the method is carried out on a watercraft, be it a surface or an underwater vehicle (manned or unmanned, i.e., autonomous or remotely controlled), it is possible to adapt the model during a maneuver. However, since it is not possible to predict how the model will change as a result, the model will usually be adapted between two maneuvers. The sensor data can be stored in a database for this purpose and used in a base station to train the model. If a test confirms an improvement in the model, the model can be distributed across all platforms that use the model. This means that the model can also be trained on new watercraft, which are often monitored and "measured" by other platforms during one of their first voyages.The platforms determine the sensor data of the new watercraft. This enables subsequent classification of the new watercraft.

[0020] The method can be used for a variety of purposes, including anti-submarine warfare (ASW), mine hunting, combat management systems (CMS), torpedoes (e.g., for target detection), reconnaissance (ACINT), and submarines.

[0021] Similarly, a computer program is disclosed, comprising instructions which, when executed by a computer, cause the computer to carry out the method described above.

[0022] Furthermore, a data processing unit for creating a reconnaissance situation picture is disclosed, wherein the data processing unit is designed to carry out the above-described method based on the sensor data.

[0023] Likewise, a system comprising the data processing unit and a plurality of sensors located in the water is disclosed, wherein the data processing unit is designed to use the sensor data of the sensors located in the water to carry out the method described above.

[0024] The data processing unit of the system can be arranged on a first platform, and the sensors located in the water can be arranged on a second platform. The second platform is designed to transmit the sensor data from the sensors located in the water to the first platform via a data transmission link, in particular a radio link or an underwater communication link. The data processing unit can also carry out the method described above based on the received sensor data. Notwithstanding this, it is also possible for additional sensors to be arranged on the first platform, and for the data processing unit to use both the sensor data from the sensors arranged on the first platform and the sensors arranged on the second platform to generate the reconnaissance situation report.The decentralized arrangement of the sensors makes it possible to detect objects from different angles, although under certain conditions objects can only be detected from a specific angle.

[0025] Furthermore, the system can include a screen layout, with the reconnaissance situational picture being displayed on the screen layout as well as a conventional representation of the sensor data. The conventional representation of the sensor data can, for example, include a DEMON analysis or a waterfall display.

[0026] Preferred embodiments of the present invention are explained below with reference to the accompanying drawings. Fig. 1: a schematic block diagram of a process for creating a reconnaissance situation report; Fig. 2: a schematic block diagram of a method for creating a reconnaissance situation report according to embodiments; Fig. 3: a schematic block diagram of a method for creating a reconnaissance situation report according to embodiments; Fig. 4: a schematic block diagram of a method for creating a reconnaissance situation report according to embodiments.

[0027] Before exemplary embodiments of the present invention are explained in more detail below with reference to the drawings, it is pointed out that identical, functionally identical or equivalent elements, objects and / or structures in the different figures are provided with the same reference numerals, so that the description of these elements shown in different exemplary embodiments is interchangeable or can be applied to one another.

[0028] Fig. 1 shows a schematic block diagram of a method for creating a reconnaissance situation report 20, wherein the reconnaissance situation report 20 is created based on sensor data 22 from sensors 24 located in the water, using a model 28 generated with machine learning 26. The training 26 initially takes place before the model 28 is used. However, it is also possible to update the model 28 during operation, i.e., to continue training it.

[0029] Either real sensor data, synthetic sensor data or a combination of real and synthetic sensor data 22 can be used as training data 30 for machine learning, i.e. the training 26.

[0030] To execute the method using the trained model 28, the model 28 can access the sensor data 22, in particular continuously. The sensor data 22 is typically digitized, i.e., sampled, as a time-value pair. Using the sensor data 22, the model 28 outputs situational image data 32. The situational image data 32 can represent a complete situational image 20 or a difference from the (temporally) previous situational image 20.

[0031] Fig. 2 shows a schematic block diagram of an embodiment of the method from Fig. 1. In this exemplary embodiment, the model 28 comprises a plurality of sub-models 28', 28'', 28''', 28'''' operating in parallel. The models 28', 28'', 28''', 28'''' can each contribute situational picture data 32 to the reconnaissance situational picture 20. For example, the sub-models 28', 28'', 28''', 28'''' can process different sensor data 22.

[0032] Fig. 3 shows a schematic block diagram of a further embodiment of the method from Fig. 1. In contrast to the embodiment from Fig. 2, two sub-models 28', 28'' are arranged consecutively. This means that the first sub-model 28' processes the sensor data 22 and outputs pre-position image data 32'. The second sub-model 28" processes the pre-position image data 32' and outputs the position image data 32. It should be noted that one and / or both sub-models 28', 28" can also have further parallel sub-models, analogous to the embodiment in Fig. 2. For example, the first sub-model 28' may comprise a plurality of sub-models. The respective pre-situation image data 32' of the parallel sub-models can be evaluated by the second sub-model 28" and output the situation image data 32. The sub-models can be trained with different training data 30', 30". This also applies to the sub-models of the exemplary embodiment from Fig. 2.

[0033] Fig. 4 shows a schematic block diagram of an embodiment of the block diagram of Fig. 1. In addition, the method comprises preprocessing 34 of the sensor data 22 to obtain preprocessed sensor data 36. The model 28 processes the preprocessed sensor data 36. Based on the preprocessed sensor data 36, ​​the model 28 outputs the situational image data 32. In addition or as an alternative to the preprocessing, the method according to this exemplary embodiment comprises postprocessing 38 to obtain postprocessed situational image data 32. The postprocessing can, for example, involve processing the results or, optionally, merging the data.

[0034] The disclosed (water) sound transducers are designed for use underwater, particularly at sea. The sound transducers can convert water sound into an electrical signal (e.g., voltage or current) corresponding to the sound pressure, the (received) water sound signal. Furthermore, it is possible for the sound transducers to convert an applied electrical voltage into water sound. The electrical voltage can follow a predetermined pattern and then be referred to as the (to be transmitted) sonar signal, while the water sound resulting from the sonar signal to be transmitted is referred to as the (transmitted) sonar signal. Examples of sonar signals are a chirp (frequency-modulated signal) or, as a special case of chirp, a sweep (linear frequency-modulated signal). The sound transducers can therefore be used as water sound receivers and / or as water sound transmitters.The sound transducers can contain a piezoelectric material, such as a piezoceramic, as their sensor material. A plurality of waterborne sound transducers or one or more waterborne sound transducers in conjunction with a signal processing unit can be referred to as a sonar system. The sound transducers can be used for (active and / or passive) sonar (sound navigation and ranging). The sound transducers are preferably not suitable for medical applications or are not used for medical applications. Likewise, the sound transducers are preferably not used for ultrasonic testing of materials or are not suitable for this purpose.

[0035] Although some aspects have been described in connection with a device, it should be understood that these aspects also represent a description of the corresponding method, so that a block or component of a device can also be understood as a corresponding method step or as a feature of a method step. Analogously, aspects described in connection with or as a method step also represent a description of a corresponding block, detail, or feature of a corresponding device.

[0036] Depending on the implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example a magnetic or optical memory, on which electronically readable control signals, e.g. a computer program, are stored, which can interact or interact with a programmable computer system (CPU and / or GPU) in such a way that the respective method is carried out. For this reason, the digital storage medium should be computer-readable. Embodiments can therefore comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.

[0037] In some embodiments, a programmable logic device (e.g., a field-programmable gate array, an FPGA) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array may interact with a microprocessor to perform any of the methods described herein. Generally, in some embodiments, the methods are performed by any hardware device. This may be general-purpose hardware such as a computer processor (CPU) or a graphics processing unit (GPU), or method-specific hardware such as an ASIC. Distributed execution across the CPU and GPU is also possible.

[0038] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the embodiments herein. List of reference symbols: 20 clarified situation report 22 Sensor data 24 sensors 26 trainings 28 models 30 training data 32 situational data 32' pre-layout data 32" post-processed situational image data 34 Preprocessing 36 preprocessed sensor signals 38 Post-processing

Claims

[1] Method for creating a reconnaissance situation picture (20), wherein the reconnaissance situation picture (20) is created based on sensor data (22) from sensors (24) located in the water by means of a model (28) generated using machine learning (26), wherein preprocessing (34) of the sensor data (22) is carried out so that the model (28) creates the reconnaissance situation picture (20) based on the preprocessed sensor data (22), wherein the preprocessing (34) comprises feature extraction so that the model (28) creates the reconnaissance situation picture (20) based on the extracted features. [2] Method according to claim 1, wherein the preprocessing (34) comprises a normalization so that the model (28) creates the reconnaissance situation picture (20) based on the normalized sensor data (22). [3] Method according to one of the preceding claims, wherein the preprocessing (34) comprises a direction formation, so that the model (28) creates the clarified situation picture (20) based on sensor data (22) from different directions of incidence. [4] Method according to one of the preceding claims, wherein the model (28) is formed with an artificial neural network as algorithm. [5] Method according to one of the preceding claims, wherein the model (28) has a feedback, so that the model (28) creates a reconnaissance situation picture (20) following the current reconnaissance situation picture (20) based on the current reconnaissance situation picture (20) and the sensor data (22). [6] Method according to one of the preceding claims, wherein the model (28) enables a prediction of the movements of the objects represented in the reconnaissance situation image. [7] Computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to one of the preceding claims. [8] Data processing unit for creating a reconnaissance situation picture (20), wherein the data processing unit is designed to carry out the method according to one of claims 1 to 6. [9] System comprising the data processing unit according to claim 8 and a plurality of sensors located in the water, wherein the data processing unit is designed to use the sensor data (22) of the sensors (24) located in the water to carry out the method according to one of claims 1 to 6. [10] System according to claim 9, wherein the data processing unit is arranged on a first platform and wherein the sensors (24) located in the water are arranged on a second platform, wherein the second platform is designed to send the sensor data (22) of the sensors (24) located in the water to the first platform via a data transmission link, in particular a radio link, wherein the data processing unit is designed to carry out the method according to one of claims 1 to 6 based on the received sensor data (22). [11] System according to one of claims 9 or 10, comprising a screen arrangement, wherein the reconnaissance situation image (20) is displayed on the screen arrangement as well as a conventional representation of the sensor data.

Images