Safety device for a self-learning measurement system in the vehicle for the detection and classification of objects in the vehicle's environment using a deep learning method

Abstract

A self-learning, safety-relevant measuring system for use in vehicles and / or motor vehicles characterized by, - that it includes a first safety device (142) and one or more neural network models (16, 136, 138) for evaluating the measurement data of at least one sensor arrangement (11, 131) of the measurement system and - that the first safety device (142) at least monitors the neural network model (138) that processes these measurement data or data derived therefrom, and - that the first safety device (142) can influence the parameters of this neural network model (138), - to avoid unwanted detection results - to retrain this neural network model (138) during operation, - including undesired detection results within the meaning of this claim - faulty and safety-relevant detection results and / or - potentially erroneous detection results and / or - faulty detection results and / or - potentially security-relevant detection results.

Description

[0001] The invention relates to an ultrasonic measuring system for use in motor vehicles, which detects objects in the environment of the vehicle, determines their position and classifies them using a neural network model and a deep learning method based thereon.

[0002] This invention thus relates to safety-relevant ultrasonic systems for automotive purposes and in particular to ultrasonic systems with deep learning networks (deep learning network models), which preferably provide real-time image identification and system design. General introduction

[0003] Assistance systems that support drivers during parking maneuvers, as well as systems for autonomous parking, are often based on ultrasonic measurement systems that inexpensively and reliably detect obstacles in front of or behind the vehicle. The use of an artificial neural network model and a deep learning process allows the evaluation of the received signals from the ultrasonic measurement system beyond the traditional determination of distances to obstacles and enables object classification.

[0004] Currently available automotive ultrasound systems enable drivers to control their vehicles using ultrasonic signals, particularly when parking. In addition, the acquisition of images, the performance of measurements, and the application of algorithms and reporting software integrated into the overall system will be addressed here. These functionalities are intended to help better understand the traffic situation and hazards in the vehicle's surroundings. The system presented here is intended to take over some of the driver's environmental analysis tasks and report the results of these analyses to the driver or other vehicle subsystems. Before starting a journey, the driver must set up the vehicle and thus the system by selecting the settings and functions to be used and executed during the journey. During the journey, the driver can preferably make modifications.This usually begins with the selection of the sensors to be used. Depending on the weather, it may be sensible to switch off certain sensors and increase the importance of others. If necessary, the vehicle manufacturer can, for example, store a different measurement protocol in the vehicle's memory depending on the type of driver. Different operating settings for the ultrasonic measuring device can also be configured. In the past, the basic configuration of prototype ultrasonic sensor systems took a lot of time to develop, since an automotive ultrasonic device typically has various control parameters. Burdening the driver with reconfiguring the ultrasonic measuring systems in their vehicle is unacceptable. It is therefore sensible for ultrasonic measuring devices in vehicles to store the parameters optimized fully automatically or semi-automatically during driving in an operating parameter memory.By appropriately presetting the ultrasound devices, it can be ensured that they begin their learning process at a reasonable starting point.

[0005] It is desirable to further advance such automation of setting up an ultrasound system for a specific traffic situation. One promising area of ​​emerging artificial intelligence is deep learning. Deep learning is a rapidly developing branch of machine learning algorithms that mimics the way the human brain works when analyzing problems. The human brain remembers what it has learned from solving a similar problem in the past and applies this knowledge to solve a new one. Research is currently underway to identify potential applications of this technology in a number of fields, including pattern recognition, natural language processing, and computer vision.Deep learning algorithms have a distinct advantage over traditional forms of computer programming algorithms because they can be generalized and trained to detect image features by analyzing image samples rather than writing custom computer code. Another application is the detection of structures, groups, or trends in multidimensional data sets. However, the environmental situation of a vehicle visualized in an ultrasound system does not seem to be readily amenable to automated image recognition. Every situation is different, and the shapes, sizes, positions, and functions of the objects vary greatly. This is especially true for pedestrians and / or cyclists. Furthermore, the quality and clarity of ultrasound images vary, even when using the same ultrasound system.This is because the body habitus of pedestrians, wind, air circulation, humidity, and air temperature all influence the ultrasonic signals returned from the vehicle's environment, which are used to generate the images. Scanning an object in the vehicle's environment at a greater distance results in severely attenuated ultrasonic signals and poorly defined geometry in the images. Accordingly, it is desirable to apply deep learning to an ultrasound system in a way that is effective for a wide range of environmental situations to accelerate the creation of meaningful ultrasonic environment maps with correctly classified abstracted objects through ultrasound image recognition. State of the art

[0006] Ultrasonic measuring systems are currently used in vehicles as part of parking assistance systems. In particular, for vehicles that park or drive fully autonomously, the precise localization of obstacles is necessary, as described, for example, in the document DE 10 2008 036 009 B4. However, this only involves locating the obstacles and not classifying them or assigning them to object lists, such as other vehicles, hedges, curbs, etc. This is necessary in order to be able to derive further possible driving maneuvers. One way to classify objects is by measuring their external dimensions and comparing these with the dimensions of reference objects in a database, while simultaneously taking the object's speed into account, as described in DE 103 36 638 A1. This method has the disadvantage that it is generally not possible to scan an arbitrarily large volume of space with arbitrary precision.It is also difficult to reliably detect objects whose position relative to the vehicle is rotated by angles unknown to the database.

[0007] This disadvantage can be overcome by neural network models, as shown in [Dror 1995]. Here, a three-layer feedforward neural network model reliably detected objects at various angles relative to the ultrasonic sensors under laboratory conditions.

[0008] Since the objects found in the environment of vehicles have more complex geometries than those used in [Dror 1995], and the signal propagation conditions are also worse than under laboratory conditions, the use of a multi-layered neural network model based on a deep learning method is proposed here. The term "deep learning" does not yet have a uniform definition [Zhang 2018]. In this paper, the term "deep learning" follows the definition given in [LeCun 2015]. According to this definition, deep learning methods are representation learning methods in which a system, e.g., a neural network model, is given raw data as input information, and the system independently learns which representation obtained from the raw data is necessary for classification.Deep learning methods use several representation layers, which correspond to intermediate layers in a neural network model.

[0009] DE 10 2017 210 151 A1 discloses a safety device that monitors a vehicle control system that is not capable of self-learning. DE 10 2017 210 151 A1 does not explain how to ensure that the parameters of the neural networks used are not modified in a safety-threatening manner during the self-learning process. Task

[0010] The proposal is therefore based on the task of developing a system that not only locates detected objects in the vehicle's surroundings, but also performs a classification and is able to reliably perform this classification for different orientations of the objects relative to the vehicle.

[0011] It is a further object of the present invention to use deep learning technology to more reliably detect the geometry in the environment of a vehicle in ultrasound images and environment maps and / or ultrasound measurement data.

[0012] Another task is to recognize not only the geometry of the arrangement of objects in the vehicle's environment, but also the view of this geometry in ultrasound images and environment maps and / or ultrasound measurement data.

[0013] A further task is to use deep learning recognition of this geometry anatomy to automate the optimization of the ultrasound system for ultrasonic measurement of the environment of a vehicle. Solution to the task

[0014] In accordance with the principles of the present invention, an ultrasound system and method are described that enable an ultrasound system to identify the geometry in an ultrasound-generated environment map by processing it with a neural network model. The neural network model is first trained by presenting it with multiple environment maps or ultrasound measurement signals that map back to a known geometric arrangement of objects. Once trained, live images captured by a user are analyzed in real time by the neural network model, thereby identifying the geometry of the arrangement in the environment maps and / or ultrasound measurement signals.The identification of the geometry and / or the objects and / or the object arrangements is used to annotate or enhance the typically already existing environment map image and / or to configure the user controls on the ultrasound system to perform an in-depth analysis of the identified geometry of the objects in the vehicle's environment.

[0015] In a system of the type described above, the proposed solution is achieved by a deep learning method based on a neural network model that uses the received signals of an ultrasonic measuring system in the vehicle as input to classify objects and determine their positions.

[0016] The proposal is first explained using an ultrasonic measurement system as an example, then applied to a measurement system. The following describes the combination of such a measurement system with an ultrasonic measurement system and a self-learning method that includes a self-learning voter based on a neural network. This principle is then applied to the combination of two measurement systems. A generalized self-learning network composed of neural networks is then represented as nodes.

[0017] We first refer to Fig. 1. In Fig. 1 shows a schematically simplified block diagram of an ultrasonic vehicle environment analysis system constructed according to the principles of the present invention. An ultrasonic transducer array (1012) is provided in a vehicle part (1010), for example, a bumper, for transmitting ultrasonic waves and receiving echo information. The ultrasonic transducer array (1012) can be a one- or two-dimensional array of ultrasonic transducer elements capable of scanning, for example, in two or preferably three dimensions, both in height (in 3D) and in azimuth. The ultrasonic transducer array (1012) is preferably coupled to a microbeam former (1014), preferably already within the vehicle part (1010). The microbeam former (1014) controls the transmission and reception of ultrasonic signals by the ultrasonic transducer elements of the ultrasonic transducer array (1012).Microbeam formers (1014) within the meaning of this disclosure are capable of at least partially aligning the sensitivity lobe of the signals received by groups or "patches" of ultrasonic transducer elements. The microbeam former (1014) is coupled to a transmit / receive switch (1016) that switches between transmit and receive and protects the main beam former (1020) from high-energy transmit signals during the transmit phase. The transmission of ultrasonic beams thus shaped under the control of the microbeam former (1014) by the ultrasonic transducer array of the ultrasonic transducer assembly (1012) is controlled by a transmit controller (1018) coupled to the transmit / receive switch (1016) and the main beam former (1020). The main beamformer (1020) receives inputs from the user control or the user control panel (1038) if necessary.The transmission characteristics controlled by the transmission controller (1018) include, for example, the amplitude, phase, and polarity of the transmission waveforms. Beams formed in the direction of pulse transmission can be directed directly forward or orthogonally to the ultrasonic transducer array (1012) by varying the phase control of the individual ultrasonic transducers, or they can be radiated at different angles for a wider field of view.

[0018] The echoes received from a neighboring group of ultrasonic transducer elements are beamformed by being appropriately delayed for each ultrasonic transducer element and then combined. In this way, a receive lobe can be generated. The partially beamformed signals generated by the microbeamformer (1014) using each ultrasonic transducer group are coupled to the main beamformer (1020). The main beamformer (1020) combines the partially beamformed signals from individual ultrasonic transducer groups into a fully beamformed, coherent echo signal. For example, the main beamformer (1020) can have 128 channels, each of which receives a partially beamformed signal from a respective group of 12 ultrasonic transducer elements.In this way, the signals received from more than 1500 ultrasonic transducer elements of a two-dimensional ultrasonic transducer array (1012) can efficiently contribute to a single beamform signal.

[0019] The coherent echo signals are subjected to signal processing by a signal processor (1026), which includes filtering with a digital filter and preferably noise reduction. The signal processor (1026) can also shift the frequency band to a lower frequency range or baseband range. The processed echo signals are then preferably demodulated into quadrature components (I and Q), which provide signal phase information.

[0020] The beamformed and processed coherent echo signals are coupled to a video processor (1052) that generates an environment map of the vehicle. The video processor (1052) preferably performs amplitude (envelope) detection of the demodulated quadrature I and Q signal components by detecting the echo signal amplitude in the form of (I 2 +Q 2 ) 1 / 2calculated. The quadrature echo signal components are also coupled to a Doppler processor (1054), which stores ensembles of echo signals from discrete points in an image field, which are then used to estimate the Doppler shift at points in the image using a fast Fourier transform (FFT) processor. For a color Doppler image, the estimated Doppler flux values ​​at each point of the surround map are filtered and converted to color values, preferably using a lookup table or a heuristic formula. The surround map image signals and the Doppler flux values ​​are coupled to a scan converter (1032), which converts the surround map and Doppler samples from their typically acquired R-θ coordinates to Cartesian (x, y) coordinates for display in a desired display format, e.g., a rectilinear display format or a sector display format.Preferably, either the environment map image or the Doppler image can be shown alone in the user display. Furthermore, it is advantageous to display both images together in a single color Doppler image.

[0021] The ultrasound images generated by the scanning transducer (1032) are coupled to an image processor (1030) and a multiplanar reformatter (1044). The multiplanar reformatter (1044) converts echoes received from points in a common plane in a volumetric region of space surrounding the vehicle into an ultrasound image of that plane. A volume renderer (1042) converts the echo signals of a 3D data set into a projected 3D image as seen from a given reference point. The 2D or 3D images are coupled to an image processor (1030) for further enhancement, buffering, and temporary storage for display on an image display (1040). A graphics display overlay containing text and other graphic information, such as B. the data of detected objects e.g. from geographical map databases, is preferably generated by a graphics processor (1036) for display with the ultrasound images.

[0022] According to the principles of the present invention, the ultrasound system includes a neural network model (1080) whose software is stored in digital memory. Deep learning neural network models include software that can be written by a software designer and is also publicly available from a number of sources. An application that can be used to create a neural network model, called "NVidia Digits," is available at https: / / developer.nvidia.com / digits. NVidia Digits is a high-level user interface that includes a deep learning framework called "Caffe," developed by the Berkeley Vision and Learning Center (http: / / caffe.berkeleyvision.org / ). A list of common deep learning frameworks suitable for use in an implementation of the present invention can be found at https: / / developer.nvidia.com / deep-learning-frameworks.Coupled to the neural network model (1080) is a training image memory (1034) in which ultrasound images of known anatomy of a patient are stored and used to train the neural network model (1080) to identify ultrasound images of the geometry of the vehicle's surroundings. Live images acquired by the ultrasound system of . Fig. 1 with known vehicle surrounding geometries are presented to the neural network model (1080) after training to identify the geometry in the images by the neural network model (1080). In a constructed implementation of the present invention, the neural network model (1080) has been trained to identify not only the respective vehicle surrounding geometry in an ultrasound image, but also the view of the vehicle surrounding geometry seen in the ultrasound image, e.g., another car, a curb, or a fire hydrant. The trained neural network model generates an object detection score for the identified objects in the vehicle surroundings and also preferably a confidence factor of what the neural network model (1080) estimates as the accuracy of its identification (e.g., 80%, 100%, or another confidence factor).Once the neural network model (1080) is 100% confident in its identification, the image ID is coupled to a user control settings control (1048) of the ultrasound system, where system settings can be specified for analysis of the identified geometry type. These settings are preferably applied to a touchscreen display control (1046), where the graphics of a touchscreen user control display can be modified accordingly and coupled to the graphics processor (1036). The graphics processor (1036) generates the corresponding control graphics, which are applied to a touchscreen display (1060) on the user control panel (1038) of the ultrasound system.

[0023] A method for training and using a neural network model to identify anatomy in ultrasound images is described in Fig. 2. In step A (1090), a neural network model is installed on a computer-based system. The neural network model may be one designed by a software architect or it may be one created using one of the deep learning frameworks described above. In step B (1092), images of the same vehicle environment situation geometry, preferably obtained from a variety of test drives for similar situations, are presented to the neural network model. To train it to form the geometry, this presentation is performed together with the known type of situation geometry and its view in the images in step C (1094). The number of training images or training data sets used is preferably hundreds or thousands to train the neural network model in the variations of such vehicle environment situation geometry.Once the trained neural network model is installed in an ultrasound system in a vehicle, in step D (1096), environment maps captured by the ultrasound system are presented to the neural network model for identification. In step E (1098), the identification generated by the neural network model is used to annotate the images, e.g., as an obstacle or curb. This information is also used to set up and self-train the ultrasound system.

[0024] Variations of the system and method described above will be readily apparent to those skilled in the art. An ultrasound system can use deep learning models to assess image quality and provide feedback to the driver, enabling the driver to obtain higher-resolution images where possible. Driver decision support can be implemented by combining historical information with the identification of the vehicle's surrounding situation geometry. This data can be processed by deep learning algorithms to provide decision support outputs.

[0025] It should be noted here that an ultrasound system suitable for use in an implementation of the present invention, and in particular the component structure of the ultrasound system of Fig. 1, e.g., in hardware, software, or a combination thereof. The various embodiments and / or components of an ultrasound system, for example, the deep learning software modules or components and controls therein, may also be implemented as part of one or more computers or microprocessors. The computer or processor may, for example, include a computing device, an input device, a display unit, and an interface for accessing the Internet, as in Fig. 1. The computer or processor may include a microprocessor. The microprocessor may, for example, be connected to a communications bus to access a data network of the vehicle and / or a data network external to the vehicle to import training images and / or training data sets. The computer or processor may also include a memory. The memory devices, such as the image memory (1028), may include random access memory (RAM) and read-only memory (ROM). The computer or processor may further include a storage device, which may be a hard disk drive or a removable storage device such as a floppy disk drive, an optical drive, a solid-state drive, and the like. The memory device may also be another similar device for loading computer programs or other instructions into the computer or processor.

[0026] As used herein, the term "computer" or "module" or "processor" or "workstation" can include any processor-based or microprocessor-based system, including systems using microcontrollers, reduced instruction set computers (RISCs), ASICs, logic circuits, and any other circuit or processor capable of performing the functions described herein. The above examples are exemplary only and are not intended to limit the definition and / or meaning of these terms in any way.

[0027] The computer or processor executes a set of instructions stored in one or more memory elements to process input data. The memory elements can also store data or other information as desired or needed. The memory element can be in the form of an information source or a physical storage element in a processing machine.

[0028] The instruction set of an ultrasound system, including those that control the acquisition, processing, and transmission of ultrasound images as described above, may contain various instructions that instruct a computer or processor as a processing engine to perform certain operations such as the methods and procedures of the various embodiments of the invention. The set of instructions may be in the form of a software program. The software may exist in various forms, for example, as system software or application software, and it may be embodied as a tangible and non-transitory computer-readable medium. It may also be implemented, in whole or in part, as a finite state machine. Furthermore, the software may be in the form of a collection of separate programs or modules, such as a neural network model module, a program module within a larger program, or a portion of a program module.The software may also incorporate modular programming in the form of object-oriented programming. The processing of input data by the processing engine may occur in response to operator commands, in response to the results of previous processing, or in response to a request made by another processing engine. A) Ultrasonic measurement system with deep learning

[0029] A proposed ultrasonic measuring system (164) for use in motor vehicles for detecting obstacles and / or objects in the surroundings of a vehicle using deep learning methods preferably comprises an ultrasonic transducer arrangement (11). This is configured to acquire first ultrasonic measured values. This acquisition occurs by emitting ultrasonic signals that are emitted as a burst, pulse, or coded. The ultrasonic transducer arrangement preferably consists of several ultrasonic transmitters and ultrasonic receivers. Ultrasonic transducers may also be considered as components of the ultrasonic transducer arrangement.which can operate as ultrasound transmitters and receivers. The ultrasound transducer arrangement (11) generates first ultrasound measurement values ​​depending on the received ultrasound signals. These are preferably ultrasound echo signals of the ultrasound signals emitted by the ultrasound arrangement. To generate a user-friendly display, an image processor (18) is preferably used, which is coupled to the ultrasound transducer arrangement (11) and can generate two- or three-dimensional displays depending on the first ultrasound measurement values ​​of the ultrasound transducer arrangement (11). The proposed ultrasound measuring system (164) further preferably comprises a first memory (16) with a first neural network model (16),which is stored in the first memory (16). The first neural network model (16) can be implemented in hardware or as software on a processor. Preferably, the first neural network model (16) is adapted and provided such that it can receive the ultrasonic measurement values ​​from the ultrasonic transducer arrangement (11), possibly after undergoing suitable processing by amplifiers, filters, etc., and can identify the objects in the vicinity of the vehicle and / or their geometric arrangement in the vicinity of the vehicle in the first ultrasonic measurement values ​​using a deep learning technique by assigning an identification and thus can generate a first object recognition result. Furthermore, the proposed device comprises a two- or three-dimensional display (19) which is used to display the two- or three-dimensional representations generated by the image processor (18).is designed or suitable or provided. The two- or three-dimensional display (19) is preferably suitable or designed or provided to at least temporarily display one of the identifications and / or first object recognition results and / or information derived therefrom. This is thus an ultrasonic measuring system (164) for use in motor vehicles for detecting obstacles and / or objects in the environment of a vehicle using deep learning methods, having an ultrasonic transducer arrangement (11) that is configured to acquire first ultrasonic measurement values ​​and comprises an image processor (18) that is coupled to the ultrasonic transducer arrangement (11) and that can generate two- or three-dimensional representations depending on the first ultrasonic measurement values ​​of the ultrasonic transducer arrangement (11), and a first memory (16) with a first neural network model (16),which is stored in the first memory (16) and is adapted to receive the ultrasonic measurement values ​​and to identify the objects and their geometric arrangement in the surroundings of the vehicle in the first ultrasonic measurement values ​​using a deep learning technique by assigning an identification and thus generating a first object recognition result, and with a two- or three-dimensional display (19) which is designed, suitable or provided for displaying the two- or three-dimensional representations generated by the image processor (18), wherein said two- or three-dimensional display (19) is designed, suitable or provided for displaying at least temporarily one of the identifications and / or first object recognition results and / or information derived therefrom.

[0030] In a first refinement of this ultrasonic measuring system (164), the first neural network model is further designed to identify the view of the arrangement of objects and / or object components in the first ultrasonic measured values ​​and / or in the two- or three-dimensional representations. The first neural network model therefore has the function of an image recognition system. In the basic version, which was described above, the recognition alternatively also takes place on the basis of the raw sensor data. It is conceivable, in such a device, on the one hand, to recognize arrangements of objects and / or objects in the surroundings of the vehicle on the basis of the raw sensor data, i.e. here the ultrasonic measured values, with the aid of a first neural network model or on the basis of a possiblyto carry out a comparison of the environment map extracted therefrom with the aid of a second neural network model and then to distinguish between the output data streams of the first neural network model and the second neural network model with the aid of a third neural network model. Preferably, first parts of the output data streams of the first and second neural network models have a first weighting and second parts of the output data streams of the first and second neural network models have a second weighting. Preferably, the first weighting is higher than the second weighting. The purpose of this is to ensure that safety-relevant, potentially critical constellations that were detected by a neural network model of the first and second neural network model always prevail. The same applies to signals in the raw sensor data that also indicate a high probability of a critical event.

[0031] The above-described ultrasonic measuring system (164) preferably further comprises a plurality of first user controls (10), wherein the setting of one or more of the first user controls (10) occurs in response to the identification of an arrangement of objects and / or object components or of objects in the first ultrasonic measured values ​​and / or in the two- or three-dimensional representations and / or in dependence on the first object recognition results.

[0032] In the previously described ultrasonic measuring system (164), the setting or adaptation of control parameters of one or more of the controllers preferably occurs in response to the identification of the arrangement of objects and / or object components in the first ultrasonic measured values ​​and / or in the two- or three-dimensional representations and / or as a function of the first object detection results. In this case, the setting of one or more of the controllers (13) of the ultrasonic measuring system or of the vehicle preferably occurs in response to the identification of objects and / or object components and / or the arrangement of objects and / or objects and / or object components in the first ultrasonic measured values ​​and / or in the two- or three-dimensional representations and / or as a function of the first object detection results.

[0033] Furthermore, the ultrasonic measurement system (164) described in this section preferably comprises a user control display (9) that displays settings made in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the first ultrasonic measurements and / or in the two- or three-dimensional representations and / or in dependence on the first object detection result.

[0034] In the ultrasonic measurement system (164) described here, the user control display (9) preferably activates and displays control options in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the first ultrasonic measurement values ​​and / or in the two- or three-dimensional representations and / or in the first object recognition results.

[0035] The ultrasound measuring system (164) described in this section preferably also comprises a first training image memory (17) for the purpose of training the first neural network model (16), which stores training images for the first neural network model (16).

[0036] In the ultrasonic measuring system (164) described in this section, the display (19) is preferably further designed to display the two- or three-dimensional representations which are at least temporarily additionally provided with the identification determined by the first neural network model (16) and / or a part of the first object recognition result. B) Method for operating an ultrasonic measuring system with deep learning

[0037] In the context of the proposal, a method was developed for detecting objects and / or object components and / or arrangements of objects and / or object components in first ultrasonic measurement values ​​and / or in the two- or three-dimensional representations that depend on first ultrasonic measurement values, and / or for determining a first object detection result based on first ultrasonic measurement values. It comprises a) installing (21) a first neural network model (16) on a computer-based system. This may, for example, be a spatially separated computer system or a microcomputer system that forms a unit with parts of the ultrasonic sensor arrangement (11), and b) training (22) the first neural network model (16) to identify the objects and / or object components and / or arrangements of objects and / or object components in the first ultrasonic measurement values ​​and / or in the two- or three-dimensional representations that depend on the first ultrasonic measurement values, and to generate a first object recognition result. In this case, previously known ultrasonic measurement values ​​and / or in the two- or three-dimensional representations that depend on the first ultrasonic measurement values ​​are input to the first neural network model.presented as input signals of the first neural network model (16). The first output signals generated by the first neural network model (16) are evaluated, for example, by a first parameter modification device (140) for the first neural network model (16). The first parameter modification device (140) thus carries out an evaluation of the first output signals of the first network model (16). Depending on whether the evaluation of the first output signals of the first network model (16) by the first parameter modification device (140) is evaluated as positive or negative, the first parameter modification device (140) carries out a modification of the parameters of the first neural network model (16) such that the previous change in the parameters of the first neural network model (16) is carried out again, preferably with the same sign, if the current comparison with a target value,in particular a target value for the evaluation result of the evaluation by the first parameter modification device (140), yields a positive result, which corresponds to an improvement in the recognition result, and that the one previous change in the parameters of the first neural network model (16) is carried out again, preferably with the opposite sign, if the current comparison with a target value, in particular a target value for the evaluation result of the evaluation by the first parameter modification device (140), yields a negative result, which corresponds to a deterioration in the recognition result. The respective comparison is preferably also carried out by the parameter modification device (140). Such learning algorithms can also be modified in such a way that not only the immediately preceding parameter change is used to determine the next change in the training process,but also other, even earlier, preceding changes. The preceding changes themselves can also be used as further input parameters of the parameter modification device (140). Furthermore, the learning process can be carried out with varying degrees of intensity depending on the previous history and on the individual parameters of the first neural network model (16) as well as on the individual output signals of the first neural network model (16). A linear and / or non-linear link between the output signals of the first neural network model (16) and, if applicable, further input signals of the parameter modification device (140), on the one hand, including any preceding values ​​of these signals, which may be temporarily stored,and the evaluation result and the resulting change in the parameters of the first neural network model (16) is conceivable and useful depending on the respective application. The evaluation result can also be a vector signal of several individual evaluation results, which allows for an improvement of the learning process. For example, it can be ensured that safety-relevant deviations from the output signals of the first neural network model (16) result in a significantly stronger adjustment of the parameters of the first neural network model (16) than non-safety-relevant deviations from the output signals of the first neural network model (16). The first neural network model (16) is fully trained when all predetermined input training data sets receive a positive evaluation, in particular by the parameter modification device (140).This means that their answer will then be within a specified range;, c) using (25) the trained, first neural network model (16) to identify objects and / or object components and / or arrangements of objects and / or object components in the first live ultrasound measurement values ​​and / or in the two- or three-dimensional live representations that depend on the first live ultrasound measurement values, whereby a first object recognition result is generated. Here, the ultrasound measuring device, of which the first neural network model (116) is a part, is used as intended to identify objects and / or object components and / or arrangements of objects and / or object components in the surroundings of the exemplary vehicle. The ultrasound sensor arrangement (11) transmits ultrasound signals and receives the echoes of these ultrasound signals as ultrasound measurement signals. Since these are received during actual operation, they are referred to here as live ultrasound measurement signals.The associated measured values ​​are referred to here as live ultrasound measured values. Based on these live ultrasound measurement signals or signals and / or information derived therefrom, the first neural network model (16) then performs the aforementioned identification of objects and / or object components and / or arrangements of objects and / or object components in the vehicle's surroundings.

[0038] Expediently, the training (22) of the first neural network model (16) also includes training (22) with previously recorded live ultrasound training values ​​(data from live ultrasound measurement values) of objects and / or object components and / or of arrangements of objects and / or object components in the surroundings of the vehicle and / or in the surroundings of a prototype vehicle and / or from a laboratory scene of such objects and / or object components and / or such arrangements of objects and / or object components. Of course, the calculation of such recorded live ultrasound training values ​​is also conceivable, for example, in order to achieve a good starting point for training the first neural network model (16).

[0039] The training (22) of the first neural network model (16) can also include training (22) with two- or three-dimensional training representations. In this case, the ultrasonic measurement signals of the ultrasonic transducer arrangement (11) are not evaluated directly, but rather the two- or three-dimensional representations typically derived therefrom, preferably pictorial ones.

[0040] The training (22) of the first neural network model (16) may further comprise training (22) with training images of a common arrangement of a plurality of subjects.

[0041] These three exemplary evaluation methods (training (22) with ultrasonic measurement signals and possibly other measurement signals, training (22) with two- or three-dimensional training representations, training (22) with training images) can, as already described above, be mixed with each other and with other training data (e.g. simulated user inputs) as training data.

[0042] The training (22) preferably comprises the identification of objects and / or object components and / or arrangements of objects and / or object components in the two- or three-dimensional training representations. This then generally enables the identification of objects and / or object components and / or arrangements of objects and / or object components in the vehicle's surroundings during subsequent operation.

[0043] The same applies to alternative or supplementary training (22), which comprises identifying objects and / or object components and / or arrangements of objects and / or object components based on the first ultrasonic measurement values ​​of the ultrasonic transducer arrangement (11). This also generally enables identification of objects and / or object components and / or arrangements of objects and / or object components in the surroundings of the vehicle during subsequent operation. The training (22) thus comprises determining a first object recognition result by the first neural network model (16). This can be done on the basis of the two- or three-dimensional training representations and / or on the basis of the first ultrasonic measurement values ​​of the ultrasonic transducer arrangement (11) and, if applicable, on the basis of additional, further measurement values.This can instead, or in parallel, also be done on the basis of derived measured values ​​and / or signals and / or information derived from the two- or three-dimensional training representations and / or from the first ultrasonic measured values ​​of the ultrasonic transducer arrangement (11) and, if applicable, from the additional, further measured values. The two- or three-dimensional training representations within the meaning of this disclosure can be displayed in real form on a display (19) or only as a virtual data set, for example in pixel form and / or vector form and / or in a compressed, for example object-oriented form. An object-oriented form typically comprises one data set per object, comprising, for example, a piece of data that specifies an object class, and, for example, a set of coordinates, for example for position and, if applicable, orientation in space, and, if applicable, speed and, if applicable, speed direction, etc.Furthermore, such a data set can, for example, specify the size of the object and / or deformations of the object compared to a reference object model.

[0044] In this sense, the first object recognition result is therefore a very general result of the first neural network model (16), which is in particular typically the essential input signal of the preferably present first parameter modification device (140), which preferably adapts the parameters of the first neural network model (16) depending on the result of the evaluation, especially during training (22), and thus carries out training (22). For this purpose, reference values ​​matching the input data presented to the first neural network model are transmitted to the first parameter modification device (140). The parameter modification device (140) compares the respective reference value for a set of input data for the first neural network model (16) with the first object recognition result of the first neural network model (16) and, based on the comparison result, generates a modification of the parameters of the first neural network model (16).The training (22) can be performed in the target system. In practice, however, it is also expedient to perform the training (22) using another device that is identical in construction or sufficiently similar in essential parts. Such another device can, for example, be a non-series laboratory setup of the ultrasound measuring device or parts thereof that fulfills and / or is intended for the intended purpose of performing the training (22) and determining the parameter data for the first neural network model (16).The parameters of the first neural network model (16) determined by means of training (22) are then stored in a memory or on another storage medium after completion of training (22) and, at a given time, for example, shortly before the end of production of another target ultrasound measuring device of the same or similar design, are then loaded into the first neural network model (16) of this target ultrasound measuring device. It is also conceivable to consider the determined parameters of the first neural network model (16) when designing such a target ultrasound measuring device and to configure the construction of the target ultrasound measuring device such that the determined parameters are fixedly specified, for example, by hardwiring circuit components of an exemplary target ultrasound measuring device. Not all parameters need to be either programmable or adjustable, or fixedly specified, on the one hand.Rather, for example, some parameters can be hard-wired and others can be designed to be programmable. This has the advantage that more complex circuitry can be limited to the absolutely necessary components.

[0045] In the proposal presented here, said training (22) can, in the case of programmable or adjustable parameters, possibly also be continued during operation (use (25)) under certain conditions, which will be discussed later, which is why the device described here is then capable of self-learning. In this respect, the training (22) in self-learning mode can also alternate with the use (25) already described above. Preferably, but not necessarily, a method according to the invention thus comprises, for example, phases of training (22) and phases of use (25), which can, for example, also alternate if necessary.However, prior to the first use (25), training (22) is typically carried out, or preferably at least programming with or adjustment according to previously determined parameter sets of the first neural network model (16), which were determined using an identical or sufficiently similar ultrasonic measuring device or a sufficiently matching simulation device. Particularly preferred is the recording of such ultrasonic measurement values ​​and the necessary training data using test vehicles in real road traffic in the target markets. From the recorded data, the necessary parameter sets of the first neural network model (16) are then calculated or otherwise determined in the laboratory. These parameter sets are used for the initial programming or initial adjustment of the target ultrasound systems, or from which the design data for the hard-wired parts of the first neural network (16) are determined.

[0046] Preferably, the identification of the objects and / or object components and / or arrangements of objects and / or object components and / or the first object recognition results of the first neural network model (16) thus performed by the first neural network model (16) is used to annotate the two- or three-dimensional representations, which depend on ultrasonic measurement values ​​of the ultrasonic sensor arrangement (11) and possibly further measurement values ​​of other sensors or signals or information derived therefrom, when displayed on a screen, for example in a displayed map of the surroundings of a vehicle for use by the driver. In particular, it is desirable to highlight particularly dangerous and / or otherwise relevant objects when displayed on a display (19) for the driver.

[0047] Thus, in one variant, the proposed method also comprises using the identification of the objects and / or object components and / or arrangements of objects and / or object components and / or the first object recognition results in such an environment map of the surroundings of the vehicle and / or to inform the driver by means of a two- or three-dimensional representation and / or to inform the driver by means of a control system and / or control unit for controlling and / or regulating autonomous driving and / or autonomous parking. C) Second measurement system with deep learning

[0048] The ultrasonic measuring system (164) previously described in section A) can be modified into a second measuring system (161) for use in motor vehicles for detecting obstacles and / or objects and / or object components and / or for detecting arrangements of obstacles and / or objects and / or object components in the environment of a vehicle using deep learning methods. The characteristic structure of the second measuring system (161) described below corresponds to that of the ultrasonic measuring system (164) described in section A). Instead of an ultrasonic sensor arrangement (11), for example of transducers, a sensor arrangement of further sensors (131) is now used. This can be a single sensor or a plurality of sensors. For example, it can be a LIDAR system or a RADAR.Instead of the first neural network model (16), the second measuring system (161) comprises a second neural network model (136). The structure of the second measuring system (161), however, is analogous to the ultrasonic measuring system (164) described in Section A). For the sake of completeness, the structure of this second measuring system (161) is described again below. Provided that the difference between the "ultrasound" measuring principle of the ultrasonic measuring system (164) described in Section A) and the measuring principle of the second measuring system (161) described below does not necessarily require a deviation in the respective structural component described, what is written in Section A) about the ultrasonic measuring system (164) also applies, at least in a figurative sense, to this second measuring system (161).

[0049] The proposed second measuring system (161) modified in this way then comprises said second sensor arrangement (131), which is configured to acquire second sensor measurement values, and an image processor (18) coupled to the second sensor arrangement (131) and capable of generating two- or three-dimensional representations as a function of the second sensor measurement values ​​of the second sensor arrangement (131), a second memory (136) with a second neural network model (136) stored in the second memory (136) and adapted to receive the second sensor measurement values ​​and to identify the objects and their geometric arrangement in the environment of the vehicle in the second sensor measurement values ​​using a deep learning technique by assigning a second identification and thus generating a second object recognition result, and a two- or three-dimensional display (19),which is designed, suitable, or intended to display the two- or three-dimensional representations generated by the image processor (18). The two- or three-dimensional display (19) is preferably suitable and / or designed and / or intended to display, at least temporarily, one of the second identifications and / or second object recognition results and / or information and / or signals derived therefrom.

[0050] The second neural network model (136) is further preferably configured to identify the view of the arrangement of objects and / or object components in the second sensor measurements of the second sensor arrangement (131) and / or in the two- or three-dimensional representations. Since in many cases the nature of the objects is not a problem, but rather their arrangement relative to one another or the change in this arrangement over time, this is advantageous. As a rule, this should therefore also be captured by the methods presented here.

[0051] The proposed second measuring system (161) preferably further comprises a plurality of second user controls (130), wherein the adjustment of one or more of the second user controls (130) occurs in response to the identification of an arrangement of objects and / or object components in the second sensor measured values ​​and / or in the two- or three-dimensional representations and / or depending on the second object recognition results. For example, the vehicle driver can mark objects they consider particularly dangerous (e.g., playing children or easily overlooked objects such as posts, etc.) in their representation. The measuring system (or the ultrasonic measuring system (164) in the case of device A) can then, firstly, track the movement of these objects identified as critical relative to the vehicle, for example, during a parking maneuver.Secondly, this user input can be used as a training signal for the first neural network model (16) in the case of an ultrasonic measuring system (164) or for the second neural network model (136) in the case of a second measuring system (161).

[0052] In the proposed second measuring system (161), the setting or adaptation of control parameters of a control of the controls or of several controls of the controls of the vehicle or of vehicle components, for example a lighting and / or a warning signal generator, preferably takes place in response to the identification of the arrangement of objects and / or object components in the second sensor measurement values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or depending on the second object recognition results of the second neural network model (136).

[0053] In the proposed second measuring system (161), the setting of one or more of the controls of the second measuring system (161) or of the vehicle or of vehicle components is preferably carried out in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the second sensor measurement values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or as a function of the second object recognition results of the second neural network model (136).

[0054] The proposed second measuring system (161) preferably further comprises a user control display (9) that displays settings made in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the second sensor measurement values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or depending on the second object recognition result of the second neural network model (136).

[0055] The user control display (9) further activates control options of the measuring device and / or the ultrasonic measuring device and / or the vehicle and / or vehicle components and parts in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the second sensor measurement values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or depending on the first object recognition results of the second neural network model (136). The user control display (9) preferably also or alternatively displays these control options.

[0056] The proposed second measurement system (161) preferably further comprises a second training image memory (137) that stores training images for the second neural network model (136). These training images can be present, for example, as previously described, in the form of pixel data, vector data, or compressed object-oriented data.

[0057] Preferably, in the case of the second measuring system (161), the display (19) is further designed to display the two- or three-dimensional representations which are at least temporarily additionally provided with the identification determined by the second neural (136) network model and / or a part of the second object recognition result of the second neural (136) network model. D) Second method for the second measuring system

[0058] The proposed method for detecting objects and / or object components and / or arrangements of objects and / or object components in second sensor measurement values ​​of a second sensor arrangement (131) and / or in the two- or three-dimensional representations that depend on second sensor measurement values ​​of the second sensor arrangement (131), and / or for determining a second object detection result based on second sensor measurement values ​​of the second sensor arrangement (131) comprises, among other things, the following steps: • The method comprises the step of installing (51) the second neural network model (136) on a computer-based system, for example a laboratory computer. • The method comprises the step of training (52) the previously installed second neural network model (136) with training data of second sensor measurements from the second sensor arrangement (131) that were synthetically determined by simulation and / or previously recorded, which have preferably already been supplemented with evaluation data manually or otherwise by evaluation. The aim of this is to identify the objects and / or object components and / or arrangements of objects and / or object components in the second sensor measurements from the second sensor arrangement (131) and / or in the two- or three-dimensional representations that depend on second sensor measurements from the second sensor arrangement (131), and thus to generate a second object recognition result from the second neural network model (136), which can be compared with the relevant evaluation data.Based on the respective comparison result, the parameters of the second neural network (136) can then be adjusted. • The method comprises the step of using (54) the trained second neural network model (136) to identify objects and / or object components and / or arrangements of objects and / or object components in the second sensor measured values ​​and / or in the two- or three-dimensional representations that depend on second sensor measured values, whereby a second object recognition result is generated. The parameters determined in the training (52) are loaded into and / or set in the second neural network model (136) of a target measuring system and / or these parameters are used in the construction of the target measuring system to permanently set at least parts of these parameters, for example by hard wiring within the device that implements the second neural network model.

[0059] The training (52) of the second neural network model (136) thus preferably comprises a training (52) with sensor live training values ​​of the second sensor arrangement (131) of objects and / or object components and / or of arrangements of objects and / or object components and / or with two- or three-dimensional training representations, with training images of a common arrangement of a plurality of subjects.

[0060] The training (52) preferably also comprises identifying objects and / or object components and / or arrangements of objects and / or object components in the two- or three-dimensional training representations, and preferably evaluating these identifications with regard to possibly previously manually or automatically determined evaluation data that are to be assigned to these two- or three-dimensional training representations. The training (52) can also comprise identifying objects and / or object components and / or arrangements of objects and / or object components based on the second sensor measurement values ​​of the second sensor arrangement (131), preferably with the aid of existing evaluation data after the evaluation of this identification has been completed, possibly with the aid of previously manually or automatically determined evaluation data that are to be assigned to these second sensor measurement values ​​of the second sensor arrangement (131).Furthermore, the training (52) can comprise determining a second object recognition result based on the two- or three-dimensional training representations and evaluating it, if necessary with the aid of previously manually or automatically determined evaluation data. The training (52) comprises determining a second object recognition result based on the second sensor measurement values ​​of the second sensor arrangement (131).

[0061] Preferably, the method comprises using the identification of the objects and / or object components and / or arrangements of objects and / or object components and / or the second object recognition results to annotate the two- or three-dimensional representations that depend on sensor measurements.

[0062] In a further refinement of this variant, the method comprises using the identification of the objects and / or object components and / or arrangements of objects and / or object components and / or the second object recognition results in an environment map of the surroundings of the vehicle and / or to inform the driver by means of a two- or three-dimensional representation and / or by means of a control system and / or control unit for controlling and / or regulating autonomous driving and / or parking. E) First combination method using the first ultrasonic measuring system and the second measuring system

[0063] The two methods B and D can be combined using a combined device comprising a first ultrasonic measuring system (164) according to section A) and a second measuring system (161) according to section C). The question here is how a common object recognition hypothesis is generated from the first object recognition result of the first ultrasonic measuring system (164) and the second object recognition result of the second measuring system (161). This generation of a common object recognition hypothesis should preferably be self-learning.The method described below thus comprises a method according to section B) and a method according to section D) for detecting objects and / or object components and / or arrangements of objects and / or object components in the second sensor measurement values ​​of the second sensor arrangement (131) and in the first ultrasound measurement values ​​of the ultrasound transducer arrangement (11) and / or based on the first object detection results of the first neural network model (16) and based on the second object detection results of the second neural network model (136). In order to generate the joint object detection hypothesis, the following additional steps are preferably performed:

[0064] A method according to section A) is carried out (90a) for the ultrasonic measuring system (164), with determination of the ultrasonic measured values ​​of the ultrasonic transducer arrangement (11) and the determination of a first object recognition result of the first neural network (16). In this document, the first object recognition result comprises identification information and / or other recognition information of objects and / or object components and / or arrangements of objects and / or object components in the first ultrasonic measured values ​​of the ultrasonic transducer arrangement (11).

[0065] Furthermore, a method according to section D) is carried out (190b) for the second measuring system (161), with determination of the second sensor measured values ​​of the second sensor arrangement (131) and the determination of a second object recognition result. In this document, the second object recognition result comprises identification information and / or other recognition information of objects and / or object components and / or arrangements of objects and / or object components in the second sensor measured values ​​of the sensor arrangement (131).

[0066] Furthermore, the first combination method described in this section E) comprises providing (91) a third neural network model (138). This third neural network model (138) subsequently functions as a self-learning voter. The third neural network model (138) generates the third neural object recognition result from the first object recognition result of the first neural network model (16) and the second object recognition result of the second neural network model (136). In this document, the joint object recognition hypothesis comprises identification information and / or other recognition information of objects and / or object components and / or of arrangements of objects and / or object components in the second sensor measurement values ​​of the sensor arrangement (131) and in the first ultrasound measurement results of the first ultrasound transducer arrangement (11).Furthermore, the joint object recognition hypothesis can comprise first training signals for the first neural network (16) and second training signals for the second neural network (136). While these first and second training signals are preferably separate, they can also be common to the first and second neural network models (16, 136), which is typically not preferred but possible.

[0067] In the first combination method presented here in section E), training (92) of the third neural network model (138) is preferably provided as a further step, as before. This training is preferably carried out using known first object recognition results of the first neural network model (16) and known second object recognition results of the second neural network model (136), which are supplemented by associated evaluation data. The third neural network model is stimulated during training (92) in order to identify the objects and / or object components and / or arrangements of objects and / or object components in the second sensor measurement values ​​and in the ultrasonic measurement values ​​and / or in the two- or three-dimensional representations that depend on the sensor measurement values ​​and / or ultrasonic measurement values, and to generate a common object recognition hypothesis.This common object recognition hypothesis can then be evaluated using the associated evaluation data. The parameters of the third neural network model (138) are then modified, for example, by a first safety device (142) depending on the result of the comparison of the common object recognition hypothesis with the associated evaluation data during training (92). The first safety device (142) can receive high-priority signals (H11, H15, H131, H135) from components of the first ultrasonic measuring system (164) and from components of the second measuring system (161).If the ultrasonic transducer arrangement (11) and / or the sensor arrangement (131) detect, by whatever means, a potentially safety-threatening situation or an increased probability of such a situation occurring in the future, it is possible to overwrite the common object detection hypothesis of the third neural network model (138) if the latter does not indicate the detection of this situation, thus ensuring safe operation. Likewise, the user should preferably be able to overwrite the common object detection hypothesis in certain cases, if necessary, by means of a user input (H142). Furthermore, a higher-priority system should preferably be able to overwrite the common object detection hypothesis in certain cases, if necessary, by means of a user input (H142).

[0068] In the event of such an overwrite, it is useful for the first safety device (142) to modify the parameters of the third neural network model (138) and thus perform a training (92) of the third neural network model (138). The special feature is that this can thus be done during operation.

[0069] The training (92) of the third neural network model (138) is preferably initially carried out, as before, using a suitable laboratory system that is sufficiently similar to the target system for correct training. The parameters thus determined for the third neural network model (138) can then later be transferred to the target system. This can again be done, as before, for example, through programming and / or configuration and / or already during the design phase of the target system.

[0070] As before, the combination method presented here in this section E) comprises using (95) the trained third neural network model (138), preferably within a target system comprising an ultrasonic measuring system (164) according to section A) and a measuring system according to section C), for identifying objects and / or object components and / or arrangements of objects and / or object components in the second sensor measurement values ​​of a sensor arrangement (131) and in the ultrasonic measurement values ​​of an ultrasonic transducer arrangement (11) and / or on the basis of the first object recognition results of the first neural network model (16) and on the basis of the second object recognition results of the second neural network model (136).

[0071] Here, too, the training (92) of the third neural network model (138) preferably also includes training with previously recorded or simulation-determined first live object recognition results of the first network model (16) of objects and / or object components and / or of arrangements of objects and / or object components, and / or with second live object recognition results of the second network model (136) of objects and / or object components and / or of arrangements of objects and / or object components, and / or with two- or three-dimensional training representations and / or with training images of a common arrangement of a plurality of subjects. This training data, typically previously determined by simulation or measurement / testing, is preferably added manually and / or automatically to evaluation data before the start of training (92).A common object recognition hypothesis of the third neural network model (138) determined on the basis of this training data can then be compared, for example, by the first security device (142) or another training device with the evaluation data associated with the respective training data. Depending on the comparison result, the first security device (142) or the other training device then modifies the parameters of the third neural network model (138) accordingly.

[0072] Preferably, the training (92) of the third neural network model (138) comprises identifying objects and / or object components and / or arrangements of objects and / or object components in the two- or three-dimensional training representations and / or identifying objects and / or object components and / or arrangements of objects and / or object components based on the first and second object recognition results and / or determining a common object recognition hypothesis based on the two- or three-dimensional training representations and / or determining a common object recognition hypothesis based on the first ultrasonic measurement values ​​of the first ultrasonic transducer arrangement (11) and the second sensor measurement values ​​of the second sensor arrangement (131). As already explained above, during the training process, evaluation data serving as reference data is assigned to this training data.

[0073] As previously described in the methods in sections B) and D), the first combination method described here in section E) also comprises using (92) the identification of the objects and / or object components and / or arrangements of objects and / or object components according to the common object recognition hypothesis to annotate the two- or three-dimensional representations, which depend on the second sensor measurement values ​​of the second sensor arrangement (131) and the ultrasound measurement values ​​of the first ultrasound transducer arrangement (11) and / or on the first object recognition result of the first neural network model (16) and on the second object recognition result of the second neural network model (136) and / or the common object recognition hypothesis.

[0074] Preferably, the first combination method described here in section E) further comprises using (95) the identification of the objects and / or object components and / or arrangements of objects and / or object components according to the common object recognition hypothesis and / or the common object recognition hypothesis in an environment map of the surroundings of the vehicle and / or to inform the driver by means of a two- or three-dimensional representation and / or to inform the driver by means of a control system and / or control device for controlling and / or regulating autonomous driving and / or parking.

[0075] To perform the training (22) of the first neural network model (16), this first combination method preferably involves comparing the common object recognition hypothesis of the third neural network model (138) with the first object recognition result of the first neural network model (16) and, if necessary, changing the parameterization of the first neural network model (16) depending on the result of this comparison. This comparison and this change are preferably carried out by a first parameter modification device (140) for the first neural network model (16).

[0076] To perform the training (52) of the second neural network model (136), this first combination method preferably involves comparing the common object recognition hypothesis of the third neural network model (138) with the second object recognition result of the second neural network model (136) and, if necessary, changing the parameterization of the second neural network model (136) depending on the result of this comparison. This comparison and this change are preferably carried out by a second parameter modification device (141) for the second neural network model (136).

[0077] As previously described, training (92) of the third neural network model (138) is preferably carried out by the first security device (142). The higher-priority signals (H131, H135, H11, H15, and H142) are expressly mentioned again in this context, since these essentially enable either self-induced training (92) using internal, higher-priority signals (H131, H135, H11, and H15) or user-induced training (92) using user-generated, higher-priority signals (H142). As previously mentioned, a higher-priority technical device can also initiate training (92) of the third neural network model (138) using higher-priority signals (H142) generated by this device.

[0078] Of course, the parameterization of the first neural network model (16) and / or the parameterization of the second neural network model (136) and / or the parameterization of the third neural network model (138) can also be changed depending on a user input (142) with the aid of a training step (22, 52, 92). The user input can be made by means of a suitable signaling (e.g., H142), e.g., to the first parameter modification device (140) for the first neural network model (16) and / or the second parameter modification device (141) for the second neural network model (136) and / or the first safety device (142) and / or the second safety device (143) to be described later and / or the fourth parameter modification device (142) to be described later for the fourth neural network model (156) to be described later.

[0079] Such a user input for generating such a higher-priority external signal (e.g. H142) can be made, for example, by means of a touch gesture on a touch pad or a display (9) designed as a touch screen, or by means of a three-dimensional gesture. Such a three-dimensional spatial gesture, preferably executed in the space in front of the display (9), or such a preferably two-dimensional touch gesture is preferably recognized by an appropriately designed gesture recognition system and converted into the said higher-priority signal (e.g. H142). In order to reduce the probability of false triggers, these gestures are preferably sequences of individual gestures that are combined to form the gestures in question and that must be performed in the correct order and at the correct speed.

[0080] In order to be able to carry out the previously described training (92) of the third neural network model (138), it is advantageous if the first combination method E) described here comprises a step of monitoring one or more of the high-priority signals (H131, H135, H11, H15, H142).As previously described, these high-priority signals (H131, H135, H11, H15, H142) can then bring about a change in the parameterization of the third neural network model (138) depending on the state of these high-priority signals (H131, H135, H11, H15, H142) via the first parameter modification device (140) for the first neural network model (16) and / or via the second parameter modification device (141) for the second neural network model (136) and / or via the first safety device (142) and / or via the second safety device (143), which will be described later, and / or the fourth parameter modification device for the fourth neural network model (156), which will be described later.Instead of or in addition to this change, a change in the representation on the display (19) can take place depending on the state of the high-priority signal (H131, H135, H11, H15, H142) and / or a change in the representation on the display (19) can take place depending on the common object recognition hypothesis.

[0081] In addition to these high-priority signals (H11, H15, H131, H135, H142), the first neural network model (16) and / or the second neural network model (136) and / or the third neural network model (138) can have further input signals that influence the first object recognition result of the first neural network model (16) and / or the second object recognition result of the second neural network model (136) and / or the common object recognition hypothesis of the third neural network model (138). Consequently, a potential method step of the first combination method described here results in a possible change to the common object recognition hypothesis and / or the first object recognition result and / or the second object recognition result depending on one or more further external parameters that are transmitted via such a further input signal.Such external parameters can be, for example, the vehicle speed, the temperature, the air pressure, the humidity, the on or off state of the driving lights and / or the parking lights and / or the indicators and / or the fog lights and / or the additional headlights, etc., icing of the road or the driveway detected by an ice sensor, rain detected by a rain sensor and / or the on or off state of the windshield wipers, the leveling of the vehicle, the rotation rates of the vehicle about its axes, the translational and / or rotational acceleration and speed values, the steering angle, the steering angular velocity, the tire pressure, the engine speed, the speed of the wheels, the braking state, etc.

[0082] The training (92) of the neural network models (16, 136, 138) preferably takes place at least always in the form of a change in the parameterization of the first neural network model (16) and / or the second neural network model (136) and / or the third neural network model (138) when the vehicle reaches a first position and when the respective neural network model has incorrectly detected or detects an object at this first position. If, for example, one of these neural network models detects an obstacle and the system then determines that the vehicle is driving through the previously detected obstacle, this obstacle was obviously not present. Thus, the previous detection was incorrect, which can be used to train (22, 52, 92) the respective neural network model.

[0083] In the same way, training (22, 52, 92) can be carried out by changing the parameterization of the first neural network model (16) and / or the second neural network model (136) and / or the third neural network model (138) when a higher priority signal (H11, H15, H131, H135) and / or a high priority signal (H142) or a user input signals a collision or collision warning or other danger.

[0084] Preferably, a signal is then sent to the user and / or an error database of the vehicle and / or another computer system in the event of a change in the parameterization of the first neural network model (16) and / or the second neural network model (136) and / or the third neural network model (138) that a critical learning step has been carried out and / or that an error has occurred. F) Second combination method using the second measuring system and a third measuring system

[0085] The combination method presented in section E) can be adapted for use with a combined device and a second measuring system (161) according to section C) and a third measuring system (162) according to section C). Here, the question again arises as to how a common object recognition hypothesis is generated from the second object recognition result of the second measuring system (161) and the third object recognition result of the third measuring system (162). This generation of a common object recognition hypothesis should again preferably be self-learning.The method described below thus comprises the parallel application of two methods according to section D) for detecting objects and / or object components and / or arrangements of objects and / or object components in the second sensor measured values ​​of a second sensor arrangement (131) of the second measuring system (161) and in third sensor measured values ​​of a third sensor arrangement (151) of the third measuring system (162) and / or on the basis of the second object detection results of a second neural network model (136) of the second measuring system (161) and / or on the basis of third object detection results of a fourth neural network model (156) of the third measuring system (162).The combination method presented in this section preferably comprises the additional step of carrying out (171) a method according to section C) for the second measuring system (161), wherein the second measuring system (161) comprises said second neural network model (136), and the step of determining the second sensor measured values ​​of the second sensor arrangement (131) and the step of determining a second object recognition result of the second neural network model (136).The combination method presented in this section F) preferably further comprises the step of carrying out (172) a method according to section C) for the third measuring system (162), wherein the third measuring system (162) comprises a fourth neural network model (156), and the step of determining the third sensor measured values ​​of a third sensor arrangement (151) and the step of determining a third object recognition result of the fourth neural network model (156).Furthermore, the second combination method presented here comprises providing (173) a fifth neural network model (158) and training (174) this fifth neural network model (158) in order to identify the objects and / or object components and / or arrangements of objects and / or object components in the second sensor measurement values ​​of the second sensor arrangement (131) and in the third sensor measurement values ​​of the third sensor arrangement (151) and / or in the two- or three-dimensional representations that depend on second and / or third sensor measurement values, and to generate a common object recognition hypothesis of the fifth neural network model (158).Analogous to the first combination method from section E), the second combination method preferably comprises using (175) the trained fifth neural network model (158) to identify objects and / or object components and / or arrangements of objects and / or object components in the second sensor measurement values ​​of the second sensor arrangement (131) and in the third sensor measurement values ​​of the third sensor arrangement (151) on the one hand and / or on the basis of the first object recognition results of the first neural network model (136) and on the basis of the second object recognition results of the second neural network model (156) on the other hand and / or to generate a common object recognition hypothesis of the fifth neural network model (158).

[0086] Preferably, the training (174) of the fifth neural network model (158) comprises training with second live object recognition results of the second network model (136) of objects and / or object components and / or of arrangements of objects and / or object components and / or training with third live object recognition results of the third network model (156) of objects and / or object components and / or of arrangements of objects and / or object components and / or training with two- or three-dimensional training representations and / or with training images of a common arrangement of a plurality of subjects. The training (174) thus proceeds in the same manner as the training (92) in the first combination method from section E).

[0087] Likewise analogous to the training (92) of the first combination method from section E), the training (174) of the fifth neural network model (158) preferably comprises identifying objects and / or object components and / or arrangements of objects and / or object components in the two- or three-dimensional training representations and / or identifying objects and / or object components and / or arrangements of objects and / or object components on the basis of the second and third object recognition results and / or determining a common object recognition hypothesis on the basis of the two- or three-dimensional training representations comprises determining a common object recognition hypothesis on the basis of the second sensor measurement values ​​of the second sensor arrangement (131) and the third sensor measurement values ​​of the third sensor arrangement (151).

[0088] The second combination method described here preferably comprises, in analogy to the first combination method of section E), using (175) the identification of the objects and / or object components and / or arrangements of objects and / or object components according to the common object recognition hypothesis to annotate the two- or three-dimensional representations, wherein these depend on the second sensor measured values ​​and the third sensor measured values ​​on the one hand and / or wherein these depend on the second object recognition result of the second neural network model (136) and on the third object recognition result of the third neural network model (156) and / or the common object recognition hypothesis.

[0089] The second combination method described here also preferably comprises, in analogy to the first combination method of section E), the use (175) of the identification of the objects and / or object components and / or arrangements of objects and / or object components according to the common object recognition hypothesis and / or the common object recognition hypothesis in an environment map of the surroundings of the vehicle and / or for informing the driver by means of a two- or three-dimensional representation and / or for informing the driver by means of a control system and / or control device for controlling and / or regulating autonomous driving and / or parking.

[0090] The second combination method described here, similarly to the first combination method of section E), preferably comprises comparing the common object recognition hypothesis with the second object recognition result of the second neural network model (136) and changing the parameterization of the second neural network model (136) depending on the result of this comparison. This comparison is preferably carried out by said second parameter modification device (141) for the second neural network model (136).

[0091] The second combination method described here, similarly to the first combination method of section E), preferably comprises comparing the common object recognition hypothesis with the third object recognition result of the fourth neural network model (156) and changing the parameterization of the fourth neural network model (156) depending on the result of this comparison. This comparison is preferably carried out by said fourth parameter modification device (144) for the fourth neural network model (156).

[0092] The second combination method described here also preferably comprises, analogously to the first combination method of section E), changing the parameterization of the second neural network model (136) and / or the parameterization of the fourth neural network model (156) and / or the parameterization of the fifth neural network model (158) as a function of a user input (H142) with the aid of a training step (52, 174). Again, the user input is made, for example, by means of a touch gesture on a touch pad or a display (9) designed as a touch screen, or by means of a three-dimensional gesture.

[0093] The second combination method described here, similar to the first combination method of Section E), also includes monitoring one or more high-priority signals (H131, H135, H151, H155) and changing the parameterization of the fifth neural network model (158) depending on the state of the high-priority signal (H131, H135, H151, H155). This again enables the independent training of the fifth neural network model (158), particularly during operation.

[0094] The second combination method described here also preferably comprises, in analogy to the first combination method of section E), monitoring a high-priority signal (H131, H135, H151, H155) and changing the representation on the display (19) depending on the state of the high-priority signal (H131, H135, H151, H155) and / or changing the representation on the display (19) depending on the common object recognition hypothesis of the fifth neural network model (158).

[0095] The second combination method described here, similarly to the first combination method of Section E), preferably comprises changing the common object detection hypothesis and / or the first object detection result and / or the second object detection result depending on one or more additional external parameters, which are again transmitted via another input signal. Such external parameters can, for example, again be the vehicle speed, temperature, air pressure, humidity, the on or off state of the driving lights and / or the parking lights and / or the turn signals and / or the fog lights and / or the auxiliary lights, etc., icing of the road or driveway detected by an ice sensor, rain detected by a rain sensor and / or the on or off state of the windscreen wipers, the leveling of the vehicle, the rotation rates of the vehicle about its axes, the translational and / or rotational acceleration and speed values, the steering angle, the steering angular speed, the tire pressure, the engine speed, the speed of the wheels, the braking state, etc.

[0096] The second combination method described here, similarly to the first combination method of section E), preferably comprises changing the parameterization of the second neural network model (136) and / or the fourth neural network model (156) and / or the fifth neural network model (158) when the vehicle reaches a first position and when the relevant neural network has or is incorrectly detecting an object at this first position. The provisions of section E) regarding the first combination method apply accordingly.

[0097] The second combination method described here, similarly to the first combination method of section E), preferably includes changing the parameterization of the second neural network (136) and / or the fourth neural network (156) and / or the fifth neural network (158) when a higher-priority signal (H151, H155, H131, H135) and / or another high-priority signal (H152) or a user input signals a collision or collision warning or other hazard. The provisions in section E) regarding the first combination method apply accordingly.

[0098] The second combination method described here also preferably comprises, analogously to the first combination method of section E), a signal to the user and / or an error database and / or a computer system in the event of a change in the parameterization of the second neural network (136) and / or the fourth neural network (156) and / or the fifth neural network (158) that a critical learning step has been carried out and / or that an error has occurred. G) Simplest self-learning system or measuring system

[0099] The combination of the measuring systems previously described in sections A) and C) together with the device structure assumed in the combination methods E) and F) can be characterized as a self-learning system and / or measuring system as follows: Such an overall measuring system represents a network of neural network models (16, 136, 138, 156, 158), hereinafter referred to as a neural network. The neural network models (16, 136, 138, 156, 158) represent the nodes of the neural network. The neural network proposed here comprises at least a first neural network model (16) with a first set of parameters and a second neural network model (136) with a second set of parameters and a third neural network model (138) with a third set of parameters.With reference to the previously presented second combination method from Section F), the neural network proposed here comprises at least a second neural network model (136) with a second parameter set, a fourth neural network model (156) with a fourth parameter set, and a fifth neural network model (158) with a fifth parameter set. Thus, there are again at least three neural network models distributed across at least two node layers of the neural network. Each neural network model of the neural network has at least a first input data stream and a first output data stream.Since the basic structure of the neural network for implementing the second combination method of Section F), as just discussed, is identical to the basic structure of the neural network for implementing the first combination method of Section E), we will now restrict ourselves to the neural network for implementing the first combination method of Section E). The results can, as will be obvious to a person skilled in the art, be transferred to the neural network for implementing the second combination method of Section F. This restriction therefore explicitly includes the application of what follows in this section to a neural network for implementing the second combination method of Section F).

[0100] The third neural network model (138) has a second input data stream. The output data stream of the first neural network model (16) is the first input data stream of the third neural network model (138). The output data stream of the second neural network model (136) is the second input data stream of the third neural network model (138). The output data stream of the third neural network (138) depends at least on its first and second input data streams. An output data stream of the third neural network (138) can then, as described in Section E, change the first parameter set of the first neural network (16) and / or the second parameter set of the second neural network (136).This is preferably done by means of said first parameter modification device (140) for the first neural network model (16) and / or by means of the second parameter modification device (141) for the second neural network model (136). H) More complex self-learning system or measuring system

[0101] Since the basic structure of the neural network for implementing the second combination method of Section F), as discussed above, is identical to the basic structure of the neural network for implementing the first combination method of Section E), we will now also limit ourselves in this section to the neural network for implementing the first combination method of Section E). The results can, as will be obvious to a person skilled in the art, be transferred to the neural network for implementing the second combination method of Section F). This limitation therefore explicitly includes the application of what follows in this section to a neural network for implementing the second combination method from Section F).

[0102] The proposed self-learning system and / or measurement system comprises a network of neural network models (16, 136, 138). This network of neural network models (16, 136, 138) is again referred to as a neural network below, as in Section G). Again, the neural network models (16, 136, 138) represent the nodes of the neural network. The neural network comprises at least a first neural network model (16) with a first parameter set, a second neural network model (136) with a second parameter set, and a third neural network model (138) with a third parameter set. Furthermore, the neural network comprises a first parameter modification device (140) for the first neural network model (16) and a second parameter modification device (141) for the second neural network model (136).Each neural network model (16, 136, 138) of the neural network has at least a first input data stream and a first output data stream. The third neural network model (138) has a second input data stream. The output data stream of the first neural network model (16) is the first input data stream of the third neural network model (138). The output data stream of the second neural network model (136) is the second input data stream of the third neural network model (138). The output data stream of the third neural network (138) depends at least on its first and second input data streams.In this case, an output data stream of the third neural network (138) can change the first parameter set of the first neural network (16) by means of the first parameter modification device (140) and / or the second parameter set of the second neural network (136) by means of the second parameter modification device (141) and / or the said output data stream of the third neural network (138) is provided to influence these changes.

[0103] Preferably, the change in the parameter sets is influenced by user inputs using input devices such as touch screens, switches, etc. Preferably, said input devices include gesture recognition devices.

[0104] Preferably, parts of the third output data stream of the third neural network model (138) correspond in a selectable manner either to parts of the first output data stream of the first network model (16) or to parts of the second output data stream of the second network model (136). This has the advantage that these parts can be exchanged for one another, for example, by the third network model (138). This selection is preferably performed by the third neural network model (138) depending on the first output data stream of the first network model (16) and the second output data stream of the second network model (136).

[0105] Preferably, the first parameters of the first network model (16) are modified by the third network model (138) if the selection was made such that parts of the third output data stream of the third neural network model (138) do not correspond to parts of the first output data stream of the first network model (16). In this case, it is assumed that the recognition by the first neural network model (16) was not optimal, and this effect is to be reduced.

[0106] Preferably, the first parameters of the first network model (16) are modified by the third network model (138) if the selection was made such that parts of the third output data stream of the third neural network model (138) correspond to parts of the first output data stream of the first network model (16). In this case, it is assumed that the recognition by the first neural network model (16) was optimal, and this effect is to be enhanced.

[0107] Preferably, second parameters of the second network model (136) are modified by the third network model (138) if the selection was made such that parts of the third output data stream of the third neural network model (138) do not correspond to parts of the second output data stream of the second network model (136). In this case, it is assumed that the recognition by the second neural network model (136) was not optimal, and this effect is to be reduced.

[0108] Preferably, second parameters of the second network model (136) are modified by the third network model (138) if the selection was made such that parts of the third output data stream of the third neural network model (138) correspond to parts of the second output data stream of the second network model (136). In this case, it is assumed that the recognition by the second neural network model (136) was optimal, and this effect is to be enhanced. F) General summary regarding an automotive ultrasonic measuring device with deep learning

[0109] This disclosure discloses an ultrasonic measuring system (164) for use in motor vehicles for detecting obstacles and / or objects in the surroundings of a vehicle using deep learning methods. The system is characterized in that it comprises a device, in particular a computer, in which a neural network model (16, 136, 138) is implemented, which is adapted by a deep learning technique to receive ultrasonic measurement values ​​and to identify the objects and their geometric arrangement in the surroundings of the vehicle in the ultrasonic measurement values ​​by assigning an identification, thus generating a first object detection result. The device has means (141, 140) for further training at least a portion (16, 136) of the neural network model (16, 136, 138) during ongoing vehicle operation.In order to exclude the training of safety-relevant incorrect behavior, the device preferably has further means (142) for preventing and / or suppressing erroneous and safety-relevant recognition results and / or potentially erroneous and / or potentially safety-relevant recognition results, for example by means of high-priority signals (H11, H15, H131, H135), user inputs (H142) and / or external signals and by checking the object recognition results and / or common object recognition hypotheses, and for identifying and retraining the relevant neural network models (16, 136, 138) during operation in the case of such undesirable recognition results. G) Automotive ultrasonic measuring system with beam forming

[0110] This disclosure discloses an ultrasonic measuring system for use in motor vehicles for detecting obstacles and / or objects in the environment of a vehicle using deep learning methods, characterized in that it comprises a microbeam former (1014) and a plurality of ultrasonic transducers within an ultrasonic transducer arrangement (1012), wherein the orientation and / or shape of the receive and / or transmit beams of the plurality of ultrasonic transducers within an ultrasonic transducer arrangement (1012) depends on signals from the microbeam former (1014). In transmit mode, the microbeam former (1014) generates its own, specifically modulated control signal for each of the plurality of ultrasonic transducers, such that the ultrasonic wave resulting in the free space in front of the ultrasonic transducers has the desired cross-sectional shape, spherical amplitude distribution, and main propagation direction.A simple example would be the generation of similar, but phase-shifted, control signals by the microbeam former (1014), which are then converted into sound signals by the ultrasonic transducers. In receive mode, the microbeam former (1014) provides a spatiotemporal inverse transformation of the vector received signal, which is composed of the received signals from the various ultrasonic transducers. In the simplest case, the received signals from the various ultrasonic transducers are specifically delayed in their respective phases by the microbeam former (1014), so that the superposition of the thus delayed received signals from the various ultrasonic transducers results in an aligned sensitivity lobe.

[0111] It is conceivable to provide a separate transmit microbeam former (1014) for the transmit channel and a separate receive microbeam former (1014) for the receive channel.

[0112] Fig. Figure 24 shows the alignment of transmitted and received signals using constructive interference of the ultrasonic waves. The more ultrasonic transducers an ultrasonic transducer array (1012) comprises, the better the daughter maxima can be suppressed and thus the signal focusing improved. It is therefore recommended to use more than 10 ultrasonic transducers, preferably more than 20, and even better more than 50 ultrasonic transducers. To achieve height resolution, it is recommended that the ultrasonic transducer array (1012) comprise ultrasonic transducers arranged in at least two rows essentially parallel to the roadway and at least five columns perpendicular to the roadway. This arrangement can, of course, only be achieved with ideally equal-sized wheels, exactly parallel bumper mounting, etc. The term "essentially" is therefore intended to include a deviation of + / - 10° from these arrangements. H) Automotive ultrasonic measuring system with Doppler processor

[0113] The above description also proposes an ultrasonic measuring system for use in motor vehicles for detecting obstacles and / or objects in the surroundings of a vehicle using deep learning methods, characterized in that it comprises a Doppler processor (1054) which stores ensembles of echo signals from discrete points and / or spatial directions, in particular in an image field, which are then used to estimate the Doppler shift, in particular at points in the image and / or in spatial directions, using a fast Fourier transform (FFT).

[0114] Preferably, a multispectral image resolved according to spatial directions is generated using a fast discrete Fourier transform. Since the transmission frequency is known, the following values ​​are then obtained for each spatial direction: amplitude of the reflection, time delay or distance of the reflection, frequency shift of the reflection, or positive or negative velocity in the sensor direction (toward the ultrasonic transducers). If past images are also evaluated, a motion vector can be assigned to prominent points in the respective image using temporal feature tracking. A particularly preferred situation is when the spacing of the ultrasonic transducers, e.g., on the bumper, allows the Doppler processor to perform the Doppler analysis specifically for the measurement signal of each ultrasonic transducer.Since the angle between the reflecting object and the respective ultrasonic transducer varies at sufficient proximity, the results of the Doppler analyses of the received signals from different ultrasonic transducers differ slightly. In this way, velocities perpendicular to the line of sight between the ultrasonic transducers and the respective object can be deduced directly from the ultrasonic transducer received signals. This can then be correlated with the velocities determined by image processing of the ultrasound image sequences. I) Automotive ultrasonic measuring system with multiplanar reformer

[0115] The above description also proposes an ultrasonic measuring system for use in motor vehicles for detecting obstacles and / or objects in the environment of a vehicle using deep learning methods, characterized in that it comprises a multiplanar reformatter (1044) which converts echoes received from points in a common plane in a volumetric region of the space in the environment of the vehicle into an ultrasonic image of that plane.

[0116] The function of the multiplanar reformator (1044) is typically closely interwoven with the function and interaction of the microbeam former (1014) and multiple ultrasonic transducers within an ultrasonic transducer array (1012). While the microbeam former (1014) controls and monitors the orientation and / or shape of the receive and / or transmit beams of the multiple ultrasonic transducers within an ultrasonic transducer array (1012) of signals from the microbeam former (1014), the multiplanar reformator (1044) selects from the incoming data stream of the receive signals from the ultrasonic transducers those signal components in a plane that is typically parallel to a vehicle plane and / or the roadway plane.To do this, it combines the received signals from at least two ultrasonic transducers mounted at different heights, for example, on the bumper, and evaluates the phase shift of the signals in such a way that only those signals containing reflections in a single plane are processed. A plane is defined as a limited solid angle range with a minimum sensitivity angle φ. min (see Fig. 25) between the vehicle and the perpendicular to the roadway and a maximum angle φ max (see Fig. 25) between the vehicle and the perpendicular to the roadway.

[0117] Here, too, it is recommended to use more than 10 ultrasonic transducers, preferably more than 20, and even better, more than 50. To achieve height resolution, it is recommended that the ultrasonic transducer array (1012) comprise ultrasonic transducers arranged in at least two rows, preferably more than five rows, essentially parallel to the roadway. This position can, of course, only be achieved with ideally equal-sized wheels, exactly parallel bumper mounting, etc. The term "essentially" should therefore also include a deviation of + / - 10° from these positions. J) Automotive ultrasonic measuring system with volume renderer

[0118] In the above description, an ultrasonic measuring system for use in motor vehicles for detecting obstacles and / or objects in the environment of a vehicle using deep learning methods was also proposed, which is characterized in that it comprises a volume renderer (1042) which converts the echo signals of a 3D data set into a projected 3D image as seen from a given reference point.

[0119] For this purpose, a computer system calculates a three-dimensional model of the vehicle's surroundings from the received data. This can then be presented to the driver, depending on the hazard situation, in such a way that they can accurately grasp the situation with maximum probability. For this purpose, it is useful for a neural network model, for example the third neural network model (138), to determine the optimal viewer position and the optimal viewing direction of the vehicle's surroundings model, for example, depending on the common object recognition hypothesis, and to initiate a corresponding display of the determined vehicle's surroundings model on the screen (19). What constitutes a good perspective can be very driver-specific and also culture-dependent.It is therefore advisable to determine a set of starting parameters for the relevant neural network model using so-called focus groups consisting of a representative cross-section of drivers in the target market and then to adapt the view to the specific driver using user inputs (H142). J) Automotive ultrasonic measuring system with safety device

[0120] The above description also proposes a measuring system for use in motor vehicles for detecting obstacles and / or objects in the surroundings of a vehicle using deep learning methods, which is characterized in that it comprises a first safety device (142) and one or more neural network models (16, 136, 138) for evaluating the measurement data of at least one sensor arrangement (11, 131) of the measuring system, and in that the first safety device (142) monitors at least the neural network model (138) which processes these measurement data or data derived therefrom.

[0121] The safety device can preferably be influenced by signals (H11, H131) from the at least one sensor arrangement (11, 131) of the measuring system. Furthermore, the safety device (142) can preferably influence the parameters of at least one neural network model (16, 136, 138). Furthermore, the safety device (142) should be capable of being influenced by a user signal (H142) or an external control signal (H142). In this way, in the event of safety-threatening incidents, the user can initiate a learning process by negatively signaling them, which prevents such occurrences from occurring in the future.

[0122] Conversely, the security device (142) preferably has the property of being able to reliably prevent signaling of the at least one neural network model (16, 136, 138) that endangers security and / or potentially endangers security. This can be achieved by plausibility checks of the detected signals as a function of high-priority additional signals (H11, H15, H131, H135, H155, H151) within the device and / or by external control signals (H142), wherein these signals can be one or more signals in time and / or space multiplex and / or information at predetermined memory locations of a computer network. Advantage

[0123] The proposed system enables the determination of the geometric arrangement and classification of objects in the vehicle's environment using an overall measurement system with self-learning measurement systems (161, 162) and a self-learning decision engine (138, 158). This learning process can also occur during normal operation, allowing the system to continuously improve. User inputs and safety signals (H11, H15, H131, H135, H142) secure the potential space of possible behavior in such a way that critical states are avoided. List of characters Fig. 1 In Fig. 1, an ultrasonic vehicle environment analysis system is shown schematically in simplified block diagram form, constructed in accordance with the principles of the present invention. Fig. Figure 2 shows a method for training and using a neural network model to identify anatomy in ultrasound images. Fig. 3 shows an alternative structure of the proposed ultrasonic measuring system (164) with a first neural network model (16) for the corresponding method according to section B). Fig. Figure 4 shows a model for training and using the first neural network model (16). Fig. Figure 5 shows a model for training and storing the parameters of the first neural network model (16). Fig. 6 shows a model for loading the stored parameters of the first neural network model (16) and then using the first neural network model (16). Fig. Figure 7 shows a model for training and using the second neural network model (136). Fig. Figure 8 shows a model for training and storing the parameters of the second neural network model (136). Fig. 9 shows a model for loading the stored parameters of the second neural network model (136) and then using the second neural network model (136). Fig. Figure 10 shows schematically the first combination method from section E). Fig. Figure 11 shows a model for training and using the third neural network model (138). Fig. Figure 12 shows a model for training and storing the parameters of the third neural network model (138). Fig. 13 shows a model for loading the stored parameters of the third neural network model (138) and then using the third neural network model (138). Fig. Figure 14 shows a model for training and using the fifth neural network model (158). Fig. Figure 15 shows a model for training and storing the parameters of the fifth neural network model (158). Fig. 16 shows a model for loading the stored parameters of the fifth neural network model (158) and then using the fifth neural network model (158). Fig. Figure 17 shows schematically the second combination method from section F). Fig. 18 shows the structure of the proposed combined overall system comprising an ultrasonic measuring system (164) and a second measuring system (161) for the corresponding first combination method according to Section E). Fig. 19 shows the structure of the proposed combined overall system comprising a second measuring system (161) and a third measuring system (162) for the corresponding first combination method according to Section F). Fig. 20 shows a model for training and using an overall system according to Fig. 19. (Can be applied to a complete system according to Fig. 18 can be easily transferred.) Fig. 21 shows a model for training and storing the parameters of the neural network models of an overall system according to Fig. 19. (Can be applied to a complete system according to Fig. 18 can be easily transferred.) Fig. 22 shows a model for loading the parameters of the neural network models of an overall system according to Fig. 19 and using the complete system prepared in this way. (Can be applied to a complete system according to Fig. 18 can be easily transferred.) Fig. Figure 23 shows the beam forming by constructive and destructive interference based on the phase information in the echo signal for ultrasonic transducers of the ultrasonic transducer array (1012) with different positions along the vehicle width. Fig. 24 Shows the limitation of the reception range to an angular range by a multiplanar reformator using triangulation based on the phase information in the echo signal for ultrasonic transducers of the ultrasonic transducer array (1012) with different height positions. Description of the figuresFigures 1 and 2

[0124] The Fig. 1 and Fig. 2 are described above in the text. Figure 3

[0125] Fig. Figure 3 shows the structure of the proposed ultrasonic measuring system (164) for use in a vehicle. The vehicle may be, for example, a motor vehicle, a motorcycle, an aircraft, a ship, or the like.

[0126] A proposed ultrasonic transducer arrangement (11) detects ultrasonic measured values ​​by emitting ultrasonic waves and receiving the echo signals. The ultrasonic transducer arrangement (11) can be a one- or two-dimensional arrangement of one or more ultrasonic transducers in order to detect measured values ​​in one, two, or three dimensions. An ultrasonic transducer of the ultrasonic transducer arrangement can, within the meaning of this proposal, consist of an ultrasonic transmitter and a separate ultrasonic receiver in spatial division multiplex or an ultrasonic transducer that transmits and receives in time division multiplex. The ultrasonic transducer arrangement (11) is connected to control electronics (12), which can optionally be capable of beam shaping, for example by phase shifting the emitted ultrasonic signals of the individual ultrasonic transducers of the ultrasonic transducer arrangement (11).The control electronics (12) controls the corresponding ultrasonic transducers of the ultrasonic transducer arrangement (11) for the preferably directed transmission or reception of the ultrasonic signals. This orientation is desirable, but not mandatory. The transmission of ultrasonic signals by the ultrasonic transducer arrangement (11) is controlled by a transmitter controller (13). This controller is connected to the control electronics (12) and sets the transmission parameters of the ultrasonic transducers of the ultrasonic transducer arrangement (11) depending on settings that are specified via the optional user settings (14) stored in an optional memory or by a higher-level control unit. Transmission parameters that can be controlled by the transmitter controller (13) include, for example, burst length, frequency, frequency response, coding, polarity, phase, or amplitude.

[0127] The echo signals received by the ultrasonic transducers of the ultrasonic transducer arrangement (11) are processed by the signal processor (15), including filtering by means of digital filters and noise suppression. The signal processor (15) can also be wholly or partially a digital signal processing device, for example a signal processor in the sense of processor technology or a device with the same function, and / or an analog circuit and / or a mixture of these two circuit variants. The received signals processed by the signal processor (15) are the input signals of the first neural network model (16), which is typically stored in a digital memory of a computer. The first neural network model (16) can also be implemented, for example, as a special microelectronic circuit or other functionally equivalent device.The neural network model (16) is the implementation of a deep learning method, as described above. The first neural network model (16) is trained using training data from various objects in various positions. This training data can be determined by measurement and / or simulation. Before training, evaluation data is assigned to the training data sets. This evaluation represents the expected evaluation by the first neural network model (16) and is used by the training device to decide whether and how the parameters of the first neural network model (16) need to be changed. This training data is stored in a digital memory (17). For training (22), methods of supervised or unsupervised learning can be applied.After completion of the training (22), the trained data for parameterizing the connections of the first neural network model (16) can be extracted (= saved) (35) into a database of neural network data. These can then be loaded into an identical ultrasonic measuring system (164) or a substantially identical ultrasonic measuring system without separate training in the production of further identical or substantially identical ultrasonic measuring systems (164), without the need for training (22) of the further identical ultrasonic measuring systems (164). An ultrasonic measuring system (164) is considered to be identical within the meaning of this disclosure if the ultrasonic measuring system (164) fulfills its intended purpose when using the network data determined in this way. The network data essentially consists of the parameters of the neural network model.It is sensible to determine this network data using a laboratory setup and then feed the network data into a series-production-ready product. This has the advantage of eliminating the time-consuming initial training phase in production. The additional, identical ultrasonic measuring systems (164) then further optimize themselves during operation from this predetermined starting point. For the purposes of this disclosure, an identical ultrasonic measuring system (164) is therefore considered to be an ultrasonic measuring system (164) that is suitable or intended to use the determined training data in the database of neural network data of the ultrasonic measuring system (164).The ultrasonic measuring system (164) used to determine the training data in the database of neural network data may, for example, be a laboratory system with significant mechanical quality defects, while the other identical ultrasonic measuring systems (164) may be production-ready, mechanically stable ultrasonic measuring systems (164) which may differ significantly in appearance and shape from the laboratory system.

[0128] The output values ​​of the first neural network model (16) are preferably the positions of the detected objects and preferably their classification, as well as, for example, a corresponding confidence value that can be assigned to the detected detection result as a whole, and / or confidence values ​​that are assigned to the individual objects. Further data can be part of the output values. Alternative hypothesis lists are also possible, wherein the hypothesis list comprises alternative detection results with different confidence values ​​for these object detection results. Each object detection result comprises one or more objects. The preferably used image processor (18) converts this information into a suitable two- or three-dimensional representation, which is output to the vehicle driver by the display (19). The information about object positions and classifications is preferably also transferred to the user control (110).The selection of a stored user control or the adjustment of control parameters of a control in response to the classification is implemented by the user control (110) and forwarded to the settings memory (14) or displayed by the user control display (111). The settings memory (14) also transmits parameters dependent on the classification to the image processor (18), so that the representation of the objects and their identification output by the display (19) can be automatically adapted accordingly. Figure 4

[0129] Fig. Figure 4 shows a model for training and using the described neural network model for determining the position and classifying objects in the vehicle's environment. In the installation step (21), a first neural network model (16) is installed on a computer system as the basis of the deep learning method. In the training step (22), the corresponding training receive signals from measured or simulated objects in as many different arrangements and positions as possible are passed to the first neural network model (16) as input values. In an identification step (23), the objects are identified for the neural network model. The actual learning takes place during the training step (22) and the identification step (23). The responses of the first neural network model (16) are compared with evaluation data, and the parameters of the first neural network model (16) are modified as appropriate.For this purpose, the largest possible amount of training data should be used.

[0130] The trained first neural network model (16) is located in the vehicle preferably on a digital memory, see. Fig. 3. Here, in the identification step (24), the live ultrasound reception signals from the operation of the ultrasound transducer array (11) are passed to the trained first neural network model in order to determine the position and classify the objects. The output values ​​of the first neural network model (16), which may include the position and classification, are then further used by the described system in the subsequent use step (25). Figure 5

[0131] shows a model for training and storing the parameters of the first neural network model (16).

[0132] In the installation step (21), a first neural network model (16) is installed on a computer system as the basis of the deep learning process. In the training step (22), the corresponding training receive signals from measured or simulated objects in as many different arrangements and positions as possible are passed to the first neural network model (16) as input values. In an identification step (23), the objects are identified for the neural network model. The actual learning takes place during the training step (22) and the identification step (23). The responses of the first neural network model (16) are compared with evaluation data, and the parameters of the first neural network model (16) are appropriately modified. For this purpose, the largest possible amount of training data should be used.

[0133] The network data (parameter data) of the trained first neural network model (16) are, in contrast to the Fig. 4 is now saved in a backup step (35). Figure 6

[0134] shows a model for loading the parameters of the first neural network model (16) stored in the saving step (35) and for subsequently using the first neural network model (16).

[0135] In a loading step (41), the parameters (network data) of the first neural network model (16) previously stored in the storage step on a first ultrasonic measuring system (164) are reloaded into the memory of the first neural network model (16) of a second ultrasonic measuring system (164) that is substantially identical to the first ultrasonic measuring system (164). This loading step (41) can also be a design step in which the structure and design of the second ultrasonic measuring system (164) are defined such that they are functionally equivalent to an ultrasonic measuring system (164) with such a memory.

[0136] The first neural network model (16) of the second ultrasonic measuring system (164) trained by means of this loading step (41) is then finally located at a typically later point in time in the vehicle, preferably on the said digital memory, see. Fig. 3, of the second ultrasonic measuring system (164). Here, the live ultrasonic reception signals from the operation of the ultrasonic transducer arrangement (11) during vehicle operation are transferred to the trained first neural network model (16) in the identification step (24), for example, to determine the position and classify the objects. The output values ​​of the first neural network model (16), which may include the position and the classification, are then preferably further used by the described system in the subsequent use step (25), as described above. Figure 7

[0137] Fig. Figure 7 shows a model for training and using the described second neural network model (136) for determining the position and classifying objects in the vehicle's surroundings using a second measuring system (161). In the installation step (51), a second neural network model (136) is installed on a computer system as the basis of the deep learning method. In a training step (52), the corresponding training received signals from measured or simulated objects in as many different arrangements and positions as possible are passed to the second neural network model (136) as input values. In an identification step (53), the objects are identified for the second neural network model (136). The actual learning takes place during the training step (52) and the identification step (53).The responses of the second neural network model (136) are compared with evaluation data, and the parameters of the second neural network model (136) are modified accordingly. For this purpose, the largest possible amount of training data should also be used.

[0138] The trained second neural network model (136) is preferably located in the vehicle on a digital memory. Here, the live measurement signals from the operation of the sensor arrangement (131) are transferred to the trained second neural network model (136) in the identification step (54), for example, to perform an alternative determination of the position and an alternative classification of the objects. The output values ​​of the second neural network model (136), which may include, for example, the alternative position and the alternative classification, are then further used by the described system in the subsequent use step (55). Figure 8

[0139] shows a model for training and storing the parameters of the second neural network model (136).

[0140] In the installation step (51), a second neural network model (136) is installed on a computer system as the basis of the deep learning process. In the training step (52), the corresponding training receive signals from measured or simulated objects in as many different arrangements and positions as possible are passed to the second neural network model (136) as input values. In an identification step (53), the objects are identified for the second neural network model (136). The actual learning takes place during the training step (52) and the identification step (53). The responses of the second neural network model (136) are compared with evaluation data, and the parameters of the second neural network model (136) are appropriately modified. For this purpose, the largest possible amount of training data should again be used.

[0141] The network data (parameter data) of the trained second neural network model (136) are, in contrast to the Fig. 7 is now saved in a backup step (65). Figure 9

[0142] shows a model for loading the parameters of the second neural network model (136) stored in the saving step (65) and for subsequently using the second neural network model (136).

[0143] In a loading step (71), the parameters (network data) of the second neural network model (136) previously stored on a first overall measuring system in the saving step (65) are reloaded into the memory of the second neural network model (136) of a second overall measuring system, which is essentially identical to the first overall measuring system. This loading step (71) can also be a design step in which the structure and design of the second overall measuring system are defined in such a way that it is functionally equivalent to an overall measuring system with such a memory.

[0144] The second neural network model (136) of the second overall measuring system, trained by means of this loading step (71), is then ultimately located, typically at a later point in time in the vehicle, preferably on the said digital memory of the second overall measuring system. Here, in the identification step (54), the live measurement signals from the operation of the sensor arrangement (131) during vehicle operation are transferred to the trained second neural network model (136) in order, for example, to perform an alternative determination of the alternative position and an alternative classification of the objects. The output values ​​of the second neural network model (136), which may include the alternative position and the alternative classification, are then preferably further used by the described system in the subsequent use step (55), as previously described. Figure 10

[0145] shows a schematic of the first combination process from Section E. Two processes are preferably carried out in parallel.

[0146] Firstly, the second neural network model (136) for the second measuring system (161) is installed (51) on a second computer system. Secondly, the first neural network model (16) for the ultrasonic measuring system (164) is installed (21) on a first computer system in an analogous manner. The first computer system and the second computer system can have the same and / or identical hardware. One or both computer systems can also be an analog circuit that executes the computing operations of the neural network models and does not necessarily have to be programmable. For the purposes of this disclosure, such an installation can also always be carried out in such a way that the structure and essential parameters of the respective neural network model are predefined, for example in the form of a suitable electronic circuit, during the design phase.For example, it could also be an analog or hybrid computer that partially predetermines the structure and / or parameterization of the respective neural network model in the form of an electrical circuit. However, a joint installation of the neural network models (16, 136, 138) in pure software form on a fully programmable computer system in Harvard and / or Von Neumann architecture is preferable and preferred.

[0147] Typically, this is followed by loading (71) previously saved parameters of the second neural network model (136) for the second measuring system (161) from the second training data memory (137) as current parameters of the second neural network model (136) for the second measuring system (161), and loading (41) preferably also previously saved parameters of the first neural network model (16) for the ultrasonic measuring system (164) from the first training data memory (17) as current parameters of the first neural network model (16). In this way, the overall measuring system is now prepared for signal processing.To process the signals, live sensor signals are now transferred (54) to the trained second neural network model (136) for the second measuring system (161) and an object identification is carried out by the trained second neural network model (136) to generate a second object recognition result and live ultrasound signals are transferred (24) to the trained first neural network model for the ultrasound measuring system (164) in the vehicle and an object identification is carried out by the trained first neural network model to generate a first object recognition result.

[0148] The live ultrasound signals differ from the ultrasound signals with which the parameters of the first neural network model (16) were determined in that they are now determined in the actual application by the ultrasound transducer arrangement (11) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0149] The live sensor signals differ from the sensor signals with which the parameters of the second neural network model (136) were determined in that they are now determined in the actual application by the sensor arrangement (131) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0150] Finally, a comparison (84) of the first object recognition results of the first object identification performed by the first neural network model (16) for the ultrasonic measuring system (164) is carried out with the alternative second object recognition results of the alternative second object identification performed by the second neural network model (136) for the second measuring system (161), and a common object recognition hypothesis is determined (84), which represents the recognition result of the entire measuring system. The common object recognition hypothesis can be a list of objects with recognition probabilities and of arrangements of such objects with recognition probabilities. It can also contain only one object and / or only one arrangement. A probability specification does not necessarily have to be included in the common object recognition hypothesis.

[0151] The learning process is now carried out as follows:A first parameter adjustment (86) of the first neural network model (16) for the ultrasonic measuring system (164) is carried out depending on the result of the comparison (84) of the first object recognition results of the first object identification carried out by the first neural network model (16) for the ultrasonic measuring system (164) with the alternative second object recognition results of the alternative second object identification carried out by the second neural network model (136) for the second measuring system (161).In this case, the parameter adaptation is generally carried out in such a way that non-consideration of parts of the first object recognition results of the first neural network model (16) in the joint object recognition hypothesis by the third neural network model (138), which carries out the comparison (84), leads to an attenuation of the corresponding parameters of the first neural network model (16) and that consideration of parts of the first object recognition results of the first neural network model (16) in the joint object recognition hypothesis by the third neural network model (138), which carries out the comparison (84), leads to an amplification of the corresponding parameters of the first neural network model (16).

[0152] A second parameter adjustment (87) of the second neural network model (136) for the second measuring system (161) is carried out depending on the result of the comparison (84) of the first object recognition results of the first object identification carried out by the first neural network model (16) for the ultrasonic measuring system (164) with the alternative second object recognition results of the alternative second object identification carried out by the second neural network model (136) for the second measuring system (161).In this case, the parameter adaptation is generally carried out in such a way that non-consideration of parts of the second object recognition results of the second neural network model (136) in the joint object recognition hypothesis by the third neural network model (138), which carries out the comparison (84), leads to an attenuation of the corresponding parameters of the second neural network model (136) and that consideration of parts of the second object recognition results of the second neural network model (136) in the joint object recognition hypothesis by the third neural network model (138), which carries out the comparison (84), leads to an amplification of the corresponding parameters of the second neural network model (136).

[0153] The third neural network model (138), which performs the comparison (84), thus makes a decision.

[0154] The final step is to use (85) the determined joint object recognition hypothesis. Figure 11

[0155] shows a model for training and using the third neural network model (138).

[0156] The method begins with the installation (91) of the third neural network model (138) for comparing the first object recognition results of the first object identification performed by the first neural network model (16) for the ultrasonic measuring system (164) with the alternative second object recognition results of the alternative second object identification performed by the second neural network model (136) for the second measuring system (161) and for determining a common object recognition hypothesis on a third computer system. The third computer system can be connected to the first and second computer systems of the Fig. 10 be identical.

[0157] As already mentioned above, the computer systems can also be one or more analog circuits that carry out the computing operations of the neural network models and do not necessarily have to be programmable. For the purposes of this disclosure, such an installation (91) can rather always be carried out in such a way that the structure and essential parameters of the respective neural network model are predetermined, for example in the form of a suitable electronic circuit during the design phase. It can therefore also be an analog or hybrid computer, for example, which partially predetermines the structure and / or parameterization of the respective neural network model in the form of an electrical circuit. However, an installation (91) of the neural network models (16, 136, 138) in the form of pure software on a fully programmable computer system in Harvard and / or Von Neumann architecture is preferable and preferred.

[0158] This is followed by training (92) of the third neural network model (138) by transferring correct and incorrect first object recognition results and correct and incorrect alternative second object recognition results to the third neural network model (138) together with data on their correctness and / or incorrectness, and determining the parameters of the third neural network model (138) depending on these data and / or changing the parameters of the third neural network model (138) depending on these data. This can be done, for example, by first training the first neural network model (16) entirely with laboratory data and then training the second neural network model (136) entirely with laboratory data. Corresponding laboratory data sets are then presented as training data sets to the first neural network model (16) and the second neural network model (136).These generate a first object recognition result and a second object recognition result for each of these training data set pairs. A target object recognition hypothesis is then calculated or otherwise determined for each of these pairs, either manually or automatically. This results in a training data set for the third neural network model (138), consisting of the respective first object recognition result and the respective second object recognition result and the respective target object recognition hypothesis, which represents the corresponding evaluation data.

[0159] With this data, the third neural network model (138) can then be trained (92).

[0160] Once training (92) has been successfully completed, the system (e.g., a laboratory device) or a sufficiently functionally similar device (e.g., a series product) can be deployed, e.g., in a vehicle. There, the first live object recognition results and alternative second live object recognition results are then transferred (93) to the trained third neural network model (138).

[0161] First live object detection results differ from first object detection results in that they were generated using first live ultrasound signals, while first object detection results can be generated using laboratory and / or simulation data, for example.

[0162] Second live object detection results differ from second object detection results in that they were generated using live sensor signals, while second object detection results can be generated using, for example, laboratory and / or simulation data.

[0163] The live ultrasound signals differ from the ultrasound signals with which the parameters of the first neural network model (16) were determined in that they are now determined in the actual application by the ultrasound transducer arrangement (11) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0164] The live sensor signals differ from the sensor signals with which the parameters of the second neural network model (136) were determined in that they are now determined in the actual application by the sensor arrangement (131) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0165] The third neural network model (138) then determines (94) a common object recognition hypothesis based on the first live object recognition results and the alternative second live object recognition results by the third neural network model (138).

[0166] Finally, the joint object detection hypothesis determined for the entire measurement system by the third neural network model (138) is used (95). This can be done, for example, by warning messages to the driver, etc. Figure 12

[0167] shows a model for training and storing the parameters of the third neural network model (138).

[0168] The method begins with the installation (91) of the third neural network model (138) for comparing the first object recognition results of the first object identification performed by the first neural network model (16) for the ultrasonic measuring system (164) with the alternative second object recognition results of the alternative second object identification performed by the second neural network model (136) for the second measuring system (161) and for determining a common object recognition hypothesis on a third computer system. The third computer system can be connected to the first and second computer systems of the Fig. 10 be identical.

[0169] As already mentioned above, the computer systems can also be one or more analog circuits that carry out the computing operations of the neural network models and do not necessarily have to be programmable. For the purposes of this disclosure, such an installation (91) can rather always be carried out in such a way that the structure and essential parameters of the respective neural network model are predetermined, for example in the form of a suitable electronic circuit during the design phase. It can therefore also be an analog or hybrid computer, for example, which partially predetermines the structure and / or parameterization of the respective neural network model in the form of an electrical circuit. However, an installation (91) in the form of pure software on a fully programmable computer system in Harvard and / or Von Neumann architecture is preferable and preferred.

[0170] This is followed by training (91) of the third neural network model (138) by transferring correct and incorrect first object recognition results and correct and incorrect alternative second object recognition results to the third neural network model (138) together with data on their correctness and / or incorrectness, and determining the parameters of the third neural network model (138) depending on these data and / or changing the parameters of the third neural network model (138) depending on these data. This can be done, for example, by first training the first neural network model (16) entirely with laboratory data and then training the second neural network model (136) entirely with laboratory data. Corresponding laboratory data sets are then presented as training data sets to the first neural network model (169) and the second neural network model (136).These generate a first object recognition result and a second object recognition result for each of these training data set pairs. From these, a target object recognition hypothesis is then calculated manually or automatically or determined in some other way. This results in a training data set for the third neural network model (136), consisting of the respective first object recognition result and the respective second object recognition result and the respective target object recognition hypothesis, which represents the corresponding evaluation data.

[0171] With this data, the third neural network model (138) can then be trained (92).

[0172] Once training is successfully completed, the system (e.g., a laboratory device) or a sufficiently functionally similar device (e.g., a series product) can be deployed, e.g., in a vehicle. There, the first live object recognition results and alternative second live object recognition results are then transferred (93) to the trained third neural network model (138).

[0173] First live object detection results differ from first object detection results in that they were generated with first live ultrasound signals, while first object detection results can be generated with laboratory and / or simulation data, for example. Second live object detection results differ from second

[0174] Object detection results in that they were generated using live sensor signals, while second object detection results can be generated using, for example, laboratory and / or simulation data.

[0175] The live ultrasound signals differ from the ultrasound signals with which the parameters of the first neural network model (16) were determined in that they are now determined in the actual application by the ultrasound transducer arrangement (11) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0176] The live sensor signals differ from the sensor signals with which the parameters of the second neural network model (136) were determined in that they are now determined in the actual application by the sensor arrangement (131) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0177] In contrast to the Fig. 11 concludes the procedure of Fig. 12 now concludes with the saving (105) of the parameter data of the trained third neural network (138) in a third training data memory (139). The thus saved parameters of the third neural network model (138) can thereby be applied to functionally similar systems by programming and / or constructing these systems or parts thereof. Figure 13

[0178] shows a model for loading the stored parameters of the third neural network model (138) and then using the third neural network model (138).

[0179] The procedures for the Fig. 12 created data. Thus, the method begins with loading (111) the previously saved parameters of the third neural network model (138) from the third training data store (139) or another data store that currently holds this data into the third neural network model (138). This third neural network model (138) can be located in a completely different, functionally similar overall system than the overall system with which the data to be loaded was created.

[0180] The third neural network model (138) then determines (94) a common object recognition hypothesis based on the first live object recognition results and the alternative second live object recognition results by the third neural network model (138).

[0181] Finally, the joint object detection hypothesis determined for the entire measurement system by the third neural network model (138) is used (95). This can be done, for example, by warning messages to the driver, etc. Figure 14

[0182] shows a model for training and using the fifth neural network model (158).

[0183] The method begins with the installation (201) of the fifth neural network model (158) for comparing the second object recognition results of the second object identification performed by the second neural network model (136) for the second measuring system (161) with the third object recognition results of the alternative third object identification performed by the fourth neural network model (156) for the third measuring system (162) and for determining a common object recognition hypothesis on a fourth computer system. The fourth computer system can be connected to the first and / or second and / or third computer system of the Fig. 10 be identical.

[0184] As already mentioned above, the computer systems can also be one or more analog circuits that carry out the computational operations of the neural network models and do not necessarily have to be programmable. For the purposes of this disclosure, such an installation (201) can rather always be carried out in such a way that the structure and essential parameters of the respective neural network model are predetermined, for example, in the form of a suitable electronic circuit during the design phase. It can therefore also be an analog or hybrid computer, for example, which partially predetermines the structure and / or parameterization of the respective neural network model in the form of an electrical circuit. However, an installation (201) of the neural network models (16, 136, 138, 156, 158) in the form of pure software on a fully programmable computer system in Harvard and / or Von Neumann architecture is preferable and preferred.

[0185] This is followed by training (202) of the fifth neural network model (158) by transferring correct and incorrect second object recognition results and correct and incorrect alternative third object recognition results to the fifth neural network model (158) together with data on their correctness and / or incorrectness, and determining the parameters of the fifth neural network model (158) depending on these data and / or changing the parameters of the fifth neural network model (158) depending on these data. This can be done, for example, by first training the second neural network model (136) entirely with laboratory data and then training the fourth neural network model (156) entirely with laboratory data. Corresponding laboratory data sets are then presented as training data sets to the second neural network model (136) and the fourth neural network model (156).These generate a second object recognition result and a third object recognition result for each of these training data set pairs. A target object recognition hypothesis is then calculated or otherwise determined for each of these pairs, either manually or automatically. This results in a training data set for the fifth neural network model (158), consisting of the respective first object recognition result and the respective second object recognition result and the respective target object recognition hypothesis, which represents the corresponding evaluation data.

[0186] With this data, the fifth neural network model (158) can then be trained (202).

[0187] Once training (202) has been successfully completed, the system (e.g., a laboratory device) or a sufficiently functionally similar device (e.g., a series product) can be deployed, e.g., in a vehicle. There, the second live object recognition results and alternative third live object recognition results are then transferred (203) to the trained fifth neural network model (158). Second live object detection results differ from second

[0188] Object recognition results in that they were generated with live sensor signals from the second sensor arrangement (131), while second object recognition results can be generated, for example, with laboratory and / or simulation data.

[0189] Third live object detection results differ from third object detection results in that they were generated with third live sensor signals of the third sensor arrangement (151), while third object detection results can be generated, for example, with laboratory and / or simulation data.

[0190] The second live sensor signals of the second sensor arrangement (131) differ from the second sensor signals of the second sensor arrangement (131) with which the parameters of the second neural network model (136) were determined in that they are now determined in the actual application by the second sensor arrangement (131) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0191] The third live sensor signals differ from the third sensor signals of the third sensor arrangement (151) with which the parameters of the fourth neural network model (136) were determined in that they are now determined in the actual application by the third sensor arrangement (151) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0192] The fifth neural network model (158) then determines (204) a common object recognition hypothesis based on the second live object recognition results and the third live object recognition results by the fifth neural network model (158).

[0193] Finally, the joint object detection hypothesis determined for the entire measurement system by the fifth neural network model (158) is used (205). This can be done, for example, by warning messages to the driver, etc. Figure 15

[0194] shows a model for training and storing the parameters of the fifth neural network model (158).

[0195] The method begins with the installation (201) of the fifth neural network model (158) for comparing the second object recognition results of the second object identification performed by the second neural network model (136) for the second measuring system (161) with the third object recognition results of the alternative third object identification performed by the fourth neural network model (156) for the third measuring system (162) and for determining a common object recognition hypothesis on a fourth computer system. The fourth computer system can be connected to the first and / or second and / or third computer system of the Fig. 10 be identical.

[0196] As already mentioned above, the computer systems can also be one or more analog circuits that carry out the computational operations of the neural network models and do not necessarily have to be programmable. For the purposes of this disclosure, such an installation (201) can rather always be carried out in such a way that the structure and essential parameters of the respective neural network model are predetermined, for example, in the form of a suitable electronic circuit during the design phase. It can therefore also be an analog or hybrid computer, for example, which partially predetermines the structure and / or parameterization of the respective neural network model in the form of an electrical circuit. However, an installation (201) of the neural network models (16, 136, 138, 156, 158) in the form of pure software on a fully programmable computer system in Harvard and / or Von Neumann architecture is preferable and preferred.

[0197] This is followed by training (202) of the fifth neural network model (158) by transferring correct and incorrect second object recognition results and correct and incorrect alternative third object recognition results to the fifth neural network model (158) together with data on their correctness and / or incorrectness, and determining the parameters of the fifth neural network model (158) depending on these data and / or changing the parameters of the fifth neural network model (158) depending on these data. This can be done, for example, by first training the second neural network model (136) entirely with laboratory data and then training the fourth neural network model (156) entirely with laboratory data. Corresponding laboratory data sets are then presented as training data sets to the second neural network model (136) and the fourth neural network model (156).These generate a second object recognition result and a third object recognition result for each of these training data set pairs. A target object recognition hypothesis is then calculated or otherwise determined for each of these pairs, either manually or automatically. This results in a training data set for the fifth neural network model (158), consisting of the respective first object recognition result and the respective second object recognition result and the respective target object recognition hypothesis, which represents the corresponding evaluation data.

[0198] With this data, the fifth neural network model (158) can then be trained (202).

[0199] Once training (202) has been successfully completed, the system (e.g., a laboratory device) or a sufficiently functionally similar device (e.g., a series product) can be deployed, e.g., in a vehicle. There, the optional transfer (203) of second live object recognition results and alternative third live object recognition results to the trained fifth neural network model (158) then takes place. This optional step serves to facilitate learning in real-world applications.

[0200] Second live object detection results differ from second object detection results in that they were generated with live sensor signals of the second sensor arrangement (131), while second object detection results can be generated, for example, with laboratory and / or simulation data.

[0201] Third live object detection results differ from third object detection results in that they were generated with third live sensor signals of the third sensor arrangement (151), while third object detection results can be generated, for example, with laboratory and / or simulation data.

[0202] The second live sensor signals of the second sensor arrangement (131) differ from the second sensor signals of the second sensor arrangement (131) with which the parameters of the second neural network model (136) were determined in that they are now determined in the actual application by the second sensor arrangement (131) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0203] The third live sensor signals differ from the third sensor signals of the third sensor arrangement (151) with which the parameters of the fourth neural network model (136) were determined in that they are now determined in the actual application by the third sensor arrangement (151) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0204] In contrast to the Fig. 14 concludes the procedure of Fig. 15 now concludes with the saving (215) of the parameter data of the trained fifth neural network (158) in a fifth training data memory (159). The parameters of the fifth neural network model (158) thus saved can thereby be used in functionally similar systems by programming and / or constructing these systems or parts thereof. Figure 16

[0205] shows a model for loading the stored parameters of the fifth neural network model (158) and then using the fifth neural network model (158).

[0206] The procedures for the Fig. 15 created data. Thus, the method begins with loading (211) the previously saved parameters of the fifth neural network model (158) from the fifth training data store (159) or another data store that currently holds this data into the fifth neural network model (158). This fifth neural network model (158) can be located in a completely different, functionally similar overall system than the overall system with which the data to be loaded was created.

[0207] The fifth neural network model (158) then determines (204) a common object recognition hypothesis based on the second live object recognition results and the third live object recognition results by the fifth neural network model (158).

[0208] Finally, the joint object detection hypothesis determined for the entire measurement system by the fifth neural network model (158) is used (205). This can be done, for example, by warning messages to the driver, etc. Figure 17

[0209] shows schematically the second combination method from section F).

[0210] First, an installation (51) of a second neural network model (136) for the second measuring system (161) is carried out on a second computer system.

[0211] Secondly, an installation (61) of a fourth neural network model (156) for the third measuring system (162) takes place on a fourth computer system.

[0212] Thirdly, an installation (201) of a fifth neural network model (158) takes place on a fifth computer system.

[0213] The second computer system and / or the fourth computer system and / or the fifth computer system can be the same and / or identical in terms of hardware. One, several, or all of the computer systems can also be an analog circuit that executes the computing operations of the neural network models (136, 156, 158) and does not necessarily have to be programmable. For the purposes of this disclosure, such an installation can rather always be carried out in such a way that the structure and essential parameters of the respective neural network model are predetermined during the design phase, for example in the form of a suitable electronic circuit. It can therefore also be an analog or hybrid computer, for example, which partially predetermines the structure and / or parameterization of the respective neural network model in the form of an electrical circuit.However, an installation in the form of pure software on a fully programmable computer system in Harvard and / or Von Neumann architecture is preferable and preferred.

[0214] Typically, the loading (71) of previously saved parameters of the second neural network model (136) for the second measuring system (161) from the second training data memory (137) follows as current parameters of the second neural network model (136) for the second measuring system (161).

[0215] Typically, this is followed by loading (72) previously saved parameters of the fourth neural network model (156) for the third measuring system (162) from the fourth training data memory (157) as current parameters of the fourth neural network model (156) for the third measuring system (162).

[0216] Typically, this is followed by loading (211) previously saved parameters of the fifth neural network model (158) from the fifth training data store (159) as current parameters of the fifth neural network model (158).

[0217] In this way, the entire measuring system is now prepared for signal processing.

[0218] To process the signals, firstly, a transfer (54) of second live sensor signals from the second sensor arrangement (131) to the trained second neural network model (136) for the second measuring system (161) and secondly a transfer (64) of third live sensor signals from the third sensor arrangement (151) to the trained fourth neural network model (1356) for the third measuring system (162)

[0219] In this case, a second object identification is carried out by the trained second neural network model (136) to generate a second object recognition result and a third object identification is carried out by the trained fourth neural network model (156) to generate a third object recognition result.

[0220] The second live sensor signals of the second sensor arrangement (131) differ from the second sensor signals of the second sensor arrangement (131) with which the parameters of the second neural network model (136) were determined in that they are now determined in the actual application by the second sensor arrangement (131) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0221] The third live sensor signals of the third sensor arrangement (151) differ from the third sensor signals of the third sensor arrangement (151) with which the parameters of the fourth neural network model (156) were determined in that they are now determined in the actual application by the third sensor arrangement (151) in the target operation, for example in the consumer's vehicle during use of the vehicle, and not in the laboratory or during test drives.

[0222] This is followed by the transfer (203) of first live object recognition results and alternative second live object recognition results to the trained third neural network model (138)

[0223] Finally, a comparison (204) of the second object recognition results of the second object identification performed by the second neural network model (136) for the second measuring system (161) with the third object recognition results of the third object identification performed by the fourth neural network model (156) for the third measuring system (162) takes place, and a common object recognition hypothesis is determined (204), which represents the recognition result of the entire measuring system. The common object recognition hypothesis can be a list of objects with recognition probabilities and of arrangements of such objects with recognition probabilities. It can also contain only one object and / or only one arrangement. A probability specification does not necessarily have to be included in the common object recognition hypothesis.

[0224] The learning process is now carried out as follows: A second parameter adjustment (87) of the second neural network model (136) for the second measuring system (161) is carried out depending on the result of the comparison (204) of the second object recognition results of the second object identification carried out by the second neural network model (136) for the second measuring system (161) with the third object recognition results of the alternative third object identification carried out by the fourth neural network model (156) for the third measuring system (162).In this case, the parameter adaptation is generally carried out in such a way that failure to take into account parts of the second object recognition results of the second neural network model (136) in the joint object recognition hypothesis by the fifth neural network model (158), which carries out the comparison (204), leads to an attenuation of the corresponding parameters of the second neural network model (136), and that failure to take into account parts of the second object recognition results of the second neural network model (136) in the joint object recognition hypothesis by the fifth neural network model (158), which carries out the comparison (204), leads to an amplification of the corresponding parameters of the second neural network model (136).

[0225] A fourth parameter adjustment (88) of the fourth neural network model (156) for the third measuring system (162) is carried out depending on the result of the comparison (204) of the second object recognition results of the second object identification carried out by the second neural network model (136) for the first measuring system (161) with the alternative third object recognition results of the alternative third object identification carried out by the fourth neural network model (156) for the third measuring system (162).In this case, the parameter adaptation is generally carried out in such a way that a non-consideration of parts of the third object recognition results of the fourth neural network model (156) in the joint object recognition hypothesis by the fifth neural network model (158), which carries out the comparison (204), leads to an attenuation of the corresponding parameters of the fourth neural network model (1356) and that a consideration of parts of the third object recognition results of the fourth neural network model (156) in the joint object recognition hypothesis by the fifth neural network model (158), which carries out the comparison (204), leads to an amplification of the corresponding parameters of the fourth neural network model (156).

[0226] The fifth neural network model (158), which performs the comparison (204), thus makes a decision.

[0227] The use (205) of the common object recognition hypothesis determined by the fifth neural network model (158) for the entire measurement system concludes the procedure. Figure 18

[0228] shows the structure of the proposed combined overall system comprising an ultrasonic measuring system (164) and a second measuring system (161) for the corresponding first combination method according to section E).

[0229] A first group of components forms the ultrasonic measuring system (164): The ultrasonic transducer arrangement (11) emits the acoustic ultrasonic signals and receives the echoes of the objects to be detected in the vehicle's surroundings. The ultrasonic transducer arrangement converts these received echo signals and other received ultrasonic signals into ultrasonic measured values ​​and makes them available to a signal processor (15), which further processes them. The signal processor (15) can be a digital programmable processor in von Neumann or Harvard architecture or in another suitable architecture. However, it can also be a suitable analog circuit for preprocessing the signals. In the case of analog neural networks, the processor can also be omitted if necessary, although this has not yet proven successful in practice.

[0230] The received signals processed by the signal processor (15) are the input signals of the first neural network model (16), which is typically stored in a computer's digital memory. The first neural network model (16) can also be implemented, for example, as a special microelectronic circuit or other functionally equivalent device. The neural network model (16) is the implementation of a deep learning method, as described above.

[0231] The parameters of the first neural network model (16) can be loaded from a first training data storage (17). The training data are preferably determined in the laboratory using a method as described above.

[0232] The ultrasonic transducer arrangement (11) is connected to control electronics (12), which can optionally be capable of beam shaping, for example by means of phase shifting the emitted ultrasonic signals of the individual ultrasonic transducers of the ultrasonic transducer arrangement (11). The control electronics (12) controls the corresponding ultrasonic transducers of the ultrasonic transducer arrangement (11) for preferably directed emission or reception of the ultrasonic signals. This orientation is desirable, but not mandatory. The emission of ultrasonic signals by the ultrasonic transducer arrangement (11) is controlled by a transmitter controller (13). This controller is connected to the control electronics (12) and sets the transmission parameters of the ultrasonic transducers of the ultrasonic transducer arrangement (11) depending on settings that are made via the optional user settings (14) stored in an optional memory.by a higher-level control unit. Transmission parameters that can be controlled by the transmitter control (13) include, for example, burst length, frequency, frequency response, coding, polarity, phase, or amplitude.

[0233] The output values ​​of the first neural network model (16) are preferably the positions of the detected objects and preferably their classification, as well as, for example, a corresponding confidence value, which can be assigned to the detected recognition result as a whole, and / or confidence values ​​that are assigned to the individual objects. These data form the first object recognition result. Further data can be part of the output values. Alternative hypothesis lists are also possible, wherein the hypothesis list comprises alternative recognition results with different confidence values ​​for these first object recognition results. Each first object recognition result comprises one or more objects.

[0234] The above-described ultrasonic measuring system (164) preferably further comprises a plurality of first user controls (10), wherein the setting of one or more of the first user controls (10) occurs in response to the identification of an arrangement of objects and / or object components or of objects in the first ultrasonic measured values ​​and / or in the two- or three-dimensional representations and / or in dependence on the first object recognition results.

[0235] In the previously described ultrasonic measuring system (164), the setting or adaptation of control parameters of one or more of the controllers preferably occurs in response to the identification of the arrangement of objects and / or object components in the first ultrasonic measured values ​​and / or in the two- or three-dimensional representations and / or as a function of the first object detection results. In this case, the setting of one or more of the controllers (13) of the ultrasonic measuring system or of the vehicle preferably occurs in response to the identification of objects and / or object components and / or the arrangement of objects and / or objects and / or object components in the first ultrasonic measured values ​​and / or in the two- or three-dimensional representations and / or as a function of the first object detection results.

[0236] The ultrasonic transducer arrangement (11) is connected to control electronics (12), which can optionally be capable of beam shaping, for example by means of phase shifting the emitted ultrasonic signals of the individual ultrasonic transducers of the ultrasonic transducer arrangement (11). The control electronics (12) controls the corresponding ultrasonic transducers of the ultrasonic transducer arrangement (11) for preferably directed emission or reception of the ultrasonic signals. This orientation is desirable, but not mandatory. The emission of ultrasonic signals by the ultrasonic transducer arrangement (11) is controlled by a transmitter controller (13). This controller is connected to the control electronics (12) and sets the transmission parameters of the ultrasonic transducers of the ultrasonic transducer arrangement (11) depending on settings that are made via the optional user settings (14) stored in an optional memory.by a higher-level control unit. Transmission parameters that can be controlled by the transmitter control (13) include, for example, burst length, frequency, frequency response, coding, polarity, phase, or amplitude.

[0237] A second group of components forms the second measuring system (161): Here, it is assumed that the second measuring system also transmits measurement signals whose echoes are received. This could be a LIDAR or RADAR system, for example. However, this is only an example. It could also be a different measuring system, such as a PIR infrared measuring system, which relies on signals already present in the vehicle's surroundings. PIR stands for "passive infrared." If applicable, the second sensor arrangement (131) transmits the measurement signals into the vehicle's surroundings and receives the response from the surroundings, specifically the signal response of the objects to be detected or characterized in the vehicle's surroundings to these transmitted measurement signals. The sensor arrangement (131) converts these received signal responses and other received interference and useful signal signals into signal measurement values ​​and makes them available to a second signal processor (135), which processes them further. The second signal processor (135) can again be a digital programmable processor in von Neumann or Harvard architecture or in another suitable architecture. However, it can also be a suitable analog circuit for preprocessing the signals. In the case of analog neural networks, the processor can also be omitted if necessary, although this has not yet proven successful in practice.In the case of a programmable processor, if the capacity is sufficient, the second signal processor (135) can form a unit with the first signal processor (15) and / or the said computer, for example.

[0238] The received signals of the sensor arrangement (131) processed by the second signal processor (135) are the input signals of the second neural network model (136), which is typically stored in a digital memory of a second computer, which may be identical to the first computer and / or the signal processor (15) and / or the second signal processor (135). The second neural network model (136) can also be implemented, for example, as a special microelectronic circuit or other functionally equivalent device. The second neural network model (136) is preferably the implementation of a deep learning method, as described above.

[0239] The parameters of the second neural network model (136) can preferably be loaded from a second training data storage (137). The training data are preferably determined in the laboratory using a method as described above.

[0240] The second sensor arrangement (131) is connected to a second control electronics unit (132), which, in the case of a LIDAR or radar system, can optionally be capable of beam shaping, for example by phase shifting the transmitted measurement signals of the individual sensor elements of the sensor arrangement (131). The second control electronics unit (132) controls the corresponding sensor elements of the sensor arrangement (11) for the preferably directed transmission or reception of the measurement signals. In the case of a RADAR system, this can be achieved, for example, by phase-array antennas. This alignment is desirable but not mandatory. The transmission of measurement signals by the sensor arrangement (131) is controlled by a second sensor control unit (133).This is connected to the second control electronics (132) and sets the transmission parameters of the sensor elements of the sensor arrangement (131) depending on settings that are specified via the optional user settings (134) stored in an optional second setting memory or by a higher-level control unit. Transmission parameters that can be controlled by the second sensor control (133) include, for example, burst length, frequency, frequency response, coding, polarity, phase, or amplitude in the case of a RADAR system as the second measuring system (161).

[0241] The output values ​​of the second neural network model (136) are preferably the positions of the detected objects and preferably their classification, as well as, for example, a corresponding confidence value, which can be assigned to the detected detection result as a whole, and / or confidence values ​​that are assigned to the individual objects. These data form the second object detection result. Further data can be part of the output values. Alternative hypothesis lists are also possible, wherein the hypothesis list comprises alternative detection results with different confidence values ​​for these second object detection results. Each second object detection result comprises one or more objects.

[0242] The previously described second measuring system (161) preferably further comprises a plurality of second user controls (140a), wherein the setting of one or more of the second user controls (140a) occurs in response to the identification of an arrangement of objects and / or object components or of objects in the second measured values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or in dependence on the second object recognition results.

[0243] In the previously described second measuring system (161), the setting or adaptation of control parameters of one or more of the controllers preferably takes place in response to the identification of the arrangement of objects and / or object components in the second measured values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or as a function of the second object detection results. In this case, the setting of one or more of the second sensor controls (133) of the second measuring system (161) or of the vehicle preferably takes place in response to the identification of objects and / or object components and / or the arrangement of objects and / or objects and / or object components in the second measured values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or as a function of the second object detection results.

[0244] The second sensor arrangement (131) is connected to second control electronics (132), which can optionally be capable of beam shaping, for example by phase shifting the transmitted measurement signals of the individual sensor elements of the sensor arrangement (131). The second control electronics (132) controls the corresponding sensor elements of the sensor arrangement (131) for the preferably directed transmission or reception of the measurement signals. This alignment is desirable but not mandatory. The transmission of measurement signals by the sensor arrangement (131) is controlled by the sensor control (133). This is connected to the second control electronics (132) and sets the transmission parameters of the sensor elements of the sensor arrangement (131) depending on settings that are specified via the optional user settings (134) stored in an optional second setting memory or by a higher-level control unit.Transmission parameters which can be controlled by the sensor control (133) are, for example, burst length, frequency, frequency response, coding, polarity, phase or amplitude.

[0245] The proposed overall sensor system further comprises a third neural network model (138), which is preferably implemented on a third computer system. The third computer system can form a unit with the first computer system of the first neural network model (16), and / or with the second computer system of the second neural network model (136), and / or with the first signal processor (15), and / or with the second signal processor (135).

[0246] The first neural network model (16) forms a first object recognition result.

[0247] The second neural network model (136) forms a second object recognition result.

[0248] The third neural network model (136) then forms a joint object recognition hypothesis based on the first object recognition result and the second object recognition result. Based on the determined joint object recognition hypothesis and the first object recognition result and the second object recognition result, the third neural network model (138) controls a first parameter modification device (140) for the first neural network model (16) and a second parameter modification device (141) for the second neural network model (136).

[0249] Both the first parameter modification device (140) and the second parameter modification device (141) can be implemented on another computer system or one of the aforementioned computer systems and / or signal processors. However, they can also be special devices, for example, special circuits.

[0250] The first parameter modification device (140) modifies the parameters of the first neural network model (16) depending on the signals of the third neural network model (138) and / or the first object recognition result and / or the second object recognition result.

[0251] The second parameter modification device (141) modifies the parameters of the second neural network model (136) depending on the signals of the third neural network model (138) and / or the first object recognition result and / or the second object recognition result. This makes the device self-learning capable.

[0252] For safety-relevant applications, it is necessary to detect drift of the third neural network model (138) and / or the other neural network models. Therefore, the proposed overall system comprises a first safety device (142). The task of the first safety device (142) is to monitor the operation of the overall system and detect obviously incorrect results of the three neural networks.For this purpose, the first safety device (142) receives high-priority direct signals from the ultrasonic transducer arrangement (11) in the form of a high-priority signal (H11) from the ultrasonic transducer arrangement (11), high-priority signals (H15) from the first signal processor (15), high-priority direct signals from the second sensor arrangement (131) in the form of a high-priority signal (H131) from the second sensor arrangement (131), high-priority signals (H135) from the second signal processor (135), and user inputs (H142) or other external high-priority signals. All of these high-priority signals are only examples. Each of these high-priority signals can be a single signal or a bundle of multiple signals in time and / or space multiplexing. These high-priority signals are preferred but not mandatory. They enable the system to train the decision-maker in the form of the third neural network model (138).

[0253] Furthermore, the overall measuring system described in this section preferably comprises a user control display (9) that displays settings made in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the first ultrasound measurement values ​​and / or in the two- or three-dimensional representations and / or depending on the first object detection result. The user control display (9) can, in principle, form a unit with the display (19).

[0254] In the ultrasonic measurement system (164) described here, the user control display (9) preferably activates and displays control options in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the first ultrasonic measurement values ​​and / or in the two- or three-dimensional representations and / or in the first object recognition results.

[0255] The ultrasound measuring system (164) described in this section preferably also comprises a first training image memory (17) for the purpose of training the first neural network model (16), which stores training images for the first neural network model (16).

[0256] In the ultrasonic measuring system (164) described in this section, the display (19) is preferably further designed to display the two- or three-dimensional representations which are at least temporarily additionally provided with the identification determined by the first neural network model (16) and / or a part of the first object recognition result.

[0257] The overall measuring system described above preferably further comprises a plurality of third user controls (145), wherein the setting of one or more of the third user controls (145) occurs in response to the identification of an arrangement of objects and / or object components or of objects in the first object recognition results and / or in the second object recognition results and / or in the two- or three-dimensional representations and / or in dependence on the common object recognition hypothesis.

[0258] The preferably used image processor (18) converts this information into a suitable two- or three-dimensional representation, which is output to the vehicle driver by the display (19). The information on object positions and classifications is preferably also transferred to the user controls (10, 140a, 145). The selection of a stored user control or the adjustment of control parameters of a control in response to the classification is implemented by the user control (10, 140a, 145) and passed on to the settings memory (14) or displayed by the user control display (19). The settings memory (14) also transmits parameters dependent on the classification to the image processor (18), so that the representation of the objects and their identification output by the display (19) can be automatically adapted accordingly.

[0259] Finally, it should be expressly noted that the transmission control of the second measuring system (161) is not absolutely necessary if the second measuring system (161) does not require the transmission of measurement signals for the measuring principle used. In this case, the second control electronics (134), the second sensor control (133), and the second setting memory (134) may even be omitted. However, these will generally still determine the reception parameters of the second sensor arrangement (131). Figure 19

[0260] shows the structure of the proposed combined overall system consisting of a second measuring system (161) and a third measuring system (162) for the corresponding first combination method according to Section F). The basic structure corresponds to that of Fig. 18 with the difference that the ultrasonic measuring system (164) is replaced by the third measuring system (162).

[0261] A first group of components forms the third measuring system (162): The third sensor arrangement (151) preferably (but not necessarily) transmits third measurement signals and receives the echoes of the objects to be detected in the vehicle's surroundings. The third sensor arrangement (151) converts these received echo signals and other received measurement signals into third measurement values ​​and makes them available to a third signal processor (155), which further processes them. The third signal processor (155) can be a digital programmable processor in von Neumann or Harvard architecture or in another suitable architecture. However, it can also be a suitable analog circuit for preprocessing the signals. In the case of analog neural networks, the third processor can also be omitted if necessary, although this has not yet proven successful in practice.

[0262] The received signals processed by the third signal processor (155) are the input signals of the fourth neural network model (156), which is typically stored in a computer's digital memory. The fourth neural network model (156) can also be implemented, for example, as a special microelectronic circuit or other functionally equivalent device. The fourth neural network model (156) is the implementation of a deep learning method, as described above.

[0263] The parameters of the fourth neural network model (156) can be loaded from a fourth training data store (157). The training data are preferably determined in the laboratory using a method as described above.

[0264] The third sensor arrangement (151) is connected to third control electronics (152), which can optionally be capable of beam shaping, for example by means of phase shifting the optionally emitted measurement signals of the individual third sensor elements of the third sensor arrangement (151). The third control electronics (152) controls the corresponding sensor elements of the third sensor arrangement (151) for the preferably directed emission or reception of the measurement signals. This orientation is desirable, but not mandatory. The emission of measurement signals by the third sensor arrangement (151) is controlled by a third sensor control (153). This is connected to the third control electronics (152) and sets the transmission parameters of the sensor elements of the sensor arrangement (151) depending on settings that are stored in an optional memory via the optional third user settings (154).by a higher-level control unit.

[0265] Transmission parameters which can be controlled by the sensor control (153) are, in the case of a RADAR or LIDAR system, for example, burst length, frequency, frequency response, coding, polarity, phase or amplitude.

[0266] The output values ​​of the fourth neural network model (156) are preferably the positions of the detected objects and preferably their classification, as well as, for example, a corresponding confidence value that can be assigned to the detected recognition result as a whole, and / or confidence values ​​that are assigned to the individual objects. These data form the third object recognition result. Further data can be part of the output values. Alternative hypothesis lists are also possible, wherein the hypothesis list comprises alternative recognition results with different confidence values ​​for these third object recognition results. Each third object recognition result comprises one or more objects.

[0267] The above-described third measuring system (162) preferably further comprises a plurality of third user controls (163), wherein the setting of one or more of the third user controls (163) occurs in response to the identification of an arrangement of objects and / or object components or of objects in the first ultrasonic measurement values ​​and / or in the two- or three-dimensional representations and / or in dependence on the first object recognition results.

[0268] In the third measuring system (162) described above, the setting or adaptation of control parameters of one or more of the controllers preferably occurs in response to the identification of the arrangement of objects and / or object components in the third measured values ​​of the third sensor arrangement (151) and / or in the two- or three-dimensional representations and / or as a function of the third object detection results. In this case, the setting of one or more of the controllers (153) of the third measuring system (162) or of the vehicle preferably occurs in response to the identification of objects and / or object components and / or the arrangement of objects and / or objects and / or object components in the third measured values ​​of the third sensor arrangement (151) and / or in the two- or three-dimensional representations and / or as a function of the third object detection results.

[0269] The third sensor arrangement (151) is connected to third control electronics (152), which, in the case of a LIDAR or radar system, can optionally be capable of beam shaping, for example by means of phase shifting the emitted ultrasonic signals of the individual sensor elements of the sensor arrangement (11). The third control electronics (152) controls the corresponding sensor elements of the third sensor arrangement (151) for the preferably directed emission or reception of the third measurement signals. This orientation is desirable, but not mandatory. The emission of third measurement signals by the third sensor arrangement (151) is controlled by a third sensor control (153). This is connected to the third control electronics (152) and sets the transmission parameters of the sensor elements of the sensor arrangement (11) depending on settings that are made via the optional user settings (154) stored in an optional third setting memory.by a higher-level control unit. Transmission parameters that can be controlled by the third transmitter controller (153) include, for example, burst length, frequency, frequency response, coding, polarity, phase, or amplitude in the case of a RADAR or LIDAR system.

[0270] A second group of components forms the second measuring system (161): Here, it is assumed that the second measuring system also transmits measurement signals whose echoes are received. This could be a LIDAR or RADAR system, for example. However, this is only an example. It could also be a different measuring system, such as a PIR infrared measuring system, which relies on signals already present in the vehicle's surroundings. PIR stands for "passive infrared."

[0271] If applicable, the second sensor arrangement (131) transmits the measurement signals into the surroundings of the vehicle and receives the response of the surroundings, specifically the signal response of the objects to be detected or characterized in the surroundings of the vehicle to these emitted measurement signals. The sensor arrangement (131) converts these received signal responses and other received interference and useful signal signals into signal measurement values ​​and makes them available to a second signal processor (135), which processes them further. The second signal processor (135) can again be a digital programmable processor in von Neumann or Harvard architecture or in another suitable architecture. However, it can also be a suitable analog circuit for preprocessing the signals. In the case of analog neural networks, the processor can also be omitted if necessary, although this has not yet proven successful in practice.In the case of a programmable processor, if the capacity is sufficient, the second signal processor (135) can form a unit with the first signal processor (15) and / or the said computer, for example.

[0272] The received signals of the sensor arrangement (131) processed by the second signal processor (135) are the input signals of the second neural network model (136), which is typically stored in a digital memory of a second computer, which may be identical to the first computer and / or the signal processor (15) and / or the second signal processor (135). The second neural network model (136) can also be implemented, for example, as a special microelectronic circuit or other functionally equivalent device. The second neural network model (136) is preferably the implementation of a deep learning method, as described above.

[0273] The parameters of the second neural network model (136) can preferably be loaded from a second training data storage (137). The training data are preferably determined in the laboratory using a method as described above.

[0274] The second sensor arrangement (131) is connected to a second control electronics unit (132), which, in the case of a LIDAR or radar system, can optionally be capable of beam shaping, for example by phase shifting the transmitted measurement signals of the individual sensor elements of the sensor arrangement (131). The second control electronics unit (132) controls the corresponding sensor elements of the sensor arrangement (11) for the preferably directed transmission or reception of the measurement signals. In the case of a RADAR system, this can be achieved, for example, by phase-array antennas. This alignment is desirable but not mandatory. The transmission of measurement signals by the sensor arrangement (131) is controlled by a second sensor control unit (133).This is connected to the second control electronics (132) and sets the transmission parameters of the sensor elements of the sensor arrangement (131) depending on settings that are specified via the optional user settings (134) stored in an optional second setting memory or by a higher-level control unit. Transmission parameters that can be controlled by the second sensor control (133) include, for example, burst length, frequency, frequency response, coding, polarity, phase, or amplitude in the case of a RADAR system as the second measuring system (161).

[0275] The output values ​​of the second neural network model (136) are preferably the positions of the detected objects and preferably their classification, as well as, for example, a corresponding confidence value, which can be assigned to the detected detection result as a whole, and / or confidence values ​​that are assigned to the individual objects. These data form the second object detection result. Further data can be part of the output values. Alternative hypothesis lists are also possible, wherein the hypothesis list comprises alternative detection results with different confidence values ​​for these second object detection results. Each second object detection result comprises one or more objects.

[0276] The previously described second measuring system (161) preferably further comprises a plurality of second user controls (140a), wherein the setting of one or more of the second user controls (140a) occurs in response to the identification of an arrangement of objects and / or object components or of objects in the second measured values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or in dependence on the second object recognition results.

[0277] In the previously described second measuring system (161), the setting or adaptation of control parameters of one or more of the controllers preferably takes place in response to the identification of the arrangement of objects and / or object components in the second measured values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or as a function of the second object detection results. In this case, the setting of one or more of the second sensor controls (133) of the second measuring system (161) or of the vehicle preferably takes place in response to the identification of objects and / or object components and / or the arrangement of objects and / or objects and / or object components in the second measured values ​​of the second sensor arrangement (131) and / or in the two- or three-dimensional representations and / or as a function of the second object detection results.

[0278] The second sensor arrangement (131) is connected to second control electronics (132), which can optionally be capable of beam shaping, for example by phase shifting the transmitted measurement signals of the individual sensor elements of the sensor arrangement (131). The second control electronics (132) controls the corresponding sensor elements of the sensor arrangement (131) for the preferably directed transmission or reception of the measurement signals. This alignment is desirable but not mandatory. The transmission of measurement signals by the sensor arrangement (131) is controlled by the sensor control (133). This is connected to the second control electronics (132) and sets the transmission parameters of the sensor elements of the sensor arrangement (131) depending on settings that are specified via the optional user settings (134) stored in an optional second setting memory or by a higher-level control unit.Transmission parameters which can be controlled by the sensor control (133) are, for example, burst length, frequency, frequency response, coding, polarity, phase or amplitude.

[0279] The proposed overall sensor system further comprises a fifth neural network model (158), which is preferably implemented on a further computer system. The further computer system can form a unit with the aforementioned computer system and / or the aforementioned signal processors. However, it can also be special devices, for example, special circuits.

[0280] The fourth neural network model (156) forms a third object recognition result.

[0281] The second neural network model (136) forms a second object recognition result.

[0282] The fifth neural network model (158) then forms a joint object recognition hypothesis based on the third object recognition result and the second object recognition result. Based on the determined joint object recognition hypothesis and the third object recognition result and the second object recognition result, the fifth neural network model (158) controls a fourth parameter modification device (144) for the fourth neural network model (156) and a second parameter modification device (141) for the second neural network model (136).

[0283] Both the fourth parameter modification device (144) and the second parameter modification device (141) can be implemented on another computer system or one of the aforementioned computer systems and / or signal processors. However, they can also be special devices, for example, special circuits.

[0284] The fourth parameter modification device (144) modifies the parameters of the fourth neural network model (156) depending on the signals of the fifth neural network model (158) and / or the third object recognition result and / or the second object recognition result.

[0285] The second parameter modification device (141) modifies the parameters of the second neural network model (136) depending on the signals of the fifth neural network model (158) and / or the third object recognition result and / or the second object recognition result. This makes the device self-learning capable.

[0286] For safety-relevant applications, it is necessary to detect drift of the fifth neural network model (158) and / or the other neural network models. Therefore, the proposed overall system includes a second safety device (143). The task of the second safety device (143) is to monitor the operation of the overall system and detect obviously incorrect results of the three neural networks.For this purpose, the second safety device (143) receives high-priority direct signals from the third sensor arrangement (151) in the form of a high-priority signal (H151) from the third sensor arrangement (151), high-priority signals (H155) from the third signal processor (155), high-priority direct signals from the second sensor arrangement (131) in the form of a high-priority signal (H131) from the second sensor arrangement (131), high-priority signals (H135) from the second signal processor (135), and user inputs (H142) or other external high-priority signals. All of these high-priority signals are only examples. Each of these high-priority signals can be a single signal or a bundle of multiple signals in time and / or space multiplexing. These high-priority signals are preferred but not mandatory. They enable the system to train the decision maker in the form of the fifth neural network model (158).

[0287] Furthermore, the overall measuring system described in this section preferably comprises a user control display (9) that displays settings made in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the first ultrasound measurement values ​​and / or in the two- or three-dimensional representations and / or depending on the first object detection result. The user control display (9) can, in principle, form a unit with the display (19).

[0288] In the third measuring system (162) described here, the user control display (9) preferably activates and displays control options in response to the identification of objects and / or object components and / or the arrangement of objects and / or object components in the third measured values ​​of the third sensor arrangement (151) and / or in the two- or three-dimensional representations and / or in the third object recognition results.

[0289] What is described in this section preferably also comprises, for the purpose of training the fourth neural network model (156), a fourth training image memory (157) which stores training images for the fourth neural network model (156).

[0290] In the third measuring system (162) described in this section, the display (19) is preferably further designed to display the two- or three-dimensional representations which are at least temporarily additionally provided with the identification determined by the fourth neural network model (156) and / or a part of the third object recognition result.

[0291] The overall measuring system described above preferably further comprises a plurality of third user controls (145), wherein the setting of one or more of the third user controls (145) occurs in response to the identification of an arrangement of objects and / or object components or of objects in the third object recognition results and / or in the second object recognition results and / or in the two- or three-dimensional representations and / or depending on the common object recognition hypothesis.

[0292] The preferably used image processor (18) converts this information into a suitable two- or three-dimensional representation, which is output to the vehicle driver by the display (19). The information about object positions and classifications is preferably also transferred to the user controls (140a, 145, 160). The selection of a stored user control or the adjustment of control parameters of a control in response to the classification is implemented by the user control (140a, 145, 160) and passed on to the third setting memory (154) or displayed by the user control display (19). The third setting memory (154) also transmits parameters dependent on the classification to the image processor (18), so that the representation of the objects and their identification output by the display (19) can be automatically adapted accordingly.

[0293] Finally, it should be expressly noted that the transmission control of the second measuring system (161) is not absolutely necessary if the second measuring system (161) does not require the transmission of measurement signals for the measuring principle used. In this case, the second control electronics (134), the second sensor control (133), and the second setting memory (134) may even be omitted. However, these will generally still determine the reception parameters of the second sensor arrangement (131).

[0294] Finally, it should be expressly noted here that the transmission control of the third measuring system (162) is not absolutely necessary if the third measuring system (162) does not require the transmission of measurement signals for the measuring principle used. In this case, the third control electronics (154), the third sensor control (153), and the third setting memory (154) may even be omitted. However, these will generally still determine the reception parameters of the third sensor arrangement (151). Figure 20

[0295] shows a model for training and using an entire system according to Fig. 19. (Can be applied to a complete system according to Fig. 18 can be easily transferred.)

[0296] The method comprises, as a first step, the implementation (171) of a procedure according to Fig. 7 for the second measuring system (161) with the determination of the second sensor measured values ​​and determination of a second object recognition result.

[0297] The method comprises, as a second step, the implementation (172) of a method according to Fig. 7 for the third measuring system (162).

[0298] The method comprises, as a third step as a result step of the second step, the determination of the third sensor measured values ​​and the determination of a third object recognition result.

[0299] The method comprises, as a fourth step, installing (173) a fifth neural network model (158) on a computer-based system. This step may optionally also include loading parameters for the fifth neural network model (158).

[0300] The method comprises, as a fifth step, training (174) the fifth neural network model (158) to identify the objects and / or object components and / or arrangements of objects and / or object components in the second object recognition result and / or in the third object recognition result

[0301] The method comprises, as a sixth step, which is a result step of the fifth step, the generation of a joint object recognition hypothesis.

[0302] The method comprises, as a seventh step, using (175) the trained fifth neural network model (158) • for identifying objects and / or object components and / or arrangements of objects and / or object components on the basis of the second object recognition result and the third object recognition result by the fifth neural network model (158) and / or • to generate a joint object recognition hypothesis based on the second object recognition result and the third object recognition result by the fifth neural network model (158). Figure 21

[0303] shows a model for training and storing the parameters of the neural network models of an entire system according to Fig. 19. (Can be applied to a complete system according to Fig. 18 can be easily transferred.)

[0304] The method comprises, as a first step, the implementation (171) of a method according to Fig. 7 for the second measuring system (161).

[0305] The method comprises, as a second step as a result step of the first step, determining the second sensor measured values ​​and determining a second object detection result.

[0306] The method comprises, as a third step, the step of carrying out (172) a method according to Fig. 7 for the third measuring system (162).

[0307] The method comprises, as a fourth step, which is a result step of the third step, the determination of the third sensor measured values ​​and the determination of a third object detection result.

[0308] The method comprises, as a fifth step, the step of installing (173) a fifth neural network model (158) on a computer-based system. This step may optionally also include loading parameters for the fifth neural network model (158).

[0309] The method comprises, as a sixth step, training (174) the fifth neural network model (158). This is done to identify the objects and / or object components and / or arrangements of objects and / or object components in the second object recognition result and / or in the third object recognition result and to generate a joint object recognition hypothesis.

[0310] The method comprises, as a seventh step, saving (185) the parameter data of the trained fifth neural network (158) in a fifth training data memory (159). Figure 22

[0311] shows a model for loading the parameters of the neural network models of an entire system according to Fig. 19 and using the complete system prepared in this way. (Can be applied to a complete system according to Fig. 18 can be easily transferred.)

[0312] The method comprises a first step of loading (71) the previously saved parameters of the second neural network model (136) for the second measuring system (161) from the second training data memory (137) as current parameters of the second neural network model (136) for the second measuring system (161).

[0313] The method comprises, as a second step, loading (192) the previously saved parameters of the fourth neural network model (156) from the fourth training data memory (157) into the fourth neural network model (156).

[0314] The method comprises, as a third step, loading (193) the previously saved parameters of the fifth neural network model (158) from the fifth training data memory (159) into the fifth neural network model (158).

[0315] The method comprises, as a fourth step, using (175) the trained fifth neural network model (158) • for identifying objects and / or object components and / or arrangements of objects and / or object components on the basis of the third object recognition result and the second object recognition result by the fifth neural network model (158) and / or • to generate a joint object recognition hypothesis based on the second object recognition result and the third object recognition result by the fifth neural network model (158). Figures 23 and 24

[0316] The Fig. 23 and Fig. 24 are described in the text above. GlossaryUltrasonic transducers

[0317] Ultrasonic transmitter and receiver combined in one component Neural network / neural network model

[0318] Wikipedia (as of January 18, 2019) defines a neural network as follows: In neuroscience, a neural network is any number of interconnected neurons that, as part of a nervous system, form a network intended to serve a specific function. In computational neuroscience, this term also refers to simplified models of biological networks. In computer science, information technology, and robotics, their structures are modeled as artificial neural networks and are technically reproduced, simulated, and modified. A system inspired by the functioning of biological neurons that serves as the basis for machine learning methods.

[0319] The nervous system of humans and animals consists of nerve cells (neurons) and glial cells, as well as an environment. The neurons are connected to each other via synapses, which can be thought of as connection points or nodes of an interneuronal network. In addition, chemical and electrical exchange takes place between neurons and neuroglia cells, particularly oligodendroglia and astroglia, which can change the weighting of signals. The "circuitry" of neurons typically has several inputs and one output. When the sum of the input signals exceeds a certain threshold, the neuron "fires" (excitation generation): An action potential (AP) is triggered at the axon hillock, formed in the initial segment, and propagated along the axon (excitation conduction). Action potentials in series are the primary output signals of neurons. These signals can be transmitted to other cells via synapses (excitation transmission).At electrical synapses, potential changes are transmitted through direct contact. At chemical synapses, these changes are converted into a transmitter quantum as a secondary signal, i.e., transmitted through neurotransmitters (transmission). In divergence, one neuron transmits signals to several other neurons, and in convergence, one neuron receives signals from several others.

[0320] Nerve cells are characterized by their cell processes, which are used to establish contact with individual other cells. As dendrites, they primarily serve to receive signals from other cells, while signals are transmitted to other cells via the neurite, also called the axon, which is surrounded by glial cells. With branches of its axon acting as axon collaterals, the signal from one neuron can be transmitted efferently to several other neurons (divergence). A neuron can also receive afferent signals from various other neurons (convergence), primarily via its dendrites as inputs. While the action potential transmitted across the membrane at electrical synapses is promptly transmitted as an electrical signal, at chemical synapses it is first converted into a secondary chemical signal at the presynaptic membrane region of a neuron.This occurs as potential-dependent neurocrine secretion through the release (exocytosis) of molecules of a signaling substance stored in synaptic vesicles. After bridging the narrow synaptic cleft by diffusion, this messenger substance acts as a neurotransmitter—or, in addition, as a neuromodulatory cotransmitter—on the membrane region of the postsynaptically associated cell, if it is receptive and equipped with suitable receptor molecules. Upon receptor binding, a transmitter is recognized and, directly (ionotropic) or indirectly (metabotropic), a temporary regional change in membrane permeability is initiated. Small ions flowing in or out through the membrane thus in turn trigger postsynaptic potential changes, as local electrical signals. The incoming signals converge on the membrane of a neuron, where they are spatially and temporally integrated and summed together.Such postsynaptic potentials (PSPs) manifest themselves differently, depending on the membrane's configuration of receptors and ion channels. As graded potentials, they can not only present signals of varying strength, but also be fundamentally different in quality: excitatory (EPSPs) stimulate the formation of an excitation; inhibitory (IPSPs) inhibit the formation of an action potential. With this form of synaptic connection as a means of chemical transmission, signals are given a prefix. Furthermore, they can be weighted at the junction, amplified or attenuated depending on the process. Frequently repeated transmission in rapid succession can lead to longer-lasting changes that strengthen synaptic transmission as long-term potentiation (LTP). At low frequencies, various types of changes can occur that lead to a permanent attenuation as long-term depression (LTD).In this way, the signal transmission process itself can shape or reshape the synaptic mode (neuronal plasticity). The network of neurons therefore does not exhibit rigid wiring, but rather a weighting of the signal pathways that depends on the previous state and changes through repeated use. There are various, now well-standardized theories about learning in neural networks. The first neural learning rule was described by Donald O. Hebb in 1949 (Hebb's learning rule); significant developments followed in the mid-1980s through the work of the Finn Teuvo Kohonen, among others. This resulted in typical properties of neural networks that apply equally to natural and artificial "neural systems". This includes the property that they can learn complex patterns without any abstraction about the rules that may underlie these patterns.This means that neural networks do not follow a logical system, but rather an intuitive pattern processing (see also Artificial Intelligence). This also means that the rules do not have to be developed before learning. On the other hand, any logic that determined the learning success cannot be subsequently determined from the neural network. This, in turn, does not mean that logical behavior and precise rules cannot be learned or applied by neural networks. They simply have to be laboriously acquired through training; for example, when learning the grammar of a language over years. Neural networks do not learn explicitly, but implicitly: A toddler first learns the grammar of their native language implicitly. As a schoolchild, they then generally learn the rules – once again – explicitly, through rules.The following applies specifically to the simulation of artificial neural networks in science and technology using special circuits or devices, or using processors with appropriate programming: The "correct" training of a neural network is a prerequisite for successful learning or for the correct processing of a pattern in a nervous system. Conversely, it is not possible to make a precise prediction about the "correct" interpretation of a pattern by a neural network unless this specific network is applied or calculated with this specific learning experience. Neural networks therefore have the problem that after the learning process, patterns that do not resemble the models implemented in the training set induce stochastic (i.e., seemingly "random") behavior of the output neurons. They do not work exactly, but approximately.It is known that the function of a neural network can also be performed by a Petri net or an HMM recognizer, and that a transformation between these mappings is possible. Therefore, it is expressly claimed that neural networks also include sub-devices with HMM recognizers and / or Petri nets. Deep Learning

[0321] Machine learning method that is based on a neural network model with many intermediate layers and independently learns which representation of the data derived from the input data is necessary to fulfill the given learning task.

[0322] Wikipedia (as of January 18, 2019) defines deep learning as follows: Deep learning refers to a class of optimization methods for artificial neural networks that have numerous hidden layers between the input and output layers, thus exhibiting a comprehensive internal structure. In extensions of learning algorithms for network structures with very few or no hidden layers, such as the single-layer perceptron, deep learning methods enable stable learning success even with numerous hidden layers. According to Jürgen Schmidhuber, "deep learning" is just a new term for artificial neural networks and first appeared in 2000 in the publication "Multi-Valued and Universal Binary Neurons: Theory, Learning and Applications."The problems solved in the early days of artificial intelligence were intellectually difficult for humans but simple for computers. These problems could be described using formal mathematical rules. The real challenge for artificial intelligence, however, lay in solving tasks that are easy for humans to perform but whose solutions are difficult to formulate using mathematical rules. These are tasks that humans solve intuitively, such as speech or facial recognition. A computer-based solution to these types of tasks involves the ability of computers to learn from experience and understand the world in terms of a hierarchy of concepts, where each concept is defined by its relationship to simpler concepts.By gathering knowledge from experience, this approach avoids the need for human operators to formally specify all the knowledge the computer needs to do its job. The hierarchy of concepts allows the computer to learn complicated concepts by composing them from simpler ones. If you draw a diagram showing how these concepts are built on top of each other, the diagram is deep, with many layers. For this reason, this approach is called "deep learning" in artificial intelligence. It is difficult for a computer to understand the meaning of raw sensory input data, such as in handwriting recognition, where text initially exists only as a collection of pixels. Converting a set of pixels into a string of numbers and letters is very complicated. Complex patterns must be extracted from raw data.Learning or evaluating this mapping would seem insurmountably difficult if it were programmed manually. One of the most common techniques in artificial intelligence is machine learning. Machine learning is a self-adaptive algorithm. Deep learning, a subset of machine learning, uses a series of hierarchical layers, or a hierarchy of concepts, to carry out the machine learning process. The artificial neural networks used are built like the human brain, with neurons connected to each other like a web. The first layer of the neural network, the visible input layer, processes raw data input, such as the individual pixels of an image. The data input contains variables that are accessible to observation—hence the "visible layer." This first layer passes its output to the next layer.This second layer processes the information from the previous layer and also passes on the result. The next layer receives the information from the second layer and processes it further. These layers are called hidden layers. The features they contain become increasingly abstract. Their values ​​are not specified in the source data. Instead, the model must determine which concepts are useful for explaining the relationships in the observed data. This continues across all levels of the artificial neural network. The result is output in the visible final layer (output layer). This breaks down the desired complex data processing into a series of nested simple mappings, each described by a different layer of the model. The sensory processing technique presented here goes far beyond these approaches.In particular, it represents the first self-learning network of multiple neural networks that is self-learning at every level of the chain. Each neural network in the network of neural networks represents a node of this network. Thus, each node of a neural network is itself a neural network. List of reference symbols 9 User control display; 10 User control; 11 Ultrasonic transducer arrangement; 12 control electronics; 13 Transmitter control; 14 setting memories; 15 signal processor; 16 first neural network model in a first memory; 17 first training data storage; 18 image processor; 19 advertisement; 21 Installation of the first neural network model (16) for the ultrasonic measuring system (164) on the computer system; 22 Transfer of echo signals from different objects in different positions to the first neural network model for the ultrasonic measuring system (164); 23 Identification of objects for the first neural network model for the ultrasound measurement system (164); 24 Transferring live ultrasound signals to the trained first neural network model for the ultrasound measuring system in the vehicle and performing an object identification by the trained first neural network model to generate a first object recognition result; 25 Using the first object identification performed by the first neural network model for the ultrasonic measuring system in the form of a first object recognition result; 35 Saving the parameter data of the trained first neural network model (16) for the ultrasound measuring system in a first training data memory (17); 41 Loading the previously saved parameters of the first neural network model (16) for the ultrasound measuring system from the first training data memory (17) as current parameters of the first neural network model (16); 51 Installation of the second neural network model (136) for the second measuring system (161) on the computer system; 52 Transferring sensor signals from different objects in different positions to the second neural network model (136) for the second measuring system (161); 53 Identification of the objects for the second neural network model (136) for the second measuring system (161); 54 Transferring live sensor signals from the second sensor arrangement (131) to the trained second neural network model (136) for the second measuring system (161) and performing an object identification by the trained second neural network model (136) to generate a second object recognition result; 55 Using the second object identification performed by the second neural network model (136) for the second measuring system (161) in the form of a second object recognition result; 61 Installation of the fourth neural network model (156) for the third measuring system (162) on the computer system; 62 Transfer of sensor signals from different objects in different positions to the fourth neural network model (156) for the third measuring system (162); 63 Identification of the objects for the fourth neural network model (156) for the third measurement system (162); 64 Transferring live sensor signals from the third sensor arrangement (151) to the trained fourth neural network model (156) for the third measuring system (162) and performing an object identification by the trained fourth neural network model (156) to generate a third object recognition result; 65 Saving the parameter data of the trained second neural network model (136) for the second measuring system (161) in a second training data memory (137); 65b Using the third object identification performed by the fourth neural network model (1356) for the third measuring system (162) in the form of a third object recognition result; 71 Loading the previously saved parameters of the second neural network model (136) for the second measuring system (161) from the second training data memory (137) as current parameters of the second neural network model (136) for the second measuring system (161); 72 Loading the previously saved parameters of the fourth neural network model (156) for the third measuring system (162) from the fourth 84 training data storage (157) as current parameters of the fourth neural network model (156) for the third measuring system (162); comparing the first object recognition results of the first object identification performed by the first neural network model (16) for the ultrasound measuring system (164) with the alternative second object recognition results of the alternative second object identification performed by the second neural network model (136) for the second measuring system (161), and determining a common object recognition hypothesis; 85 Use of the joint object recognition hypothesis; 86 first parameter adaptation of the first neural network model (16) for the ultrasonic measuring system (164) depending on the comparison result of the first object recognition results of the first object identification carried out by the first neural network model (16) for the ultrasonic measuring system (164) with the alternative second object recognition results of the alternative second object identification carried out by the second neural network model (136) for the second measuring system (161) in step 84; 87 second parameter adaptation of the second neural network model (136) for the second measuring system (161) depending on the comparison result of the first object recognition results of the first object identification carried out by the first neural network model (16) for the ultrasonic measuring system (164) with the alternative second object recognition results of the alternative second object identification carried out by the second neural network model (136) for the second measuring system (161) in step 84; 88 fourth parameter adaptation of the fourth neural network model (156) for the third measuring system (162) depending on the comparison result of the second object recognition results of the second object identification carried out by the second neural network model (136) for the second measuring system (161) with the third object recognition results of the alternative third object identification carried out by the fourth neural network model (156) for the third measuring system (162) in step 204; 91 Installation of the third neural network model (138) for comparing the first object recognition results of the first object identification carried out by the first neural network model (16) for the ultrasonic measuring system (164) with the alternative second object recognition results of the alternative second object identification carried out by the second neural network model (136) for the second measuring system (161) and for determining a common object recognition hypothesis on the computer system; 92 Training the third neural network model (138) by transferring correct and incorrect first object recognition results and correct and incorrect alternative second object recognition results to the third neural network model (138) together with data on their correctness and / or incorrectness and determining the parameters of the third neural network model (138) depending on these data and / or changing the parameters of the third neural network model (138) depending on these data; 93 Transfer of first live object detection results and alternative second live object detection results to the trained third neural network model (138); 94 Determination of a joint object recognition hypothesis based on the first live object recognition results and the alternative second live object recognition results by the third neural network model (138); 95 Use of the common object recognition hypothesis determined by the third neural network model (138) for the entire measurement system; 105 Saving the parameter data of the trained third neural network (138) in a third training data storage (139); 111 Loading the previously saved parameters of the third neural network model (138) from the third training data memory (139) into the third neural network model (138); 130 second user control, which may be identical to the first user control (10) when operated in time division multiplex; 131 second sensor arrangement; 132 second control electronics; 133 second sensor control; 134 second setting memory; 135 second signal processor, which may be identical to the first signal processor (15) if the task is solved in time division multiplex; 136 second neural network model in a second memory; 137 second training data storage; 138 third neural network model; 139 third training data storage; 140 first parameter modification device for the first neural network model (16); 140a second user control, which may be identical to the first user control (10) and / or the third user control (160) when operated in time division multiplex; 141 second parameter modification device for the second neural network model (136); 142 first safety device; 143 second safety device; 144 fourth parameter modification device for the fourth neural network model (156); 145 third user control; 151 third sensor arrangement; 152 third control electronics; 153 third sensor control; 154 third setting memory; 155 third signal processor, which can be identical to the first signal processor (13) and / or second signal processor (135) if the task is solved in time division multiplex; 156 fourth neural network model; 157 fourth training data storage; 158 fifth neural network model; 159 fifth training data storage; 160 third user control, which may be identical to the first user control (10) and / or the second user control (140a) when operated in time division multiplex; 161 second measuring system; 162 third measuring system; 163 fourth parameter modification device for the fourth neural network model (156); 164 Ultrasonic measuring system; 171 Conduct of a procedure under Fig. 7 for the second measuring system (161) with determination of the second sensor measured values ​​and determination of a second object recognition result; 172 Conduct of a procedure under Fig. 7 for the third measuring system (162) with determination of the third sensor measured values ​​and determination of a third object recognition result; 173 Installing a fifth neural network model (158) on a computer-based system; 174 Training the fifth neural network model (158) to identify the objects and / or object components and / or arrangements of objects and / or object components in the second object recognition result and / or in the third object recognition result and generating a joint object recognition hypothesis; 175 Using the trained fifth neural network model (158) to identify objects and / or object components and / or arrangements of objects and / or object components based on the second object recognition result and the third object recognition result by the fifth neural network model (158) and / or to generate a joint object recognition hypothesis based on the second object recognition result and the third object recognition result by the fifth neural network model (158); 185 Saving the parameter data of the trained fifth neural network (158) in a fifth training data storage (159); 192 Loading the previously saved parameters of the fourth neural network model (156) from the fourth training data memory (157) into the fourth neural network model (156); 193 Loading the previously saved parameters of the fifth neural network model (158) from the fifth training data storage (159) into the fifth neural network model (158); 201 Installation of the fifth neural network model (158) for comparing the second object recognition results of the first object identification carried out by the second neural network model (136) for the second measuring system (161) with the third object recognition results of the alternative third object identification carried out by the fourth neural network model (1356) for the third measuring system (162) and for determining a common object recognition hypothesis on the computer system; 202 Training the fifth neural network model (158) by passing correct and incorrect second object detection results 203 and correct and incorrect third object recognition results to the fifth neural network model (158) together with data on their correctness and / or incorrectness, and determining the parameters of the fifth neural network model (158) depending on these data and / or changing the parameters of the fifth neural network model (158) depending on these data; transferring second live object recognition results and third live object recognition results to the trained fifth neural network model (158); 204 Determining a joint object detection hypothesis based on the second live object detection results and the third live object detection results by the fifth neural network model (158); 205 Use of the common object recognition hypothesis determined by the fifth neural network model (158) for the entire measurement system; 211 Loading (211) the previously saved parameters of the fifth neural network model (158) from the fifth training data storage (159) or another data storage that currently holds this data into the fifth neural network model (158). This fifth neural network model (158) can be located in a completely different, functionally similar overall system than the overall system with which the data to be loaded was created; 215 Saving (215) the parameter data of the trained fifth neural network (158) in a fifth training data memory (159). The parameters of the fifth neural network model (158) thus saved can thereby be used in functionally similar systems by programming and / or constructing these systems or parts thereof; 1010 vehicle part; 1012 ultrasonic transducer assembly; 1014 microbeam formers; 1016 transmit / receive switch; 1018 transmission control; 1020 main beam former; 1026 signal processor; 1028 image memory; 1030 image processor; 1032 scanning converters; 1034 training image memory; 1036 graphics processor; 1038 User control panel or user control panel; 1040 image display; 1042 volume renderers; 1044 Multiplanar Reformator; 1046 touchscreen display control; 1048 User Control Setting Control; 1052 video processor; 1054 Doppler processor; 1060 touchscreen display; 1080 neural network model; 1090 Installation of the neural network model; 1092 Presenting images and / or measurement data sets of the same vehicle environment situation geometry, preferably obtained from a large number of test drives for similar situations, to the neural network model; 1094 Training the neural network model to train it to form the geometry by presenting the known type of situation geometry and its view in the images and / or measurement data sets together with the presentation in step (1092). 1096 Presentation of environment maps and / or measurement data sets recorded by the ultrasound system to the neural network model for identifying the vehicle environment situation geometry as an object recognition result; 1098 Use of the identification generated by the neural network model. H11 high priority signal of the ultrasonic transducer arrangement (11) to the first safety device (142); H15 high priority signal from the first signal processor (15) to the first safety device (142) H131 high priority signal of the second sensor arrangement (131) to the first safety device (142); H135 high priority signal from the second signal processor (135) to the first safety device (142); H142 User input or other external high priority signal H151 high priority signal of the third sensor arrangement (151) to the second safety device (143); H155 high priority signal from the third signal processor (155) to the second safety device (143); List of cited writings DE 10 2008 036 009 B4 DE 103 36 638 A1 [Dror 1995] IE Dror, M. Zagaeski, CF Moss, “Three-dimensional target recognition via sonar: a neural network model,” Neural Networks, vol. 8, no. 1, pp. 149-160, 1995 [LeCun 2015] Y. LeCun, Y. Begio, G. Hinton, “Deep Learning,” Nature, vol. 521, pp.436-444, 2015 [Zhang 2018] W. J. Zhang, G. Yang, Y. Lin, C. Ji and M. M. Gupta, „On Definition of Deep Learning,“ 2018 World Automation Congress (WAC), Stevenson, WA, 2018, pp. 1-5, 2018

Claims

[1] A self-learning safety-relevant measuring system for use in vehicles and / or motor vehicles characterized by , - that it comprises a first safety device (142) and one or more neural network models (16, 136, 138) for evaluating the measurement data of at least one sensor arrangement (11, 131) of the measuring system and - that the first safety device (142) monitors at least the neural network model (138) which processes these measurement data or data derived therefrom, and - that the first safety device (142) can influence the parameters of this neural network model (138), - in case of undesired detection results - to retrain this neural network model (138) during operation, - whereby undesired recognition results within the meaning of this claim - incorrect and security-relevant detection results and / or - potentially erroneous detection results and / or - incorrect recognition results and / or - potentially security-relevant detection results. [2] Measuring system according to the preceding claim characterized by , - that the safety device can be influenced by signals (H11, H131) of the at least one sensor arrangement (11, 131) of the measuring system. [3] Measuring system according to one or more of the preceding claims characterized by , - that the safety device (142) can be influenced by a user signal (H142) or an external control signal (H142). [4] Measuring system according to one or more of the preceding claims characterized by , - that the at least one neural network model (138) generates an object recognition hypothesis and - that the safety device (142) can reliably prevent safety-endangering and / or potentially safety-endangering signaling of the at least one neural network model (138) by overwriting the object recognition hypothesis of the at least one neural network model (138).

Images