Control procedure, control unit, environmental sensing system and mobile and / or stationary equipment

Abstract

Control procedure (S) for an environment detection system (10) for a vehicle as a mobile installation (1), comprising the steps: - optical detection (S1) of at least one scene (50S) from an environment (50) of the facility (1), - Evaluating (S2) the recorded scene (50S) and deriving (S3) a driving situation as an operating situation of the facility (1), - intervention (S4) in the operation of the device (1) depending on the operating situation and thereby adjusting (S5) the movement or driving behavior of the device (1), at which: - the evaluation (S2) is based on a physically motivated multiple-path decomposition of machine vision in a plurality of independent evaluation processes (S2-1, S2-2, S2-3) and - the independent assessment processes (S2-1, S2-2, S2-3) each have at least one path - for appearance-based object detection and classification (S2-1), and - for a flow-based motion analysis (S2-2), and - for a multi-view-based, three-dimensional scene reconstruction (S2-3) include, characterized by the fact that The evaluation (S2) includes a downstream neural network (S2-5), a DNN or a layer thereof.

Description

State of the art

[0001] The present invention relates to a control method, in particular for an environment detection system, a control unit for an environment detection system, an environment detection system as such and a mobile and / or stationary device and in particular a vehicle or motor vehicle.

[0002] In mobile and / or stationary systems, especially vehicles, environment detection systems as well as operating or driver assistance systems are increasingly being used to support the operator, and especially the driver, in operating and driving the respective system. One objective here is the increasing focus on so-called automated driving.

[0003] The prerequisite for this is that the respective controlling intervention in the operation of the respective facility can be based on data that are characteristic of the scene in the vicinity of the facility and of its temporal changes.

[0004] Due to the complexity of the situation to be assessed, conventional methods for environmental interpretation and / or evaluation and scene recognition are insufficient in terms of their reliability compared to the effort involved, even when complex computer vision (CV) processes and devices are used, for example using so-called machine learning or deep learning processes and neural networks.

[0005] The document US 2010 / 0315505 A1 discloses a method for detecting the movement of dynamic objects.

[0006] The document WO 2016 / 156236 A1 discloses a method for determining a future position of a moving object in a detected vehicle environment Disclosure of the invention

[0007] The control method according to the invention for an environment detection system and in particular for an operating and / or driver assistance system with the features of claim 1 has the advantage that a much higher degree of reliability is achieved with a comparatively manageable effort when evaluating scenes of an environment.This is achieved according to the invention with the features of claim 1 in that a control method for an environment detection system and in particular for an operating assistance system and / or for a driver assistance system for a mobile and / or stationary device and in particular for a vehicle is created, which control method comprises the steps of (i) optically detecting at least one scene from an environment of the device, (ii) evaluating the detected scene and deriving therefrom an operating and in particular driving situation of the device and - depending on the operating situation - (iii) controlled intervention in the operation of the device and thereby adapting the operation and in particular the movement or driving behavior of the device. According to the invention, the evaluation is carried out on the basis of a physically motivated multi-path decomposition of machine vision in a plurality of mutually independent evaluation processes.Furthermore, the independent evaluation processes of the evaluation each include a path at least for an appearance-based object detection and classification and for a flow-based motion analysis.

[0008] The subclaims show preferred developments of the invention.

[0009] The present invention is applicable in the field of environment detection systems, but also in connection with mobile units or devices, for example in vehicles, and in these cases with so-called driver assistance systems or operational assistance systems. However, the present invention is also particularly applicable in the field of stationary units or devices, and can also serve there for environment detection and / or operational assistance. In this respect, the terms "environment detection system," "operational assistance system," and "driver assistance system," on the one hand, and "mobile device or unit," "stationary device or unit," on the other, are always meant together, and the respective technical aspects are always applicable to all systems and devices.

[0010] In connection with the multi-path structure, the sub-processes of evaluating the captured scene can be executed in parallel and / or simultaneously.

[0011] The reliability of the procedure according to the invention can be further increased by, according to an advantageous development of the control method according to the invention, the independent evaluation processes comprise a path for a multi-view-based, three-dimensional scene reconstruction, in particular using stereo views and / or motion-based scene reconstructions.

[0012] In another development of the control method according to the invention, the evaluation comprises a downstream process and in particular a downstream neural network, a DNN and / or a layer thereof.

[0013] The downstream process, the downstream neural network, the DNN and / or the layer can advantageously be configured for three-dimensional scene reconstruction.

[0014] Additionally or alternatively, the downstream process, and in particular the downstream neural network, the DNN, and / or the processing layer, can be configured based on aspects of three-dimensional Euclidean geometry relative to two-dimensional projective geometry and / or based on a sensor data fusion process, in particular by utilizing sensor data sources from optical sensors, in particular from a camera arrangement with one or more cameras, from lidar sensors, from radar sensors, from infrared sensors, and / or from acoustic sensors, in particular from ultrasonic sensors. As a further additional or alternative source of information, environmental data can also be obtained from digital maps and / or other digital data sources.

[0015] In another additional or alternative embodiment of the control method according to the invention, it is provided that the evaluation comprises an upstream process and in particular an upstream neural network, a DNN and / or a layer thereof, which are set up in particular for the semantic segmentation of the captured scene and / or for the pre-control or control of the multiple paths of the evaluation and in particular of the evaluation processes.

[0016] It can further be provided that the upstream process and in particular the upstream neural network, the DNN and / or the layer are set up to detect and evaluate periodic structures of a scene and in particular periodically arranged objects and / or static, moving and / or potentially moving structures of a scene and in particular static, moving and / or potentially moving objects, in particular to distinguish them from one another.

[0017] In another advantageous embodiment of the control method according to the invention, it is provided that the process of appearance-based object detection and classification is carried out on the basis of a machine learning method, in conjunction with a training method and / or with the inclusion of a feedforward control by the upstream and / or downstream process, the respective neural network, the DNN and / or the layer.

[0018] Furthermore, the present invention relates to a control unit for an environment detection system, in particular for an operating and / or driving assistance system for a mobile and / or stationary device, and in particular for a vehicle. According to the present invention, the control unit according to the invention is configured to execute an embodiment of the control method according to the invention for an environment detection system, operating and / or driving assistance system.

[0019] The control unit can be designed partially or completely as a computer implementation, in particular as a computer-aided method or computer program.

[0020] Alternatively or additionally, the control unit may be partially or completely formed by hardware components, in particular by ASSPs and / or ASICs.

[0021] When implementing the method in hardware, reliability can be further increased by redundantly implementing the individual paths. It is particularly advantageous if the redundant parts of the implementation are designed specifically for each of the individual paths to keep the overall complexity of the redundant implementation as low as possible.

[0022] According to a further aspect of the present invention, an environment detection system, operating and / or driver assistance system for a mobile and / or stationary device, and in particular for a vehicle, is also provided. The environment detection system is designed with an embodiment of the control unit according to the invention.

[0023] Furthermore, the present invention relates to a mobile and / or stationary device, and in particular a vehicle or motor vehicle. This device has, for example, a drive and / or steering system and is equipped with an environment detection system designed according to the invention, which serves to control the drive, brake, and / or steering system. Short description of the characters

[0024] Embodiments of the invention are described in detail with reference to the accompanying figures. Fig. 1A shows schematically in the manner of a plan view an embodiment of the vehicle according to the invention, in which an embodiment of the control method according to the invention can be used in connection with an environment detection system according to the invention and its control unit. Fig. 1B to 4 schematically show embodiments of the control method according to the invention. Fig. 5 to 8 show views of scenes from a vehicle environment in which various evaluation criteria are used, which can be the result of an embodiment of the control method according to the invention. Fig. 9 shows schematically the structure of a neural network which can be used in an embodiment of the control method according to the invention. Fig. 10 schematically shows a conventional control method for an environment detection system. Preferred embodiments of the invention

[0025] The following are based on the Fig. 1A to 10, exemplary embodiments of the invention and the technical background are described in detail. Identical and equivalent elements and components, as well as those with identical or equivalent functions, are designated by the same reference numerals. The detailed description of the designated elements and components is not reproduced in every case where they appear.

[0026] The features and other properties presented can be isolated from one another in any form and combined with one another in any way without departing from the essence of the invention.

[0027] Before going into details of the present invention, for comparison purposes, based on Fig. 10 schematically explains the structure of the conventional approach for a conventional environment detection system 30'. As already explained above, an operational assistance system and / or a driver assistance system can also be considered instead of an environment detection system.

[0028] In Fig. 10 schematically shows a conventional vehicle 1' with drive 2 and steering 3. Using a conventional imaging process, for example, a conventional camera arrangement 20, the environment, i.e., the surroundings of the vehicle 1', are optically captured in step T1 and represented in a time-discrete manner as a sequence of images, i.e., as a sequence of two-dimensional projections.

[0029] In the conventional embodiment according to Fig. 10, the evaluation T2 of the scene 50S and the derivation T3 of a driving situation of the vehicle 1' - or generally of an operating situation - can be carried out in connection with a DNN (deep neural network) and corresponding learning methods with an end-to-end structure.

[0030] The aim is to ultimately plan and effect an intervention T4 in the operation of the vehicle 1' - or generally of the mobile device - on the basis of the evaluation T2 and the derivation T3, in order to thereby achieve an adaptation T5 of the operating behavior, i.e. in particular the movement behavior and / or driving behavior of the vehicle 1'.

[0031] Due to the complexity of real-life situations in scene 50S in the environment of vehicle 1', incorrect or omitted interventions in the driving operation of vehicle 1' often occur due to the limited training framework.

[0032] According to the invention, this is to be prevented by the measures described above and below.

[0033] Fig. 1A shows schematically in the manner of a plan view an embodiment of the vehicle 1 according to the invention, in which an embodiment of the control method S according to the invention can be used in connection with an environment detection system 10 according to the invention or driver assistance system 10 and its control unit 30.

[0034] Although the present invention can be viewed in connection with any other mobile and / or stationary devices 1, for example, in connection with general work machines, robots, transport vehicles, multi-camera systems for environmental monitoring, or the like, the application in a vehicle 1, and in particular in a motor vehicle, is described in detail here by way of example. However, this example is not intended to be limiting.

[0035] The Fig. The vehicle 1, schematically illustrated in FIG. 1A, constructed according to the invention, has wheels 4 that can be driven by a drive 2 and steered via a steering system 3. For the—optionally additional—control of the drive 2 of the steering system 3, an environment detection system or driver assistance system 10 is provided with a control unit 30 with a structure according to the invention. The control unit 30 is operatively connected via a control and detection line 31 to the drive 2, the steering system 3, and a camera arrangement 20 with a first camera 21 and a second camera 22.

[0036] In addition or as an alternative to the camera arrangement 20, other sensor systems can also be used which are useful for detecting objects 53 in a scene 50S in the environment 50. In the Fig. In the situation illustrated in Figure 1A, the surroundings 50 are partially covered by the field of view 51 of the first camera 21 and the field of view 52 of the second camera 22, so that the object 53 in the surroundings 50 is stereoscopically captured by the camera arrangement 20. Due to the operation of the camera arrangement 20, the time-continuous three-dimensional scene 50S in the surroundings 50 is mapped onto a sequence of time-discrete two-dimensional images and fed via the control and detection line 31 to the control unit 30 of the surroundings detection system or driver assistance system 10 for evaluation.

[0037] The evaluation result is output by the control unit 30 via the control and detection line in the form of control signals to the drive 2 and / or the steering 3 in order to influence the operation of the vehicle 1, for example by adjusting the drive and / or the steering.

[0038] Fig. 1B to 4 schematically show embodiments of the control method S according to the invention.

[0039] This is Fig. 1B is constructed in the manner of a flow chart and shows step S1 of optically and / or visually capturing the scene 50S in the environment 50, step S2 of evaluating the scene 50S, step S3 of deriving a driving situation based on the evaluation of the scene 50S, step S4 of intervening in the operation of the vehicle 1, for example by generating and transmitting control parameters to the drive 2, the brake and / or to the steering 3, and step S5 of adapting the operating behavior and in particular the movement behavior and / or driving behavior of the vehicle 1.

[0040] The Fig. The process S described in 1B can be processed as a sub-process, for example in connection with a control loop.

[0041] Fig. 2 shows a schematic view of another embodiment of the control method S according to the invention, in which the segmentation or decomposition of step S2 of evaluating a scene 50S in the environment 50 is already taken into account. This step S2 is carried out in conjunction with the control unit 30 according to the present invention, whereas step S1 of optically and / or visually capturing the scene 50S is carried out on the basis of the camera arrangement 20, and steps S4 of intervening in the operation and S5 of adapting the operation of the vehicle 1 in conjunction with the drive 2 of the steering system 3 are implemented. Step S3 of deriving an operating situation or a driving situation can also be carried out in conjunction with the control unit 30 according to the present invention.

[0042] Fig. 3 shows in detail the decomposition, decomposition or segmentation of the steps S2 of evaluating the scene 50S and S3 of deriving the operating situation.

[0043] In the foregoing, the real scene 50S in the environment 50 is initially represented by S0 as a three-dimensional and time-continuous structure. This is captured by a first sub-step S1-1 of optical and / or visual detection S1 of the scene 50S by sensor means, in particular by the camera arrangement 20, and is represented in the second sub-step S1-2 as a two-dimensional and time-discrete structure and fed to the input side of the evaluation step S2.

[0044] The complexity of the evaluation S2 is divided by decomposition into less complex and physically motivated subprocesses, here S2-1 of appearance-based object detection and classification, S2-2 of flow-based motion analysis, namely based on optical flow, and S2-3 of multi-view-based and three-dimensional scene reconstruction. These subprocesses S2-1 to S2-3 are supplemented by an input-side upstream process S2-4 based on an input-side neural network and by an output-side downstream process S2-5, also based on an output-side neural network.

[0045] The input-side neural network S2-4 carries out a pre-control of the multiple paths with the processes S2-1 to S2-3, in particular with regard to the classification of the objects and their movement, in particular in distinction to periodic structures or periodically arranged objects in the scene 50S.

[0046] The output neural network S2-5, in cooperation with the third evaluation process S2-3 of the multi-view-based and three-dimensional scene reconstruction, maps the image onto a three-dimensional representation, whereby, on the basis of physical and geometric principles, the information lost during the projection from the three-dimensional and time-continuous representation to the two-dimensional and time-discrete representation is reconstructed or supplemented.

[0047] In sub-step S2-6, the three-dimensional and time-discrete reconstructed scene representation is then available, on the basis of which - with the aid of further data - a driving situation is derived in step S3, with the subsequent intervention S4 in the operation and adaptation S5 of the operation of the vehicle 1 as a mobile device.

[0048] In Fig. 3, the input-side neural network S2-4 is shown in dashed lines; it therefore represents only one option for a preferred embodiment of the method S according to the invention and the corresponding environment detection system or driver assistance system 10 according to the invention.

[0049] In the embodiment according to Fig. 4, in the embodiment of the method S according to the invention, only one output-side neural network S2-5 is provided. Furthermore, the evaluation step S2 is decomposed into only two substeps, namely the first evaluation process S2-1 of appearance-based object detection and classification and the second evaluation process S2-2 of flow-based motion analysis.

[0050] The embodiment of the control method S according to the invention according to Fig. 4 can be advantageously implemented with reduced effort. As a result, for example, in connection with the intervention S4 in the operation and the adjustment S5 of the operation of vehicle 1, the possibility arises of issuing a warning S5-1 to the driver of vehicle 1 and / or issuing S5-2 a trigger pulse to apply the brake for emergency braking, whereby the brake can be considered part of drive 2.

[0051] Fig. 5 to 8 show views of scenes 50S from a vehicle environment 50, in which various evaluation criteria are used, which can be the result of an embodiment of the control method according to the invention.

[0052] In these figures, a person is included as object 53 in the environment 50 of the vehicle 1 and is in the fields of view 51 and 52 of the cameras 21 and 22 of the camera arrangement 20 according to Fig. 1 recorded.

[0053] In connection with the control method S according to the invention for the environment detection system or driver assistance system 10 for the vehicle 1, understood as a mobile device, initially according to Fig. 5 a two-dimensional image from scene 50S in environment 50.

[0054] In connection with Fig. 6, after evaluation by the control method S according to the invention for an environment detection system or driver assistance system 10, an object classification and detection using a so-called bounding box 54 results, namely in connection with a typical object recognition with a trained object detector, which, in connection with the present invention, implements the path for the appearance-based object detection and classification S2-1. Based on such detection and classification, a corresponding emergency braking action can then be generated and initiated.

[0055] Fig. 7 shows the process for a typical false detection as it occurs in the prior art, namely in connection with a bounding box 55 for the mirror image on a wet road surface in the vicinity of the vehicle 1.

[0056] According to the invention, such false detections can be avoided if a three-dimensional reconstruction of the situation is used on the basis of physical and geometric principles, as is the case in connection with Fig. 8, in connection with a three-dimensional construct assigned to the object 53, namely the person, which is spatially and temporally assigned in the object 53 and prevents, according to the invention, false detection in connection with reflections on the road surface, etc.

[0057] Fig. 9 shows schematically the structure of a neural network which can be used in an embodiment of the control method according to the invention.

[0058] These and other features and characteristics of the present invention are further explained with reference to the following statements: Video-based systems, e.g., for operational assistance or driver assistance systems 10 for vehicles 1, can be used in a variety of applications for sensing and interpreting the surroundings 50. For example, video systems in the motor vehicle 1 can be used to implement driver assistance functions such as lane departure warning or traffic sign recognition.

[0059] The use of complex sensor fusion systems is necessary to increase the reliability of the overall system and to reduce the security effort required to release the system by considering largely independent individual sensor paths.

[0060] The concept of video-based environment interpretation requires detection and classification of the objects 53 relevant for the respective application, such as lane lines, vehicles or pedestrians.

[0061] For this task, trained classifiers can be used, which are based on predefined or fully trained image features using deep learning (DL).

[0062] In the area of ​​deep learning methods, implementations based on convolutional neural networks (CNN) are particularly noteworthy, as they form the basis for a variety of tasks in the field of machine vision, which can also be referred to as computer vision (CV).

[0063] In addition to the performance of CNN-based methods, the power of the corresponding development tools and the comparatively easy portability to graphics cards (GPU) bring further advantages for the video system architecture.

[0064] The present invention demonstrates improvements regarding deficiencies of such methods in DL and CNN as end-to-end methods for safety-critical applications and proposes, among other things, a multi-path concept in machine vision and as a viable CV architecture.

[0065] One aspect of the present invention is alternatively or additionally based on a simplified stochastic model for video-based environmental perception in complex, real-world scenarios.

[0066] The viability of a video-based system concept to solve this task as well as the practical feasibility of the concept are primarily determined by the mastery of so-called core events, which are also referred to as corner cases.

[0067] These are rare, safety-critical events that the video system must detect and reliably resolve with a very high detection probability. The detection probability depends on the criticality, i.e., the hazard potential of the respective function under consideration. It is defined as part of the functional safety analysis for motor vehicles, e.g., according to ISO 26262.

[0068] Fig. 10 shows a video system related to the conventional control T with end-to-end architecture (end-to-end: E2E), which realizes a function F based on the input video data D by a large, deep neural network or DNN.

[0069] The E2E-DNN forms a black box, which is adapted to the respective task based on training data.

[0070] The internal representation of the data as well as the semantic meaning of the inputs and outputs of the DNN layers are not explicitly defined, but result from the data-driven training process of the network.

[0071] If one considers a safety-critical function, such as automatic emergency braking, based on a video system, the basic system requirement is to detect each object relevant to the function in question, e.g. pedestrians, vehicles, obstacles, etc., sufficiently early and at the same time not to cause or allow any false triggering of the system.

[0072] From the point of view of signal theory, this requires a detector system with false alarm probability P f = 0 and object detection probability P d = 1.

[0073] Such a system is not physically feasible, so that in practice one has to weigh up the system benefit for operating points (Pf, Pd) with P f = ε = 10 -n ≈ 0 and with P d = 1-δ ≈ 1 must be met.

[0074] The proof that a video system achieves such an operating point under real, complex operating conditions requires the recording and analysis of practically all function-relevant environmental scenarios, i.e., among other things, the local constellation of static and dynamic objects as well as global influencing factors such as lighting conditions, weather conditions, degree of contamination of the camera optics, etc.

[0075] From the perspective of probability theory, the core events already mentioned represent rare events, i.e. compound events which are defined by the simultaneous occurrence of various, stochastically independent elementary events which, individually or in combination, can lead to critical system behavior with regard to the function under consideration.

[0076] For such a core event as a composite event, the probability P c their occurrence the connection: Pc=Prob{ω1,ω2,…,ωN}=∏iNProb{ωi}.

[0077] Given that the elementary events considered here, e.g. a collision-relevant object constellation, a non-cooperative texture of the objects, a camera orientation with direct glare from extraneous light, a structured contamination of the optics, etc., have comparatively low probabilities of occurrence, the joint probability P c exponentially towards zero, as the following estimate shows: Prob{ωi}<10−M⇒Pc<10−N⋅M→0.

[0078] This means that the P c reciprocal proportional data volume for training an E2E system grows exponentially with exponent N·M, which cannot generally be represented under real boundary conditions and permissible assumptions.

[0079] A core aspect of the present invention is alternatively or additionally to break down an existing and given complex task in the environment interpretation into manageable subtasks or subsystems by means of a specific, physically motivated system decomposition or decomposition, thereby bringing about a reduction which is characterized by a significantly lower complexity of the respective subsystem compared to the overall system and by a clear physically and / or semantically defined subfunction.

[0080] The present invention proposes, among other things, a basic video system concept which, according to the above considerations, is necessary for the realization of future cognitive systems in safety-critical applications, such as automated driving, service robotics or security and surveillance technology.

[0081] Fig. 2 shows a video system constructed according to the invention with explicit decomposition of the scene 50S, which realizes a function F on the basis of the input video data D.

[0082] The present invention comprises the CV multi-path concept (1) according to Fig. , which essentially represents a physically motivated decomposition of the video system. The video-based acquisition S1 for representing the environment 50 is associated with a loss of information due to the projection of the three-dimensional world onto a sequence of two-dimensional images. Even in the case of an ideal perspective mapping without taking into account real model deviations of the optics / imager system, the representation of the Euclidean three-dimensional environment by a sequence of two-dimensional projective images is associated with the loss of fundamental invariants, since distances, distance ratios, angles, and areas are not preserved in the mapping. The temporal / dynamic scene content orThe corresponding motion information is generally not completely preserved by the temporal discretization of the 2-dimensional image sequence, since dynamic scene contents can only be reconstructed without errors if the sampling theorem is observed by a sufficiently high frame rate.

[0083] These fundamental relationships motivate the physically based multi-path concept for CV systems proposed here. This concept provides specific CV paths for the most complete reconstruction of environmental information.

[0084] The path S2-2 represents an optical flow based motion analysis and addresses the reconstruction of the camera's own motion as well as the relative movements of the dynamic objects in the considered scene 50S.

[0085] The path S2-3 represents a three-dimensional scene reconstruction based on a plurality of viewpoints or views (multi view based 3D scene reconstruction) and reconstructs the 3D structure of the viewed scene 50S using structure-from-motion or stereo video methods.

[0086] It is crucial that both path S2-2 of motion analysis and path S2-3 of scene reconstruction can be realized independently of training data and solely on the basis of physically sound model assumptions, which results in the potential for generic object recognition.

[0087] The path S2-1 represents appearance-based object detection and classification, is generally implemented on the basis of training methods from the field of machine learning, and reconstructs a substantial part of the semantics of the observed scene 50S by detecting objects 53 and assigning them to specific object classes.

[0088] The layers S2-4 and S2-5, which are implemented using DNN, have a significant influence on the performance and robustness of the CV system.

[0089] The so-called first, input-side or bottom-up DNN S2-4 realizes a semantic segmentation of the observed scene 50S, which can be used for the initialization and specific control of the CV paths S2-1 to S2-3.

[0090] An advantageous feature is the pixel-by-pixel marking or labeling of image information, which can be used directly to feed forward the CV classification path S2-1 for object recognition as well as for model adaptation of the generic paths S2-2 and S2-3.

[0091] The second, output-side or top-down DNN S2-5 realizes a data fusion of the CV paths S2-1 to S2-3, which in particular achieves a cross-plausibility check of the individual paths so that only consistent, fused data reaches the top-down feedback channel.

[0092] The proposed CV system concept leads to a decomposition of the core events introduced above.

[0093] If we assume, for the sake of simplicity, that the critical elementary events under consideration can be divided into two categories, where the first category K1 contains critical elementary events for the consistent scene reconstruction and the second category K2 contains critical elementary events for the function to be realized, this leads to a decomposition of the form: Pc=Prob{ω1,ω2,…,ωN}=∏iNProb{ωi}=∏i∈K1NProb{ωi}×∏i∈K2NProb{ωi}=Pc1×Pc2.

[0094] This means that the effort to secure and validate the function can be significantly reduced, since the reciprocal probabilities P c1 , P c2 of occurrence with |K1|+|K2|=N and Pci<10−|Ki|M are significantly larger than P c . In the case |K1|=|K2|=N / 2 results in the limiting case of equally probable elementary events Pc1=Pc2=√Pc.

[0095] Assuming that the hedging effort for the considered function is inversely proportional to P c or P ci is, this leads for example for 1 / P c = 10 6 h to a practically manageable order of magnitude of 1 / P ci = 10 3 h to secure the structured CV system.

[0096] This basic concept of system decomposition can be directly generalized to further levels of decomposition, so that even complex functions can be managed through sufficiently deep modularization.

[0097] However, it should be noted that the fulfillment of the assumptions made for stochastic modeling must also be demonstrated on the basis of empirical test data, so that a limited decomposition depth should be aimed for in the practical implementation of CV system decomposition.

[0098] The invention could be advantageously integrated into future automotive and robotics systems, e.g., automated vehicles and service robots, and in particular, lawnmowers, agricultural robots, logistics robots, and the like.

[0099] To supplement this, an example of implementation is shown below: The multi-path CV concept, which forms a core aspect of an embodiment of the presented invention, is used here for the robust realization of an automatic emergency braking system.

[0100] Some of the system components are in Fig. 4 shown.

[0101] Fig. Figure 5 shows a typical pedestrian protection scene according to Euro-NCAP (Euro-NCAP: European New Car Assessment Program).

[0102] This scenario often presents conventional approaches to video-based object detection with robustness problems due to reflections on the wet road surface.

[0103] Fig. 6 shows a typical object detection with a so-called image capture box 54, which is also called a bounding box, based on a trained object detector.

[0104] This detector implements the path S2-3 of the appearance-based object detection according to the Fig. 3 and Fig. 4.

[0105] Fig. Figure 7 shows a typical misdetection caused by the reflection of the pedestrian as object 53 on the wet road surface, here represented by the bounding box 55.

[0106] Fig. Figure 8 shows a three-dimensional reconstruction of the scene 50S under consideration, which is represented here by the impact or collision time (Time-to-Contact: TTC, time until contact with or impact on the object), whereby the respective impact or collision time of an object is represented by differently hatched areas in Fig. 8 is shown.

[0107] This path S2-1 of object detection according to Fig. 4 depicts the path of the three-dimensional reconstruction of the scene 50S from multiple viewpoints or views (multi view based 3D scene reconstruction) Fig. 3.

[0108] The algorithms used here for 3D reconstruction and TTC estimation are known as such.

[0109] This path enables a generic detection of the pedestrian as object 53 or generally of the object 53 relevant for braking by spatially grouping points with a critical impact or collision time TTC of less than 1 s, which is not distorted by the mirroring.

[0110] Fig. Figure 9 shows a simple neural network S2-5 for fusion of paths S2-1 and S2-2 for object detection and motion analysis according to Fig. 4.

[0111] In its simplest form, this network S2-5 implements an OR connection to warn the driver and an AND connection of the paths to trigger the brake for emergency braking.

[0112] However, under real operating conditions, such a system would potentially lead to a large number of false warnings, since not every detected object with a low TTC requires braking intervention.

[0113] The inclusion of the global context of the scene 50S as well as a situation-adaptive realization of the driver warning and the braking intervention can be realized by a correspondingly complex DNN S2-5.

[0114] The parameters of this network S2-5 are determined in a training process based on annotated image data.

Claims

[1] Control method (S) for an environment detection system (10) for a vehicle as a mobile device (1), comprising the steps: - optical detection (S1) of at least one scene (50S) from an environment (50) of the device (1), - Evaluation (S2) of the captured scene (50S) and deriving (S3) a driving situation as the operating situation of the device (1), - depending on the operating situation, controlled intervention (S4) in the operation of the device (1) and thereby adaptation (S5) of the movement or driving behaviour of the device (1), at which: - the evaluation (S2) is carried out on the basis of a physically motivated multi-path decomposition of machine vision in a plurality of independent evaluation processes (S2-1, S2-2, S2-3) and - the independent evaluation processes (S2-1, S2-2, S2-3) each have a path of at least - for appearance-based object detection and classification (S2-1), and - for a flow-based movement analysis (S2-2), and - for a multi-view based, three-dimensional scene reconstruction (S2-3), characterized by , that the evaluation (S2) comprises a downstream neural network (S2-5), a DNN or a layer thereof. [2] Control method (S) according to claim 1, in which the downstream process, the downstream neural network (S2-5), the DNN and / or the layer for processing - based on aspects of three-dimensional Euclidean geometry in relation to two-dimensional projective geometry and / or - based on a sensor data fusion process, and / or from digital maps or other digital data sources, are set up. [3] Control method (S) according to one of the preceding claims, in which the evaluation (S2) has a preceding process which - for semantic segmentation of the captured scene and / or - for pre-control or control of the multiple paths of the evaluation processes (S2-1, S2-2, S2-3) of the evaluation (S2) are set up. [4] Control method (S) according to claim 3, wherein the upstream process, the upstream neural network (S2-4), the DNN and / or the layer are arranged - periodic structures of a scene (50S) and / or - to detect and evaluate static, moving and / or potentially moving structures of a scene (50S). [5] Control method (S) according to one of the preceding claims, in which the process of appearance-based object detection and classification (S2-1) is carried out on the basis of a machine learning method, in conjunction with a training method and / or with the inclusion of a feedforward control by the upstream and / or downstream process, the respective neural network (S2-4), the DNN and / or the layer. [6] Control unit (30) for an environment detection system (10) for a vehicle as a mobile device (1), which is designed to carry out a control method for an environment detection system (10) according to one of claims 1 to 5. [7] Environment detection system (10) for a vehicle as a mobile device (1) with a control unit (30) according to claim 6. [8] Vehicle as a mobile device (1), with a drive (2) and / or with a steering system (3) and with an environment detection system (10) according to claim 7 for controlling the drive (2) and / or the steering system (3).

Images