Methods for providing a distributed perception system, in particular one based on distributed neural networks, and a corresponding perception system

Abstract

The invention relates to a method for providing a distributed perception system (100), in particular based on distributed neural networks (DNNs), which shares neural activations between different agents (101), wherein moving or parked vehicles and / or infrastructure participants can serve as agents (101) of the distributed perception system (100), comprising: - Providing a common neural network (DNN) for the different agents (101) through individual neural networks (DNNs) of the different agents (101), - Determining a common coordinate system (S) for the individual neural networks (DNNs) of the different agents (101), - Sharing of activations (A) among the individual neural networks (DNNs) of the different agents (101), in particular using the common coordinate system (S).

Description

[0001] The invention relates to a method for providing a distributed perception system, in particular one based on distributed neural networks, and to a corresponding perception system. The invention further relates to a corresponding neural network and a corresponding agent, in particular in the form of a vehicle, for providing a corresponding perception system. The invention also relates to a method for training a corresponding neural network.

[0002] In a highly dynamic or congested environment, such as a traffic intersection, it can be difficult or even impossible to gain sufficient situational awareness for safe decision-making when viewing the environment from only one location. A key problem is that relevant areas of the environment may be obscured by opaque objects or remain undetected due to inadequate sensors. In a specific traffic scenario at a congested intersection, some road users might be obscured from certain angles by a large object (e.g., a bus). Delayed detection of these obscured road users leads to an increased risk of accidents, which must be mitigated.

[0003] Some methods that recognize the aforementioned problem propose sharing raw data, mostly in the form of sensor data or representations thereof (e.g., features extracted by a neural network). Such methods require significant processing power from each agent.

[0004] Other methods propose sharing previously made predictions. Such methods require less computing power from each agent than the previously mentioned methods, but at the expense of perception accuracy. Another disadvantage of the late fusion approach is the significant delay from sensor data to the final result, as fusion can only occur once the predictions from the individual agents are complete and transmitted.

[0005] A well-known document, DE 10 2008 042 565 A1, suggests the use of external sensors.

[0006] It is therefore an object of the present invention to overcome at least one of the disadvantages described above, at least partially. In particular, it is an object of the invention to provide a method for providing a distributed perception system, especially one based on distributed neural networks, which enables improved neural capacity, improved performance, and reduced processing effort. Preferably, it is an object of the invention to provide a method for providing a distributed perception system, especially one based on distributed neural networks, which enables improved, extended perception functions and the safe use of perception functions, particularly for assisted, automated, or autonomous driving.Furthermore, it is an object of the invention to provide a corresponding neural network and a corresponding agent, in particular in the form of a vehicle, for providing a corresponding perception system. It is also an object of the invention to provide a method for training a corresponding neural network.

[0007] The problem is solved by a method for providing a distributed perception system, in particular one based on distributed neural networks, with the features of the independent method claim, and a corresponding perception system with the features of the dependent system claim. Furthermore, the invention relates to a corresponding neural network and a corresponding agent, in particular in the form of a vehicle, for providing a corresponding perception system with the features of the corresponding dependent claims. The invention also relates to a method for training a corresponding neural network with the features of the corresponding dependent claim.Features and details described in connection with the different embodiments and / or aspects of the invention naturally also apply in connection with the other embodiments and / or aspects, and vice versa, so that with regard to the disclosure of the individual embodiments and / or aspects, mutual reference is always made or can be made.

[0008] The invention provides: A method for providing a distributed perception system, in particular one based on distributed neural networks, which shares neural activations between different agents, where moving or parked vehicles and / or infrastructure participants can serve as agents of the distributed perception system, exhibiting: - Providing a common neural network for the different agents through individual (preferably deep) neural networks of the different agents, - Determining a common coordinate system for the individual neural networks of the different agents, - Sharing, especially synchronized sharing, of activations among the individual neural networks of the different agents, especially using the common coordinate system.

[0009] A distributed perception system is proposed, based on neural networks of different agents. Advantageously, the perception system is designed such that neural activations (during perceptual functions) can be shared between different agents.

[0010] Within this perceptual system, neural networks deployed in different agents can form a shared, large (virtual) neural network. This shared neural network can implicitly access the sensor input of all agents, thus providing an expanded field of view. This expanded field of view reduces or even eliminates obscured (unvisible or undetectable) areas. The expanded field of view enables a multitude of perspectives on jointly perceived objects.

[0011] The distributed perceptual system can provide increased neural capacity and thus improved perceptual performance. The collective perceptual system can synergistically deliver improved perceptual performance compared to the individual performance of unconnected neural networks.

[0012] In an automotive application, the various agents can be mobile perception systems, e.g., vehicles (both in traffic and parked), and / or stationary perception systems, e.g., parked vehicles and / or infrastructure devices (such as traffic lights, which may preferably be installed at dangerous and / or complex traffic locations).

[0013] The various agents can have different sensor modalities, such as radar, sonar and / or lidar, e.g. cameras (video, infrared, etc.).

[0014] The proposed distributed perception system can thus provide an extension of the existing perception systems in vehicles.

[0015] Each agent can work independently and, through shared activations via the distributed perception system, have an expanded field of view.

[0016] The distributed perception system can be used for various perception tasks, such as object recognition and classification (2D / 3D bounding boxes), pose or orientation recognition, semantic 2D and 3D segmentation, free space recognition, object prediction, etc.

[0017] Advantageously, the different agents can use a common coordinate system for a common perception output.

[0018] The connection between the neural networks of the different agents can be established via connections, especially wireless connections, between different hidden layers and / or nodes of the connected neural networks.

[0019] In inference (a joint forward pass of the data through the connected neural networks), all available sensor data can be propagated to all participating agents via neural activations.

[0020] An overall configuration of the distributed perception system can include dynamic or moving and / or static agents.

[0021] The various agents can enter and leave the distributed perception system in near real time.

[0022] Examples of at least partially dynamic systems could be: - Intersections with passing vehicles that can connect to infrastructure-based perception systems, - Residential areas with passing vehicles that can connect to infrastructure-based perception systems and / or parked vehicles, - different vehicles traveling in the same area and in the same direction, e.g. on a motorway, - Parking garages, parking lots, garages, with vehicles entering or exiting and with parked vehicles that can connect to infrastructure-based perception systems and / or parked vehicles.

[0023] Distributed perception systems can be controlled in different ways: - At least partially local or static systems can be controlled centrally, e.g., by fixed (infrastructure-based) perception systems, to which dynamic agents can connect. In such systems, a local agent or the local neural network, which performs the master tasks, can determine a common coordinate system and / or an inference frequency that can be adopted by the dynamic agents. - At least partially mobile systems can be formed between dynamic agents, e.g., on highways. The agents can agree on a common coordinate system with respect to their relative position. This coordinate system does not need to be static with respect to the environment (the static world), since the agents are moving. This coordinate system can be updated repeatedly, e.g., inference by inference.

[0024] In principle, mixed static and / or mobile systems are conceivable.

[0025] To protect the distributed perception system against security threats, local plausibility checks can be implemented.

[0026] In the automotive sector, manufacturer-, hardware-, and / or software-specific neural networks can be used. However, it is also conceivable that neural networks of a specific standard could be provided and / or used.

[0027] Training the neural networks on the distributed perception system can advantageously follow a two-stage approach: After initial training of the neural networks on the individual perception systems, fine-tuning of the connected neural networks can be performed on the distributed perception system.

[0028] In local or static systems, fine-tuning can be performed on the infrastructure-based local agent or on the local neural network, which takes over the master functions.

[0029] Furthermore, it can be provided that the shared neural network is provided by a connection, particularly a wireless one, between the individual neural networks of the different agents. It is also conceivable that the shared neural network is provided by a connection, particularly a wireless one, between corresponding layers and / or nodes of the individual neural networks of the different agents. In this way, sharing activations to provide an expanded field of view between spaced (moving and / or static) agents can be enabled.

[0030] Furthermore, it can be implemented that the sharing of activations among the individual neural networks of the different agents is carried out at corresponding layers and / or nodes. In this way, activations at comparable locations can be used to provide an expanded field of view.

[0031] Advantageously, the distributed perception system can be used for assisted, automated, or autonomous driving. Thus, in highly dynamic or crowded environments, the distributed perception system can enable improved perception and safer decision-making because the environment can be perceived by different agents. Relevant areas of the environment that might be obscured for individual agents by opaque objects can be reliably perceived using the distributed perception system.

[0032] Advantageously, the distributed perception system can be used for at least one of the following perception tasks: - Object detection, - Object classification, - Pose recognition, - Direction detection, - semantic 2D or 3D segmentation, - Clearance detection, and / or - Object prediction, - Traffic situation recognition, - Route planning, and / or - Providing map data, etc.

[0033] To prevent manipulation, at least one local security check and / or plausibility check for activations can be performed before or during the use of the distributed perception system. A local security check and / or plausibility check can be implemented, for example, through the following measures: - a comparison of threshold values ​​for identical or comparable activations, - an evaluation of temporal changes in threshold values ​​for identical or comparable activations, and / or - a comparison with available sensor data, etc.

[0034] For the sake of simplicity, it can be stipulated that a shared activation is then considered safe and / or plausible if it is initiated by at least two agents.

[0035] Advantageously, sensor data can be used as input data in the individual neural networks of the different agents, originating from at least one of the following sensors: - an optical sensor, - a camera, - a video camera, - an infrared sensor, - a radar sensor, - a UWB sensor, - a lidar sensor, - an ultrasonic sensor, and / or - an acoustic sensor.

[0036] In this way, improved, extended environmental monitoring can be enabled through various sensors.

[0037] Advantageously, the distributed perception system can be formed at at least one of the following locations: - a geographical section, - a driving section, - at, especially known, danger points, - at accident scenes, - at an intersection, - a residential area, - a parking lot, - a parking garage, - a garage, - a tunnel system, and / or - a motorway, etc.

[0038] In a (at least partially) static system, it is conceivable that the common coordinate system is determined by a static agent, e.g., an infrastructure participant, such as at an intersection, traffic light, tunnel, building, signage, garage, etc. This allows for a technically simple solution for determining the common coordinate system.

[0039] In a (at least partially dynamic) system, it is conceivable that the common coordinate system is chosen by connected agents, for example, by vehicles traveling in close proximity and / or in the same direction, presumably over a certain period of time. In this way, a flexible solution for dynamically determining the common coordinate system can be provided.

[0040] Advantageously, the various agents can dynamically connect to and / or disconnect from the distributed perception system. This enables flexible, needs-based use of the distributed perception system.

[0041] Advantageously, the perceptual functions of the individual neural networks of the different agents can use the common coordinate system. In this way, activations at appropriate locations within the individual neural networks can be shared.

[0042] One advantage is that, before using the distributed perception system, input data from the individual neural networks of the different agents can be transferred into the common coordinate system. This allows for the rapid application of activations at appropriate points within the individual neural networks.

[0043] Suitable neural networks can be selected depending on the following properties of the different agents: - technical design, - Hardware equipment, - Software equipment, - Sensor equipment, and / or - Communication equipment, etc.

[0044] Advantageously, suitable neural networks for use in the distributed perception system can be selected according to a specific standard.

[0045] According to a further aspect, the invention provides: A distributed perception system, in particular one based on distributed neural networks, which is provided by a method according to one of the preceding claims. The distributed perception system offers the same advantages described above in connection with the method. These advantages are fully referenced herein.

[0046] According to a further aspect, the invention provides: A neural network for an agent, in particular in the form of a vehicle, wherein the agent forms part of a distributed perception system, in particular one based on distributed neural networks, which is provided by a method according to one of the preceding claims, comprising a perception function configured to provide and / or use the common coordinate system. The neural network offers the same advantages described above in connection with the method. These advantages are fully referenced herein.

[0047] According to a further aspect, the invention provides: An agent, in particular in the form of a vehicle, for a distributed perception system, in particular one based on distributed neural networks, comprising a neural network according to the preceding claim. The agent enables the same advantages to be achieved as described above in connection with the method. These advantages are fully referenced herein.

[0048] According to another aspect, the invention provides: A method for training a neural network for a distributed, in particular distributed neural network-based, perception system, which is provided by a method according to one of the preceding claims, comprising: - Individual training of neural networks of the different agents for a specific use, - Joint training of neural networks of the different agents of the distributed perception system.

[0049] In this way, neural networks can be trained according to a two-stage approach: After initial training of the neural networks, fine-tuning of the connected neural networks can be performed on the distributed perception system. For (at least partially local or static) systems, the fine-tuning can be performed on the infrastructure-based local agent or on the local neural network, which assumes the master functions.

[0050] Further advantages, features, and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can each be advantageous individually or in any combination. The drawings show: Fig. 1. An exemplary traffic situation, Fig. 2 a schematic representation of a proposed procedure, and Fig. 3 a schematic representation of a distributed perception system.

[0051] The Fig. Figures 1 to 3 serve to explain a procedure which was developed to provide a distributed perception system 100, in particular based on distributed neural networks DNNs, which shares neural activations between different agents 101.

[0052] In mobile applications, moving or parked vehicles and / or infrastructure participants can serve as agents 101 of the distributed perception system 100.

[0053] The distributed perception system 100 can in principle be formed by dynamic and / or static agents 101.

[0054] Stationary infrastructure participants can advantageously be deployed at dangerous and / or complex traffic locations. In principle, stationary agents can be deployed at the following locations: - at frequent accident sites, - at an intersection, - a residential area, - a parking lot, - a parking garage, - a garage, - a tunnel system, and / or - a motorway.

[0055] As it is Fig. 1 and Fig. As indicated in point 3, the perception system 100 can be used for at least one of the following perception tasks WA: - Object detection, - Object classification, - Pose recognition, - Direction detection, - semantic 2D or 3D segmentation, - Clearance detection, and / or - Object prediction, - Traffic situation recognition, - Route planning, and / or - Providing map data.

[0056] As it is Fig. 1 and Fig. As indicated in point 3, the individual neural networks DNNs of the different agents can use 101 sensor data as input data ED, which originates from at least one of the following sensors: - an optical sensor, - a camera, - a video camera, - an infrared sensor, - a radar sensor, - a UWB sensor, - a lidar sensor, - an ultrasonic sensor, and / or - acoustic sensor.

[0057] As it is Fig. As indicated in points 1 to 3, the procedure has the following steps: - Providing a common neural network DNN for the different agents 101 through individual (preferably deep) neural networks DNNs of the different agents 101, - Determining a common coordinate system S for the individual neural networks DNNs of the different agents 101, - Sharing (especially synchronized sharing) of activations A among the individual neural networks DNNs of the different agents 101, especially using the common coordinate system S.

[0058] As it is Fig. As illustrated in Figure 1, a distributed perception system 100 is provided, which is based on neural networks (DNNs) of different agents 101. Advantageously, the perception system 100 is implemented in such a way that, in particular, the individual neural networks (DNNs) of the different agents 101 are implemented such that neural activations A can be shared between different agents 101.

[0059] Within the distributed perception system 100, neural networks (DNNs) deployed in different agents 101 can form a common large (virtual) neural network (DNN). This common neural network (DNN) can implicitly access the input data (ED) of all agents 101.

[0060] The distributed perception system 100 can provide an enlarged field of view.

[0061] As it is Fig. Figure 1 shows that the distributed perception system can make 100 hidden (unseeable or unperceivable) areas visible to some agents.

[0062] Furthermore, the enlarged field of vision can enable a variety of perspectives on objects perceived together.

[0063] The distributed perception system 100 can provide increased neural capacity and improved perceptual performance.

[0064] The combined perceptual system 100 can provide improved perceptual performance in a synergistic way, compared to the individual performance of the unconnected neural networks DNNs.

[0065] As it is Fig. As illustrated in Figure 1, the proposed distributed perception system 100 can provide an extension of the existing perception systems in the individual vehicles.

[0066] Each agent 101 can work independently and, if desired, have an enlarged field of view available through shared activations A via the distributed perception system 100.

[0067] As mentioned above, the different agents 101 can use a common coordinate system S for a common perception output.

[0068] The common coordinate system S can be determined by a static agent 101.

[0069] The common coordinate system S can also be selected by connected agents 101, e.g. depending on the current positions of the connected agents 101.

[0070] The connection between the neural networks DNNs of the different agents 101 can be established, for example, via connections, especially wireless connections, between different hidden layers and / or nodes of the connected neural networks.

[0071] In an inference that can provide for a joint forward pass of the input data ED through the connected neural networks DNNs, all available sensor data can be propagated to all participating agents 101 via neural activations A.

[0072] The various agents 101 can join and leave the distributed perception system 100 almost in real time.

[0073] Examples of at least partially dynamic perception systems could be the following: - Intersections with passing vehicles that can connect to infrastructure-based perception systems, - Residential areas with passing vehicles that can connect to infrastructure-based perception systems and / or parked vehicles, - different vehicles traveling in the same area and in the same direction, e.g. on a motorway, - Parking garages, parking lots, garages, with vehicles entering or exiting and with parked vehicles that can connect to infrastructure-based perception systems 100 and / or parked vehicles.

[0074] The distributed perception system 100 can be controlled statically and / or dynamically: - At least partially local or static systems can be controlled centrally, e.g., by fixed (infrastructure-based) perception systems, to which moving agents 101 can connect. In such systems, the local agent 101 or the local neural network, which performs the master tasks, can determine a common coordinate system S and / or an inference frequency (for synchronization) that is adopted by the dynamic agents. - At least partially mobile systems can be formed between dynamic agents 101, e.g., on highways. The agents 101 can agree on a common coordinate system S, for example, with respect to their relative positions. This common coordinate system S can move relative to the environment because the agents 101 are moving. This common coordinate system S can be updated repeatedly, e.g., inference by inference.

[0075] In principle, mixed statically and / or dynamically controlled systems are conceivable.

[0076] To protect the distributed perception system 100 against security threats, local plausibility checks can be implemented.

[0077] In the automotive sector, manufacturer-, hardware-, and / or software-specific deep neural networks (DNNs) can be used. However, it is also conceivable that neural networks (DNNs) of a specific standard could be provided and / or used.

[0078] The training of the neural networks DNNs on the distributed perception system 100 can advantageously follow a two-stage approach: After an initial training of the neural networks DNNs of the individual perception systems, a fine-tuning of the connected neural networks DNNs on the distributed perception system 100 can be carried out.

[0079] In local or static systems, fine-tuning can be performed on the infrastructure-based local agent 101 or on the local neural network DNNs, which can take over the master functions.

[0080] The above explanation of the embodiments describes the present invention exclusively by way of examples. Of course, individual features of the embodiments can be freely combined with one another, provided they are technically feasible, without departing from the scope of the present invention. Reference symbol list 100 Perception System 101 Agent DNN common neural network DNNs individual neural networks A Activations S common coordinate system t time ED input data WA Perception Tasks

Claims

[1] Method for providing a distributed perception system (100), in particular based on distributed neural networks (DNNs), which shares neural activations between different agents (101), wherein moving or parked vehicles and / or infrastructure participants can serve as agents (101) of the distributed perception system (100), comprising: - Providing a common neural network (DNN) for the different agents (101) by individual neural networks (DNNs) of the different agents (101), - Determining a common coordinate system (S) for the individual neural networks (DNNs) of the different agents (101), - Sharing activations (A) among the individual neural networks (DNNs) of the different agents (101), in particular using the common coordinate system (S). [2] Method according to claim 1, wherein the common neural network (DNN) is provided by a, in particular wireless, connection between the individual neural networks (DNNs) of the different agents (101), and / or wherein the common neural network (DNN) is provided by a, in particular wireless, connection between corresponding layers and / or nodes of the individual neural networks (DNNs) of the various agents (101). [3] Method according to one of the preceding claims, wherein the sharing of activations (A) among the individual neural networks (DNNs) of the different agents (101) is performed at corresponding layers and / or nodes (P). [4] Method according to one of the preceding claims, wherein the distributed perception system (100) is used for assisted, automated or autonomous driving, and / or wherein the perception system (100) is used for at least one of the following perception tasks (WA): - object recognition, - object classification, - Pose detection, - Direction detection, - semantic 2D or 3D segmentation, - Free space detection, and / or - object prediction, - Traffic situation detection, - Route planning, and / or - Providing map data. [5] Method according to one of the preceding claims, wherein at least one local security check and / or plausibility check for activations (A) is carried out during or before using the distributed perception system (100), and / or wherein a shared activation (A) is assessed as safe and / or plausible if it is initiated by at least two agents (101). [6] Method according to one of the preceding claims, wherein in the individual neural networks (DNNs) of the different agents (101) sensor data are used as input data (ED) which originate from at least one of the following sensors: - an optical sensor, - a camera, - a video camera, - an infrared sensor, - a radar sensor, - a UWB sensor, - a lidar sensor, - an ultrasonic sensor, and / or - an acoustic sensor. [7] Method according to one of the preceding claims, wherein the distributed perception system (100) is formed at at least one of the following locations: - a geographical section, - a driving section, - at, especially known, danger points, - at accident scenes, - an intersection, - a residential area, - a parking space, - a parking garage, - a garage, - a tunnel system, and / or - a highway. [8] Method according to one of the preceding claims, wherein the common coordinate system (S) is determined by a static agent (101), and / or wherein the common coordinate system (S) is selected by connected agents (101). [9] Method according to one of the preceding claims, wherein the various agents (101) dynamically connect to the distributed perception system (100), and / or wherein the various agents (101) dynamically leave the distributed perception system (100). [10] Method according to one of the preceding claims, wherein perception functions of the individual neural networks (DNNs) of the different agents (101) use the common coordinate system (S), and / or wherein, before using the distributed perception system (100), input data (ED) of the individual neural networks (DNNs) of the various agents (101) are transferred into the common coordinate system (S). [11] Method according to one of the preceding claims, where the individual neural networks (DNNs) are selected depending on the following properties of the different agents (101): - technical design, - hardware equipment, - software equipment, - Sensor equipment, and / or - Communication equipment, and / or wherein the individual neural networks (DNNs) for use in the distributed perception system (100) are selected according to a specific standard. [12] Distributed perception system (100), in particular based on distributed neural networks (DNNs), which is provided by a method according to one of the preceding claims. [13] Neural network (DNNs) for an agent (101), in particular in the form of a vehicle, wherein the agent (101) forms part of a distributed perception system (100), in particular based on distributed neural networks (DNNs), which is provided by a method according to one of the preceding claims, comprising a perception function which is designed to provide and / or use the common coordinate system (S). [14] Agent (101), in particular in the form of a vehicle, for a distributed perception system (100), in particular based on distributed neural networks (DNNs), comprising a neural network (DNNs) according to the preceding claim. [15] Method for training a neural network (DNNs) for a distributed perception system (100), in particular based on distributed neural networks (DNNs), which is provided by a method according to one of the preceding claims, comprising: - individually training neural networks (DNNs) of the different agents (101) for an intended use, - joint training of neural networks (DNNs) of the different agents (101) of the distributed perception system (100).

Images