Cloud platform for automated mobility and computer-implemented method for providing cloud-based data enrichment to automated mobility

Abstract

Cloud platform (20) for automated mobility (16, 18, 19) comprising environmental sensing devices (10), wherein the environmental sensing devices (10) each comprise: a housing (11); sensors (12, 13) arranged in the housing (11), comprising optical sensors, sound sensors, radar sensors, or a combination of the aforementioned sensors (12, 13), wherein the housing (11) comprises openings (14) for the sensors (12, 13) for sensing the environment; a computer (15) arranged in the housing (11) and connected to the sensors (12, 13) for signal transmission; and a first interface to the cloud platform; and a main computer (21), wherein the environmental sensing devices (10) and the main computer (21) are connected to form a computer network of the cloud platform (20), in which the environmental sensing devices (10) are connected to each other and to the main computer (21) for data transmission and are connected via the cloud platform storage space,Computing power and computer programs are provided, • a memory (22),o in which first instructions of a first computer program are stored, wherein the first instructions cause the computers (15) of the environmental sensing devices (10) to detect, classify, locate and / or track objects (16) from the respective environment based on the signals of the sensors (12, 13) and the computers (15) to provide the detections, localizations and / or tracking to the main computer (21) as data via the respective first interfaces of the environmental sensing devices (10), when the first computer program is running on the computers (15) without being installed on the local computers (15) of the environmental sensing devices (10), ando in which second instructions of a second computer program are stored, wherein the second instructions cause,that the main computer (21) synchronizes and / or combines the data from the computers (15) of the environmental sensing devices (10) and obtains a data enrichment for the second environmental sensing from individual sensings of the respective environmental sensing devices (10) when the second computer program is running on the main computer (21), and a second interface (23) to provide the second environmental sensing for the automated mobility (18, 19).

Description

[0001] The invention relates to a cloud platform for automated mobility and a computer-implemented method for providing cloud-based data enrichment to automated mobility.

[0002] EP 2 818 922 B1 discloses a traffic monitoring device. Such traffic monitoring devices, also known as PoliScan Speed, are in particular systems for speed monitoring.

[0003] DE 10 2019 210 933.0 discloses an environmental sensing device for automated mobility comprising a housing, sensors arranged in the housing, the housing comprising openings for the sensors to sensing the environment, an evaluation unit arranged in the housing and connected to the sensors for signal transmission, the evaluation unit comprising a working memory for executing a computer program for classifying, localizing, and / or tracking objects in the environment, the computer program comprising commands that cause the evaluation unit to obtain the objects and their respective state of motion from the sensor signals when the computer program is running on the evaluation unit, and an interface for providing the objects and their state of motion to the automated mobility system. The invention further relates to a system comprising the environmental sensing devices.

[0004] In automated mobility, obstructions and / or complex traffic situations can create hazardous areas where the sensor set of an autonomous vehicle cannot detect all vulnerable road users with 100% certainty. The underlying problem is that environmental perception, especially of objects (static and dynamic), in the vicinity of an autonomous vehicle cannot guarantee 100% certainty in certain situations. These situations can include: • Intersections with limited visibility must be navigated with autonomous vehicles (city driving, university areas on campus); • Risks that cannot be assessed must be covered: passenger transport in situations with an increased number of people / intoxicated persons, e.g. at city festivals; • Securing autonomous vehicles in residential areas with risky play streets; • Non-rational pedestrians and cyclists, e.g. pedestrian crossing in Tokyo, bicycle couriers in New York; • Roadworks: Traffic flow is altered and there are special obstacles in the lane. Unknown vehicles, such as excavators and other construction vehicles, may be in the lane. Construction workers may be standing near or in the lane, in areas where speeds of 50-100 km / h are normally permitted.

[0005] Another problem is that, presumably, for the full introduction of Level 5 vehicles, all other road users will first need to be integrated into the automated mobility / autonomous world.

[0006] Furthermore, in difficult situations, including driving in narrow streets, the location of vehicles is made more difficult.

[0007] One objective of the present invention was to provide a solution to the problems mentioned above.

[0008] The invention solves this problem through a cloud-based computing platform. According to the invention, stationary or mobile perception modules, also known as environmental sensing devices, are networked together. These environmental sensing devices detect, classify, locate, and / or track road users, such as pedestrians, vehicles, commercial vehicles, and bicycles, as well as environmental objects, such as lane markings, trees, road boundaries, or construction sites, using radar, lidar, cameras, ultrasound, infrared, or acoustic sensors. Through networking via the cloud, ADAS / AD systems or map providers are provided with improved information regarding expected traffic situations. In particular, information is provided on situations that pose a risk to a driver.

[0009] According to one aspect, the invention provides a cloud platform for automated mobility. The cloud platform comprises a computer network of environmental sensing devices. Each environmental sensing device comprises a housing and sensors arranged within the housing. The housing includes openings for the sensors to detect the environment. Furthermore, the environmental sensing devices comprise a computer arranged within the housing, connected to the sensors for signal transmission, and a first interface to the cloud platform. The cloud platform further comprises a main computer and a memory. The memory stores initial instructions of a first computer program.The first commands cause the computers of the environmental sensing devices to detect, classify, locate, and / or track objects in their respective environments based on sensor signals. When the first computer program is running, the computers provide this detection, localization, and / or tracking data to the main computer via their respective interfaces. Furthermore, the memory contains second commands for a second computer program. These second commands cause the main computer to synchronize and / or combine the data from the computers of the environmental sensing devices and to enrich the data from individual sensing devices for the second environmental sensing process. When the second computer program is running on the main computer, the cloud platform also includes a second interface to provide the second environmental sensing data for automated mobility.

[0010] According to another aspect, the invention provides a computer-implemented method for providing cloud-based data enrichment for improved environmental perception to automated mobility systems. The method comprises the following steps: • Detecting, classifying, locating and / or tracking objects in their respective environments using environmental detection devices distributed throughout a transport infrastructure, • Providing the received data to a main computer of a cloud platform, • Synchronizing and / or combining the data in the main computer, • Obtaining a second environmental survey from the individual surveys of the respective environmental survey devices and • Providing the second environmental sensing system to automated mobility.

[0011] The transport infrastructure encompasses either a national regional transport infrastructure or a global transport infrastructure. With regard to the global transport infrastructure, the environmental monitoring devices are not only located in one country, but are distributed across countries worldwide. This creates a globally operating cloud platform.

[0012] According to one aspect of the invention, a cloud platform according to the invention is used to carry out the method.

[0013] Advantageous embodiments of the invention will become apparent from the dependent claims, the drawing and the description of preferred embodiments.

[0014] Automated mobility encompasses the ecosystem of automated driving, from partial automation, such as driving with driver assistance systems, to full automation, i.e., driverless driving. Participants in automated mobility include vehicles of all kinds, especially road vehicles, people transport systems, such as robot taxis, passenger drones, and automated people movers, automated transit networks, such as personal / group rapid transit systems, drivers, passengers, non-mobilized road users, such as pedestrians and cyclists, and infrastructure elements, such as environmental sensing devices or intelligent traffic management systems. Intelligent traffic management systems include lighting systems, such as traffic lights, and communicate with the cloud platform, such as environmental sensing devices and / or the main computer, for example, via V2X technology.

[0015] The cloud platform provides storage space, computing power, and / or computer programs as a service. The computer programs are made available to the computer network without needing to be installed on the local computers of the environmental monitoring devices.

[0016] The computer network is a connection between the environmental monitoring devices and the main computer of the cloud platform. Within the network, the environmental monitoring devices are interconnected and connected to the main computer for data transmission. The computer network may, for example, have a star topology with the main computer as the central distribution point. The computer network can be a wireless network, such as an ad-hoc network, or a Powerline network, where the network is established using an existing electrical power grid.

[0017] Since the environmental monitoring devices are distributed throughout the transport infrastructure, each device collects data from a specific location, for example, where it is stationary. This means that the second environmental monitoring system advantageously incorporates data from several different fields of view of the environmental monitoring devices.

[0018] The main computer provides the automated mobility system, for example vehicles, pedestrians, cyclists, passengers in public transport vehicles and other participants, with a second environmental sensing system, which is an improved environmental sensing system based on the individual sensing data from the environmental sensing devices.

[0019] According to one aspect of the invention, the main computer provides the second environmental data set to the individual environmental data set devices, which then communicate with participants in the automated mobility system. According to another aspect of the invention, the main computer provides the second environmental data set to the participants in the automated mobility system.

[0020] The housing of the environmental device protects the sensors, the evaluation unit, the interfaces, and other electronic components, such as the power supply, against the ingress of foreign objects, dust, and / or liquids. For example, the housing is encapsulated and sealed on the outside.

[0021] The sensors include optical cameras, infrared cameras, time-of-flight sensors or lidar, acoustic sensors such as microphones, radar sensors and ultrasonic sensors.

[0022] The main computer and the subcomputers include, for example, processors (CPUs, GPUs), ICs, ASCIIs, FGPAs, and other logic components to execute the primary and secondary computer programs. The hardware / microarchitecture of the main computer and / or the subcomputers is designed for a computing power of several hundred tera-ops. Furthermore, the subcomputers include interfaces to sensors, such as FPD-Link or GMSL-Link interfaces.

[0023] The first, second, and subsequent instructions of the first and second computer programs are sections of software code written, for example, in the programming language C++, Java, or Python.

[0024] By classifying, localizing, and / or tracking objects, a perception of the respective environment of an environmental sensing device is obtained. Generally, according to one aspect of the invention, the perception comprises the following structure: First, the various sensor signals or sensor data are synchronized, for example, three-dimensional lidar and / or radar point clouds with two-dimensional pixel images from a camera. In each dataset of one of the sensors, or in a combined dataset of all sensors, object features are then determined, for example, using machine learning methods, such as convolutional networks trained on image segmentation and / or object recognition, and / or classical image processing algorithms, such as color thresholding, gradient detection, or Sobel filters.Object features include geometric features for object description, such as bounding boxes, L-shapes for describing vehicles, ellipsoids, and clusters. The detected features are fused in a further stage and tracked over time, i.e., tracked using known tracking algorithms, for example the Joint Integrated Probabilistic Data Association (JIPDA) algorithm (see https: / / ieeexplore.ieee.org / document / 1020938), the Gaussian Mixture Probability Hypothesis Density Filter (PHD) algorithm (see https: / / ieeexplore.ieee.org / document / 1710358), or the convolutional network for three-dimensional object detection, tracking, and motion prediction disclosed in http: / / openaccess.thecvf.com / content_cvpr_2018 / papers / Luo_Fast_and_Furious_CVPR_2018_paper.pdf.The tracking results are combined with detected features of the static environment, such as lane lines, obstacles, lane boundaries or construction zones, to obtain a description of the scenery, which then represents the environment perception.

[0025] For the classification, localization, and / or tracking of objects, the first computer program executes, for example, an algorithm that generates object features from raw data or individual sensor images. Various features are fused and tracked over time. Furthermore, according to one aspect of the invention, the first commands include instructions for the computers to transform the objects detected by the sensors from the coordinate system of the respective sensor into the world coordinate system. This enables the environmental sensing devices to understand and perceive the surrounding environment.

[0026] The environmental monitoring devices thus enable real-time processing of the sensor signals.

[0027] The objects and their state of motion are provided, for example, via object lists. For each object, relative to a world coordinate system, its Cartesian coordinates x, y, z, velocity components v_x, v_y, v_z, rotational velocity components ω_x, ω_y, ω_z, azimuth angle φ, and elevation angle θ are provided. Based on the provided state of motion, the objects are categorized as dynamic or static.

[0028] According to a further aspect of the invention, the second computer program comprises instructions that cause the main computer to generate comprehensive forecasts of movement patterns of automated mobility participants and of traffic events based on the data from the environmental sensing devices when the second computer program is running on the main computer. Since the environmental sensing devices are distributed throughout the traffic infrastructure, each device collects data from a specific position, for example, where it is stationary. Thus, the forecasts advantageously incorporate data from several different fields of view of the environmental sensing devices.

[0029] According to one aspect of the invention, the forecasts are long-term forecasts. The cloud platform's storage system collects and stores data from the computers of the environmental monitoring devices over extended periods, such as years or decades. The main computer analyzes this long-term data to generate long-term forecasts of movement patterns of automated mobility users and traffic events.

[0030] According to one aspect of the invention, the second computer program includes instructions that cause the main computer to evaluate the long-term data, for example by means of exponential smoothing, trend forecasting, moving averages, extrapolation or regression, in order to obtain the forecasts.

[0031] The main computer evaluates the long-term data, for example, heuristically, deterministically, or using machine learning algorithms to comprehensively analyze decision trees or recurrent artificial neural networks.

[0032] This allows the cloud platform to learn, for example, that high traffic volumes occur during specific events. The cloud platform then makes this future knowledge available to participants in automated mobility, enabling them to better prepare for traffic. Specific events include public holidays, trade fairs or exhibitions, sporting events, and the start and / or end of school holidays. Furthermore, the cloud platform learns how individual participants and / or groups of participants in automated mobility, such as pedestrians or drivers, behave in specific scenarios. One such scenario is the approach of an emergency vehicle with its sirens blaring. Through long-term forecasts, the cloud platform has learned that vehicles should form an emergency lane in such a scenario.Based on long-term forecasts, the cloud platform learns, for example, that on every second Monday of a month, or at another specific time, a cyclist A enters a roundabout B. This information is then provided to vehicles C approaching roundabout B at these times, enabling them to react accordingly when entering the roundabout, for example, by initiating specific braking maneuvers manually or automatically / autonomously.

[0033] In a further embodiment of the invention, environmental monitoring devices of the cloud platform are also positioned at schools. Based on forecasts, for example, from recordings made by cameras arranged in the environmental monitoring devices, the cloud platform learns, for instance, that at a certain time on weekdays, such as 1 p.m., schoolchildren leave the school and head towards a nearby bus stop. This information is then provided to approaching vehicles, for example, via V2X communication, which then know in advance that there is an increased number of schoolchildren around the bus stop in question.The vehicles, meaning human drivers or ADAS / AD systems that assist or perform the driving task automatically / autonomously, thus have the future knowledge that at the affected position the driving style must be adjusted accordingly to be attentive, slow and / or with high reaction speed.

[0034] According to another aspect of the invention, the second computer program includes instructions that cause the main computer to create risk models for automated mobility based on data from the environmental sensing devices when the second computer program is running on the main computer. These risk models model situations and corresponding responses to critical conditions. Using these risk models, the cloud platform warns participants in the automated mobility system of impending potential critical situations. This enables participants to react more effectively in such situations, thereby increasing safety in automated mobility.

[0035] According to one aspect of the invention, the risk models are created based on the forecasts.

[0036] For example, the main computer uses sensor data from environmental monitoring devices to create statistical distributions, such as pedestrian and / or bicycle traffic at specific times and locations, in order to derive potential hazards. For instance, the main computer uses the environmental monitoring device data to determine the probability of a pedestrian crossing a roadway at a specific location, such as the probability of a pedestrian crossing a street at a traffic light-controlled pedestrian crossing even though the light is red. The cloud platform then provides this information to vehicles approaching this pedestrian crossing, warning them that there is a significant probability of a pedestrian crossing the road even though the traffic light is green for the vehicle.Accordingly, the cloud platform also provides pedestrians intending to cross at a green light with the probability that an approaching vehicle will cross at a red light from the vehicle's perspective. Further violations of traffic regulations are also incorporated into the risk models. According to another aspect of the invention, the main computer determines the probability that a vehicle could collide with a pedestrian or cyclist at a specific time and location, and uses this information to warn the vehicles involved in advance.

[0037] According to another aspect of the invention, the inventive method provides forecasts, for example long-term forecasts, and / or risk models.

[0038] According to another aspect of the invention, the computers of the environmental sensing devices are designed to fuse the sensor signals and / or perform plausibility checks. The fusion of individual sensor data results in a more detailed and higher-quality environmental sensing compared to individual sensor data. During the development of the environmental sensing device, it was surprisingly found that fusing data from a 360° lidar scanner with camera object detection yields the best results. A more cost-effective alternative is a sensor set consisting of ultrasonic sensors, radar sensors, and camera sensors. The plausibility check provides multiple redundancies in the environmental sensing device.

[0039] According to another aspect of the invention, the first computer program comprises a first machine learning algorithm, and the computers of the environmental sensing devices execute the first machine learning algorithm for the classification, localization, and / or tracking of objects when the first computer program is running on the computers. Additionally or alternatively, the second computer program comprises a second machine learning algorithm, and the main computer executes the second machine learning algorithm for the synchronization and / or combination of the data from the computers when the second computer program is running on the main computer.

[0040] Machine learning is a technology that teaches computers and other data processing devices to perform tasks by learning from data, rather than being programmed for the tasks. For example, the machine learning algorithm is a convolutional neural network trained on semantic image recognition. With regard to object tracking, the convolutional neural network is advantageously a recurrent convolutional network, meaning a network with recurrent layers, such as LSTM units (Long Short-Term Memory units).

[0041] According to the invention, the environmental sensing devices comprise optical sensors, sound sensors, radar sensors, or a combination of the aforementioned sensors. Optical sensors capture images of the environment. Sound sensors additionally or alternatively detect ambient noise, such as emergency vehicle sirens, children's cries, bicycle bells, or the sounds of other vehicles. This is particularly advantageous at urban intersections and / or in residential areas with shared spaces. Noise detection is also beneficial in electrified automated mobility systems, which are quieter than automated mobility systems based on combustion engines. Radar sensors enable the detection of objects in the environment even under poor visibility conditions, such as rain or snow.

[0042] According to another aspect of the invention, the environmental sensing devices comprise at least one camera sensor, lidar sensor, radar sensor, ultrasonic sensor, and / or acoustic sensor. The camera sensor is, for example, a sensor from a monocular or stereo camera or a time-of-flight sensor. The lidar sensor is, for example, a 360° semiconductor laser scanner. The acoustic sensor is, for example, a microphone. The camera sensor photographically captures the environment and the objects within it. A stereo camera or a time-of-flight sensor provides depth information. The lidar sensor is characterized by its relatively high resolution. In one embodiment of the invention, the first machine learning algorithm, for example, a convolutional network, is trained on semantic image recognition based on the signals from the camera sensor, lidar sensor, and / or radar sensor.The acoustic sensor detects participants in automated mobility at locations that are not visible, such as intersections.

[0043] According to another aspect of the invention, the first computer program instructs the computers to group, associate, track, and / or process the objects from images and / or point clouds of the sensors. For example, the images and / or point clouds are processed using sensor fusion methods, such as track-to-track fusion based on normalized covariance matrices. Tracks are recorded separately for each sensor. These tracks are then fused into system tracks. This improves object recognition. A point cloud comprises the spatial coordinates of points. 3D radar or lidar data are typically 3D point clouds.

[0044] According to another aspect of the invention, the second interface is a radio interface through which the cloud platform communicates with automated vehicles and / or an automated mobility infrastructure, such as environmental sensing devices. For example, the second interface is a V2X interface. According to another aspect of the invention, the environmental sensing devices comprise a V2X control unit and the radio interface. The radio interface is designed, for example, for 5G radio technology. For example, the environmental sensing devices locate an autonomous vehicle. The autonomous vehicle receives its own absolute position, determined by the environmental sensing devices, via the radio interface and thus has ground-truth information. The autonomous vehicle also has a V2X communication module for communicating with the environmental sensing devices.Depending on the range of V2X communication, the autonomous vehicle already has an object list of existing objects in risky areas (intersections, pedestrian zones, construction sites, etc.). The vehicles receive an object list of all moving and stationary objects in their vicinity from the cloud platform. V2X communication can take place via Bluetooth or Wi-Fi, for example.

[0045] According to another aspect of the invention, the environmental detection devices are arranged at universities, trade fair grounds, airports, urban areas, intersections, roundabouts, road crossings, schools, residential areas, and / or construction sites to provide automated vehicles with a list of objects, including their movement status, via a cloud platform before the automated vehicles reach the environmental detection devices. Thus, the environmental detection devices are positioned particularly at critical locations, such as critical intersections or roundabouts, and detect critical situations. Critical situations include, for example, a pedestrian obscured by a vehicle who intends to cross a zebra crossing, a cyclist obscured by other vehicles at an intersection, or complex intersections and roundabouts.Critical locations and situations pose a challenge for ADAS and AD systems because these areas are not fully visible to the ADAS and AD systems. The second environmental sensing provided by the cloud platform gives these systems an improved perspective on critical locations, enabling them to react more effectively in hazardous situations. This enhances the safety of automated mobility.

[0046] According to another aspect of the invention, the environmental sensing devices include a cooling device for circulating a cooling medium within the housing. The cooling medium is a liquid, for example water, or a gas, for example air. The cooling device is, for example, a pump or a fan. According to another aspect of the invention, cooling is achieved using active and / or passive cooling methods. This, for example, increases the reliability of the sensors in hot weather.

[0047] According to another aspect of the invention, the environmental monitoring devices include integrated cleaning devices for the sensors and / or the computers. This allows for the removal of contamination. Alternatively or additionally, contamination can be removed by service personnel.

[0048] According to another aspect of the invention, the environmental monitoring devices include means for online calibration. For example, the sensors, sensor parameters, and / or the computers are calibrated online. Online calibration means that components and / or parameters of the environmental monitoring devices are continuously checked, particularly during operation of the environmental monitoring devices. According to another aspect of the invention, the environmental monitoring devices include an interface, for example, a radio interface, to a calibration device that performs the calibration.

[0049] According to another aspect of the invention, the environmental detection devices have the form of a cylinder. For example, the environmental detection devices are column-shaped. For example, the environmental detection columns are similar in shape, geometry, and construction to or corresponding with the traffic monitoring device as disclosed in EP 2 818 922 B1, or similar to or corresponding with speed camera columns.

[0050] According to another aspect of the invention, the environmental monitoring devices are designed for stationary or mobile operation. For example, the environmental monitoring devices can be mounted on a trailer.

[0051] According to another aspect of the method according to the invention, the main computer executes the second computer program. The second computer program comprises the second machine learning algorithm. The second machine learning algorithm causes the main computer to synchronize and / or combine the data received from the environmental sensing devices to generate predictions, including predictions of movement patterns of automated mobility participants and traffic events, and / or to generate risk models, when the second computer program is running on the main computer. The second machine learning algorithm is trained on the data received from the environmental sensing devices in the cloud.

[0052] During training, the second machine learning algorithm configures its architecture to minimize a cost function specific to the training and the data. This optimizes the inference of the second machine learning algorithm. For example, the second machine learning algorithm is an artificial neural network that, in a supervised learning process, adjusts the weights of neuronal connections gradient-based in forward and backward feeding.

[0053] Training on data obtained from environmental monitoring devices has the advantage that the second machine learning algorithm is trained on a large amount of data, particularly long-term data. The data from each environmental monitoring device represents a specific, for example, stationary, field of view of the automated mobility and collects data over extended periods. This improves the learning of spatial and / or temporal relationships. Cloud-based training has the advantage that, apart from the computer network provided by the cloud platform, the computers of the individual environmental monitoring devices, and the main computer, no additional hardware is required for data collection, processing, and analysis.

[0054] The invention is explained using the figures Fig. 1: Exemplary embodiment of a cloud platform according to the invention, Fig. 2: Example of a critical situation, Fig. 3: further embodiment of a cloud platform according to the invention, Fig. 4: Example of an environmental perception and Fig. 5: Schematic representation of a method according to the invention, with examples shown.

[0055] In the figures, identical reference symbols denote identical or functionally similar reference parts. Only the reference parts relevant to the respective understanding are shown in the figures.

[0056] The in Fig. 1 environmental detection devices shown 10 are designed as environmental monitoring columns. They are housed in a casing. 11 the environmental detection devices 10 Each unit contains a sensor set for environmental monitoring. The sensor set includes, for example, camera sensors. 12 and lidar sensors 13 The sensors 12 and 13are behind openings 14 of the case 11 arranged for environmental monitoring. A computer 15 receives the signals from the sensors as input. 12 and 13 The computer outputs... 15 Lists of recorded objects 16 with their respective states of motion to a main computer 21 a cloud platform 20 Ready. The environmental monitoring devices 10 include an interface 17 , via which the lists of recorded objects 16 the main computer 21 be provided. The interface 17 For example, a radio interface. Via the interface 17 can the environmental monitoring pillars 10 also directly with participants in automated mobility, for example vehicles 18 , communicate, for example via V2V or V2X communication. The cloud platform 20 includes an interface17a , to obtain data from the environmental monitoring devices 10 to obtain.

[0057] The cloud platform includes several environmental monitoring devices. 10 , which are distributed throughout a transport infrastructure, either stationary or mobile. The environmental monitoring devices 10 are, for example, located at critical points, such as complex intersections like in Fig. 3. Furthermore, the cloud platform includes 20 the main computer 21 , a cloud-based storage 22 The computers 15 the environmental detection devices 10 form with the main computer 21 a cloud-based computer network.

[0058] The storage 22 the cloud platform 20 includes program instructions that cause the computers to 15 based on the signals from the sensors 12 , 13 the objects 16detect, classify, locate and / or track and report the detections, locations and / or tracking to the main computer 21 provide as data. The storage 22 It also includes program instructions that cause the main computer to 21 the computer data 15 synchronized and / or combined and from individual recordings of the respective environmental recording devices 10 a data enrichment for the second environmental survey is received.

[0059] The cloud platform 20 It also includes a second interface 23 . Via the second interface 23 The cloud platform 20 the second environmental assessment of participants in automated mobility, for example vehicles 18 , ready. The second interface 23 For example, it's a wireless interface designed for V2V or V2X communication. That means the cloud platform...20 Transmits data according to a protocol for V2V or V2X communication.

[0060] Fig. Figure 2 shows an example of a critical situation. The pedestrian 19 intends to cross a roadway. The pedestrian 19 is from the perspective of the approaching vehicle 18 from the pedestrian immediately next to 19 The parked vehicle was obscured. The driver of the vehicle 18 sees the pedestrian 19 No. The pedestrian is also not detectable by the vehicle's ADAS or AD sensors. 18 According to the invention, the pedestrian 19 from an environmental monitoring device installed at the zebra crossing 10 detected. Via the cloud platform 20 The vehicle receives 18 the information that the pedestrian is behind the parked vehicle 19 is located, who could cross the road.

[0061] At the in Fig. 3 cloud platforms shown 20 For automated mobility, for example, four environmental detection devices are needed. 10 Located at an urban intersection with several pedestrian crossings. The data from the environmental monitoring devices. 10 are in the main computer 21 merged. Via the second interface 23 The cloud platform communicates 20 with autonomous vehicles 18 and with pedestrians 19 The autonomous vehicles 18 include V2X control units and corresponding interfaces to connect to the cloud platform 20 and / or the environmental detection devices 10 to communicate. This enables the autonomous vehicles to 18 of all moving and static objects in the environment 16 Information about the respective states of motion of these objects 16 from expanded fields of vision.

[0062] Fig. Figure 4 shows an environmental perception of an environmental detection device. 10 , which connect to the cloud platform 20 is transferred to connect environmental perceptions with other environmental detection devices. 10 to provide improved environmental perception. Various objects were classified into specific object classes. For example, vehicles, pedestrians, cyclists, signs, trees, roadway, footpath, and road markings were recognized.

[0063] The process steps V1, namely detecting, classifying, locating and / or tracking objects in their respective environments, are carried out by environmental detection devices distributed throughout a transport infrastructure. 10 , V2, namely providing the received data to the main computer 21 the cloud platform 20 , V3, namely synchronizing and / or combining the data in the main computer 21, V4, namely obtaining a second environmental data set from the individual data sets of the respective environmental data set devices, and V5, namely providing the second environmental data set to the automated mobility system, are in a process flow in Fig. 5 shown. Reference symbol list 10 Environmental detection device 11 cases 12 Sensor 13 Sensor 14 openings 15 computers 16 objects 17 Interface 17a interface 18 vehicles 19 pedestrians 20 Cloud Platform 21 main computers 22 storage 23 second interface QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] EP 2818922 B1 [0002, 0049] DE 102019210933

[0003]

Claims

[1] Cloud platform (20) for automated mobility (16, 18, 19) comprehensive • a computer network of environmental monitoring devices (10), wherein the environmental monitoring devices (10) each comprise ◯ a housing (11), ◯ Sensors (12, 13) arranged in the housing (11), comprising optical sensors, sound sensors, radar sensors or a combination of the aforementioned sensors (12, 13), wherein the housing (11) comprises openings (14) for the sensors (12, 13) for sensing the environment, ◯ a computer (15) arranged in the housing (11) and connected to the sensors (12, 13) to transmit a signal and ◯ a first interface to the cloud platform, • a main computer (21), • a storage (22), ◯ in which the first instructions of a first computer program are stored, wherein the first instructions cause the computers (15) of the environmental sensing devices (10) to detect, classify, locate and / or track objects (16) from the respective environment based on the signals of the sensors (12, 13), and the computers (15) to provide the detections, localizations and / or tracking to the main computer (21) as data via the respective first interfaces of the environmental sensing devices (10) when the first computer program is running on the computers (15), and ◯ in which second instructions of a second computer program are stored, wherein the second instructions cause the main computer (21) to synchronize and / or combine the data of the computers (15) of the environmental monitoring devices (10) and to obtain a data enrichment for the second environmental monitoring from individual recordings of the respective environmental monitoring devices (10) when the second computer program is running on the main computer (21), and • a second interface (23) to provide the second environmental sensing of automated mobility (18, 19). [2] Cloud platform (20) according to claim 1, wherein the second computer program comprises instructions that cause the main computer (21) to generate forecasts, including forecasts of movement patterns of participants in automated mobility (18, 19) and of traffic events, based on the data from the computers (15) of the environmental sensing devices (10), when the second computer program is running on the main computer (21). [3] Cloud platform (20) according to claim 1 or 2, wherein the second computer program comprises instructions that cause the main computer (21) to create risk models for automated mobility (18, 19) based on the data from the computers (15) of the environmental sensing devices (10) when the second computer program is running on the main computer (21). [4] Cloud platform (20) according to one of claims 1 to 3, wherein the first computer program comprises instructions that cause the computers (15) of the environmental sensing devices (10) to fuse and / or validate the signals of the sensors (12, 13) when the first computer program is running on the computers (15). [5] Cloud platform (20) according to any one of claims 1 to 4, wherein the first computer program comprises a first machine learning algorithm and the computers (15) of the environmental sensing devices (10) execute the first machine learning algorithm for classifying, localizing and / or tracking the objects (16) when the first computer program is running on the computers (15), and / or the second computer program comprises a second machine learning algorithm and the main computer (21) executes the second machine learning algorithm for synchronizing and / or combining the data of the computers (15), for generating forecasts including forecasts of movement patterns of participants in automated mobility (18, 19) and of traffic events and / or for generating risk models when the second computer program is running on the main computer (21). [6] Cloud platform (20) according to any one of claims 1 to 5, wherein the first computer program comprises instructions that cause the computers (15) of the environmental sensing devices (10) to bundle, associate, time track and / or process the objects (16) from images and / or point clouds of the sensors (12, 13) when the first computer program is running on the computers (15). [7] Cloud platform (20) according to any one of claims 1 to 6, wherein the second interface (23) is a radio interface by means of which the cloud platform (20) communicates with automated vehicles (18) and / or an automated mobility infrastructure (18, 19). [8] Cloud platform (20) according to any one of claims 1 to 7, wherein the environmental sensing devices (10) are arranged on university, trade fair, airport, urban sites, at intersections, at roundabouts, at road crossings, at schools, in residential areas and / or at construction sites to provide automated vehicles (18) via the cloud platform (20) with a list of objects (16) including their movement state before the automated vehicles (18) reach the environmental sensing devices (10). [9] Computer-implemented method for providing cloud-based data enrichment for improved environmental sensing to automated mobility (18, 19) comprising the steps • Detecting, classifying, locating and / or tracking objects (16) in their respective environments by environmental detection devices (10) distributed throughout a transport infrastructure (V1), • Providing the received data to a main computer (21) of a cloud platform (20) (V2), • Synchronizing and / or combining the data in the main computer (21) (V3), • Obtaining a second environmental survey from the individual surveys of the respective environmental survey devices (10) (V4) and • Providing the second environmental sensing to automated mobility (18, 19) (V5). [10] Computer-implemented method according to claim 9, wherein a cloud platform (20) according to any one of claims 1 to 8 is used to carry out the method. [11] Computer-implemented method according to claim 9 or 10, wherein the main computer (21) executes a second computer program comprising a second machine learning algorithm which causes the main computer (21) to synchronize and / or combine the data obtained from the environmental sensing devices (10) for the purpose of generating forecasts, including forecasts of movement patterns of participants in automated mobility and of traffic events, and / or for generating risk models when the second computer program is running on the main computer (21), wherein the second machine learning algorithm is trained on the data obtained from the environmental sensing devices (10) in the cloud.

Images