Autonomous vehicle steering based on clearance process

Abstract

The present invention relates to a method for confirming a step-by-step clearance of a vehicle (12) for automatically steering the vehicle along a specified driving route (38) from a current vehicle position to a destination. The vehicle (12) has a sensor system (20, 22) with at least one optical camera (20) for monitoring an ambient environment (24) in a driving direction (26) of the vehicle (12). The method includes the steps of providing a current video image (40) from the optical camera (20) at the current vehicle position, confirming a clearance for automatically steering the vehicle (12) along the specified driving route (38) based on an automatic comparison of image information (54) from a monitoring area (46) of the video image (40) with reference image information (66) from the monitoring area (46) based on a previously captured comparison image, and outputting the confirmed clearance for automatically steering the vehicle (12) along the specified driving route (38) at a step corresponding to the current video image (40). The present invention also relates to a method for clearance-based automatic steering of a vehicle (12).

Description

[Technical Field]

[0001] The present invention relates to a method for determining a stepwise clearance of a vehicle for automatic steering along a predetermined driving route from a current vehicle position to a destination point, wherein the vehicle includes a sensor system having at least one optical camera for monitoring the surrounding environment in the direction of travel of the vehicle.

[0002] Furthermore, the present invention relates to a method for clearance-based automatic steering of a vehicle along a predetermined driving route from a current vehicle position to a destination point, wherein the vehicle includes a sensor system having at least one optical camera for monitoring the surrounding environment in the vehicle's direction of travel, and the method includes determining a step-by-step clearance for automatic steering of the vehicle along the predetermined driving route using the method.

[0003] The invention also relates to a driver assistance system for a vehicle, which driver assistance system is designed to carry out the above method.

[0004] The present invention also relates to a cloud-based driver assistance system having at least one vehicle and a cloud-based clearance server, wherein the at least one vehicle and the cloud-based clearance server are connected to each other via a data connection, and the cloud-based driver assistance system is designed to perform the above method. [Background technology]

[0005] Various types of assistance systems are already used in modern vehicles to make driving safer and more relaxing. These include emergency braking systems, lane keeping systems, distance control systems or intersection congestion warning systems, to name just a few. Such assistance systems are partly subsumed under the term ADAS (Advanced Driver Assistance Systems).

[0006] Additionally, applications in the field of "autonomous driving" are continuously being improved to further rest the vehicle driver, leading to a point where the vehicle drives fully autonomously and the vehicle driver is no longer required in the traditional sense. The long-term goal is to achieve Level 5 autonomous driving in the sense of SAE standard J3016, which relates to autonomous driving without intervention by the vehicle driver. Currently, the first Level 4 systems for specific applications are being prepared for release.

[0007] For all of these applications, it is important to detect the vehicle's surroundings as reliably as possible. This applies to both static and dynamic objects. In particular, the detection of dynamic objects is typically based on the vehicle's own detection of the surroundings using a corresponding sensor system. It will be considered here that any type of surroundings sensor used, such as an optical camera, a LiDAR-based surroundings sensor, a radar sensor, or even an ultrasonic sensor, has certain advantages and disadvantages that affect the detection of the surroundings. Thus, optical cameras are distinguished by high resolution and long range, enabling reliable identification of objects in the vehicle's surroundings. However, they have weaknesses in determining the distance to objects, and their performance can deteriorate in certain environmental conditions, such as fog or rain. In contrast, LiDAR-based surroundings sensors are very reliable in determining the distance to objects and are highly resistant to various environmental conditions, but are relatively expensive and have low horizontal angular resolution. Reliable identification of objects in the vehicle's surroundings is also currently impossible. The description of LiDAR-based surroundings sensors applies equally to radar sensors. And although ultrasonic sensors are only suitable for short distances and have very little directional characteristics, they are particularly cost-effectively available and are already widespread in current vehicles. Affordability, among other things, is important for the mass distribution of driver assistance systems, which currently precludes the use of LiDAR-based ambient environment sensors.

[0008] Sensor systems with various types of ambient environment sensors that can be provided cost-effectively overall are already known in the prior art for use in various assistance systems. In this case, for example, an optical front camera mounted behind the vehicle's windshield is combined with ultrasonic sensors mounted along the vehicle's sides. However, this solution is also typically insufficient to provide the desired performance and reliability for autonomous driving, for example, according to Level 4. The performance of the sensor system for ambient environment detection can be increased by using an increasing number of ambient environment sensors, for example, multiple optical cameras. However, such sensor systems are not suitable for current mass-produced vehicles due to the cost of the sensor system and the additional demands for processing the sensor data provided by these ambient environment sensors.

[0009] This also applies to applications in essentially known surroundings. Thus, for example, automated parking systems are known in which a vehicle independently drives itself to a parking space, for example, within a property. Such systems currently require monitoring by the vehicle driver. This monitoring can also occur from outside the vehicle. This corresponds to autonomous driving according to Level 2. However, autonomous driving in the sense of Level 4 cannot yet be implemented in this manner. This also applies to current valet parking systems. Techniques exist for improving the performance of ambient environment detection by using additional external ambient environment sensors. Such external ambient environment sensors are part of the infrastructure within the area in which the vehicle is operated. However, these external ambient environment sensors are also associated with high additional costs. This can also occur only with the consent of the infrastructure owner, e.g., the property owner, and is difficult to implement, especially in public areas. In addition, liability issues between the vehicle manufacturer and the provider of the external ambient environment sensors are difficult to clarify.

[0010] To simplify the requirements for data processing within the vehicle, it is possible in principle to perform autopilot in a cloud-based manner, where sensor information is transmitted from the vehicle's environment detection sensors to a cloud server and processed there. However, this is not practical due to the amount of data that needs to be transmitted.

[0011] Therefore, in practice, in a vehicle automatic steering, it is typical for the vehicle driver to monitor the maneuver and provide clearance so that the vehicle can continue to steer. This can occur, for example, in the form of continuous confirmation by the vehicle driver, with the maneuver being stopped in the absence of confirmation. Alternatively or additionally, the vehicle driver can check the information provided for the automatic steering, for example, before or during the maneuver. The vehicle driver can also provide clearance from outside the vehicle, but this requires that the vehicle driver is at least located near the vehicle. Summary of the Invention [Problem to be solved by the invention]

[0012] Proceeding from the above-mentioned prior art, the present invention is therefore based on the object of specifying a method for cloud-based automated steering of a vehicle along a predetermined driving route, a method for clearance-based automated steering of a vehicle along a predetermined driving route, a driver assistance system for a vehicle performing one of the above methods, and a corresponding cloud-based driver assistance system that enables efficient cloud-based steering of a vehicle with a high level of reliability, in particular for limited applications such as autonomous parking of a vehicle.

[0013] This object is achieved according to the invention by the features of the independent claims. Advantageous embodiments of the invention are specified in the dependent claims. [Means for solving the problem]

[0014] According to the present invention, therefore, a method is specified for determining a step-by-step clearance of a vehicle for automatic steering along a predetermined driving route from a current vehicle position to a destination point, wherein the vehicle includes a sensor system having at least one optical camera for monitoring the surrounding environment in the vehicle's direction of travel, the method comprising the steps of providing a current video image of the optical camera at the current vehicle position, determining a clearance for automatic steering of the vehicle along the predetermined driving route based on an automatic comparison of an item of image information from a control area of ​​the video image with an item of reference image information from the control area derived from a previously recorded comparison image, and outputting the determined clearance for automatic steering of the vehicle along the predetermined driving route for the step corresponding to the current video image.

[0015] The present invention further specifies a method for clearance-based automatic steering of a vehicle along a predetermined driving route from a current vehicle position to a destination point, wherein the vehicle includes a sensor system having at least one optical camera for monitoring the surrounding environment in the vehicle's direction of travel, and the method includes determining a step-by-step clearance for automatic steering of the vehicle along the predetermined driving route using the above method.

[0016] According to the invention, a driver assistance system for a vehicle is also specified, which driver assistance system is designed to carry out the above method.

[0017] According to the present invention, a cloud-based driver assistance system is also specified having at least one vehicle and a cloud-based clearance server, the at least one vehicle and the cloud-based clearance server being connected to each other via a data connection, and the cloud-based driver assistance system is designed to perform the above method.

[0018] The basic concept of the present invention is therefore to enable a stepped clearance of a vehicle for automatic steering of the vehicle along a predetermined driving route derived from the vehicle's sensor system, where monitoring of the vehicle's surrounding environment in the direction of travel is performed based on an optical camera. The stepped clearance can be automatically generated so that clearance-based automatic steering can be performed. The clearance can be generated based on an automatic comparison of items of image information from a control area of ​​a video image with corresponding items of reference image information from the control area. These clearances can therefore be used, for example, like a typical clearance previously generated by a vehicle driver to enable automatic steering of the vehicle. The vehicle can therefore be automatically steered within or through the control area. By repeatedly executing the method, repeated checks of the driving route to the destination can occur, and the vehicle can be automatically steered to the destination.

[0019] Communication between the vehicle and the cloud-based clearance server via the data connection can be performed particularly efficiently here, since the amount of data to be transmitted is reduced due to the observation of only the control area. The transmission of items of image information to the clearance server is limited to the control area and can therefore be accelerated due to the reduced amount of data to be transmitted. Processing of the corresponding items of image information can also be performed in a short time due to the observation of only the control area. As a result, low latency times can be achieved in the overall data processing, from providing the current video image to receiving the clearance at the vehicle.

[0020] In particular, in certain applications, a driving route having a sufficient length can therefore be detected and cleared, so that autonomous driving, for example, in the sense of Level 4, can be achieved. This also applies, for example, to applications for autonomous parking of a vehicle, where the driving route is predetermined or pre-determined, for example, so that only a small area of ​​the surrounding environment is monitored. A low driving speed can also be predetermined in such applications, so that the control area only changes slowly along the driving route.

[0021] The method can be particularly advantageously implemented in current applications of autonomous driving according to, for example, level 2, which require monitoring by a vehicle driver. Confirmation of the vehicle driver's monitoring by a corresponding clearance for moving the vehicle along a predetermined driving route can here be replaced by an automatically determined clearance. Thus, existing driver assistance systems for autonomous driving according to level 2 can easily be extended for autonomous driving according to, for example, level 4, without greater intervention being required for the corresponding application for autonomous driving. It is only necessary to additionally implement the described method for determining a staged clearance and use receipt of a clearance for moving the vehicle along a predetermined driving route from a clearance server as confirmation of the vehicle driver's monitoring.

[0022] The vehicle may be any vehicle designed for automated or autonomous operation.

[0023] Autopilot refers to at least partially autonomous movement of a vehicle. For this purpose, a driving route may be predetermined, for example, by a journey along a driving route by a human vehicle driver, so that the driving route can be learned. The predetermined driving route connects the current vehicle position and a destination point. The predetermined driving route specifies a route course for reaching the destination position. Due to the predetermined driving route, items of reference image information are available for comparison with image information of the current video image.

[0024] The sensor system includes at least one optical camera for monitoring the vehicle's surroundings in the direction of travel. The optical camera can be mounted, for example, behind the vehicle's windshield, to monitor the surroundings in front of the vehicle, which corresponds to typical forward driving. Therefore, an optical camera oriented toward the rear of the vehicle is needed in the case of reversing driving. Such an optical camera typically has a field of view of 90° or 120°, up to 180°, so as to be able to detect at least a medium-range nearby driving route, for example, up to several tens of meters.

[0025] The sensor system may include additional ambient environment detection sensors, for example ultrasonic sensors, to perform further monitoring of the vehicle's ambient environment. Reliability when maneuvering the vehicle may thus be additionally improved.

[0026] The provision of a current video image of an optical camera at the current vehicle position corresponds to the provision of an image or a sequence of images in the form of a video. The current video image may be coded in any manner or may exist in RAW format. The properties of the provided current video image are essentially determined by the properties of the optical camera.

[0027] Determining clearance for automated steering of a vehicle along a predetermined driving route based on an automatic comparison of an item of image information from a control area with an item of reference image information from the control area derived from a previously recorded comparison image allows differences between the items of image information of the two control areas to be automatically identified. The item of reference image information for the control area represents the vehicle without any obstacles relevant to it. An implicit clearance is thus assumed for the item of reference image information. If there is a deviation of the received item of image information from the item of reference image information, the presence of an obstacle can be inferred, and as a result, clearance is not generated. Nevertheless, the type of deviation can be determined so that clearance can be granted in the event of a deviation. Because the control area is defined by the driving route, the automatic comparison can determine whether the vehicle can be driven through the control area.

[0028] The items of reference image information may be stored in the vehicle or in a cloud-based clearance server, depending on the design of the method. The method may be performed in the vehicle, e.g., essentially autonomously, i.e., the items of reference image information are stored in the vehicle. The items of reference image information may also be generated by the respective vehicle itself. Alternatively, the items of reference image information may be transmitted from a cloud-based clearance server to the vehicle and permanently provided there. Alternatively, the items of reference image information may be dynamically provided by the cloud-based clearance server, e.g., depending on the location of the vehicle. Parts of the method may also be performed in the cloud-based clearance server. For this purpose, for example, the items of image information may be transmitted from the vehicle to the cloud-based clearance server.

[0029] The control area corresponds to an area in which clearance is determined by automatic comparison of an item of image information of a video image with an item of reference image information of a comparison image. The control area may include the entire video image or only a section thereof. Different areas of the current video image may form the control area here. The control area may also vary, for example, depending on driving parameters, and for example, the control area may be selected to be different sizes at different speeds.

[0030] The clearance for automatically steering the vehicle along the predetermined driving route represents the clearance for traveling within the observed control area. Thus, the vehicle may automatically steer within or through the control area. Repeated checks of the entire driving path to the destination may occur due to repeated execution of the method.

[0031] Clearance-based autopiloting of a vehicle along a predetermined driving route according to a received clearance relates to autonomous driving of a vehicle. The vehicle may perform lateral and longitudinal control here. Autopiloting of the vehicle is performed here based on a determined clearance, and the vehicle uses the determined clearance to autopilot along the predetermined driving route. The clearance is granted incrementally according to each video image provided by an optical camera.

[0032] The method may, on the one hand, be performed in the vehicle itself, i.e., in the driver assistance system. Alternatively, the method may be performed using a cloud-based driver assistance system formed by at least one vehicle together with a cloud-based clearance server. Parts of the method are thus performed in the respective vehicle, while other steps of the method are performed by a clearance server located in the cloud.

[0033] A cloud-based clearance server is essentially any server that is connected to corresponding vehicles via a data connection. It is only important that the clearance server executes the necessary method steps assigned to it. The clearance server typically includes processing means, storage means, and communication means, the communication means being used to establish the data connection.

[0034] The data connection is used to transmit data, in this case, items of image information and clearances from the control area, between at least one vehicle and the clearance server. Thus, the data connection is essentially any communication connection between the vehicle and the clearance server, which may include essentially any combination of transmission media and protocols. It represents a typical communication means and, as such, is not critical to the functioning of the cloud-based driver assistance system. Any data connection may be used here, and in the currently typical packet-oriented data transmission, any transmission path may be used for data transmission without the at least one vehicle or clearance server having any influence over the data transmission. Such influence typically cannot be exercised.

[0035] A video image may be, for example, a single image, also known as a frame, recorded and provided by an optical camera. Alternatively, a video image may be a moving image based on a sequence of single frames, i.e., a video sequence is observed.

[0036] The item of image information may include pixels of the current video image in the control area, and thus a portion of the currently provided video image. Alternatively or additionally, the item of image information may include or be based on pixels of the current video image in the control area that have been processed or pre-processed in any manner. In this manner, for example, relevant portions of the current video image in the control area may be determined and compared. The item of image information may include information, for example semantic information, regarding the content of the current video image in the control area.

[0037] In an advantageous design of the invention, the method includes a step of matching an item of image information from a control area with an item of reference image information from the control area. In practice, localization errors may occur or the localization of the vehicle may not be sufficiently accurate, which may hinder or even make impossible the comparison of the received item of image information with the item of reference image information for each current control area. The localization of the vehicle may be improved by matching the items of image information. Thus, the items of image information may be reliably compared by matching the items of image information, and the occurrence of errors may be reduced. The matching of the items of image information with the items of reference image information occurs before the determination of clearance based on the automatic comparison of the received item of image information with the item of reference image information. Such a method is also known by the name "visual odometry."

[0038] In an advantageous embodiment of the invention, matching an item of image information from a control area with an item of reference image information from the control area comprises determining a current vehicle position. Matching the item of image information with the item of reference image information can be efficiently performed based on the vehicle position.

[0039] In an advantageous embodiment of the present invention, determining the current vehicle position includes determining the current vehicle position based on receiving signals from a satellite navigation system, particularly using a differential global positioning system, and / or providing vehicle odometry information, and / or identifying landmarks along the vehicle's predetermined driving route, and / or determining the current vehicle position using a system for visual simultaneous positioning and mapping. Various of the above methods for determining the vehicle's position can be used alone or in combination to determine the current vehicle position. This relates to both the vehicle currently being steered or steered and providing items of reference image information. The use of signals from satellite navigation systems is widespread, particularly for vehicle navigation. The use of a differential global positioning system can increase the accuracy of position determination from an accuracy of a few meters to an accuracy of less than half a meter. Vehicle odometry information can also be provided by odometry sensors, for example, by wheel rotation sensors (wheel tics) and by a sensor for detecting the current steering angle. The odometry information allows for highly accurate determination of the vehicle's position change. Because odometry sensors typically have short measurement cycles and are therefore faster than, for example, satellite navigation systems, the vehicle position can be determined particularly reliably by combining both types of positioning. Landmarks can contribute to the precise determination of the vehicle's position and improve the positioning. For example, Aruco codes are known as landmarks. Visual simultaneous localization and mapping, also known as "visual simultaneous localization and mapping" (V-SLAM), is based on positioning derived from time-offset camera images, which allows the vehicle's position changes to be determined.

[0040] In an advantageous embodiment of the present invention, providing vehicle odometry information includes providing visual odometry information for the vehicle based on ground structures and / or providing sensor signals from at least one odometry sensor of the vehicle. Ground structures, such as pavement with specific stones, can be identified in high detail, especially over short distances, enabling reliable position determination based on the shape of those structures. High accuracy in position determination can be achieved based on typical structure sizes of paving stones, for example, in the range of approximately 10-20 cm. Feature movement, e.g., pixel movement, can be reliably detected based on the ground structures, i.e., pixel flow is detected. Deviations in pixel flow represent relief differences and can be evaluated very reliably. Pixel flow is robust to typical error sources, such as color, lighting, or ground moisture.

[0041] In an advantageous embodiment of the invention, if clearance is lacking, the method includes transmitting an item of image information from the control area and an item of reference image information from the control area to an operator to determine clearance. Thus, even if there is an error in determining the clearance, i.e., if the clearance is not automatically granted based on a comparison of the item of image information with the item of reference image information from the control area, clearance can still occur if the operator determines that the vehicle can be maneuvered within the control area. Therefore, vehicle driver intervention is not required. The operator can, in principle, be located anywhere. For example, the operator can be directly connected to the clearance server or can determine the clearance "remotely." However, in principle, it may be sufficient if the vehicle occupant acts as the operator. Thus, for example, the item of image information from the control area and the item of reference image information from the control area can be transmitted to a user interface of the vehicle, and the operator can check and grant the clearance via the user interface.

[0042] In an advantageous embodiment of the invention, the method includes, upon clearance by the operator, adapting an item of reference image information based on an item of image information from the control area. Thus, when clearance is performed by the operator, the system can be adapted by adapting the item of reference image information so that clearance can occur automatically within the observed control area upon a new comparison of the item of image information with the item of reference image information. A machine learning process preferably takes place. If the items of reference image information are stored on a cloud-based clearance server, all connected vehicles can jointly contribute to improving and adapting the items of reference image information.

[0043] In an advantageous embodiment of the present invention, the method includes a step of driving along a driving route to learn the driving route, in particular as a trajectory, including providing video images of an optical camera along the driven driving route and saving items of image information from a control area as items of reference image information from the control area, or at least details of the video images as comparison images. The driving along the driving route, and thus the learning of the driving route, can be performed using any vehicle. As soon as the driving route is driven for the first time, the items of reference image information are ready for the corresponding vehicle. If a clearance server is used, the items of reference image information can also be provided to other vehicles so that the method for determining stepwise clearances and for automatically steering along the predetermined driving route can be performed by all vehicles.

[0044] The more frequently a driving route is driven, the better the quality of the items of reference image information, and the more frequently and reliably automatic checks and clearances can be performed. In addition to the driving route, the trajectory includes motion information for maneuvering, such as speed or acceleration.

[0045] In an advantageous embodiment of the invention, the method includes transmitting an item of image information from the controlled area as an item of reference image information from the controlled area, or at least details of the video image as a comparison image, to a cloud-based clearance server. The items of reference image information are thus stored on the cloud-based clearance server so that they may be provided to various vehicles. The items of reference image information may also be provided collectively by various vehicles in the cloud-based clearance server. The items of reference image information may be items of image information forwarded by vehicles to the cloud-based clearance server, or the cloud-based clearance server may perform processing of the transmitted items of image information, for example, to determine the items of reference image information from the comparison image.

[0046] Items of image information may be stored as items of reference image information, and / or at least details of the video images may be stored as comparison images, so that these comparison images (details thereof) can be used as items of image information, or items of image information can be generated therefrom in any desired format at any time.

[0047] In an advantageous embodiment of the invention, the method comprises determining an item of image information from a control area based on a flow of image elements, in particular pixels, based on at least two single video images from an optical camera. Such an item of image information can generate a three-dimensionality of the item of image information that improves the clearance determination based on an automatic comparison of the item of image information from the control area with a reference item of image information from the control area. The principle is based on image elements at different distances that undergo different relative position changes in relation to the optical camera when the position of the vehicle changes.

[0048] In an advantageous embodiment of the present invention, the method includes a step for identifying a driving path within a current video image and a step for identifying a control area along the driving path, particularly within the driving path, depending on the vehicle's driving route and / or driving parameters. The identification of the driving path within the current video image can be performed depending on the vehicle's position and alignment, for example, in the form of a projection or overlay of the driving path within the current video image. The driving path can be defined as a single line, particularly as a center line, or with lateral boundaries. The driving path can preferably be identified depending on the driving route within the current video image, so that on the one hand the amount of data to be processed can be reduced and on the other hand the driving path ensures that control areas are selected in a manner that is relevant to the vehicle's operation. Processing of data that is not relevant to the vehicle's operation is omitted. Therefore, depending on the design of the method, the data transmission requirements between the corresponding vehicle and the cloud-based clearance server can be further reduced.

[0049] The identification of the control area along the driving corridor can be performed, for example, using a window having predetermined dimensions and a predetermined position associated with an optical camera. Thus, the window can be moved along the driving corridor as the vehicle moves. In a driving corridor defined by horizontal boundaries, the control area can be identified between these boundaries and two distance limits. The identification of the control area can be performed according to the driving route, so that the control area can change accordingly with the driving route, i.e., different areas of the current video image can be identified as control areas. The identification of the control area can be performed according to the driving parameters of the vehicle, so that the control area can change depending on the driving parameters. Thus, the control area can be selected to be different sizes, for example, at different speeds.

[0050] In an advantageous embodiment of the present invention, the method includes determining an item of brightness information of a current video image of the optical camera, and identifying a control area occurs by further taking into account the determined item of brightness information of the current video image, and / or the method includes determining an item of image information from the control area by taking into account the determined item of brightness information of the current video image. Due to the identification of the control area based on the determined brightness information, for example, the control area can be selected so that it contains only useful items of image information. Processing of the current video image of the optical camera can thus be simplified and therefore accelerated. Based on the determined brightness information of the current video image, for example, the item of image information from the control area can be partially filtered if there is no distinguishable image content due to insufficient light or overexposure. Brightness can also occur due to static lighting and also based on the light cone of the vehicle, the vehicle's own vehicle, or even a third-party vehicle. Identification of the control area based on the determined brightness information of the current video image may be performed directly in the vehicle, whereby the amount of data of the item of image information to be transmitted from the control area may be reduced during subsequent transmission to the cloud-based clearance server. Alternatively, after transmission of the item of image information from the control area to the clearance server, the control area may be adapted.

[0051] In an advantageous embodiment of the present invention, the method includes supplementing the item of image information from the control area and / or the item of reference image information from the control area with different weather conditions, particularly ground moisture, snow, or hail. Ground moisture may occur, for example, in the form of localized puddles. In addition, a wet underlying surface may have a different color than a dry underlying surface. In particular, uneven levels of moisture on the underlying surface may result in color patterns that can result in problems, such as incorrect identification of objects. This therefore applies to puddles. Color changes on the underlying surface may also result in overall problems, for example, due to the aforementioned moisture or snow or ice coverage. Due to the wide variety of potential different weather conditions and the resulting color changes, light differences may occur between the current video image and the comparison image, making the automatic comparison of the received item of image information with the item of reference image information sensitive to errors and causing problems when determining clearance, i.e., clearance cannot be easily granted. The neural network is preferably used to learn and apply the completion of the image information item or the reference image information item for different weather conditions. Training can be performed for the reference image information item based on multiple training data, i.e., various image information items as reference image information items. Complementation can be trained globally, i.e., independently of a specific control area, or individually for each of the different control areas. Complementation can be performed for the image information item from the control area directly in the vehicle, or in the case of transmission of the image information item to the clearance server, it can be performed or prompted by the clearance server after transmission. Thus, this applies, for example, to provided reference image information items from the control area, which can be based on image information items provided by various vehicles.

[0052] In an advantageous embodiment of the present invention, the method includes complementing an item of image information from a control region and / or an item of reference image information from a control region for shadows. Shadows can occur in sunlight or globally depending on the lighting conditions on any object, causing changes in the color of the underlying surface. Therefore, color patterns can occur, which can result in problems, such as incorrect identification of the object. Due to the various potential shapes of shadows and the resulting color changes, light differences can occur between the current video image and the comparison image. Due to this, automatic comparison of the received item of image information with the item of reference image information is typically sensitive to errors and can cause problems in determining clearance, i.e., clearance cannot be easily granted. A neural network is preferably used to learn and apply the complementation of the item of image information or the item of reference image information for shadows. Training can be performed for the item of reference image information based on multiple training data, i.e., various items of image information as items of reference image information. Complementation can be trained globally, i.e., independently of a specific control region, or individually for each different control region. Completion can be performed for an item of image information from a controlled area directly in the vehicle, or in the case of transmission of an item of image information to a clearance server, it can be performed or prompted by the clearance server after transmission. This therefore applies, for example, to provided reference image information items from a controlled area, which may be based on items of image information provided by various vehicles. Here, it is possible to distinguish between stationary and moving objects for shadow completion. In the case of stationary objects such as buildings, signs, or trees, compensation can be performed reliably even if the object moves, for example, due to wind. In the case of moving objects such as a passing person or a flying bird, compensation may be required because their shadows typically move outside the controlled area in a short period of time.

[0053] In an advantageous embodiment of the invention, the complementation of the item of image information from the control area and / or the item of reference image information from the control area with respect to shadows comprises a time-dependent complementation of the item of image information from the control area and / or the item of reference image information from the control area with respect to shadows. The local time determines the sun altitude information, from which the predicted shadows can be determined in detail. Thus, for example, different shadows based on the same object can be taken into account in the morning or evening. Together with the date, precise information is also generated regarding the sun altitude, e.g., the length of the shadow. In addition, the position information can be used to perform complementation of any position of the vehicle and, for example, to automatically determine the local time.

[0054] In an advantageous embodiment of the present invention, the method includes transmitting items of image information from a control area to a cloud-based clearance server and transmitting a determined clearance for moving the vehicle along a predetermined driving route from the cloud-based clearance server to the vehicle, where previously recorded comparison images are stored in the cloud-based clearance server, and determining the clearance includes determining the clearance within the cloud-based clearance server. The method is therefore performed using a cloud-based driver assistance system. The transmission of the items of image information from the control area to the clearance server occurs via a data connection in a fundamentally arbitrary manner. To ensure reliable operation of the respective vehicle, the data connection preferably allows for low-latency data transmission. The clearance determination is here outsourced to the cloud-based clearance server, and its resources can be used as a result. Therefore, fewer resources are to be stored in the vehicle. In particular, items of reference image information can advantageously be provided in the cloud-based clearance server, since in this manner transmission of the items of reference image information to the vehicle can be omitted. Only items of image information from the control area of ​​the video image will be transmitted to determine the clearance. Due to the transmission of the determined clearance for moving the vehicle along the predetermined driving route from the clearance server to the vehicle, the corresponding clearance for traveling along the observed control area can be used therein. The amount of data to be transmitted can be kept small by clever selection of the control area.

[0055] In an advantageous embodiment of the invention, the method includes compressing the item of image information from the controlled area before transmitting the item of image information to the cloud-based clearance server, and decompressing the item of image information from the controlled area after transmitting the item of image information to the cloud-based clearance server. Data transmission over the data connection can be further reduced by compressing the item of image information. Various methods for compressing items of image information are known per se and therefore need not be described in detail here.

[0056] In an advantageous embodiment of the invention, the items of reference image information are stored on a cloud-based clearance server, and the method includes receiving from the cloud-based clearance server items of reference image information from the control area derived from previously recorded comparison images, and the clearance determination includes determining the clearance in the vehicle. The items of reference image information are thus made available to all vehicles on the cloud-based clearance server and can be transmitted from the cloud-based clearance server to the vehicles. The items of reference image information can be transmitted "offline," i.e., independent of the vehicle's current maneuvering along the predetermined driving route. Alternatively, the items of reference image information for a driving route can be transmitted upon reaching a point on the driving route. Still alternatively, the items of reference image information can be transmitted individually for each clearance determination.

[0057] In an advantageous embodiment of the present invention, the method includes an additional step of requesting the transmission of items of reference image information from the cloud-based clearance server from the control area derived from a previously recorded comparison image. It can thus be ensured that only the items of reference image information are needed. For example, upon reaching the start of the driving route, items of reference image information for maneuvering the vehicle along the entire driving route can be requested. Alternatively, for each clearance, a corresponding item of reference image information can be requested from the cloud-based clearance server on a current basis. Furthermore, for example, reference information stored in the vehicle can be monitored in relation to their timeliness, for example, according to the storage date. If the stored reference information is no longer up-to-date, a request to transmit these items of reference image information can be sent to the cloud-based clearance server to replace the items of reference image information that no longer match the current items of reference image information. If the items of reference image information are stored locally in the vehicle, for example, a request can be sent if clearance cannot be granted upon automatic comparison of the items of image information for the control area with the items of reference image information. If a more recent item of reference image information exists, the comparison can therefore be performed again.

[0058] In an advantageous embodiment of the invention, the method includes compressing the items of reference image information from the controlled area before transmitting the items of reference image information from the cloud-based clearance server, and decompressing the items of reference image information received from the controlled area after receiving the items of reference image information from the cloud-based clearance server. Data transmission over the data connection can be further reduced by compressing the items of reference image information. Various methods for compressing items of image information are known per se and therefore need not be described in detail here. In order to save storage space, decompression of the received items of reference image information can be performed only if necessary, i.e., the items of reference image information received from the cloud-based clearance server are initially stored in a compressed state.

[0059] In an advantageous embodiment of the present invention, a method for clearance-based autopiloting of a vehicle is implemented based on control of the vehicle by an external server, control of the vehicle based on a method for visual automatic localization and mapping (V-SLAM), or control of the vehicle based on a method for automatic localization and mapping using at least one ambient environment sensor for generating a point cloud of the vehicle's surroundings, in particular at least one radar and / or LiDAR-based ambient environment sensor. Using the corresponding method, the vehicle can already actually autopilot along a predetermined driving route, so that extensions can be implemented without much effort using the method for determining the vehicle's incremental clearance for autopiloting and also using the method for clearance-based autopiloting of the vehicle. Depending on the method and its embodiment, the driving route or the corresponding trajectory can be determined within the vehicle, in particular in the corresponding driver assistance system, or on the server side.

[0060] In an advantageous embodiment of the present invention, a method for clearance-based automated steering of a vehicle along a predetermined driving route from a current vehicle position to a destination point is performed for automated valet parking or training parking using a previously learned trajectory for driving along the driving route. The corresponding application for the current vehicle is already partially used, so that an extension can be implemented without much effort using functions for determining the vehicle's step-by-step clearance for automated steering and also functions for clearance-based automated steering of the vehicle. The driving route or the corresponding trajectory can be determined within the vehicle, specifically the corresponding driver assistance system, or on the server side, depending on the specific application and its implementation.

[0061] In an advantageous embodiment of the present invention, the method includes receiving a vehicle position, particularly based on receiving Global Navigation Satellite System (GNSS) signals, and selecting a driving route based on the received vehicle position. The vehicle position is not used here for navigation, but rather to identify a driving route so that clearances can be determined along the driving route. Thus, for example, associated reference image information items can be loaded from a memory in the vehicle, and the reference image information items can be deployed, or the reference image information items for the driving route can be transmitted by a cloud-based clearance server where they are stored and received by the vehicle or driver assistance system. Currently, the systems NAVSTAR GPS (Global Positioning System), GLONASS (Global Navigation Satellite System), Galileo, and Beidou operate as global navigation satellite systems.

[0062] The features and advantages of the described method can be easily transferred to the described system, and vice versa. The individual steps of the method can also be performed in any order, essentially. The method is not limited to the sequence of method steps described as an example, as long as this is clear from the description for those skilled in the art.

[0063] The present invention will be described in more detail below based on preferred embodiments and with reference to the accompanying drawings. Each of the features shown may constitute an aspect of the present invention both individually and in combination. Features of different exemplary embodiments may be transferred from one exemplary embodiment to another. [Brief explanation of the drawings]

[0064] [Figure 1]1 is a schematic diagram of a cloud-based driver assistance system, here represented by way of example by a vehicle and a cloud-based clearance server, the vehicle and the cloud-based clearance server being connected to each other via a data connection, the vehicle including a sensor system having at least one optical camera for monitoring the surrounding environment in the direction of travel of the vehicle according to a first preferred embodiment. [Figure 2] FIG. 2 shows a schematic diagram of the functionality of the cloud-based driver assistance system from FIG. 1 for performing a method for automatically steering a vehicle along a predetermined driving route using a cloud-based clearance server, together with the driving route and control area. [Figure 3] 2 is a schematic diagram of the position determination of the vehicle from FIG. 1 based on various types of position determination; [Figure 4] FIG. 1 is a detailed schematic diagram of simultaneous visual localization based on different recognition pixels and mapping-based vehicle localization. [Figure 5] 1 is a schematic diagram of a control region in an exemplary current video image taking into account lighting conditions; [Figure 6] FIG. 10 is a further schematic diagram of a control region within an exemplary current video image taking into account lighting conditions. [Figure 7] FIG. 1 is a detailed schematic diagram of the effect of different weather conditions and resulting ground moisture resulting in different ground colorations. [Figure 8] FIG. 1 is a detailed schematic diagram of the effect of different weather conditions and resulting ground moisture resulting in different ground coloration in the case of uneven drying of the ground. [Figure 9] 1 is a schematic illustration of different weather conditions and the resulting effect of puddle formation. [Figure 10] FIG. 10 is a schematic diagram of the effect of a falling shadow resulting in a different ground coloration relative to the no-shadow area. [Figure 11] FIG. 10 is a further schematic illustration of the effect of a falling shadow resulting in a different ground coloration relative to the no-shadow area. [Figure 12]1 is a schematic diagram of an exemplary surrounding environment for a vehicle having a homogeneous smooth ground structure; [Figure 13] 1 is a schematic diagram of an exemplary vehicle environment having a rough, porous ground structure; [Figure 14] 1 is a schematic diagram of an exemplary surrounding environment of a vehicle having a ground structure with a first type of ground paving; [Figure 15] 2 is a schematic diagram of an exemplary vehicle environment having a ground structure with a second type of ground paving; [Figure 16] 2 is a flowchart of a method for determining a vehicle's step-by-step clearance for automatic steering along a predetermined driving route from a current vehicle position to a destination point, consistent with the cloud-based driver assistance system of the first embodiment of FIG. 1 . [Figure 17] FIG. 10 is a schematic diagram of an exemplary surrounding environment of a vehicle with a ground structure and pixel flow having a third type of ground paving. [Figure 18] 1 is a flowchart of a second embodiment of a method for determining vehicle step clearances for automated steering along a predetermined driving route from a current vehicle position to a destination point. [Figure 19] 10 is a flowchart of a third embodiment of a method for determining vehicle step clearances for automated steering along a predetermined driving route from a current vehicle position to a destination point. DETAILED DESCRIPTION OF THE INVENTION

[0065] FIG. 1 shows a cloud-based driver assistance system 10 according to a first preferred embodiment.

[0066] 1 and 2 with one vehicle 12, the cloud-based driver assistance system 10 may include additional vehicles 12 not shown. The cloud-based driver assistance system 10 further includes a cloud-based clearance server 14.

[0067] The vehicles 12 include an assistance system 16 for autonomously parking the vehicles 10, for example, for which a driving route 38 is predetermined or pre-determined. The assistance system 16 includes a control unit 18 that executes the method. The assistance system 16 additionally includes sensor systems 20, 22 having an optical camera 20 and a plurality of ultrasonic sensors 22 mounted on each vehicle 12 for monitoring a surrounding environment 24 of the vehicle 12. The optical camera 20 is mounted behind the windshield of the vehicle 12 for monitoring the surrounding environment 24 in a direction of travel 26 ahead of the vehicle 12. The control unit 18 and the sensor systems 20, 22 are connected to each other via a data bus 28. The data bus 28 may be embodied in accordance with standards typical in the automotive field, such as, for example, CAN, LIN, LON, or FlexRay.

[0068] Vehicle 12 includes a communications unit 30 for communicating with cloud-based clearance server 14. Cloud-based clearance server 14 includes corresponding communications means 32 for communicating with vehicle 12. Accordingly, a data connection may be established between vehicle 12 and cloud-based clearance server 14 via communications unit 30 and communications means 32. The data connection is essentially any communications connection between vehicle 12 and clearance server 14, which may include a dynamic combination of essentially any transmission media and protocols.

[0069] Cloud-based clearance server 14 is essentially an optional server connected to vehicles 12 via a data connection. Clearance server 14 includes processing means 34 and storage means 36 in addition to communication means 32 in a typical manner. Storage means 36 stores programs for execution by processing means 34.

[0070] The cloud-based driver assistance system 10 is designed to implement a method, described hereinafter, for cloud-based automated steering of the vehicle 12 along a predetermined driving route 38 from a current vehicle position to a destination point. The method is based on determining a step-by-step clearance of the vehicle 12 for automated steering along the predetermined driving route 38. Such a driving route 38 is shown, by way of example, in FIG. 2 .

[0071] The cloud-based driver assistance system 10 performs the method as trained parking using a previously learned trajectory for driving along the driving route 38, and control of the vehicle 12 is generally performed based on methods for visual automatic localization and mapping (V-SLAM).

[0072] The method begins at step S100 with providing a current video image 40 from the optical camera 20 at the current vehicle position. Such a video image 40 is shown by way of example in FIG.

[0073] The current video image 40 is provided here as a single image, also known as a frame, from the optical camera 20, e.g., recorded and provided by the optical camera 20. Alternatively, the video image 40 may be a moving image based on a sequence of single frames, i.e., a video sequence is observed. The current video image 40 may be encoded in any manner or provided in RAW format. The properties of the provided current video image 40 are dictated by the properties of the optical camera 20.

[0074] Step S110 relates to identifying a driving path 42 within the current video image 40. The identification of the driving path 42 within the current video image 40 is visualized here in FIG. 2 as a projection or overlay of the driving path 42 within the current video image 40. The driving path 42 is defined here using lateral boundaries 44 between which the driving route 38 of the vehicle 12 extends.

[0075] Step S120 involves determining an item of brightness information of the current video image 40 from the optical camera 20. Step S120 is optional.

[0076] Step S130 relates to identifying a control area 46 along the driving corridor 42 as a function of the driving route 38 and / or driving parameters of the vehicle 12.

[0077] The control area 46 is a window having dimensions and a position relative to the optical camera 30 along the predetermined driving path 38. During driving of the vehicle 12, the control area 46 moves with the vehicle 12 along the driving path 42. The control area 46 is defined laterally between the boundary lines 44. Further limits of the control area 46 are defined here by forward and rearward distance limits 48.

[0078] The location of the control area 46 within each video image 40 depends on the driving route 38, for example, on a straight section or a curve of the driving route 38. The size and location of the control area 46 may be adapted depending on the driving parameters of the vehicle 12. For example, the control area 46 may be different sizes at different speeds of the vehicle 12.

[0079] Optionally, in accordance with optional step S120, the identification of the control area 46 may be performed upon further consideration of the determined brightness information of the current video image 40. Due to the further identification of the control area 46 based on the determined brightness information, the control area 46 may be selected so that it contains the most useful potential item of image information 54. In FIGS. 5 and 6, the current video image 40 is shown as it may be provided when driving in a parking garage. An illuminated area 50 and an unilluminated area 52 occur in each case. In FIG. 5, the control area 46 is selected as an example within the driving corridor 42, not explicitly shown in FIG. 5, so that it is located in the illuminated area 50.

[0080] Step S140 involves determining, based on at least two single video images 40 from the optical camera 20, an item of image information 54 from the control area 46 based on the flow of image elements 56, in particular pixels.

[0081] The principle is illustrated in Fig. 4 and is based on image elements 56 at different distances that undergo different relative position changes when the position of the vehicle 12 changes. For this purpose, two image elements 56 associated with a box-shaped object 58 and a ground area 60, respectively, are shown in Fig. 4 by way of example. At time t1, the two image elements 56 are located in the field of view 62 of the optical camera 20 on a straight line 64. The vehicle 12 now moves in the driving direction 26. At time t2, the two image elements 56 are located at different angles in the field of view 62 of the optical camera 20, as a result of which the positions of the two image elements 56 can be determined and the two image elements 56 become distinguishable.

[0082] Optionally, in response to optional step S120, the determination of the item of image information 54 from the control area 46 may be performed upon further consideration of the determined brightness information of the current video image 40.

[0083] Thus, for example, within control region 46, image points of video image 40 may be determined as items of image information 54, and the image points of video image 40 may be partially filtered based on the determined brightness information of current video image 40 if there is no distinguishable image content due to either lack of light or overexposure. This is the case, for example, for unlit region 52 within current video image 40 shown in Figures 5 and 6.

[0084] Step S150 involves transmitting the item of image information 54 from the control area 46 to the clearance server 14.

[0085] Prior to transmission of the items of image information 54, they are compressed. Transmission of the items of image information 54 from the control area 46 to the clearance server 14 occurs over a data connection in essentially any manner. Following transmission of the items of image information 54, they are again decompressed. Various methods for compressing items of image information 54 are known per se and therefore need not be described in detail here.

[0086] Step S160 relates to complementing the items of image information 54 from the control area 46 and / or the items of reference image information 66 from the control area 46 with respect to different weather conditions, in particular ground moisture, snow, or hail. Thus, a wet underlying surface 68 may have a different color than a dry underlying surface 70, as shown in FIG. 7. Thus, a ground area 60, as shown in FIG. 8, may include an irregular pattern with wet and dry underlying surfaces 68 and 70. Ground moisture may additionally occur in the form of localized puddles 72, as shown in FIG. 9. In the ground area 60, a clear visual difference may be seen between the puddles 72 shown therein and the surrounding wet underlying surface 68.

[0087] The neural network is preferably used to learn and apply completion of items of image information 54 or items of reference image information 66 for different weather conditions. Training may be performed for items of reference image information 66 based on multiple training data, i.e., items of image information 54 transmitted to clearance server 14. Complementation may be trained globally, i.e., independent of a particular control area 46.

[0088] Step S170 relates to complementing the items of image information 54 from the control area 46 and / or the items of reference image information 66 from the control area 46 with respect to the shadow 74. The complementation of the items of image information 54 and the items of reference image information 66 with respect to the shadow 74 is performed according to time to take into account the occurrence of different shadows 74 at different times. The local time together with the date indicates sun altitude information, from which the predicted shadow 74 can be determined in terms of alignment and length. Accurate information regarding the sun altitude results, from which the shadow 74 can be characterized. Additionally, the position information can be used to perform complementation of any position of the vehicle 12 and, for example, to automatically determine the local time.

[0089] Shadows 74 are shown by way of example in Figures 10 and 11. In Figure 10, the shadow 74 of a traffic sign, and therefore a non-moving object, can be seen in the ground area 60, which will be interpolated. In Figure 11, the shadow 74 of a passing person, and therefore a moving object, can be seen in the ground area 60. In this case, interpolation does not have to be performed, deriving from the idea that the moving object is expected to move out of the control area 46 soon.

[0090] The neural network is preferably used to learn and apply the completion of items of image information 54 or items of reference image information 66 for shadows 74. Training may be performed for items of reference image information 66 based on multiple training data, i.e., items of image information 54 transmitted to clearance server 14. The completion may be trained globally, i.e., independently of a particular control region 46.

[0091] Step S180 involves matching the items of image information 54 from the control area 46 with the items of reference image information 66 from the control area 46. To this end, the current vehicle position of the vehicle 12 is determined.

[0092] 3, the position of vehicle 12 along driving route 38 can only be determined with the accuracy indicated by outer window 76. A more accurate determination of the position of vehicle 12 is desirable, for example, within inner window 78.

[0093] The position of the vehicle 12 within the exterior window 76 may be achieved using, for example, position determination based on received satellite position signals of a global navigation satellite system, as performed using the assistance system 16 for autonomous parking of the vehicle 10.

[0094] To improve position determination, odometry information of the vehicle 12 provided by wheel rotation sensors 80 on the wheels of the vehicle 12 is used. Additionally, visual odometry information of the vehicle 12 based on ground structure can be used. Various ground structures are shown as examples in FIGS. 12-15. The figures show a ground area 60, such as might be found in a factory hall, as shown based on a pallet 88. The ground area from FIG. 12 is smooth and has no discernible structure. This is not typically expected during use of the vehicle 12 in typical road traffic. FIG. 13 shows a ground area 60 with a rough, porous ground structure, such as might be found in asphalt or similar materials. FIG. 14 shows a ground area 60 with a uniform pavement, which therefore has uniform joints 90 in a two-dimensional arrangement. FIG. 15 also shows a ground area 60 with pavement having joints 90 in a two-dimensional arrangement. The pavements in FIGS. 14 and 15 are different, and therefore the joints 90 are positioned differently. Due to the typical structure size of, for example, paving stones, which is in the range of approximately 10-20 cm, a high accuracy can be achieved in position determination, for example, with the pavements of Figures 14 and 15 .

[0095] 17 additionally illustrates pixel flow in observing a ground structure. FIG. 17 shows a video image 40 having a ground area 60 including pavement with a joint 90. While driving in the direction of travel 26, individual features move within horizontal flow lines 92, i.e., the features approach the optical camera 20 in the direction of the arrow. When there is a disturbance, for example due to an obstacle, a deviation movement 94 occurs, as shown in FIG. 17.

[0096] Odometer information for vehicle 12 is also transmitted to cloud-based clearance server 14.

[0097] Cloud-based clearance server 14 additionally includes a computing unit 82 that performs visual simultaneous localization and mapping, known as "visual simultaneous localization and mapping" (V-SLAM), and is based on position fixes derived from time-offset images from which changes in position of vehicle 12 can be determined. For this purpose, video images 40 are captured in input data station 84 and filtered using a preliminary filter after receipt in cloud-based clearance server 14. Processing by computing unit 82 then occurs.

[0098] Additionally, the computing unit 82 combines position information from a position determination based on received satellite position signals of a global navigation satellite system, as well as position information from a position determination based on odometry information of the vehicle 12 and visual odometry information of the vehicle 12.

[0099] Step S190 relates to determining clearance based on an automatic comparison of the received item of image information 54 from the control area 46 with an item of reference image information 66 from the control area 46. The item of image information 54 and the item of reference image information 66 relate to the same location, i.e., the currently observed control area 46. The item of reference image information 66 was previously determined from a previously recorded comparison image and stored in the clearance server 14. Preferably, the item of reference image information 66 is based on a plurality of previously recorded comparison images, thus when the vehicle 12 or a further vehicle 12 has transmitted an item of image information 54 for the corresponding control area 46 to the clearance server 14 and no obstacles have been determined.

[0100] Thus, the items of reference image information 66 include items of image information 54 for which it has already been established that the vehicle 12 can drive onto the driving corridor 42 in a direction toward the control area 46 and / or can drive into the control area 46. Thus, the items of reference image information 66 represent an obstacle-free control area 46. If there is a deviation of the received items of image information 54 from the items of reference image information 66, the presence of an obstacle can be inferred. Otherwise, no obstacle is present. A difference between the items of image information 54 and the items of reference image information 66 therefore has the consequence that clearance is not granted.

[0101] In this case, according to step S200, such a negative clearance results in the transmission of the item of received image information 54 from the control area 46 and the item of reference image information 66 from the control area 46 to an operator, who determines the clearance, thus resulting in a manual clearance.

[0102] If clearance is granted in step S190, step S200 is skipped.

[0103] Step S210 relates to the adaptation of items of reference image information 66 based on items of received image information 54 in the event of a positive clearance by the operator. When clearance is performed by the operator, the items of reference image information 66 can thus be adapted so that clearance can preferably already occur automatically in the event of a new comparison of the same items of image information 54 with items of reference image information 66 in the observed control area 46. The adaptation of items of reference image information 66 preferably occurs based on a machine learning process.

[0104] Step S210 may be performed independently of step S200, i.e., even if clearance based on an automatic comparison of an item of received image information 54 from the control area 46 with an item of reference image information 66 from the control area 46 occurs in accordance with step S190, matching of the item of reference image information 66 may be performed based on the item of received image information 54.

[0105] Step S220 relates to transmitting a clearance from the clearance server 14 to the vehicle 12 for moving the vehicle 12 along the predetermined driving route 38. The clearance is transmitted from the clearance server 14 to the vehicle 12 via a data connection. The transmission of the clearance represents the clearance for traveling through the observed control area 46.

[0106] Step S230 relates to maneuvering the vehicle 12 along the predetermined driving route 38 in accordance with the received clearance. Maneuvering the vehicle 12 along the predetermined driving route 38 is performed as autonomous driving of the vehicle 12 in accordance with the received clearance. The vehicle 12 now performs lateral and longitudinal control to follow the predetermined driving route 38, which causes the vehicle 12 to first drive toward and then into the control area 46.

[0107] Cloud-based autopilot involves at least partially autonomous movement of the vehicle 12. A driving route 38 may be predetermined for this purpose, for example, by driving along the driving route 38 by a human vehicle driver who may learn the driving route 38. The predetermined driving route 38 connects the current vehicle position and a destination point. The predetermined driving route 38 indicates a route course for reaching the destination position.

[0108] By repeatedly performing this method, continuous checking of the driving path 42 can occur and the vehicle 12 can automatically navigate along the driving path 42. Areas of the driving path 42 possibly located between the vehicle 12 and the control area 46 have already been previously observed and cleared as the control area 46.

[0109] For example, in the case of current Level 2 applications requiring vehicle operator supervision, the received clearance may now be replaced by confirmation of vehicle operator supervision.

[0110] Prior to the first implementation of the method, an item of reference image information 66 must be generated and stored in the storage means 36 of the cloud-based clearance server 14. This may occur in the context of driving along a driving route 38 to learn the driving route 38, particularly as a trajectory. The trajectory includes the driving route 38 as well as motion information for the maneuver, such as speed or acceleration. Based on the method described above, video images 40 from the optical camera 20 are provided along the driven driving route 38, and items of image information 54 from each control area 46 along the driving corridor 42 are stored in the clearance server 14 as items of reference image information 66 from the control area 46. Details of the video images 40 may be transmitted as items of image information 54 to the clearance server 14 as comparison images. Driving along the driving route 38, and thus learning the driving route 38, may be performed using any vehicle 12.

[0111] Figure 18 relates to a second embodiment having a vehicle 12 executing the method shown in Figure 18 for clearance-based autopilot of the vehicle 12 along a predetermined driving route 38 from a current vehicle position to a destination point. The method includes a method for determining incremental clearances for autopiloting the vehicle 12 along the predetermined driving route 38.

[0112] The method is performed by a driver assistance system 10, which in this embodiment is installed in a vehicle. Processing of the items of image information 54 and automatic comparison of the items of image information 54 with items of reference image information 66 in the control area occurs locally within the driver assistance system 10. The items of reference image information 66 are stored in the vehicle 10.

[0113] The driver assistance system 10 of the second embodiment is not shown separately but is similar to that of the first embodiment, and as such it will be described hereinafter with reference to the first embodiment, the description focusing on the differences between the two driver assistance systems 10.

[0114] The driver assistance system 10 includes an assistance system 16 installed in a vehicle 12 for autonomously parking the vehicle 10, for example, for which a driving route 38 is predetermined or pre-determined. The assistance system 16 includes a control unit 18 that executes the method. The assistance system 16 additionally includes sensor systems 20, 22 having an optical camera 20 and a plurality of ultrasonic sensors 22 mounted on each vehicle 12 for monitoring a surrounding environment 24 of the vehicle 12. The optical camera 20 is mounted behind the windshield of the vehicle 12 for monitoring the surrounding environment 24 in a driving direction 26 ahead of the vehicle 12. The control unit 18 and the sensor systems 20, 22 are connected to each other via a data bus 28. The data bus 28 may be embodied in accordance with standards typical in the automotive field, such as CAN, LIN, LON, or FlexRay.

[0115] The driver assistance system 10 is designed to implement a hereinafter described method for clearance-based automated steering of the vehicle 12 along a predetermined driving route 38 from a current vehicle position to a destination point. Such a driving route 38 is shown, by way of example, in FIG. 2.

[0116] The method begins at step S300 with providing a current video image 40 from the optical camera 20 at the current vehicle position. The above description of the first embodiment with step S100 applies.

[0117] Step S310 relates to determining an item of brightness information of the current video image 40 from the optical camera 20. Step S310 corresponds to step S120 above and is again optional.

[0118] Step S320 relates to identifying a control area 46 along the driving corridor 42 as a function of the driving route 38 and / or driving parameters of the vehicle 12. The description above regarding the corresponding step S130 of the first embodiment also applies here.

[0119] Step S330 relates to determining an item of image information 54 from the control area 46 based on the flow of image elements 56, in particular pixels, based on at least two single video images 40 from the optical camera 20. Step S330 corresponds to the corresponding step S140 of the first embodiment.

[0120] Step S340 relates to supplementing the items of image information 54 from the control area 46 and / or the items of reference image information 66 from the control area 46 with different weather conditions, in particular ground moisture, snow or hail.

[0121] Step S350 relates to complementing the items of image information 54 from the control area 46 and / or the items of reference image information 66 from the control area 46 for the shadow 74.

[0122] With respect to the completion of steps S340 and S350, the above description regarding steps S160 and S170 applies correspondingly.

[0123] Step S360 relates to matching items of image information 54 from the control area 46 with items of reference image information 66 from the control area 46. To this end, the current vehicle position of the vehicle 12 is determined. Reference is made to the above description regarding step S180.

[0124] Step S370 relates to determining clearance based on an automatic comparison of the received item of image information 54 from the control area 46 with the item of reference image information 66 from the control area 46. In this embodiment, the comparison is performed by the control unit 18 in the vehicle 12. Furthermore, clearance is determined as described in corresponding step S190.

[0125] If clearance is not granted, in step S380, transmission of the items of received image information 54 from the control area 46 and the items of reference image information 66 from the control area 46 to an operator occurs, who determines the clearance. Thus, manual clearance occurs. In this embodiment, the items of image information 54 from the control area 46 and the items of reference image information 66 from the control area 46 are transmitted to a user interface of the vehicle 12 for the occupant of the vehicle 12 as the operator. The operator may check and grant clearance via the user interface.

[0126] If clearance is granted in step S370, step S380 is skipped.

[0127] Step S390 relates to the adaptation of items of reference image information 66 based on items of image information 54 upon clearance by the operator. Here, in accordance with above step S210, items of reference image information 66 stored in the vehicle are adapted so that clearance can preferably already occur automatically upon a new comparison of the same items of image information 54 with items of reference image information 66 in the observed control area 46. The adaptation of items of reference image information 66 also preferably occurs here on the basis of a machine learning process.

[0128] Step S390 may be performed independently of step S380, i.e., even if clearance based on an automatic comparison of an item of received image information 54 from the control area 46 with an item of reference image information 66 from the control area 46 occurs in accordance with step S190, additional matching of the item of reference image information 66 may be performed based on the item of image information 54.

[0129] Step S400 relates to maneuvering the vehicle 12 along the predetermined driving route 38 in accordance with the clearance. The above description of the corresponding step S230 applies.

[0130] By repeatedly performing the method, continuous clearance can occur and the vehicle 12 can automatically navigate along the driving route 38. Each area ahead of the vehicle 12 is previously observed and cleared as a control area 46.

[0131] Prior to a first implementation of the method, an item of reference image information 66 must be generated and stored in the vehicle 12. This may occur in the context of driving along a driving route 38 to learn the driving route 38, particularly as a trajectory. The trajectory includes the driving route 38 as well as motion information for maneuvering, such as speed or acceleration. Based on the method described above, video images 40 from the optical camera 20 are provided along the traveled driving route 38, and items of image information 54 from each control area 46 along the driving corridor 42 are stored as items of reference image information 66 from the control area 46.

[0132] Figure 19 relates to a third embodiment having a cloud-based driver assistance system 10 implementing the method shown in Figure 18 for clearance-based autopilot of a vehicle 12 along a predetermined driving route 38 from a current vehicle position to a destination point. The method includes a method for determining incremental clearances for autopiloting the vehicle 12 along the predetermined driving route 38.

[0133] The cloud-based driver assistance system 10 for performing the method of the third embodiment corresponds to that of the first embodiment, for which reason a more detailed description in this respect will be omitted.

[0134] The cloud-based driver assistance system 10 of the third embodiment is designed to implement the hereinafter described method of the third embodiment for cloud-based automatic steering of the vehicle 12 along a predetermined driving route 38 from a current vehicle position to a destination point. The method is based on determining a step-by-step clearance of the vehicle 12 for automatic steering along the predetermined driving route 38 and corresponds in large part to the method of the first embodiment, and for that reason the description will focus on the differences between the two methods.

[0135] The method of the third embodiment begins in step S500 with providing a current video image 40 from the optical camera 20 at the current vehicle position. The above description of the first embodiment with step S100 applies.

[0136] Step S510 relates to identifying a driving path 42 within the current video image 40. The above description regarding the corresponding step S110 of the first embodiment applies.

[0137] Step S520 relates to identifying a control area 46 along the driving corridor 42 as a function of the driving route 38 and / or driving parameters of the vehicle 12.

[0138] In this embodiment, the identification of the control region 46 occurs without considering any item of brightness information from the current video image 40. Otherwise, the above description regarding the corresponding step S130 of the first embodiment applies.

[0139] Step S530 relates to determining an item of image information 54 from the control area 46 based on the flow of image elements 56, in particular pixels, based on at least two single video images 40 from the optical camera 20. Step S530 corresponds to the corresponding step S140 of the method of the first embodiment.

[0140] Step S540 relates to requesting transmission from the cloud-based clearance server 14 of an item of reference image information 66 from the control area 46 derived from a previously recorded comparison image. A corresponding message is sent by the assistance system 16 of the vehicle 12 via the vehicle's communication unit 30 to the clearance server 14 via a data connection with a request for the item of reference image information 66 for the control area 46. The requested item of reference image information 66 is stored in the cloud-based clearance server 14.

[0141] Step S550 relates to receiving an item of reference image information 66 from the control area 46 from the clearance server 14. In accordance with the request in step S540, the item of reference image information 66 for the control area 46 of the previously recorded comparison image is transmitted from the cloud-based clearance server 14 to the vehicle 12.

[0142] To this end, items of reference image information 66 from the control area 46 are compressed prior to transmission by the cloud-based clearance server 14 and decompressed prior to reception in the vehicle 12 by the control unit 18 .

[0143] Step S560 relates to matching the items of image information 54 from the control area 46 with the items of reference image information 66 from the control area 46. To this end, the current vehicle position of the vehicle 12 is determined. Reference is made to the above description regarding step S180.

[0144] Step S570 relates to determining clearance based on an automatic comparison of an item of image information 54 from the control area 46 with an item of reference image information 66 received from the control area 46. The item of image information 54 and the item of reference image information 66 relate to the same location, i.e., the currently observed control area 46. In this embodiment, the comparison is performed by the control unit 18 in the vehicle 12. Furthermore, clearance is determined as described in corresponding step S190.

[0145] If clearance is not granted, then in step S580, transmission of the received item of image information 54 from the control area 46 and the item of reference image information 66 from the control area 46 to an operator occurs, who determines the clearance. A manual clearance thus occurs. In this embodiment, the item of image information 54 from the control area 46 is transmitted via a data connection to the cloud-based clearance server 14, and from there, together with the item of reference image information 66 from the control area 46 obtained directly from the storage means 36, to a remotely connected operator who determines the clearance.

[0146] If clearance is granted in step S570, step S580 is skipped.

[0147] Step S590 relates to the matching of items of reference image information 66 upon positive clearance by the operator based on items of received image information 54. The above description regarding step S210 applies accordingly.

[0148] Step S590 may be performed independently of step S580, i.e. even if clearance based on an automatic comparison of the received image information 54 items from the control area 46 with the reference image information 66 items from the control area 46 has already occurred in accordance with step S190, additional matching of the reference image information 66 items may be performed based on the image information 54 items in that each image information 54 item is transmitted to the cloud-based clearance server 14.

[0149] Step S600 relates to maneuvering vehicle 12 along predetermined driving route 38 in accordance with the received clearance. The description above regarding corresponding step S230 applies.

[0150] Furthermore, the statements made regarding the first embodiment for generating and storing items of reference image information 66 in the storage means 36 of the cloud-based clearance server 14 also apply here. [Explanation of symbols]

[0151] 10 Cloud-based driver assistance systems 12 vehicles 14. Cloud-based clearance server 16 Support Systems 18 Control Unit 20 Optical camera, sensor system 22 Ultrasonic sensors and sensor systems 24 Surrounding environment 26 Direction of travel 28 Data Bus 30 Communication Unit 32 Means of communication 34 Processing means 36 Memory means 38 Driving Routes 40 video images 42 Driver's aisle 44 Borderline 46 Control Area 48 Distance Limit 50 lighting area 52 Non-illuminated area 54 Image information items 56 Image element, pixel 58 Box-shaped object 60 ground area 62 field of view 64 straight line 66 Reference Image Information Item 68 Wet Subsurface 70 Dry Subsurface 72 Puddle 74 Shadow 76 Exterior window 78 Inner Window 80 Wheel rotation sensor, odometer sensor 82 computing units 84 Input Data Station 86 Spare Filter 88 Palettes 90 Joint 92 Horizontal streamlines 94 Deviant Movement

Claims

1. A method for determining a stepwise clearance of a vehicle (12) for automated steering along a predetermined driving route (38) from a current vehicle position to a destination point, the vehicle (12) including a sensor system (20, 22) having at least one optical camera (20) for monitoring a surrounding environment (24) in a direction of travel (26) of the vehicle (12); providing a current video image (40) from said optical camera (20) at said current vehicle position; determining a clearance for automatically steering the vehicle (12) along the predetermined driving route (38) based on an automatic comparison of an item of image information (54) from a control area (46) of the video image (40) with an item of reference image information (66) from the control area (46) derived from a previously recorded comparison image; outputting the determined clearance for automatically steering the vehicle (12) along the predetermined driving route (38) for a step corresponding to the current video image (40); A method comprising:

2. 2. The method of claim 1, wherein the method includes aligning the items of the image information (54) from the control area (46) with the items of the reference image information (66) from the control area (46).

3. 3. The method of claim 2, wherein the step of aligning the items of image information (54) from the control area (46) with the items of reference image information (66) from the control area (46) includes determining a current vehicle position.

4. The determination of the current vehicle position comprises: Receiving signals from satellite navigation systems, in particular using the Differential Global Positioning System, and / or providing items of odometer information for said vehicle (12); and / or Identifying landmarks along the predetermined driving route of the vehicle; and / or Determining the current vehicle position using a system for simultaneous visual localization and mapping - Patent Application 20070122997 4. The method of claim 3, further comprising determining the current vehicle position based on:

5. The provision of items of odometered information of the vehicle (12) comprises: Providing items of visual odometry information for the vehicle (12), particularly based on ground structures; and / or 5. The method of claim 4, including providing a sensor signal from at least one odometry sensor (80) of the vehicle (12).

6. 6. The method according to claim 1, wherein if there is no clearance, the method comprises transmitting the item of image information (54) from the control area (46) and the item of reference image information (66) from the control area (46) to an operator to determine the clearance.

7. 7. The method of claim 6, wherein the method includes matching the items of the reference image information (66) based on the items of the image information (54) from the control area (46) upon clearance by the operator.

8. The method comprises the steps of: driving along the driving route (38) to learn the driving route (38), in particular as a trajectory; providing video images (40) from the optical camera (20) along the traveled driving route (38); and 8. A method according to any one of claims 1 to 7, characterized in that it comprises the step of saving an item of image information (54) from said control area (46) as a comparison image to an item of reference image information (66) from said control area (46) or at least as a detail of said video image (40).

9. 9. The method of claim 8, wherein the method includes transmitting the item of image information (54) from the control area (46) and an item of reference image information (66) from the control area (46) as a comparison image or at least details of the video image (40) to a cloud-based clearance server (14).

10. 10. The method according to any one of claims 1 to 9, characterized in that the method comprises a step of determining the item of image information (54) from the control area (46) based on a flow of image elements (56), in particular pixels, based on at least two single video images (40) from the optical camera (20).

11. The method comprises: identifying a driving path (42) within the current video image (40); and 11. The method according to claim 1, further comprising the step of identifying the control area (46) along, in particular within, the driving corridor (42) depending on the driving route (38) and / or driving parameters of the vehicle (12).

12. The method includes determining an item of brightness information of the current video image (40) from the optical camera (20); the identification of the control area (46) occurs upon further consideration of the determined brightness information item of the current video image (40); and / or The method according to any one of claims 1 to 11, characterized in that the method comprises the step of determining the item of image information (54) from the control area (46) taking into account the determined item of brightness information of the current video image (40).

13. 13. The method according to any one of claims 1 to 12, characterized in that the method comprises a step of complementing the items of image information (54) from the control area (46) and / or the items of reference image information (66) from the control area (46) for different weather conditions, in particular ground moisture, snow or hail.

14. The method according to any one of claims 1 to 13, characterized in that the method comprises a step of complementing the items of the image information (54) from the control area (46) and / or the items of the reference image information (66) from the control area (46) for shadows (74).

15. 15. The method of claim 14, wherein the complementing of the items of the image information (54) from the control area (46) and / or the items of the reference image information (66) from the control area (46) for a shadow (74) comprises time-dependent complementing of the items of the image information (54) from the control area (46) and / or the items of the reference image information (66) from the control area (46) for a shadow (74).

16. The method comprises: transmitting the item of image information (54) from the control area (46) to a cloud-based clearance server (14); and transmitting the determined clearance for moving the vehicle (12) along the predetermined driving route (38) from the cloud-based clearance server (14) to the vehicle (12); 16. The method of any one of claims 1 to 15, wherein the previously recorded comparison image is stored on the cloud-based clearance server (14), and wherein the determination of the clearance includes determining the clearance within the cloud-based clearance server (14).

17. The method includes compressing the item of image information (54) from the controlled area (46) prior to transmission of the item of image information (54) to the cloud-based clearance server (14); and 17. The method of claim 16, further comprising extracting the transmitted item of image information from the control area after the transmission of the item of image information to the cloud-based clearance server.

18. The items of reference image information (66) are stored on a cloud-based clearance server (14), and the method includes: receiving, from the cloud-based clearance server (14), the item of reference image information (66) from the control area (46) derived from the previously recorded comparison image; The method of any one of claims 1 to 17, characterized in that the determination of the clearance comprises determining the clearance within the vehicle (12).

19. 20. The method of claim 18, wherein the method includes the additional step of requesting the transmission from the cloud-based clearance server of the item of reference image information from the control area that originates from the previously recorded comparison image.

20. The method includes compressing the item of reference image information (66) from the control area (46) prior to the transmission of the item of reference image information (66) to the cloud-based clearance server (14); and 20. The method of claim 18 or 19, comprising extracting the received items of reference image information (66) from the controlled area (46) after receiving the items of reference image information (66) from the cloud-based clearance server (14).

21. 21. A method for clearance-based automated steering of a vehicle (12) along a predetermined driving route (38) from a current vehicle position to a destination point, the vehicle (12) including a sensor system (20, 22) having at least one optical camera (20) for monitoring the surrounding environment (24) in the direction of travel (26) of the vehicle (12), the method comprising: determining incremental clearances for automated steering of the vehicle (12) along the predetermined driving route (38) using the method of any one of claims 1 to 20.

22. The method for clearance-based autopilot of the vehicle (12) comprises: Control of said vehicle (12) by an external server, in particular for automated valet parking type 2, Controlling the vehicle (12) based on a method for visual automatic localization and mapping (V-SLAM); or Control of the vehicle (12) based on a method for automatic localization and mapping using at least one ambient environment sensor to generate a point cloud of the vehicle's environment, in particular using at least one radar sensor and / or one LiDAR-based ambient environment sensor.

22. The method of claim 21, wherein the method is performed based on

23. 23. The method of claim 21 or 22, wherein the method for clearance-based automated steering of a vehicle (12) along a predetermined driving route (38) from a current vehicle position to a destination point is performed for automated valet parking or for practice parking using a previously learned trajectory for driving along the driving route (38).

24. The method comprises receiving a position of the vehicle (12) based in particular on receiving signals of a Global Navigation Satellite System (GNSS), The method according to any one of claims 21 to 23, characterized in that the method comprises selecting a driving route based on the received position of the vehicle (12).

25. A driver assistance system (10) for a vehicle (12), said driver assistance system (10) being designed to carry out a method according to any one of claims 1 to 20.

26. 21. A cloud-based driver assistance system (10) having at least one vehicle (12) and a cloud-based clearance server (14), wherein the at least one vehicle (12) and the cloud-based clearance server (14) are connected to each other by a data connection, and wherein the cloud-based driver assistance system (10) is designed to perform the method of any one of claims 1 to 20.

Images