Automatic maneuvering of a vehicle on the basis of a clearance process

Abstract

The invention relates to a method for ascertaining a stepwise clearance for a vehicle (12) to automatically maneuver the vehicle along a specified driving route (38) from a current vehicle position to a destination. The vehicle (12) has a sensor system (20, 22) with at least one optical camera (20) for monitoring the surroundings (24) in the driving direction (26) of the vehicle (12). The method has the steps of providing a current video image (40) of the optical camera (20) at the current vehicle position, ascertaining the clearance to automatically maneuver the vehicle (12) along the specified driving route (38) on the basis of an automatic comparison of image information (54) from a monitored region (46) of the video image (40) with reference image information (66) from the monitored region (46) on the basis of a previously captured comparison image, and outputting the ascertained clearance to automatically maneuver the vehicle (12) along the specified driving route (38) in a step which corresponds to the current video image (40). The invention likewise relates to a method for a clearance-based automatic maneuvering of a vehicle (12).

Description

[0001] Automatic maneuvering of a vehicle based on clearances

[0002] The present invention relates to a method for determining step-by-step releases for a vehicle for automatic maneuvering along a predetermined route from a current vehicle position to a destination point, wherein the vehicle has a sensor system with at least one optical camera for monitoring an environment in the direction of travel of the vehicle.

[0003] Furthermore, the present invention relates to a method for the release-based, automatic maneuvering of a vehicle along a predetermined travel route from a current vehicle position to a destination point, wherein the vehicle has a sensor system with at least one optical camera for monitoring an environment in the direction of travel of the vehicle, wherein the method comprises determining step-by-step releases for the automatic maneuvering of the vehicle along the predetermined travel route using the above method.

[0004] The present invention also relates to a driving assistance system for a vehicle, wherein the driving assistance system is designed to carry out the above method.

[0005] The present invention also relates to a cloud-based driving assistance system having at least one vehicle and a cloud-based release server, wherein the at least one vehicle and the cloud-based release server are connected to one another via a data connection, and the cloud-based driving assistance system is designed to carry out the above method.

[0006] Various assistance systems are already being used in current vehicles to make driving safer and more relaxing. These include emergency braking systems, lane keeping systems, distance control systems, and cross-traffic alert systems, to name just a few. Such assistance systems are sometimes referred to collectively as ADAS (Advanced Driver Assistance Systems). In addition, applications in the field of "autonomous driving" are being continuously developed to increasingly relieve the burden on the driver, culminating in ferry operations in which the vehicle drives completely autonomously and a driver in the traditional sense is no longer required. The longer-term goal is to achieve autonomous driving as defined by the SAE standard J3016 at Level 5, where Level 5 refers to autonomous driving without driver intervention. The first Level 4 systems are currently being brought to market maturity for specific use cases.

[0007] For all of these applications, it is important to detect the vehicle's surroundings as reliably as possible. This applies to both static and dynamic objects. The detection of dynamic objects, in particular, is typically based on the vehicle itself detecting the surroundings using an appropriate sensor system. It should be noted that each type of environmental sensor used, such as optical cameras, LiDAR-based environmental sensors, radar sensors, or even ultrasonic sensors, has certain advantages and disadvantages that influence the detection of the environment. Optical cameras, for example, are characterized by high resolution and a long range, enabling the reliable detection of objects in the vehicle's surroundings. However, they have weaknesses when determining distances to objects, and their performance can decrease under certain environmental conditions such as fog or precipitation.LiDAR-based environmental sensors, on the other hand, are very reliable in determining the distance to objects and exhibit a high tolerance for various environmental conditions. However, they are comparatively expensive and have low horizontal angular resolution. Reliable detection of objects in the vehicle's surroundings is currently not possible. The same applies to radar sensors as to LiDAR-based environmental sensors. Ultrasonic sensors, on the other hand, are only suitable for short ranges and have very low directional characteristics, while they are particularly inexpensive and are already widely used in current vehicles. For the mass adoption of driver assistance systems, a low price is important, which currently precludes the use of LiDAR-based environmental sensors.Sensor systems with multiple types of environmental sensors are already known in the prior art for use in various assistance systems, and these can be provided cost-effectively. For example, an optical front camera mounted behind a vehicle windshield is combined with ultrasonic sensors mounted along the vehicle's sides. However, this solution is typically insufficient to provide the desired performance and reliability, for example, for Level 4 autonomous driving. With an increasing number of environmental sensors, such as multiple optical cameras, the performance of the sensor systems for environmental detection can be increased.However, such sensor systems are not suitable for current production vehicles due to the cost of the sensors and the additional requirements for processing the sensor data provided by these environmental sensors.

[0008] This also applies to use cases in known environments. For example, automatic parking systems are known in which vehicles drive independently to their parking space within a property. Such systems currently require monitoring by the driver. This monitoring can also take place from outside the vehicle. This corresponds to autonomous driving according to Level 2. However, autonomous driving in the sense of Level 4 cannot yet be realized. This also applies to current valet parking systems. There are approaches to improving the performance of environmental detection through additional, external environmental sensors. Such external environmental sensors are part of an infrastructure in the area in which the vehicle is maneuvered. However, these external environmental sensors also involve high additional costs.This can also only be done with the consent of an infrastructure owner, such as a landowner, and is particularly difficult to implement in public areas. Furthermore, the question of responsibility between the vehicle manufacturer and a provider of external environmental sensors is difficult to clarify.

[0009] To simplify the data processing requirements in the vehicle, it is theoretically possible to perform autonomous maneuvering in a cloud-based manner. This involves transmitting sensor information from the vehicle's environmental sensors to a cloud server and processing it there. However, this is not practical due to the amount of data to be transmitted.

[0010] In practice, therefore, when a vehicle is maneuvering autonomously, it is common for the driver to monitor the maneuvering and issue authorizations so that the vehicle can continue maneuvering. This can be done, for example, in the form of continuous confirmation from the driver, with the maneuvering being stopped if confirmation is not received. Alternatively or additionally, the driver can check the information provided for autonomous maneuvering, for example, before or during the maneuvering. While the driver can also issue authorizations from outside the vehicle, it is necessary that the driver be at least in the vicinity of the vehicle.

[0011] Based on the above-mentioned prior art, the invention is therefore based on the object of specifying a method for cloud-based, automatic maneuvering of a vehicle along a predetermined driving route, a method for release-based, automatic maneuvering of a vehicle along a predetermined driving route, a driving assistance system for a vehicle which carries out one of the above methods, and a corresponding cloud-based driving assistance system which enable efficient cloud-based maneuvering of the vehicle with a high level of reliability, in particular for limited applications such as autonomous parking of the vehicle.

[0012] The object is achieved according to the invention by the features of the independent claims. Advantageous embodiments of the invention are specified in the subclaims.

[0013] According to the invention, a method is thus provided for determining step-by-step authorizations for a vehicle to automatically maneuver along a predetermined route from a current vehicle position to a destination point, wherein the vehicle has a sensor system with at least one optical camera for monitoring an environment in the direction of travel of the vehicle, comprising the steps of providing a current video image from the optical camera at the current vehicle position, determining the authorization for automatically maneuvering the vehicle along the predetermined route based on an automatic comparison of image information from a control area of ​​the video image with reference image information from the control area based on a previously recorded comparison image, and outputting the determined authorization for automatically maneuvering the vehicle along the predetermined route for a step that corresponds to the current video image.

[0014] According to the invention, a method is further specified for the release-based, automatic maneuvering of a vehicle along a predetermined travel route from a current vehicle position to a destination point, wherein the vehicle has a sensor system with at least one optical camera for monitoring an environment in the direction of travel of the vehicle, wherein the method comprises determining step-by-step releases for the automatic maneuvering of the vehicle along the predetermined travel route using the above method.

[0015] According to the invention, a driving assistance system for a vehicle is also specified, wherein the driving assistance system is designed to carry out the above method.

[0016] According to the invention, a cloud-based driving assistance system is also specified with at least one vehicle and a cloud-based release server, wherein the at least one vehicle and the cloud-based release server are connected to one another via a data connection, and the cloud-based driving assistance system is designed to carry out the above method.

[0017] The basic idea of ​​the present invention is therefore to enable step-by-step authorization for a vehicle to maneuver automatically along a predetermined route based on the vehicle's sensors, wherein the vehicle's surroundings are monitored in the direction of travel based on the optical camera. Step-by-step authorizations can be generated automatically so that authorization-based, automatic maneuvering can be carried out. The authorizations can be generated based on the automatic comparison of the image information from the control area of ​​the video image with the corresponding reference image information from the control area. These authorizations can thus be used, for example, like previously usual authorizations generated by the vehicle driver, to enable the automatic maneuvering of the vehicle. The vehicle can thus maneuver automatically into or through the control area.By iteratively performing the procedure, an iterative check of the route to the destination point can be carried out, and the vehicle can maneuver automatically to the destination point.

[0018] Communication between the vehicle and the cloud-based approval server via the data connection can be carried out particularly efficiently, as the amount of data to be transmitted is reduced by viewing only the control area. The transmission of image information to the approval server is limited to the control area and can thus be accelerated due to the reduced amount of data to be transmitted. Furthermore, by viewing only the control area, the relevant image information can be processed in a short time. As a result, low latency times can be achieved throughout the entire data processing process, from the provision of the current video image to the receipt of the approval in the vehicle.

[0019] In particular, in certain applications, a route of sufficient length can be recorded and released, enabling, for example, autonomous driving at Level 4. This applies, for example, to applications for autonomous vehicle parking, where a route is predefined or, for example, determined in advance, so that only a small area of ​​the environment needs to be monitored. In such applications, a low driving speed can also be specified so that the control area changes only slowly along the route.

[0020] The method can be implemented particularly advantageously, for example, in current applications for Level 2 autonomous driving that require monitoring by the vehicle driver. In this case, confirmation of monitoring by the vehicle driver can be replaced by the automatically determined approval to move the vehicle along the specified route. Accordingly, existing driver assistance systems for Level 2 autonomous driving can be easily expanded to autonomous driving, for example, according to Level 4, without major interventions in the corresponding autonomous driving application being necessary. It is only necessary to additionally implement the described method for determining step-by-step approvals and to use the receipt of the approval to move the vehicle along the specified route from the approval server as confirmation of monitoring by the vehicle driver.

[0021] The vehicle can be any vehicle designed for self-driving or autonomous driving.

[0022] Automatic maneuvering involves at least partially autonomous movement of the vehicle. For this purpose, the route can be predefined, for example, by a human driver driving the route so that the route can be learned. The predefined route connects the current vehicle position and the destination. The predefined route specifies a route for reaching the destination. The predefined route provides reference image information for comparison with the image information of the current video image.

[0023] The sensor system comprises at least one optical camera for monitoring the area surrounding the vehicle in the direction of travel. The optical camera can, for example, be mounted behind a windshield of the vehicle to monitor the area in front of the vehicle, which corresponds to typical forward driving. Accordingly, when reversing, an optical camera directed toward the rear of the vehicle is required. Such optical cameras typically have a large field of view of 90° or 120° up to 180°, so that the optical camera can capture the driving route at least at a close to medium distance, for example, up to a few tens of meters.

[0024] The sensor system can include additional environmental sensors, such as ultrasonic sensors, to provide additional monitoring of the vehicle's surroundings. This can further improve the reliability of the vehicle's maneuvering.

[0025] Providing the current video image from the optical camera at the current vehicle position corresponds to providing an image or sequence of images in the form of a video. The current video image can be encoded in any way or be in a raw format. Properties of the provided current video image are essentially defined by properties of the optical camera.

[0026] Determining the authorization for the vehicle to maneuver automatically along the specified route based on an automatic comparison of image information from a control area with reference image information from a control area, based on a previously recorded comparison image, makes it possible to automatically detect differences between the image information of the two control areas. The reference image information for the control area represents this area without any obstacles relevant to the vehicle. For the reference image information, an implicit authorization is therefore assumed. If the received image information deviates from the reference image information, the presence of an obstacle can be assumed, so that no authorization is generated. In order to be able to issue authorization despite a deviation, the type of deviation can be determined.Since the control area is defined by the driving route, the automatic comparison can determine whether the vehicle can be moved through the control area.

[0027] Depending on the design of the method, the reference image information can be stored in the vehicle or in the cloud-based release server. The methods can, for example, be carried out essentially autonomously in the vehicle, i.e. the reference image information is stored in the vehicle. The reference image information can also be generated by the respective vehicle itself. Alternatively, the reference image information can be transmitted from the cloud-based release server to the vehicle and made available there permanently. Alternatively, the reference image information can be provided dynamically by the cloud-based release server, for example depending on the position of the vehicle. Parts of the method can also be carried out in the cloud-based release server. For this purpose, the image information can then, for example, be transmitted from the vehicle to the cloud-based release server.

[0028] The control area corresponds to an area for which the release is determined by automatically comparing the image information of the video image with the reference image information of the comparison image. The control area can encompass the entire video image or just a portion of it. Different areas of the current video image can form the control area. The control area can also change depending on driving parameters, for example, the control area can be selected to be different sizes for different speeds.

[0029] The authorization for the vehicle to maneuver automatically along the specified route represents authorization to enter the control area under consideration. The vehicle can thus maneuver automatically into or through the control area. By iteratively performing the procedure, an iterative check of the entire driving corridor up to the destination can be performed.

[0030] The release-based, automatic maneuvering of the vehicle along the specified route according to the received release relates to autonomous driving of the vehicle. The vehicle can perform lateral and longitudinal control. The automatic maneuvering of the vehicle is carried out based on the determined releases, which allow the vehicle to maneuver independently along the specified route. The releases are issued incrementally depending on the respective video images provided by the optical camera.

[0031] On the one hand, the method can be carried out in the vehicle itself, i.e., in the driving assistance system. Alternatively, the method can be carried out using a cloud-based driving assistance system formed by the at least one vehicle together with the cloud-based approval server. Parts of the method are thus carried out in the respective vehicle, and the other steps of the method are carried out by the approval server located in the cloud.

[0032] The cloud-based approval server is essentially any server connected to the corresponding vehicle via a data connection. The only important thing is that the required process steps assigned to the approval server are executed. The approval server typically includes processing resources, storage resources, and communication resources, with the communication resources being used to establish the data connection.

[0033] The data connection is used to transmit data, in this case the image information from the control area and the release, between the at least one vehicle and the release server. The data connection is therefore, in principle, any communication connection between the vehicle and the release server, which can comprise a combination of any transmission media and protocols. It represents a common means of communication and, as such, is irrelevant to the function of the cloud-based driver assistance system. Any data connection can be used here; with the packet-oriented data transmission that is common today, any transmission path can be used for data transmission without the at least one vehicle or the release server having to influence it. Usually, such influence cannot be exerted either.

[0034] The video image can, for example, be a single image, also known as a frame, captured and provided by the optical camera. Alternatively, the video image can be a moving image based on a sequence of individual frames, i.e., video sequences are viewed.

[0035] The image information can include pixels of the current video image in the control area, i.e., a portion of the currently provided video image. Alternatively or additionally, the image information can include or be based on pixels of the current video image in the control area that have been processed or preprocessed in any way. This allows, for example, relevant parts of the current video image in the control area to be determined and compared. The image information can include information, for example, semantic information, relating to the content of the current video image in the control area.

[0036] In an advantageous embodiment of the invention, the method comprises a step for matching the image information from the control area with the reference image information from the control area. In practice, localization errors can occur, or the localization of the vehicle is not sufficiently accurate, which can make a comparison of the received image information with the reference image information for the current control area difficult or even impossible. By matching the image information, the localization of the vehicle can be improved. By matching the image information, the image information can thus be reliably compared, and the occurrence of errors can be reduced. The matching of the image information with the reference image information takes place before a release is determined based on an automatic comparison of the received image information with the reference image information.Such procedures are also known as “visual odometry”.

[0037] In an advantageous embodiment of the invention, matching the image information from the control area with the reference image information from the control area involves determining a current vehicle position. Based on the vehicle position, matching the image information with the reference image information can be performed efficiently.

[0038] In an advantageous embodiment of the invention, determining a current vehicle position comprises determining the current vehicle position based on receiving signals from a satellite navigation system, in particular using a Differential Global Positioning System, and / or providing odometry information of the vehicle, and / or detecting landmarks along the vehicle's predefined route, and / or determining the current vehicle position using a system for visual, simultaneous positioning and mapping. Various of the above methods for determining the vehicle's position can be used alone or in combination to determine the current vehicle position. This applies both to the currently maneuvering or maneuvering vehicle and to the provision of reference image information.The use of signals from a satellite navigation system is widespread, particularly for vehicle navigation. By using a Differential Global Positioning System, the accuracy of positioning can be increased from an accuracy of a few meters to an accuracy of less than half a meter or less. The vehicle's odometry information can be provided by odometry sensors, for example a wheel rotation sensor (wheel tics) or a sensor for detecting the current steering angle. This odometry information enables very accurate determination of changes in the vehicle's position. Since odometry sensors typically have short measurement cycles and are therefore faster than, for example, satellite navigation systems, a combination of both types of positioning can determine the vehicle's position particularly reliably.Landmarks can help determine the exact position of a vehicle and improve positioning. Aruco codes, for example, are known as landmarks. Visual simultaneous localization and mapping (V-SLAM) is based on determining a position based on temporally offset camera images, which allows a change in the vehicle's position to be determined.

[0039] In an advantageous embodiment of the invention, the provision of odometry information of the vehicle comprises providing visual odometry information of the vehicle, in particular based on a ground structure and / or providing sensor signals from at least one odometry sensor of the vehicle. Ground structures such as pavement with certain stones can be recognized with a high degree of detail, particularly at short distances, and enable reliable position determination based on the shape of their structure. Due to typical structure sizes of, for example, paving stones in the range of approximately 10 to 20 centimeters, high levels of accuracy in position determination can be achieved. Based on the ground structure, a movement of features, for example pixels, can be reliably detected, i.e. a pixel flow is detected.Deviations in pixel flow are representative of relief differences and can be evaluated very reliably. Pixel flow is tolerant of typical error sources such as color, lighting, or soil moisture.

[0040] In an advantageous embodiment of the invention, if clearance is lacking, the method comprises a step for transmitting the image information from the control area and the reference image information from the control area to an operator to determine clearance. Thus, even if an error occurs when determining clearance, i.e. if clearance is not automatically granted based on a comparison of the image information with the reference image information from the control area, clearance can still be granted if the operator determines that the vehicle can be maneuvered into the control area. Intervention by the driver is therefore not necessary. In principle, the operator can be located anywhere. For example, the operator can be directly connected to the clearance server or determine clearance remotely. In principle, however, it can be sufficient for a passenger of the vehicle to act as the operator.For example, the image information from the control area and the reference image information from the control area can be transmitted to a user interface of the vehicle, and the operator can check and grant the release via the user interface.

[0041] In an advantageous embodiment of the invention, the method, upon approval by the operator, comprises adapting the reference image information based on the image information from the control area. Thus, if approval is granted by the operator, the system can be adapted by adapting the reference image information so that approval can be granted automatically in the control area under consideration upon a renewed comparison of the image information and the reference image information. A machine learning process is preferably used. If the reference image information is stored on the cloud-based approval server, all connected vehicles can jointly contribute to improving and adapting the reference image information.

[0042] In an advantageous embodiment of the invention, the method comprises a step of driving the route to learn the route, in particular as a trajectory, comprising providing video images from the optical camera along the driven route and storing image information from the control area as reference image information from the control area or at least a section of the video images as comparison images. Driving the route and thus learning the route can be performed with any vehicle. Thus, as soon as a route has been driven for the first time, the reference image information is available for the corresponding vehicle.When using the approval server, the reference image information can also be made available to other vehicles, allowing all vehicles to perform the process for determining step-by-step approvals and for autonomous maneuvering along a specified route. The more frequently the route has been traveled, the better the quality of the reference image information, and the more frequently and reliably automatic checking and approval can be performed. In addition to the route, the trajectory includes movement information for maneuvering, such as speeds or accelerations.

[0043] In an advantageous embodiment of the invention, the method comprises transmitting the image information from the control area as reference image information from the control area or at least a section of the video images as comparison images to a cloud-based release server. The reference image information is thus stored in the cloud-based release server so that it can be made available to different vehicles. The reference image information can also be made available jointly by different vehicles in the cloud-based release server. The reference image information can be the image information as transmitted from the vehicles to the cloud-based release server, or the cloud-based release server can process the transmitted image information and, for example, determine the reference image information from the comparison images.

[0044] The image information can be stored as reference image information and / or at least the section of the video images as comparison images, so that these (sections of the) comparison images can be used as image information, or the image information can be generated therefrom in a desired form at any time.

[0045] In an advantageous embodiment of the invention, the method comprises determining the image information from the control area based on a flow of image elements, in particular pixels, based on at least two individual video images from the optical camera. Such image information can generate a spatiality of the image information, which improves the determination of the clearance based on an automatic comparison of the image information from the control area with the reference image information from the control area. The principle is based on the fact that image elements at different distances experience different relative position changes to the optical camera when the vehicle changes position.In an advantageous embodiment of the invention, the method comprises steps for identifying a driving corridor in the current video image and for identifying the control area along the driving corridor, in particular within the driving corridor, depending on the driving route and / or driving parameters of the vehicle. Identifying the driving corridor in the current video image can be performed, for example, in the form of a projection or an overlay of the driving corridor into the current video image, depending on the position and orientation of the vehicle. The driving corridor can be defined as a single line, in particular as a center line, or with lateral boundary lines.The driving corridor can preferably be identified based on the driving route in the current video image, thus reducing the amount of data to be processed and ensuring that the control area is selected in a way that is relevant for maneuvering the vehicle. Processing of data that is not relevant for maneuvering the vehicle can be omitted. Accordingly, depending on the design of the method, data transmission requirements between the corresponding vehicle and the cloud-based approval server can be further reduced.

[0046] The control area can be identified along the driving corridor, for example, using a window with predetermined dimensions and a predetermined position relative to the optical camera. This window can be moved along the driving corridor as the vehicle moves. For a driving corridor defined by lateral boundary lines, the control area can be identified between these boundary lines and two distance limits. The control area can be identified depending on the driving route, so that the control area can change with the route, i.e., different areas of the current video image are identified as a control area. The control area can be identified depending on the vehicle's driving parameters, so that the control area can change depending on the driving parameters.For example, the control area can be selected to have different sizes for different speeds. In an advantageous embodiment of the invention, the method comprises determining brightness information of the current video image from the optical camera, and identifying the control area takes place with additional consideration of the determined brightness information of the current video image and / or the method comprises determining the image information from the control area taking into consideration the determined brightness information of the current video image. By identifying the control area based on the determined brightness information, the control area can, for example, be selected such that it contains only meaningful image information. This can simplify and thus accelerate the processing of the current video image from the optical camera.Based on the determined brightness information of the current video image, for example, the image information from the control area can be partially filtered if no distinguishable image content is available due to a lack of light or overexposure. The brightness can be created by static lighting or based on light cones from vehicles, the ego vehicle, or even third-party vehicles. Identifying the control area based on the determined brightness information of the current video image can be carried out directly in the vehicle, which can reduce the amount of data from the image information to be transmitted from the control area during subsequent transmission to the cloud-based release server. Alternatively, the control area can be adapted after the image information from the control area has been transmitted to the release server.

[0047] In an advantageous embodiment of the invention, the method comprises compensating the image information from the control area and / or the reference image information from the control area for different weather conditions, in particular ground moisture, snow, or hail. Ground moisture can occur, for example, in the form of local puddles. Furthermore, a moist surface can have a different color than a dry surface. Thus, in particular, uneven moistening of the surface can result in color patterns that can lead to problems, such as incorrect object detection. The same applies to puddles. A change in the color of the surface as a whole can also lead to problems, for example due to the aforementioned moistening or a coating of snow or ice.Due to the wide variety of possible, different weather conditions and the resulting color changes, optical differences can arise between the current video image and the comparison image, which usually makes the automatic comparison of the received image information with the reference image information error-prone and can cause problems when determining the approval, i.e., the approval cannot be granted easily. Preferably, a neural network is used to learn and apply the compensation of the image information or the reference image information for different weather conditions. Training can be carried out for the reference image information based on a plurality of training data, i.e., different image information as reference image information. Compensation can be trained generally, i.e.Independently of a specific control area, or individually for different control areas. Compensation can be performed directly in the vehicle for the image information from the control area, or, in the case of image information being transmitted to the release server, it can be performed or initiated after the transmission from the release server. The same applies to the reference image information provided from the control area, which can be based, for example, on image information provided by different vehicles.

[0048] In an advantageous embodiment of the invention, the method comprises compensating the image information from the control area and / or the reference image information from the control area for shadows. Shadows can appear on any object in sunlight or generally depending on the lighting situation, causing the color of the background to change. This can result in color patterns that can lead to problems, for example incorrect detection of objects. Due to the wide variety of possible shadow shapes and the resulting color changes, optical differences can arise between the current video image and the comparison image, which usually makes the automatic comparison of the received image information with the reference image information prone to errors and can cause problems when determining the release, i.e. the release cannot be granted without further ado.Preferably, a neural network is used to learn and apply the compensation of the image information or the reference image information for shadows. The training can be carried out for the reference image information based on a plurality of training data, i.e. different image information as reference image information. The compensation can be trained generally, i.e. independently of a specific control area, or individually for different control areas. The compensation can be carried out for the image information from the control area directly in the vehicle, or in the case of transmission of the image information to the release server, it can be carried out or initiated after the transmission from the release server. The same applies to the provided reference image information from the control area, which can be based, for example, on image information provided by different vehicles.For shadow compensation, a distinction can be made between stationary and moving objects. For stationary objects such as buildings, signs, or trees, compensation can be performed reliably, even if the objects are moving, for example, due to wind. For moving objects such as people walking by or birds flying overhead, compensation may not be necessary, as these shadows typically move out of the control area in a short period of time.

[0049] In an advantageous embodiment of the invention, compensating the image information from the control area and / or the reference image information from the control area for shadows comprises a time-dependent compensation of the image information from the control area and / or the reference image information from the control area for shadows. A local time indicates sun position information, from which an expected shadow can be determined in detail. This allows different shadows based on the same objects to be taken into account, for example, in the morning or evening. Together with a date, this results in precise information regarding the sun position, for example, the length of shadows. Position information can also be used to perform the compensation for any position of the vehicle and, for example, to automatically determine a local time.

[0050] In an advantageous embodiment of the invention, the method comprises steps for transmitting the image information from the control area to a cloud-based approval server, and transmitting the determined approval for moving the vehicle along the predetermined route from the cloud-based approval server to the vehicle, and wherein the previously recorded comparison image is stored in the cloud-based approval server and determining the approval comprises determining the approval in the cloud-based approval server. This method is carried out accordingly with the cloud-based driving assistance system. The transmission of image information from the control area to the approval server takes place via the data connection in principle in any desired manner. In order to maneuver the respective vehicle reliably, the data connection preferably enables data transmission with low latency.The determination of the approval is outsourced to the cloud-based approval server so that resources can be used there. Accordingly, few resources need to be kept available in the vehicles. In particular, the reference image information can be advantageously provided in the cloud-based approval server, as this eliminates the need to transmit the reference image information to the vehicle(s). Only the image information from the control area of ​​the video image needs to be transmitted to determine the approval. By transmitting the determined approval for moving the vehicle along the specified route from the approval server to the vehicle, the corresponding approval for entering the control area in question can be used there. By cleverly selecting the control area, the data volume to be transmitted can be kept small.

[0051] In an advantageous embodiment of the invention, the method comprises compressing the image information from the control area before transmitting the image information to the cloud-based release server and decompressing the image information from the control area after transmitting the image information to the cloud-based release server. By compressing the image information, data transmission over the data connection can be further reduced. Various methods for compressing image information are known per se and therefore need not be discussed in detail here.

[0052] In an advantageous embodiment of the invention, the reference image information is stored in a cloud-based release server, and the method comprises a step for receiving the reference image information from the control area based on the previously recorded comparison image from the cloud-based release server, wherein determining the release comprises determining the release in the vehicle. The reference image information can therefore be made available on the cloud-based release server for all vehicles and transmitted from there to the vehicles. The reference image information can be transmitted offline, i.e., independently of any current maneuvering of the vehicle along the predetermined route. Alternatively, the reference image information for a route can be transmitted upon reaching a point on the route. Further alternatively, the reference image information for determining each release can be transmitted individually.

[0053] In an advantageous embodiment of the invention, the method comprises an additional step for requesting the transmission of reference image information from the control area based on the previously recorded comparison image from the cloud-based release server. This ensures that only required reference image information is transmitted. For example, upon reaching the starting point of the route, the reference image information for maneuvering the vehicle along the entire route can be requested. Alternatively, the corresponding reference image information can be requested from the cloud-based release server for each release. Furthermore, reference information stored in the vehicle can be monitored for its timeliness, for example, depending on a storage date.If the stored reference information is no longer current, a request can be sent to the cloud-based approval server to transfer this reference image information to replace the outdated reference image information with current reference image information. If the reference image information is stored locally in the vehicle, for example, a request can be sent if the automatic comparison of the image information with the reference image information for the control area fails to grant approval. If more current reference image information is available, the comparison can be performed again.

[0054] In an advantageous embodiment of the invention, the method comprises compressing the reference image information from the control area before transmitting the reference image information from the cloud-based release server and decompressing the received reference image information from the control area after receiving the reference image information from the cloud-based release server. By compressing the reference image information, data transmission via the data connection can be further reduced. Various methods for compressing image information are known per se and therefore need not be discussed in detail here. To save storage space, the decompression of the received reference image information can be performed only when needed, i.e. the reference image information received from the cloud-based release server is initially stored in a compressed state.

[0055] In an advantageous embodiment of the invention, the method for release-based, automatic maneuvering of the vehicle is carried out based on a control of the vehicle by an external server, in particular for automatic valet parking type 2, a control of the vehicle based on a method for visual, automatic localization and mapping, V-SLAM, or a control of the vehicle based on a method for automatic localization and mapping using at least one environmental sensor for generating a point cloud of the surroundings of the vehicle, in particular using at least one radar sensor and / or a LiDAR-based environmental sensor.With corresponding methods, vehicles can already maneuver autonomously along predefined routes in practice, so that an extension with the method for determining step-by-step authorizations for the vehicle to maneuver autonomously, as well as with the method for authorization-based, autonomous maneuvering of a vehicle, can be implemented with little effort. Depending on the method and its design, the route or a corresponding trajectory can be determined in the vehicle, specifically the corresponding driver assistance system, or on the server side.

[0056] In an advantageous embodiment of the invention, the method for release-based, automatic maneuvering of a vehicle along a predetermined route from a current vehicle position to a destination is designed for automatic valet parking or for trained parking with a previously learned trajectory for driving along the route. Corresponding applications are already partially in use for current vehicles, so that an extension with functions for determining step-by-step releases for the vehicle for automatic maneuvering as well as for release-based, automatic maneuvering of a vehicle can be implemented with little effort. The route or a corresponding trajectory can be determined in the vehicle, specifically the corresponding driver assistance system, or on the server side, depending on the specific application and its design.In an advantageous embodiment of the invention, the method comprises receiving a position of the vehicle, in particular based on the reception of signals from a global navigation satellite system (GNSS), and the method comprises selecting a driving route based on the received position of the vehicle. The position of the vehicle is not used for navigation here, but rather to identify the driving route in order to be able to determine the approvals along this driving route. This allows, for example, the associated reference image information to be loaded from a memory in the vehicle, the reference image information to be decompressed, or the reference image information for the driving route to be transmitted from the cloud-based approval server in which it is stored and received by the vehicle or the driving assistance system.The global satellite navigation systems currently in operation are NAVSTAR GPS (Global Positioning System), GLONASS (Global Navigation Satellite System), Galileo and Beidou.

[0057] Features and advantages of the described method can be readily transferred to the described system and vice versa. Individual steps of the method can also be performed in any order. The method is not limited to the sequence of steps described by way of example, as long as it is obvious to the person skilled in the art from the description.

[0058] The invention will be explained in more detail below with reference to preferred embodiments and the accompanying drawings. The features presented may represent an aspect of the invention both individually and in combination. Features of various embodiments are transferable from one embodiment to another.

[0059] It shows

[0060] Fig. 1 is a schematic view of a cloud-based driving assistance system, which is shown here by way of example with a vehicle and a cloud-based release server, wherein the vehicle and the cloud-based release server are connected to each other via a data connection, and wherein the vehicle has a sensor system with at least one optical camera for monitoring an environment in the direction of travel of the vehicle, according to a first preferred embodiment,

[0061] Fig. 2 is a schematic representation of the function of the cloud-based driving assistance system of Fig. 1 for carrying out a method for automatically maneuvering the vehicle along a predetermined driving route using the cloud-based release server, with a driving route and a control area,

[0062] Fig. 3 is a schematic representation of a position determination of the vehicle from Fig. 1 based on different types of position determination,

[0063] Fig. 4 shows a detailed schematic representation of a positioning of the vehicle based on a visual, simultaneous positioning and mapping using differently perceived pixels,

[0064] Fig. 5 is a schematic representation of a control area in an exemplary, current video image taking into account a lighting situation,

[0065] Fig. 6 shows a further schematic representation of a control area in an exemplary, current video image taking into account a lighting situation,

[0066] Fig. 7 a detailed, schematic representation of an effect of different weather conditions and resulting soil moisture, which leads to a different soil coloration,

[0067] Fig. 8 is a schematic representation of an effect of different weather conditions and resulting soil moisture, which leads to different soil coloring when the soil dries unevenly,

[0068] Fig. 9 is a schematic representation of an effect of different weather conditions and resulting puddle formation,

[0069] Fig. 10 is a schematic representation of an effect of shadows, which leads to different ground colouring compared to an area without shadow,

[0070] Fig. 11 is another schematic representation of an effect of shadows, which leads to different ground colouring compared to an area without shadow,

[0071] Fig. 12 is a schematic representation of an exemplary environment of the vehicle with a homogeneous, smooth floor structure,

[0072] Fig. 13 is a schematic representation of an exemplary environment of the vehicle with a coarse-pored soil structure,

[0073] Fig. 14 is a schematic representation of an exemplary environment of the vehicle with a floor structure with a first type of floor paving,

[0074] Fig. 15 is a schematic representation of an exemplary environment of the vehicle with a floor structure with a second type of floor paving,

[0075] Fig. 16 is a flowchart of a method for determining step-by-step releases for a vehicle for autonomous maneuvering along a predetermined route from a current vehicle position to a destination point, in accordance with the cloud-based driving assistance system of the first embodiment of Fig. 1, Fig. 17 is a schematic representation of an exemplary environment of the vehicle with a floor structure with a third type of floor paving and a pixel flow,

[0076] Fig. 18 is a flowchart of a method of a second embodiment for determining step-by-step releases for a vehicle for automatic maneuvering along a predetermined route from a current vehicle position to a destination point, and

[0077] Fig. 19 is a flowchart of a method of a third embodiment for determining step-by-step releases for a vehicle for automatic maneuvering along a predetermined route from a current vehicle position to a destination point.

[0078] Figure 1 shows a cloud-based driving assistance system 10 according to a first preferred embodiment.

[0079] The cloud-based driving assistance system 10 is illustrated in Figures 1 and 2 with a vehicle 12, but may include additional vehicles 12 not shown here. The cloud-based driving assistance system 10 further includes a cloud-based release server 14.

[0080] The vehicle 12 comprises a support system 16 for autonomous parking of the vehicle 10, in which a driving route 38 is predetermined or, for example, determined in advance. The support system 16 comprises a control unit 18 that carries out the method. The support system 16 also comprises a sensor system 20, 22 with an optical camera 20 and a plurality of ultrasonic sensors 22, which are attached to the respective vehicle 12, for monitoring an environment 24 of the vehicle 12. The optical camera 20 is attached behind a windshield of the vehicle 12 for monitoring the environment 24 in a direction of travel 26 in front of the vehicle 12. The control unit 18 and the sensor systems 20, 22 are connected to one another via a data bus 28. The data bus 28 can, for example, be designed according to a standard commonly used in the automotive sector, such as CAN, LIN, LON, or FlexRay.The vehicle 12 comprises a communication unit 30 for communicating with the cloud-based release server 14. The cloud-based release server 14 comprises corresponding communication means 32 for communicating with the vehicle 12. Accordingly, a data connection between the vehicle 12 and the cloud-based release server 14 can be established via the communication unit 30 and the communication means 32. The data connection is, in principle, any communication connection between the vehicle 12 and the release server 14, which can comprise a dynamic combination of, in principle, any transmission media and protocols.

[0081] The cloud-based release server 14 is any server connected to the vehicle 12 via the data connection. The release server 14 typically includes, in addition to the communication means 32, processing means 34 and storage means 36. The storage means 36 stores a program for execution by the processing means 34.

[0082] The cloud-based driving assistance system 10 is designed to carry out the method described below for the cloud-based, automatic maneuvering of the vehicle 12 along the predetermined route 38 from a current vehicle position to a destination. The method is based on determining step-by-step authorizations for the vehicle 12 to maneuver automatically along the predetermined route 38. Such a route 38 is shown as an example in Figure 2.

[0083] The cloud-based driving assistance system 10 performs the method as trained parking with a previously learned trajectory for driving along the driving route 38, wherein the control of the vehicle 12 is generally performed based on a method for visual, automatic localization and mapping, V-SLAM.

[0084] The method begins in step S100 with providing a current video image 40 from the optical camera 20 at the current vehicle position. Such a video image 40 is shown as an example in Figure 2. The current video image 40 is provided here to the optical camera 20, for example, as a single image, also known as a frame, which is recorded and provided by the optical camera 20. Alternatively, the video image 40 can be a moving image based on a sequence of individual frames, i.e., video sequences are viewed. The current video image 40 can be encoded in any desired way or be in a raw format. Properties of the provided current video image 40 are defined by properties of the optical camera 20.

[0085] Step S110 involves identifying a driving corridor 42 in the current video image 40. Identifying the driving corridor 42 in the current video image 40 is visualized here as a projection or overlay of the driving corridor 42 into the current video image 40 in Figure 2. The driving corridor 42 is defined here by lateral boundary lines 44, between which the driving route 38 for the vehicle 12 runs.

[0086] Step S120 relates to determining brightness information of the current video image 40 of the optical camera 20. Step S120 is optional.

[0087] Step S130 relates to identifying a control area 46 along the travel corridor 42 depending on the travel route 38 and / or travel parameters of the vehicle 12.

[0088] The control area 46 is a window with dimensions and a position relative to the optical camera 30 along the specified travel corridor 38. As the vehicle 12 travels, the control area 46 moves along the travel corridor 42 with the vehicle 12. The control area 46 is defined laterally between the boundary lines 44. Further boundaries of the control area 46 are defined here by a front and a rear distance limit 48.

[0089] The position of the control area 46 in the respective video image 40 depends on the driving route 38, for example, on a straight section of the driving route 38 or on a curve. The dimensions and position of the control area 46 can be adapted depending on the driving parameters of the vehicle 12. For example, the control area 46 can be of different sizes for different speeds of the vehicle 12. Optionally, depending on the optional step S120, the identification of the control area 46 can be carried out with additional consideration of the determined brightness information of the current video image 40. By additionally identifying the control area 46 based on the determined brightness information, the control area 46 can be selected such that it contains the most meaningful image information 54 possible.Figures 5 and 6 show current video images 40, such as those that might be provided when driving in a parking garage. Illuminated areas 50 and unlit areas 52 are shown in each case. In Figure 5, the control area 46 in the driving corridor 42 (not explicitly shown in Figure 5) is selected, by way of example, to be located in the illuminated area 50.

[0090] Step S140 relates to determining image information 54 from the control area 46 based on a flow of image elements 56, in particular pixels, based on at least two individual video images 40 of the optical camera 20.

[0091] The principle is illustrated in Figure 4 and is based on the fact that image elements 56 at different distances experience different relative position changes when the position of the vehicle 12 changes. For this purpose, two image elements 56 belonging to a box-shaped object 58 and to the ground area 60 are shown as examples in Figure 4. At a time t1, the two image elements 56 lie on a straight line 64 in the field of view 62 of the optical camera 20. The vehicle 12 is moving in the direction of travel 26. At time t2, the two image elements 56 lie at different angles in the field of view 62 of the optical camera 20, so that the positions of the two image elements 56 can be determined and the two image elements 56 can be distinguished.

[0092] Optionally, depending on the optional step S120, the determination of the image information 54 from the control area 46 can be carried out with additional consideration of the determined brightness information of the current video image 40.

[0093] Thus, for example, within the control area 46 pixels of the

[0094] Video image 40 is determined as image information 54, wherein the pixels of the video image 40 are partially filtered based on the determined brightness information of the current video image 40 if no distinguishable image content is present due to a lack of light or overexposure. This applies, for example, to the unlit areas 52 in the current video images 40 shown in Figures 5 and 6.

[0095] Step S150 relates to transmitting the image information 54 from the control area 46 to the release server 14.

[0096] Before being transmitted, the image information 54 is compressed. The transmission of the image information 54 from the control area 46 to the release server 14 takes place via the data connection in, in principle, any desired manner. Following the transmission of the image information 54, it is decompressed again. Various methods for compressing image information 54 are known per se and therefore need not be discussed in detail here.

[0097] Step S160 involves compensating the image information 54 from the control area 46 and / or the reference image information 66 from the control area 46 for different weather conditions, in particular ground moisture, snow, or hail. Thus, a moist ground 68 may have a different color than a dry ground 70, as shown in Figure 7. Accordingly, the ground area 60, as shown in Figure 8, may have an irregular pattern with a moist ground 68 and a dry ground 70.

[0098] Soil moisture may also occur in the form of local puddles 72, as shown in Figure 9. On the soil area 60, a clear visual difference can be seen between the puddle 72 shown there and the surrounding moist ground 68.

[0099] Preferably, a neural network is used to learn and apply the compensation of the image information 54 or the reference image information 66 for different weather conditions. The training can be carried out for the reference image information 66 based on a plurality of training data, i.e., image information 54 that has been transmitted to the release server 14. The compensation can be trained generally, i.e., independently of a specific control area 46. Step S170 relates to compensating the image information 54 from the control area 46 and / or the reference image information 66 from the control area 46 for shadows 74. The compensation of the image information 54 and the reference image information 66 for the shadows 74 is carried out time-dependently in order to take into account the formation of different shadows 74 at different times.A local time together with a date provides information about the position of the sun, from which an expected shadow 74 can be determined in terms of orientation and length. This results in precise information about the position of the sun, from which the shadow 74 can be characterized. Additionally, position information can be used to perform compensation for any position of the vehicle 12 and, for example, to automatically determine a local time.

[0100] Shadows 74 are shown as examples in Figures 10 and 11. In Figure 10, shadows 74 of traffic signs can be seen on the ground area 60, i.e., shadows 74 of stationary objects that are to be compensated. In Figure 11, a shadow 74 of a passing person, i.e., a moving object, can be seen on the ground area 60. In this case, compensation cannot be performed because the moving object is expected to move out of the control area 46 shortly.

[0101] Preferably, a neural network is used to learn and apply the compensation of the image information 54 or the reference image information 66 for the shadows 74. The training can be performed for the reference image information 66 based on a plurality of training data, ie, image information 54 that has been transmitted to the release server 14. The compensation can be trained generally, ie, independently of a specific control area 46.

[0102] Step S180 relates to matching the image information 54 from the control area 46 with the reference image information 66 from the control area 46. For this purpose, a current vehicle position of the vehicle 12 is determined.

[0103] As schematically illustrated in Figure 3, the position of the vehicle 12 along the route 38 can only be determined with an accuracy indicated by an outer window 76. A more precise determination of the position of the vehicle 12 is desirable, for example, in the inner window 78.

[0104] The position of the vehicle 12 in the outer window 76 can be achieved, for example, with a position determination based on received satellite position signals of a global satellite navigation system, as carried out with the support system 16 for autonomous parking of the vehicle 10.

[0105] To improve position determination, odometry information of the vehicle 12, provided by wheel rotation sensors 80 on the wheels of the vehicle 12, is used. In addition, visual odometry information of the vehicle 12 based on a ground structure can be used. Various ground structures are shown as examples in Figures 12 to 15. Figure 1 shows a ground area 60, such as occurs in factory halls, as indicated by the pallets 88. The ground area in Figure 12 is smooth and has no discernible structure. This is not usually to be expected when vehicles 12 are used in normal road traffic. Figure 13 shows a ground area 60 with a coarse-pored ground structure, such as can occur with asphalt or similar materials.Figure 14 shows a floor area 60 with a uniform pavement having correspondingly uniform joints 90 in a two-dimensional arrangement. Figure 15 also shows a floor area 60 with a pavement having joints 90 in a two-dimensional arrangement. The pavements in Figures 14 and 15 are different, so the joints 90 are arranged differently. Due to typical structural sizes of, for example, paving stones in the range of approximately 10 to 20 centimeters, high accuracy in position determination can be achieved, for example, with the pavement in Figures 14 and 15.

[0106] Figure 17 additionally illustrates a pixel flow when observing ground structures. Figure 17 shows a video image 40 with a ground area 60 comprising pavement with joints 90. When driving in the direction of travel 26, individual features move along the planar flow lines 92, i.e., the features approach the optical camera 20 in the direction of the arrow. In the event of a disturbance, for example, caused by an obstacle, a deviating movement 94 results, as indicated accordingly in Figure 17. The odometry information of the vehicle 12 is also transmitted to the cloud-based release server 14.

[0107] The cloud-based release server 14 also includes a computing unit 82 that performs visual, simultaneous positioning and mapping. Visual, simultaneous positioning and mapping is known as Visual Simultaneous Localization and Mapping (V-SLAM) and is based on positioning based on temporally offset images, which allows a change in the position of the vehicle 12 to be determined. For this purpose, the video images 40 are captured in an input data station 84 after being received by the cloud-based release server 14 and filtered with a prefilter. They are then processed by the computing unit 82.

[0108] In addition, the computing unit 82 combines the position information of the position determination based on received satellite position signals of a global satellite navigation system and based on the odometry information of the vehicle 12 and the visual odometry information of the vehicle 12.

[0109] Step S190 relates to determining a release based on an automatic comparison of the received image information 54 from the control area 46 with reference image information 66 from the control area 46. The image information 54 and the reference image information 66 relate to the same position, namely the currently viewed control area 46. The reference image information 66 was determined based on a previously recorded comparison image and stored in the release server 14. Preferably, the reference image information 66 is based on several previously recorded comparison images, i.e., if the vehicle 12 or another vehicle 12 had transmitted image information 54 for the corresponding control area 46 to the release server 14 and no obstacles were detected.

[0110] The reference image information 66 thus contains image information 54 for which it has already been determined that the vehicle 12 can travel in the travel corridor 42 toward the control area 46 and / or into the control area 46. The reference image information 66 thus represents the control area 46 without obstacles. If the received image information 54 deviates from the reference image information 66, the presence of an obstacle can be assumed. Otherwise, no obstacle is present. Differences between the image information 54 and the reference image information 66 therefore result in no clearance being granted.

[0111] In this case, according to step S200, i.e., if the release is negative, the received image information 54 from the control area 46 and the reference image information 66 from the control area 46 are transmitted to an operator who determines the release. Thus, a manual release occurs.

[0112] If the release is granted in step S190, step S200 is skipped.

[0113] Step S210 relates to adapting the reference image information 66 based on the received image information 54 upon positive approval by the operator. Thus, if approval is granted by the operator, the reference image information 66 can be adapted such that, in the observed control area 46, approval can preferably already occur automatically upon a renewed comparison of the same image information 54 and reference image information 66. The adaptation of the reference image information 66 is preferably based on a machine learning process.

[0114] Step S210 can be performed independently of step S200, ie even if the release is based on the automatic comparison of the received image information 54 from the control area 46 with the reference image information 66 from the control area 46 according to step S190, an adaptation of the reference image information 66 can be performed based on the received image information 54.

[0115] Step S220 involves transmitting the authorization to move the vehicle 12 along the predetermined route 38 from the authorization server 14 to the vehicle 12. The authorization is transmitted from the authorization server 14 to the vehicle 12 via the data connection. The transmission of the authorization represents authorization to enter the control area 46 under consideration.

[0116] Step S230 relates to maneuvering the vehicle 12 along the predetermined

[0117] Driving route 38 according to the received authorization. Maneuvering of the vehicle 12 along the predetermined driving route 38 is carried out according to the received authorization as autonomous driving of the vehicle 12. The vehicle 12 performs lateral and longitudinal control to follow the predetermined driving route 38, whereby it initially drives toward the control area 46 and then into the control area 46.

[0118] Cloud-based, automatic maneuvering involves at least partially autonomous movement of the vehicle 12. For this purpose, the route 38 can be predetermined, for example, by a human driver driving the route 38, which allows the route 38 to be learned. The predetermined route 38 connects the current vehicle position and the destination point. The predetermined route 38 specifies a route for reaching the destination position.

[0119] By iteratively performing the procedure, the driving corridor 42 can be continuously checked, and the vehicle 12 can maneuver independently along the driving corridor 42. Areas of the driving corridor 42 that may be located between the vehicle 12 and the control area 46 have already been considered and released in advance as a control area 46.

[0120] For example, in current Level 2 applications that require monitoring by the driver, the received release can replace confirmation of monitoring by the driver.

[0121] Before the method is carried out for the first time, the reference image information 66 must be generated and stored in the storage means 36 of the cloud-based approval server 14. This can be done as part of driving the driving route 38 to learn the driving route 38, in particular as a trajectory. In addition to the driving route 38, the trajectory includes movement information for maneuvering, e.g., speeds or accelerations. Based on the method described above, video images 40 from the optical camera 20 are provided along the driven driving route 38, and image information 54 from a respective control area 46 along a driving corridor 42 is stored in the approval server 14 as reference image information 66 from the control area 46. As image information 54, a section of the video images 40 can be transmitted to the approval server 14 as comparison images.Driving the route 38 and thus learning the route 38 can be carried out with any vehicle 12.

[0122] Figure 18 relates to a second embodiment with a vehicle 12 that performs a method illustrated in Figure 18 for the release-based, automatic maneuvering of a vehicle 12 along a predetermined route 38 from a current vehicle position to a destination point. The method includes a method for determining step-by-step releases for the automatic maneuvering of the vehicle 12 along the predetermined route 38.

[0123] The method is carried out by a driving assistance system 10. In this embodiment, the driving assistance system 10 is installed in the vehicle. Processing of image information 54 and the automatic comparison of the image information 54 with the reference image information 66 in the control area takes place locally in the driving assistance system 10. The reference image information 66 is stored in the vehicle 10.

[0124] The driving support system 10 of the second embodiment is not shown separately, but is similar to that of the first embodiment, and will therefore be described below with reference to the first embodiment, with the description focusing on differences between the two driving support systems 10.

[0125] The driving assistance system 10 is installed in the vehicle 12 and includes a support system 16 for autonomous parking of the vehicle 10, in which a driving route 38 is predetermined or, for example, determined in advance. The support system 16 includes a control unit 18 that carries out the method. The support system 16 also includes a sensor system 20, 22 with an optical camera 20 and a plurality of ultrasonic sensors 22, which are attached to the respective vehicle 12, for monitoring an environment 24 of the vehicle 12. The optical camera 20 is attached behind a windshield of the vehicle 12 for monitoring the environment 24 in a direction of travel 26 in front of the vehicle 12. The control unit 18 and the sensor systems 20, 22 are connected to one another via a data bus 28. The data bus 28 can, for example, be designed according to a standard commonly used in the automotive sector, such as CAN, LIN, LON, or FlexRay.The driving assistance system 10 is designed to carry out the method described below for the release-based, automatic maneuvering of the vehicle 12 along the predetermined route 38 from a current vehicle position to a destination point. Such a route 38 is illustrated by way of example in Figure 2.

[0126] The method begins in step S300 by providing a current video image 40 from the optical camera 20 at the current vehicle position. The above statements regarding the first embodiment apply to step S100 there.

[0127] Step S310 relates to determining brightness information of the current video image 40 of the optical camera 20. Step S310 corresponds to the above step S120 and is also optional here.

[0128] Step S320 relates to identifying a control area 46 along the travel corridor 42 depending on the travel route 38 and / or driving parameters of the vehicle 12. Here, too, the above statements regarding the corresponding step S130 of the first embodiment apply.

[0129] Step S330 relates to determining image information 54 from the control area 46 based on a flow of image elements 56, in particular pixels, based on at least two individual video images 40 of the optical camera 20. Step S330 corresponds to the corresponding step S140 of the first embodiment.

[0130] Step S340 relates to compensating the image information 54 from the control area 46 and / or reference image information 66 from the control area 46 for different weather conditions, in particular ground moisture, snow, or hail.

[0131] Step S350 relates to compensating the image information 54 from the control area 46 and / or the reference image information 66 from the control area 46 for shadows 74.

[0132] With regard to the compensation of steps S340 and S350, the above statements regarding steps S160 and S170 apply accordingly. Step S360 concerns matching the image information 54 from the control area 46 with the reference image information 66 from the control area 46. For this purpose, a current vehicle position of the vehicle 12 is determined. Reference is made to the above statements regarding step S180 there.

[0133] Step S370 relates to determining a release based on an automatic comparison of the received image information 54 from the control area 46 with reference image information 66 from the control area 46. In this embodiment, the comparison is performed by the control unit 18 in the vehicle 12. Otherwise, the release is determined as described in the corresponding step S190.

[0134] If no approval is granted, in step S380, the received image information 54 from the control area 46 and the reference image information 66 from the control area 46 are transmitted to an operator, who determines approval. Thus, a manual approval is performed. In this embodiment, the image information 54 from the control area 46 and the reference image information 66 from the control area 46 are transmitted to a user interface of the vehicle 12 for an occupant of the vehicle 12 as the operator. The operator can check and grant approval via the user interface.

[0135] If the release is granted in step S370, step S380 is skipped.

[0136] Step S390 relates to adapting the reference image information 66 based on the image information 54 upon authorization by the operator. Here, in accordance with the above step S210, the reference image information 66 stored in the vehicle is adapted such that, in the observed control area 46, authorization can preferably already occur automatically upon a renewed comparison of the same image information 54 and reference image information 66. Here, too, the adaptation of the reference image information 66 is preferably based on a machine learning process.

[0137] Step S390 can be carried out independently of step S380, ie even if the release has already been carried out based on the automatic comparison of the received image information 54 from the control area 46 with the reference image information 66 from the control area 46 according to step S190, an additional adaptation of the reference image information 66 can be carried out based on the image information 54.

[0138] Step S400 involves maneuvering the vehicle 12 along the predetermined route 38 in accordance with the authorization. The above explanations regarding the corresponding step 230 apply.

[0139] By iteratively performing the method, continuous clearance can be achieved, and the vehicle 12 can maneuver independently along the route 38. Areas in front of the vehicle 12 are each considered in advance as control areas 46 and cleared.

[0140] Before the method is carried out for the first time, the reference image information 66 must be generated and stored in the vehicle 12. This can be done as part of driving the driving route 38 to learn the driving route 38, in particular as a trajectory. In addition to the driving route 38, the trajectory includes movement information for maneuvering, e.g., speeds or accelerations. Based on the method described above, video images 40 from the optical camera 20 are provided along the driven driving route 38, and image information 54 from a respective control area 46 along a driving corridor 42 is stored as reference image information 66 from the control area 46.

[0141] Figure 19 relates to a third embodiment with a cloud-based driving assistance system 10 that executes a method illustrated in Figure 18 for the release-based, automatic maneuvering of a vehicle 12 along a predetermined route 38 from a current vehicle position to a destination point. The method includes a method for determining step-by-step releases for the automatic maneuvering of the vehicle 12 along the predetermined route 38.

[0142] The cloud-based driving assistance system 10 for carrying out the method of the third embodiment corresponds to that of the first embodiment, which is why further explanations are omitted. The cloud-based driving assistance system 10 of the third embodiment is designed to carry out the method of the third embodiment described below for cloud-based, automatic maneuvering of the vehicle 12 along the predetermined travel route 38 from a current vehicle position to a destination point. The method is based on determining step-by-step releases for the vehicle 12 for automatic maneuvering along the predetermined travel route 38 and largely corresponds to the method of the first embodiment, which is why the description focuses on the differences between the two methods.

[0143] The method of the third embodiment begins in step S500 by providing a current video image 40 from the optical camera 20 at the current vehicle position. The above statements regarding the first embodiment apply, including step S100 therein.

[0144] Step S510 relates to identifying a driving corridor 42 in the current video image 40. The above statements regarding the corresponding step S110 of the first embodiment apply.

[0145] Step S520 relates to identifying a control area 46 along the travel corridor 42 depending on the travel route 38 and / or travel parameters of the vehicle 12.

[0146] In this embodiment, the control area 46 is identified without taking into account brightness information of the current video image 40. Otherwise, the above statements apply to the corresponding step S130 of the first embodiment.

[0147] Step S530 relates to determining image information 54 from the control area 46 based on a flow of image elements 56, in particular pixels, based on at least two individual video images 40 of the optical camera 20. Step S530 corresponds to the corresponding step S140 of the method of the first embodiment. Step S540 relates to requesting the transmission of reference image information 66 from the control area 46 based on a previously recorded comparison image from the cloud-based release server 14. The support system 16 of the vehicle 12 sends a corresponding message via the vehicle's communication unit 30 over the data connection to the release server 14, requesting the reference image information 66 for the control area 46. The requested reference image information 66 is stored in the cloud-based release server 14.

[0148] Step S550 relates to receiving the reference image information 66 from the control area 46 from the release server 14. According to the request in step S540, the reference image information 66 for the control area 46 of the previously recorded comparison image is transmitted from the cloud-based release server 14 to the vehicle 12.

[0149] For this purpose, the reference image information 66 from the control area 46 is compressed before transmission from the cloud-based release server 14 and decompressed after receipt in the vehicle 12 by the control unit 18.

[0150] Step S560 relates to matching the image information 54 from the control area 46 with the reference image information 66 from the control area 46. For this purpose, a current vehicle position of the vehicle 12 is determined. Reference is made to the above explanations regarding step S180 there.

[0151] Step S570 relates to determining a clearance based on an automatic comparison of the image information 54 from the control area 46 with the received reference image information 66 from the control area 46. The image information 54 and the reference image information 66 relate to the same position, namely the currently viewed control area 46. In this embodiment, the comparison is performed by the control unit 18 in the vehicle 12. Otherwise, the clearance is determined as described in the corresponding step S190.

[0152] If no release is granted, in step S580, the received image information 54 from the control area 46 and the reference image information 66 from the control area 46 are transmitted to an operator, who determines release. Thus, a manual release occurs. In this embodiment, the image information 54 from the control area 46 is transmitted via the data connection to the cloud-based release server 14, and from there, together with the reference image information 66 from the control area 46, which is taken directly from the storage means 36, is transmitted to a remotely connected operator, who determines release.

[0153] If the release is granted in step S570, step S580 is skipped.

[0154] Step S590 relates to adapting the reference image information 66 based on the received image information 54 upon positive approval by the operator. The above statements regarding step S210 apply accordingly.

[0155] Step S590 can be performed independently of step S580, ie even if the release is already carried out based on the automatic comparison of the received image information 54 from the control area 46 with the reference image information 66 from the control area 46 according to step S190, an additional adaptation of the reference image information 66 can be carried out based on the image information 54 by transmitting the respective image information 54 to the cloud-based release server 14.

[0156] Step S600 involves maneuvering the vehicle 12 along the predetermined route 38 in accordance with the received authorization. The above explanations regarding the corresponding step 230 apply.

[0157] Furthermore, the statements relating to the first embodiment for generating and storing the reference image information 66 in the storage means 36 of the cloud-based release server 14 also apply here.

[0158] 10 cloud-based driving support system

[0159] 12 vehicles

[0160] 14 cloud-based sharing servers

[0161] 16 Support system

[0162] 18 Control unit

[0163] 20 optical camera, sensors

[0164] 22 Ultrasonic sensor, sensor technology

[0165] 24 Surroundings

[0166] 26 Direction of travel

[0167] 28 data bus

[0168] 30 Communication unit

[0169] 32 means of communication

[0170] 34 processing agents

[0171] 36 storage media

[0172] 38 Driving route

[0173] 40 video images

[0174] 42 travel corridor

[0175] 44 boundary line

[0176] 46 Control Area

[0177] 48 Distance limit

[0178] 50 illuminated area

[0179] 52 unlit area

[0180] 54 Image information

[0181] 56 picture elements, pixels

[0182] 58 box-shaped object

[0183] 60 floor area

[0184] 62 field of view

[0185] 64 straight line

[0186] 66 reference image information

[0187] 68 damp ground

[0188] 70 dry ground

[0189] 72 puddle

[0190] 74 Shadow outer window inner window Wheel rotation sensor, odometry sensor Processing unit Input data station Pre-filter Palette Joint Planar flow line Deviating movement

Claims

Patent claims 1. A method for determining step-by-step releases for a vehicle (12) for automatic maneuvering along a predetermined route (38) from a current vehicle position to a destination point, wherein the vehicle (12) has a sensor system (20, 22) with at least one optical camera (20) for monitoring an environment (24) in the direction of travel (26) of the vehicle (12), comprising the steps Providing a current video image (40) of the optical camera (20) at the current vehicle position, Determining the authorization for automatic maneuvering of the vehicle (12) along the predetermined route (38) based on an automatic comparison of image information (54) from a control area (46) of the video image (40) with reference image information (66) from the control area (46) based on a previously recorded comparison image, and Outputting the determined release for automatic maneuvering of the vehicle (12) along the predetermined route (38) for a step corresponding to the current video image (40).

2. Method according to claim 1, characterized in that the method comprises a step of matching the image information (54) from the control area (46) with the reference image information (66) from the control area (46).

3. Method according to claim 2, characterized in that the matching of the image information (54) from the control area (46) with the reference image information (66) from the control area (46) comprises determining a current vehicle position.

4. The method according to claim 3, characterized in that determining a current vehicle position comprises determining the current vehicle position based on receiving signals from a satellite navigation system, in particular using a Differential Global Positioning System, and / or providing odometry information of the vehicle (12), and / or detecting landmarks along the predetermined route (38) of the vehicle (12), and / or determining the current vehicle position using a system for visual, simultaneous positioning and mapping.

5. The method according to claim 4, characterized in that the provision of odometry information of the vehicle (12) comprises providing visual odometry information of the vehicle (12), in particular based on a ground structure and / or providing sensor signals from at least one odometry sensor (80) of the vehicle (12).

6. Method according to one of the preceding claims, characterized in that in the event of a lack of release, the method comprises a step for transmitting the image information (54) from the control area (46) and the reference image information (66) from the control area (46) to an operator for determining the release.

7. The method according to claim 6, characterized in that, upon release by the operator, the method comprises adapting the reference image information (66) based on the image information (54) from the control area (46).

8. Method according to one of the preceding claims, characterized in that the method comprises a step for driving the driving route (38) for learning the driving route (38), in particular as a trajectory, comprising Providing video images (40) of the optical camera (20) along the traveled route (38), and Storing image information (54) from the control area (46) as reference image information (66) from the control area (46) or at least a section of the video images (40) as comparison images.

9. Method according to the preceding claim 8, characterized in that the method comprises transmitting the image information (54) from the Control area (46) as reference image information (66) from the control area (46) or at least a section of the video images (40) as comparison images to a cloud-based release server (14).

10. Method according to one of the preceding claims, characterized in that the method comprises determining the image information (54) from the control area (46) based on a flow of image elements (56), in particular pixels, based on at least two individual video images (40) of the optical camera (20). 11 . Method according to one of the preceding claims, characterized in that the method comprises steps for Identifying a driving corridor (42) in the current video image (40), and Identifying the control area (46) along the travel corridor (42), in particular in the travel corridor (42), depending on the travel route (38) and / or travel parameters of the vehicle (12).

12. Method according to one of the preceding claims, characterized in that the method comprises determining brightness information of the current video image (40) of the optical camera (20), and the identification of the control area (46) takes place with additional consideration of the determined brightness information of the current video image (40) and / or the method comprises determining the image information (54) from the control area (46) with consideration of the determined brightness information of the current video image (40).

13. Method according to one of the preceding claims, characterized in that the method comprises compensating the image information (54) from the control area (46) and / or the reference image information (66) from the control area (46) for different weather conditions, in particular ground moisture, snow, or hail.

14. Method according to one of the preceding claims, characterized in that the method comprises compensating the image information (54) from the control area (46) and / or the reference image information (66) from the control area (46) for shadows (74).

15. The method according to claim 14, characterized in that the compensating of the image information (54) from the control area (46) and / or the reference image information (66) from the control area (46) for shadows (74) comprises a time-dependent compensation of the image information (54) from the control area (46) and / or the reference image information (66) from the control area (46) for shadows (74).

16. Method according to one of the preceding claims, characterized in that the method comprises steps for Transferring the image information (54) from the control area (46) to a cloud-based release server (14), and Transmitting the determined authorization for moving the vehicle (12) along the predetermined travel route (38) from the cloud-based authorization server (14) to the vehicle (12), and wherein the previously recorded comparison image is stored in the cloud-based authorization server (14) and determining the authorization comprises determining the authorization in the cloud-based authorization server (14).

17. The method according to the preceding claim 16, characterized in that the method comprises compressing the image information (54) from the control area (46) before transmitting the image information (54) to the cloud-based release server (14) and decompressing the transmitted image information (54) from the control area (46) after transmitting the image information (54) to the cloud-based release server (14).

18. Method according to one of the preceding claims, characterized in that the reference image information (66) is stored in a cloud-based release server (14) and the method comprises a step for Receiving the reference image information (66) from the control area (46) based on the previously recorded comparison image from the cloud-based release server (14), wherein determining the release comprises determining the release in the vehicle (12).

19. Method according to the preceding claim 18, characterized in that the method comprises an additional step of requesting the transmission of the reference image information (66) from the control area (46) based on the previously recorded comparison image from the cloud-based release server (14).

20. The method according to any one of the preceding claims 18 or 19, characterized in that the method comprises compressing the reference image information (66) from the control area (46) before transmitting the reference image information (66) from the cloud-based release server (14) and decompressing the received reference image information (66) from the control area (46) after receiving the reference image information (66) from the cloud-based release server (14).

21. Method for the release-based, automatic maneuvering of a vehicle (12) along a predetermined travel route (38) from a current vehicle position to a destination point, wherein the vehicle (12) has a sensor system (20, 22) with at least one optical camera (20) for monitoring an environment (24) in the direction of travel (26) of the vehicle (12), wherein the method comprises determining step-by-step releases for the automatic maneuvering of the vehicle (12) along the predetermined travel route (38) using the method according to one of the preceding claims 1 to 20.

22. Method according to the preceding claim 21, characterized in that the method for release-based, automatic maneuvering of the vehicle (12) is carried out based on a control of the vehicle (12) by an external server, in particular for automatic valet parking type 2, a control of the vehicle (12) based on a method for visual, automatic localization and mapping, V-SLAM, or a control of the vehicle (12) based on a method for automatic localization and mapping using at least one environmental sensor for generating a point cloud of the surroundings of the vehicle, in particular using at least one radar sensor and / or a LiDAR-based environmental sensor.

23. Method according to one of the preceding claims 21 or 22, characterized in that the method for release-based, automatic maneuvering of a vehicle (12) along a predetermined travel route (38) from a current vehicle position to a destination point is designed for automatic valet parking, or for trained parking with a previously learned trajectory for driving along the travel route (38).

24. Method according to one of the preceding claims 21 to 23, characterized in that the method comprises receiving a position of the vehicle (12), in particular based on the reception of signals from a global satellite navigation system, GNSS, and the method comprises selecting a driving route based on the received position of the vehicle (12).

25. Driving assistance system (10) for a vehicle (12), wherein the driving assistance system (10) is designed to carry out the method according to one of the preceding claims 1 to 20.

26. Cloud-based driving assistance system (10) with at least one vehicle (12) and a cloud-based release server (14), wherein the at least one vehicle (12) and the cloud-based release server (14) are connected to one another via a data connection, and the cloud-based driving assistance system (10) is designed to carry out the method according to one of the preceding claims 1 to 20.

Images