Methods for controlling the driving operation of a vehicle

Abstract

The invention relates to a method for controlling the driving operation of a vehicle. According to the invention, a monitoring area in front of the vehicle is captured using sensor data to identify relevant objects (5, 6), and the vehicle's surroundings are captured to generate environmental images (7). Based on the sensor data and the environmental images (7), the relevant objects (5, 6) are optically identified and classified into known relevant objects (5) and unknown relevant objects (6). A degree of traversability (9) is assigned to the known relevant objects (5). The unknown relevant objects (6) are further processed by inputting their image into a processing unit (9) configured with the degree of traversability (9) of objects. The processing unit (9) identifies the unknown relevant objects (6) and assigns them a degree of traversability (9).This degree of traversability (9) for the unknown relevant objects (6) is provided to a control unit, and the vehicle's driving operation is controlled by the control unit using the degree of traversability (9) of all relevant objects (5, 6).

Description

[0001] The invention relates to a method for controlling the driving operation of a vehicle.

[0002] Vehicles increasingly utilize driver assistance systems to control their operation, supporting or relieving the driver in specific driving situations or during certain vehicle movements. These systems are now used in a wide variety of forms, such as Electronic Stability Programs (ESP) or Electronic Stability Control (ESC), emergency braking systems, lane keeping assist, overtaking assist, turning assist, hill start assist, traffic jam assist, parking assist, or longitudinal control systems like adaptive cruise control (ACC), which allows the driver to set a desired distance between their vehicle (often referred to as the "ego vehicle") and the vehicle ahead, depending on the vehicle's speed.

[0003] In connection with controlling the driving operation of an ego-vehicle, a method for determining a drivable free space for autonomous vehicles is known from DE 11 2019 000 048 T5. This drivable free space can indicate where the vehicle can maneuver without colliding with objects, structures, and / or the like. In exceptional cases, such as in emergency situations, a drivable boundary can be crossed to enter a potentially inaccessible area, such as a sidewalk or a lawn.

[0004] Furthermore, German patent DE 10 2021 207 093 A1 describes a method for providing classified digital images for an automatic machine learning system and for updating machine-readable program code. This method defines a multitude of classes that characterize different object types. For example, one such class characterizes objects that can be driven over, and a second such class characterizes objects that cannot.

[0005] The invention is based on the objective of providing a method for controlling the driving operation of a vehicle in which the vehicle is operated in a simple and reliable manner while adhering to safety aspects when objects appear in the front field of the vehicle.

[0006] This problem is solved according to the invention by a method for controlling the driving operation of a vehicle with the features listed in claim 1.

[0007] Advantageous further developments of the method according to the invention are part of the further patent claims.

[0008] In the inventive method for controlling the driving operation of a vehicle, during the vehicle's journey, the front of the vehicle, and in particular the area in front of the vehicle in the direction of travel, is first sensed by means of at least one sensor of the vehicle, thereby acquiring corresponding sensor data from the at least one sensor of the vehicle. Based on this sensor data, the relevant objects in the front of the vehicle, and in particular in front of the vehicle in the direction of travel, are then determined. Furthermore, the vehicle's surroundings are optically captured by means of at least one camera of the vehicle for the continuous generation of environmental images.The identified relevant objects in the foreground of the vehicle, and especially in the direction of travel ahead, are now visually identified using sensor data and environmental images. This visual identification of the relevant objects in the foreground of the vehicle, and especially in the direction of travel ahead, takes place in a vehicle control unit using point clouds derived from the sensor data. The relevant objects are identified by the vehicle control unit as elevations in the point clouds and are then identified by matching the respective point cloud to the environmental images from at least one camera.The positions of the relevant objects, and thus the elevations in the point clouds, are determined using coordinates such that the coordinates of the point clouds and the elevations identified therein are compared with the coordinates of the environmental images generated by at least one camera as a reference.

[0009] Based on the images of the surroundings provided by at least one camera, these identified relevant objects are divided into known and unknown relevant objects and thus classified accordingly. A known and predefined degree of traversability is assigned to the known relevant objects. This means that the control unit assigns a previously stored and predefined degree of traversability or traversability status to the identified known relevant objects, such as traffic signs, vehicles, sidewalks, curbs, etc.For example, the identified known relevant objects are assigned the traversability status or degree of traversability "easily traversable" or the traversability status or degree of traversability "difficult to traverse" or the general traversability status or degree of traversability "traversable yes" and thus a positive traversability status or the general traversability status or degree of traversability "not traversable" and thus a negative traversability status.

[0010] The identified but unknown relevant objects are further processed, whereby the unknown relevant objects can be processed successively and thus sequentially, or several unknown relevant objects can be processed simultaneously. For further processing of the unknown relevant objects, the unknown relevant objects are supplied in the form of images to a specially configured computing unit or database in the vehicle, which has been configured with model parameters characterizing the degree of traversability of objects.This computing unit or database, which processes relevant objects classified as unknown, therefore contains a specially configured processing model as an image-to-text model that solely contains and processes the degree of traversability of model objects. For example, the computing unit or database can assign either the traversability degree "traversable" or the traversability degree "non-traversable" to these model objects.For this purpose, a descriptive text is assigned to the image of each relevant object classified as unknown. Based on this text, the properties of the respective relevant object classified as unknown are derived by the processing unit or database, or determined using the content stored in the processing unit or database. Optionally, the name of the respective relevant object classified as unknown can also be determined and output by the processing unit in the vehicle in a suitable manner.

[0011] The computing unit transmits the degree of traversability assigned to the unknown relevant objects to the vehicle's control unit, so that the control unit knows the degree of traversability for all relevant objects in the front of the vehicle and especially in the direction of travel in front of the vehicle.

[0012] Based on the degree of traversability for all relevant objects in the front of the vehicle and especially in the direction of travel in front of the vehicle, as available in the vehicle's control unit, the vehicle's driving operation is controlled by the control unit using the degree of traversability of all relevant objects.

[0013] For example, the information on the degree of traversability available for all relevant objects in the front of the vehicle and especially in the direction of travel in front of the vehicle can be used to control evasive maneuvers, parking maneuvers or similar driving maneuvers of the vehicle.

[0014] In a preferred embodiment of the invention, the relevant point clouds representing objects are generated directly from a respective sensor of the vehicle as sensor data, or they are generated indirectly based on the sensor data. That is, the relevant point clouds representing objects can either be generated directly by the vehicle's sensors themselves, for example, by radar sensors and / or ultrasonic sensors and / or lidar sensors. Or, the relevant point clouds representing objects can be derived from the sensor data of optical sensors such as cameras and their camera images by generating the point clouds from the difference between at least two camera images.

[0015] In a preferred embodiment of the invention, the computing unit is configured with a small number of model parameters of the processing model that characterize the degree to which objects can be traversed. That is, in the method according to the invention, a "small," efficient, and lean processing model with a significantly small number of model parameters is used. This processing model is trained by a master unit, which has a large number of model parameters, with respect to the processing procedure in order to replicate the behavior of the master unit. In particular, the processing model imitates the processing pattern or processing scheme, the intermediate steps or intermediate representations during model calculation and thus during processing, as well as the output values ​​of the master unit.The efficient processing model, specifically trained for the task of determining the degree of traversability of objects, is used in the computing unit for further processing of the unknown relevant objects.

[0016] In this further processing of the unknown relevant objects, the representation of the respective unknown relevant objects is entered into the processing unit configured with the object traversability level and thus into the processing model present there. The length of the data input to be entered into and processed by the processing model can be kept short, especially with regard to the image input of the unknown relevant objects into the processing unit and thus into the processing model present there.

[0017] Since the input to the processing unit focuses on the desired identification of a specific and already localized relevant object within the respective environment image, or in front of the vehicle, and especially in the direction of travel ahead of the vehicle, the complexity and data volume during data input to the processing unit can be kept low and significantly reduced compared to other processing models. In particular, because of the prior knowledge that only a specific image of an object is relevant for processing, the subsequent processing of unknown relevant objects allows the representation of each unknown relevant object to be entered as a uniform representation into the processing unit configured with the object's traversability level.This means that the mapping of the respective unknown relevant objects does not need to be divided during data input into the computing unit, which significantly saves computing power and significantly increases the processing speed in the computing unit or in the processing model due to the considerably smaller number of calculations.

[0018] The method according to the invention is used in particular in a vehicle assistance system that performs an independent movement process of the vehicle and, in particular, longitudinal guidance or control of the vehicle and / or lateral guidance or control of the vehicle, such as an ACC (Adaptive Cruise Control) system. For this purpose, the vehicle assistance system comprises at least one control unit, at least one sensor system with at least one sensor for generating sensor data, at least one camera for generating images of the surroundings, and a processing unit for determining the degree to which objects can be driven over. The data provided to the control unit regarding the degree to which relevant objects can be driven over are then processed accordingly by the control unit and used to control the assistance system.

[0019] The aforementioned control units, control devices, or control modules of the assistance system may include a data processing device or a processor configured to perform one of the described features of the invention. For this purpose, the processor may include at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit) may be used as the microprocessor. Furthermore, the processor may include program code configured to execute one of the described features of the invention when executed by the processor. The program code may be stored in a data memory of the processor.The processor setup can be integrated, for example, on at least one circuit board and / or on at least one SoC (System on Chip).

[0020] Furthermore, a vehicle having at least one such assistance system is also claimed. The invention thus also includes a vehicle with an assistance system according to the invention, wherein the vehicle with the assistance system according to the invention can be designed in particular as a motor vehicle or motor car, especially as a passenger car, or as a truck or as a passenger bus.

[0021] The method according to the invention takes into account certain traffic situations where, due to specific circumstances, there is a particular need for assistance when operating the vehicle. According to the invention, these special vehicle movements are thus advantageously controlled in a simple manner while adhering to safety aspects.

[0022] An embodiment of the invention is described below. For this purpose, the Fig. 1 A schematic representation of certain process steps in carrying out the process according to the invention.

[0023] The embodiment described below is a preferred embodiment of the invention. In this embodiment, the described components each represent individual features of the invention that can be considered independently of one another. Each of these features further develops the invention independently and can therefore be considered part of the invention individually or in a combination other than that shown. Furthermore, the described embodiment can also be supplemented by other features of the invention already described.

[0024] In the Fig.Figure 1 shows a schematic overview of the essential process steps of the method according to the invention. In this process, a vehicle equipped with various sensor systems for data acquisition and assistance systems to support the driver moves along a route.

[0025] In the first process step 1, the area in front of the vehicle in the direction of travel is scanned, for example, using radar sensors of a radar system, to acquire data. This generates successive point clouds with measurement points as sensor data. These point clouds reveal specific elevations that are evaluated as relevant objects 5, 6 in the surrounding area, and especially in the direction of travel of the vehicle. Simultaneously, while the vehicle is moving, a camera on the vehicle captures images 7 of the surrounding area, which also depict certain relevant objects 5, 6. By comparing a "sensor image" obtained from the sensor data with the corresponding image 7 of the surrounding area, the relevant objects 5, 6 in the surrounding area of ​​the vehicle, and especially in the direction of travel, are assigned and mapped.These relevant objects 5, 6 are distinguished by an evaluation unit in the vehicle into known relevant objects 5 and unknown relevant objects 6 and classified accordingly. Known relevant objects 5 include, in particular, other vehicles, traffic signs, traffic islands, sidewalks, or traffic light poles. The degree of traversability 9 of these known relevant objects 5 is also known; that is, whether and with what difficulty these known relevant objects 5 can be traversed.For example, the known relevant objects 5 are assigned either the traversability level 9 "traversable yes" as traversability 10, for example, traffic islands or green spaces that can be driven over by a vehicle in an emergency, or the traversability level 9 "traversable no" as non-traversability 11, for example, traffic lights or traffic signs that cannot be driven over by a vehicle even in an emergency. The known relevant objects 5, with their respective traversability level 9, are then fed to a control unit of the vehicle, for example, a control unit of a vehicle assistance system.

[0026] In a further process step 2, the unknown relevant objects 6 are identified and classified by supplying them in image form as a uniform representation to a specially configured database as a computing unit 8 in the vehicle. Examples of unknown relevant objects 6 include fire hydrants, discarded items such as ladders or toolboxes, temporary lane guidance signs such as traffic cones, etc.

[0027] In process step 3, the identified and classified unknown relevant objects 6 are processed in the specially configured database, which is implemented as a processing unit 8 in the vehicle. This database, specifically trained and configured to assign and output the degree of traversability of objects, enables rapid processing after the input of the unknown relevant objects 6, due to its specialization and focus on this single task. The database, as the processing unit 8 in the vehicle, also assigns a degree of traversability 9 to each unknown relevant object 6, indicating whether and with what difficulty these unknown relevant objects 6 can be traversed.For example, unknown relevant objects 6 are assigned either the traversability level 9 "traversable yes" as traversability 10, such as ladders or toolboxes that can be driven over by a vehicle in an emergency, or the traversability level 9 "traversable no" as non-traversability 11, such as fire hydrants that cannot be driven over by a vehicle even in an emergency. The now identified and classified unknown relevant objects 6, along with their respective traversability level 9, are then fed to the vehicle's control unit, for example, the aforementioned control unit of a vehicle assistance system.Optionally, the now identified and classified unknown relevant objects 6 can also be output to a driver of the vehicle by an assistance system or a display system with their respective designation and, for example, also displayed to a driver of the vehicle with their respective designation.

[0028] In process step 4, the traversability grade 9, now assigned to all relevant objects 5 and 6—that is, either traversability grade 9 "traversable yes" as traversability 10 or traversability grade 9 "traversable no" as non-traversability 11 for the relevant objects 5 and 6—is used by at least one of the vehicle's assistance systems to control the vehicle's driving operation. For example, this information about traversability grade 9 can be used, within the context of assisted driving, automated driving, or autonomous driving, to calculate and execute evasive maneuvers in the event of an emergency while the vehicle is driving. REFERENCE MARK LIST 1 Data collection 2. Identification of Objects 3. Processing in computing unit 4th edition: Degree of traversability 5 well-known objects 6 unknown objects 7. Surroundings 8 computing units 9 Output level Drive-over capability 10 Output level Drive-over capability yes 11 Output level Drive-over capability no QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 11 2019 000 048 T5

[0003] DE 10 2021 207 093 A1

[0004]

Claims

[1] Method for controlling the driving operation of a vehicle comprising the following process steps: a) Detection of a monitoring area in the direction of travel in front of the vehicle using at least one sensor of the vehicle to identify relevant objects (5, 6) based on the sensor data of the at least one sensor, b) optical detection of the vehicle's surroundings using at least one camera of the vehicle to generate images of the surroundings (7), c) optical identification of the relevant objects (5, 6) in the direction of travel in front of the vehicle using the sensor data and the environmental images (7), d) Classification of the identified relevant objects (5, 6) based on the sensor data and / or based on the mapping of the identified relevant objects into known relevant objects (5) and unknown relevant objects (6), e) Assignment of a degree of traversability stored in a control unit (9) for the known relevant objects (5), f) Further processing of the unknown relevant objects (6) by inputting the representation of the unknown relevant objects (6) into a computing unit (8) configured to estimate the degree of traversability (9) of objects, g) Identification of the unknown relevant objects (6) and assignment of a degree of traversability (9) for the unknown relevant objects (6) by the computing unit (8) and provision of the degree of traversability (9) for the unknown relevant objects (6) to the control unit, h) Control of the vehicle's driving operation by the control unit using the degree of traversability (9) of all relevant objects (5, 6). [2] Method according to claim 1, characterized by, that point clouds representing objects are generated from a respective sensor of the vehicle as sensor data, or that point clouds representing objects are generated based on the sensor data. [3] Method according to claim 1 or 2, characterized by , that the computing unit (8) is configured with a small number of model parameters characterizing the degree of traversability (9) of objects. [4] Method according to any one of claims 1 to 3, characterized by , that the computing unit (8) takes over the processing procedure of the master unit in determining the degree of traversability (9) of objects from a master unit having a multitude of model parameters, and that the computing unit (8) applies the taken-over processing procedure in the further processing of the unknown relevant objects (6). [5] Method according to any one of claims 1 to 4, characterized by, that in the further processing of the unknown relevant objects (6) by the computing unit (8) only the mapping of the respective unknown relevant objects (6) into the model parameters configured with the degree of traversability (9) of objects and imitating the processing procedure of the master unit is entered. [6] Method according to claim 5, characterized by , that during the further processing of the unknown relevant objects (6) a uniform and undivided representation of the respective unknown relevant objects (6) is entered into the computing unit (8). [7] Method according to any one of claims 1 to 6, characterized by , that the driving operation of the vehicle is used by the control unit with reference to the degree of traversability (9) of all relevant objects (5, 6) to control evasive maneuvers of the vehicle and / or to control parking operations of the vehicle. [8] Method according to any one of claims 1 to 7, characterized by , that the relevant objects (5, 6) are assigned either the degree of traversability (9) “traversable yes” as traversability (10) or the degree of traversability (9) “traversable no” as non-traversability (11). [9] Assistance system of a vehicle (1) comprising at least one control unit, at least one sensor system with at least one sensor for generating sensor data, at least one camera for generating environmental images (7) and a computing unit (8) for determining the degree of drivability (9) of objects (5, 6), in which a method according to one of claims 1 to 8 is used. [10] Vehicle (1) with at least one assistance system operated according to claim 9.

Images