Method for controlling driving operation of vehicle

Abstract

The invention relates to a method for controlling the driving operation of a vehicle. According to the invention, a monitoring area in front of the vehicle is detected to acquire a relevant target (5, 6) from sensor data, and the environment of the vehicle is detected to generate an environment image (7). According to the sensor data and the environment image (7), relevant objects (5, 6) are optically identified and classified into known relevant objects (5) and unknown relevant objects (6). A travelable level (9) is assigned to a known target (5). The unknown target (6) is further processed by inputting a picture of the unknown target into a calculation unit (9), which is configured with the degree of travel (9) of the target. An unknown target of interest (6) is identified by a computing unit (9) and a travelable level (9) is assigned to the unknown target of interest. The drivable level (9) for the unknown target (6) is provided to a control unit, and the driving operation of the vehicle is controlled by the control unit taking into account the drivable levels (9) of all the targets (5, 6).

Description

Technical Field

[0001] This invention relates to a method for controlling the movement and operation of a vehicle. Background Technology

[0002] In vehicles, driver assistance systems are increasingly used to control vehicle operation and assist or reduce the driver's workload in specific driving situations or during specific movements of the vehicle. Today, such systems are applied in vehicles in various ways, such as as electronic stability programs like ESP or ESC, emergency braking assist systems, lane keeping assist systems, overtaking assist systems, steering assist systems, launch assist systems, traffic jam assist systems, parking assist systems, or for longitudinal control of the vehicle, such as using distance assist systems like ACC, where a desired distance between the vehicle and the vehicle ahead can be set based on the vehicle's speed, typically indicated by the vehicle's own speed.

[0003] In relation to the control of the vehicle's operation, a method for determining the passable free space of an autonomous vehicle is known from DE 11 2019 000 048 T5. This passable free space can provide indication of where the vehicle can maneuver without colliding with objects, buildings, and / or similar structures. In exceptional circumstances, such as in an emergency, the vehicle can cross drivable boundaries to enter potentially impassable spaces, such as sidewalks or grassy areas.

[0004] Furthermore, a method for providing classified digital records for a system used for automated machine learning and updating machine-readable program code is disclosed in DE 10 2021 207 093 A1. Multiple categories are pre-defined to characterize different target types. For example, a first category represents drivable targets, and a second category represents insurmountable targets. Summary of the Invention

[0005] The object of the present invention is to provide a method for controlling the driving operation of a vehicle, wherein the vehicle operates reliably in a simple manner while complying with safety requirements when a target is present in the area in front of the vehicle.

[0006] The objective is achieved according to the invention by a method for controlling the driving and operation of a vehicle having the features described in claim 1.

[0007] Advantageous improvements to the method according to the invention are integral parts of the other claims.

[0008] In the method for controlling vehicle operation according to the invention, in order to determine relevant targets in the area in front of the vehicle, especially along the direction of travel, during vehicle operation, the area in front of the vehicle, especially along the direction of travel, is first detected by at least one sensor of the vehicle, thereby acquiring corresponding sensor data from the at least one sensor. Then, relevant targets / important targets in the area in front of the vehicle, especially along the direction of travel, are determined based on these sensor data. Furthermore, the vehicle environment is optically detected by at least one camera of the vehicle to continuously generate an environmental image. The determined relevant targets in the area in front of the vehicle, especially along the direction of travel, are then optically identified based on the sensor data and the environmental image. This optical identification of relevant targets in the area in front of the vehicle, especially along the direction of travel, is performed in the vehicle control unit, taking into account the point cloud obtained from the sensor data. The relevant targets are determined by the vehicle control unit as protruding portions in the point cloud and identified by the environmental image assigned to the corresponding point cloud by the at least one camera. The position of the relevant target and thus the protrusion in the point cloud is determined by means of coordinates in the following manner: the coordinates of the point cloud and the protrusion determined therein are compared with the coordinates of the environment image generated by at least one camera as a reference.

[0009] Based on images of identified relevant targets provided by at least one camera in the environmental imagery, these identified relevant targets are divided into known relevant targets and unknown relevant targets, and thus classified accordingly. Known relevant targets are assigned drivability levels that are known and predetermined for these targets. That is, the control unit assigns a drivability level, or drivability state, stored in the control unit, to the identified known relevant targets, such as traffic signs, vehicles, sidewalks, curbs / roadside stones, etc. For example, the identified known relevant targets are assigned a drivability state or drivability level of "drivable - easy" or a drivability state or drivability level of "drivable - difficult", or are assigned a uniform drivability state or drivability level of "drivable - yes" and therefore a positive drivability state, or a uniform drivability state or drivability level of "drivable - no" and therefore a negative drivability state.

[0010] Further processing is performed on identified but classified as unknown related targets. This processing can be performed sequentially and therefore sequentially on unknown related targets, or simultaneously on multiple unknown related targets. For further processing of unknown related targets, these targets are fed as images to a specially configured computing unit or database within the vehicle, which is configured with model parameters characterizing the drivability level of the targets. Therefore, the computing unit or database processing unknown related targets contains a specially configured image-to-text processing model that includes only the drivability level of the model object and processes and evaluates only the drivability level of the model object. For example, the computing unit or database can assign a drivability level of "drivable" or "not drivable" to the model object. To this end, descriptive text is assigned to the images of the corresponding unknown related targets, and the characteristics of the corresponding unknown related targets are derived from the computing unit or database using this text, or determined while considering the content stored in the computing unit or database. Optionally, the corresponding labels for relevant targets classified as unknown can also be determined and output by the computing unit in the vehicle in a suitable manner.

[0011] The computing unit transmits the drivability rating assigned to the unknown relevant targets to the vehicle control unit, so that the control unit knows the drivability rating for all relevant targets in the area in front of the vehicle, especially in front of the vehicle along the direction of travel.

[0012] Based on the drivability levels of all relevant targets in the area in front of the vehicle, particularly those in front of the vehicle along the direction of travel, which are stored in the vehicle control unit, the control unit controls the vehicle's driving operation by taking into account the drivability levels of all relevant targets.

[0013] For example, information on the drivability level of all relevant targets in the area in front of the vehicle, especially along the direction of travel in front of the vehicle, can be used to control vehicle avoidance, vehicle parking, or similar vehicle driving operations.

[0014] In a preferred embodiment of the invention, a point cloud representing the relevant target is directly generated as sensor data from the vehicle's corresponding sensors, or the point cloud representing the relevant target is directly generated based on the sensor data. That is, the point cloud representing the relevant target can be directly generated by the vehicle's sensors themselves, such as by the vehicle's radar sensors and / or ultrasonic sensors and / or lidar sensors. Alternatively, the point cloud representing the relevant target is derived using sensor data from an optical sensor, such as a camera, and its camera images, in that the point cloud is generated by calculating the difference between at least two camera images.

[0015] In a preferred embodiment of the invention, the computing unit is configured with a small number of model parameters representing the drivability level of the target. That is, a "small," efficient, and streamlined processing model with a particularly small number of model parameters is used in the method according to the invention. This processing model is trained in terms of processing procedures by a master control unit having multiple model parameters, so as to reproduce the characteristics of the master control unit. Specifically, the processing model simulates the processing style or mode of the master control unit, intermediate steps or intermediate diagrams during model computation and thus processing, and output values. The efficient processing model, trained accordingly for a specific task of determining the drivability level of the target, is used in the computing unit for further processing of unknown related targets.

[0016] When further processing unknown related targets, the corresponding image of the unknown related target is input into the processing model, which is configured with a target traversability level. This allows for a relatively small retention of the length of the data input to the processing model, particularly when inputting the image of the unknown related target into the processing model.

[0017] Because the "input computing unit" focuses on "identifying and locating relevant targets that are fixed in the corresponding environmental image or in the area in front of the vehicle, especially along the direction of travel and in front of the vehicle," the complexity or data volume when inputting data into the computing unit can be kept low and significantly reduced compared to other processing models. In particular, due to the fundamental knowledge that "only specific images of the target are relevant for processing," when further processing unknown relevant targets, images of the corresponding unknown relevant targets can be input as a unified image into the computing unit configured with the target's drivability level. That is, images of the corresponding unknown relevant targets do not need to be segmented when inputting data into the computing unit, thus significantly saving computational power and significantly increasing the processing speed in the computing unit or processing model due to the significantly less computation.

[0018] Here, the method according to the invention is particularly applied to vehicle assistance systems that perform independent and autonomous movement of a vehicle, particularly longitudinal guidance or adjustment and / or lateral guidance or adjustment, such as a vehicle ACC system (Adaptive Cruise Control system). For this purpose, the vehicle assistance system includes: at least one control unit; at least one sensor system having at least one sensor for generating sensor data; at least one camera for generating an environmental image; and a computing unit for determining the drivability level of a target. Data related to the drivability level of the relevant target, provided to the control unit, is processed accordingly by the control unit and used to control the assistance system.

[0019] The control unit, controller, or control module of the auxiliary system may have a data processing device or a processor device configured to execute one of the inventive features. For this purpose, the processor device may have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field-Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Processing Unit) may be used as the microprocessor. Furthermore, the processor device may have program code configured to execute one of the inventive features when executed by the processor device. The program code may be stored in the data memory of the processor device. The processor device may, for example, be integrated on at least one circuit board and / or at least one SoC (System-on-Chip).

[0020] Furthermore, the present invention relates to a vehicle having at least one such assistance system. Therefore, the present invention also includes a vehicle having an assistance system according to the invention, wherein the vehicle having the assistance system according to the invention can be particularly designed as a motor vehicle or automobile, particularly as a passenger car, commercial vehicle, or bus.

[0021] The method according to the invention considers specific traffic conditions where, due to special circumstances, there are special auxiliary needs during vehicle operation. Therefore, the invention advantageously controls these special movements of the vehicle in a simple manner while adhering to safety requirements. Attached Figure Description

[0022] Embodiments of the present invention are described below. Therefore... Figure 1 A schematic diagram illustrating specific steps in performing the method according to the invention is shown. Detailed Implementation

[0023] The embodiments described below are preferred embodiments of the present invention. In the embodiments, the described components are individual, independent features of the present invention, which also independently improve the present invention and can therefore be considered as part of the present invention individually or in combinations different from those shown. Furthermore, the embodiments can also be supplemented by other features among the already described features of the present invention.

[0024] exist Figure 1 The diagram illustrates the main steps of the method according to the invention. Here, a vehicle, including various sensor systems with sensors for detecting data and auxiliary systems for assisting the driver, moves along a travel distance.

[0025] In the first step 1, to detect data, for example using radar sensors of a radar system, the environment located in front of the vehicle along the direction of travel, serving as a monitoring area, is sensed, thereby generating a continuous point cloud as sensor data with measurement points. Specific raised portions appear in these point clouds, which are considered relevant targets 5 and 6 in the environmental area, especially along the vehicle's direction of travel. Simultaneously, an environmental image 7 of the vehicle environment is detected by a vehicle camera during vehicle movement, which also shows the specific relevant targets 5 and 6. By comparing the "sensor image" obtained from the sensor data with the corresponding environmental image 7 adapted thereto, the relevant targets 5 and 6 in the vehicle's environmental area, especially along the vehicle's direction of travel, are assigned and mapped. These relevant targets 5 and 6 are distinguished by an evaluation unit within the vehicle as known relevant targets 5 and unknown relevant targets 6, and are classified accordingly. Here, known relevant targets 5 can be considered in particular other vehicles, traffic signs, safety islands, pedestrian crossings, or traffic light poles. Among these known relevant targets 5, their drivability level 9 is also known, that is, whether these known relevant targets 5 can be drivable and with what difficulty. For example, a known relevant target 5 is assigned a drivability level 9 "drivable - yes" as drivability 10, such as a safety island or green space that can be crossed by a vehicle in an emergency, or a drivability level 9 "drivable - no" as inaccessibility 11, such as a traffic light or traffic sign that cannot be crossed by a vehicle in an emergency. The known relevant target 5, with its corresponding drivability level 9, is transmitted to the vehicle control unit, such as the controller of a vehicle assistance system.

[0026] In another step 2, the unknown relevant target 6 is identified and classified by sending the target as a unified / complete image to a specially configured database in the computing unit 8 within the vehicle. Examples of unknown relevant targets 6 include fire hydrants, placed objects such as ladders or toolboxes, temporary lane signs such as roadblocks, etc.

[0027] In step 3, the identified and categorized unknown related targets 6 are processed in a database specifically configured as the computing unit 8 in the vehicle. The database of the computing unit 8 in the vehicle, which is detailed and trained by specifying and outputting drivability levels for the targets (due to its specialized research and focus on this task), can process the unknown related targets 6 rapidly upon input. Here, the database of the computing unit 8 in the vehicle also assigns drivability levels 9 to the unknown related targets 6, that is, whether these unknown related targets 6 can be drivable and with what difficulty. For example, assigning drivability level 9 "drivable is" to the unknown related targets 6 as drivability 10, such as a ladder or toolbox that can be drivable by a vehicle in an emergency, or assigning drivability level 9 "drivable is not" as indrivability 11, such as a fire hydrant that cannot be drivable by a vehicle in an emergency. The now identified and categorized unknown related targets 6 are also sent to the vehicle control unit, such as the so-called controller of the vehicle assistance system, with their corresponding drivability level 9. Optionally, the now identified and classified unknown related targets 6 may be additionally output to the vehicle driver by the vehicle's assistance system or display system along with their corresponding labels, and displayed to the vehicle driver, for example, with their corresponding labels.

[0028] In step 4, the drivability level 9 now assigned to all relevant targets 5 and 6—that is, the drivability level 9 for relevant targets 5 and 6, "drivable is" as drivability 10 or "drivable is not" as inability 11—is used by at least one vehicle assistance system to control the vehicle's driving operation. For example, this information on drivability level 9, within the scope of assisted driving operation, automatic driving operation, or autonomous driving operation, can be used to calculate and execute vehicle avoidance in the event of an emergency during vehicle operation.

[0029] List of reference numerals in the attached diagram:

[0030] 1. Data Detection

[0031] 2. Identify the target

[0032] 3. Processing is performed in the computing unit.

[0033] 4. Output can exceed the level

[0034] 5. Known Targets

[0035] 6. Unknown goals

[0036] 7. Environmental Images

[0037] 8 Computing Units

[0038] 9. Output level and driving performance

[0039] 10 output levels can drive beyond - Yes

[0040] 11 Output level can exceed - no

Claims

1. A method for controlling a driving operation of a vehicle, the method comprising the following steps: a) detecting a monitoring area in front of the vehicle in the driving direction by means of at least one sensor of the vehicle in order to determine a relevant object (5, 6) from sensor data of the at least one sensor, b) optically detecting the environment of the vehicle by means of at least one camera of the vehicle in order to generate an environment image (7), c) optically recognizing the relevant object (5, 6) in front of the vehicle in the driving direction from the sensor data and the environment image (7), d) classifying the recognized relevant object (5, 6) into a known relevant object (5) and an unknown relevant object (6) from the sensor data and / or from a picture of the recognized relevant object, e) assigning a passability rating (9) stored in a control unit to the known relevant object (5), f) further processing the unknown relevant object (6) by inputting a picture of the unknown relevant object (6) into a computing unit (8) configured for evaluating a passability rating (9) of the object, g) recognizing the unknown relevant object (6) by the computing unit (8) and assigning a passability rating (9) to the unknown relevant object (6), the passability rating (9) for the unknown relevant object (6) being provided to the control unit, h) controlling the driving operation of the vehicle by the control unit taking into account the passability ratings (9) of all relevant objects (5, 6). A point cloud representing the object is generated as sensor data by the respective sensor of the vehicle, or the point cloud representing the object is generated from the sensor data. The computing unit (8) is configured with a small number of model parameters characterizing the passability rating (9) of the object. The computing unit (8) receives a processing program of a master control unit from the master control unit having a plurality of model parameters when determining the passability rating (9) of the object, and the computing unit (8) uses the received processing program when further processing the unknown relevant object (6). When further processing the unknown relevant object (6) by the computing unit (8), only the picture of the respective unknown relevant object (6) is input into the computing unit (8) configured with model parameters characterizing the passability rating (9) of the object and simulating the processing program of the master control unit. When further processing the unknown relevant object (6), a uniform and undivided picture of the respective unknown relevant object (6) is input into the computing unit (8). When controlling the driving operation of the vehicle by the control unit taking into account the passability ratings (9) of all relevant objects (5, 6), the driving operation of the vehicle is taken into account for controlling the avoidance of the vehicle and / or for controlling a parking process of the vehicle. The passability rating (9) "passable" is assigned to the relevant object (5, 6) as a passability (10) or the passability rating (9) "not passable" is assigned as a non-passability (11). At least one control unit; 2. The method of claim 1, wherein, At least one sensor system having at least one sensor for generating sensor data; 3. The method according to claim 1 or 2, characterized in that, ​ 4. The method according to any one of claims 1 to 3, characterized in that, ​ 5. The method according to any one of claims 1 to 4, characterized in that, ​ 6. The method of claim 5, wherein, ​ 7. The method according to any one of claims 1 to 6, characterized in that, ​ 8. The method according to any one of claims 1 to 7, characterized in that, ​ 9. An assistance system of a vehicle (1), comprising: ​ ​ at least one camera for generating an environmental image (7); a computing unit (8) for determining a drivable grade (9) of an object (5, 6), wherein the method according to any one of claims 1 to 8 is implemented.

10. A vehicle (1) having at least one assistance system according to claim 9.

Images