Computer-implemented method for controlling a vehicle, and vehicle

Abstract

The invention relates to a computer-implemented method (100) for controlling a vehicle (200), comprising the steps of: providing (110) a three-dimensional recording (111) of surroundings of the vehicle (200); grouping (120) objects (121) in the recording (111); determining (130) an object (121) of the grouped objects as a critical object if the object size of the object (121) is smaller than a size threshold value, wherein the size threshold value is determined on the basis of an interest size of a region of interest (122) of the vehicle; outputting (140) object information characterising the critical object (121); determining (150) a control instruction for controlling the vehicle (200) on the basis of the object information.

Description

[0001] Computer-implemented method for controlling a vehicle and vehicle

[0002] The invention relates to a computer-implemented method for controlling a vehicle, in particular a working machine, and to such a vehicle.

[0003] Vehicles use various systems and methods for collision warning and avoidance. These vehicles typically employ one or more sensors to create images of their surroundings. Since these images are usually generated from 2D sensors, a three-dimensional model is created from them.

[0004] Classification approaches based on 3D point clouds from such 3D scans are computationally very complex. If the analysis detects that one or more relevant points are present in a predefined area, the zone is considered occupied and safety control measures can be initiated. These measures can include braking, emergency braking, and similar actions.

[0005] Particularly in the area of ​​agricultural machinery, which travels on uneven surfaces such as dirt roads, fields, and similar terrain, such irregularities trigger warnings and / or the aforementioned control instructions. Such a warning or control instruction based on background conditions is undesirable. Warnings based on background and surface conditions distract the driver and can even lead to the vehicle being partially shut down.

[0006] EP 2 831 840 B1 discloses a method in which objects in an image containing depth information are divided into regions of interest. A relatively central location of each region of interest is transformed into a single-image geometry. The corresponding part of an associated single image is searched radially outward from the relatively central location along a plurality of radial search paths. Along these paths, the associated image pixels are filtered with an edge-preserving smoothing filter to locate an edge of the associated object. The edge positions for each of the radial search paths are combined in an edge profile vector, which enables the object to be distinguished.

[0007] It is an object of the present invention to provide a computer-implemented method and a vehicle that improve upon at least one or more of the aforementioned disadvantages. In particular, it is an object of the present invention to provide a computer-implemented method that allows for a computationally simpler evaluation of recordings and meaningfully determines warnings and safety-critical control instructions.

[0008] The task, according to a first aspect, is solved by a computer-implemented method for controlling a vehicle. The vehicle can be at least partially, and in particular fully, autonomously operated. The vehicle can be a car, a truck, a work machine, or the like. The work machine can be a vehicle used for agriculture or similar purposes.

[0009] The procedure includes the following steps:

[0010] - Providing a three-dimensional image of the vehicle's surroundings;

[0011] - Grouping objects in the recording;

[0012] - Determining one of the grouped objects as a critical object if an object size of the object is smaller than a size threshold.

[0013] The size threshold is determined based on an area of ​​interest of the vehicle.

[0014] The procedure further includes:

[0015] - Outputting object information that characterizes the critical object;

[0016] - Determining a control instruction to control the vehicle based on the object information; the procedure may further include controlling the vehicle based on the control instruction.

[0017] The method advantageously solves the problem by determining whether an object is critical based on the size of the region of interest and the size threshold. The proposed method ensures that only critical objects are used for control instructions. Such critical objects can be people, animals, or other safety-relevant objects that should be identified and considered in the control process. Conversely, walls or a truck are not considered critical, and therefore, for example, no emergency braking is required for a wall. Similarly, fault lines, such as those found in gravel pits, can be excluded. The object information can further be used to classify the critical objects. This classification is computationally intensive, so the proposed method effectively reduces the number of objects required for classification.

[0018] Grouping objects in a recording can involve identifying and summarizing points and / or data objects in a three-dimensional space that share similar features or properties. This grouping can be based on predetermined distances, such as Euclidean distances. The resulting groups, often called clusters, can represent objects that are similar in their features. This approach is particularly advantageous in 3D sensing for identifying areas of interest or objects within the sensor data.

[0019] Classification can be the assignment of data objects to predefined categories or classes based on their characteristics. In the context of 3D sensor data, this can mean assigning a class to each group of points in the sensor data, such as "vehicle," "pedestrian," or "building." This classification can be performed using machine learning or AI models trained on historical data. Classification enables a detailed interpretation and analysis of the 3D sensor data. The variable of interest can be a current variable or a variable representing a planned movement trajectory of the vehicle.

[0020] Providing the recording may involve capturing the recording, for example by means of one or more sensors of the vehicle.

[0021] Providing the recording can include saving the recording and accessing the saved recording. The vehicle may have storage for this purpose.

[0022] The vehicle's surroundings can be or include the area in front of, behind, to the left, and / or to the right of the vehicle. The surroundings can also be the area along the planned trajectory of movement.

[0023] After grouping the objects, their positions within the image can be determined. The size of the object being viewed can then be compared to the size of the respective grouped object.

[0024] The object information can only characterize the critical objects.

[0025] The size threshold can be dynamically adjusted based on the size of interest, and / or the size threshold can be dynamically adjusted to determine the critical object. The size threshold can be smaller than the size of interest. For example, the size threshold could be 50%.

[0026] The size threshold can be limited at the lower end. For example, the size threshold cannot be smaller than a predetermined height threshold for a human. This means the size threshold cannot be less than 1.50 m. The lower limit of the size threshold can be predetermined based on the vehicle's driving situation. Alternatively, the lower limit can be based on the size of the sensor's field of view. In this case, the lower limit can be half or up to 90% of the maximum vertical and / or horizontal extent of the sensor's visibility range.

[0027] The procedure may include identifying an object and / or section within the recording as a non-critical object if its size is greater than or equal to the size threshold. The procedure may further include excluding the non-critical object from the object information.

[0028] Outputting the object information can further include classifying the critical object, where the object information characterizes the classified critical object. Consequently, the object information can then only characterize the classified critical object; the critical object itself can be omitted.

[0029] The quantity of interest can be determined based on one or more vehicle parameters, weather parameters, surface information of the terrain the vehicle is traversing, sensor information, and / or a planned trajectory of the vehicle. Vehicle parameters can include speed, steering angle, and / or vehicle type. Surface information can characterize the terrain as a road, gravel path, country lane, field, meadow, or similar.

[0030] The procedure may include further:

[0031] - Determining a subset of points in the image as the background if the subset of points simultaneously satisfies a first and a second condition.

[0032] The first condition can define that the subset of points is approximable to a plane. The second condition can define that the subset of points within the plane is distributable with a distribution greater than or equal to a distribution threshold. Determining the subset of points and related features can be performed independently of the object information, if required. Advantageously, determining the subset of points allows for the easy identification of surfaces such as roads, substrates, walls, and the like. Consequently, this subset of points can be given lower priority or even ignored during further processing for vehicle control, simplifying data handling. This also reduces the probability of false warnings.

[0033] The procedure may include further:

[0034] - Outputting background information characterizing the specific subset of points.

[0035] Determining the steering instruction can be further based on background information. The background can be defined, for example, as a break line or an ascending or descending section of road.

[0036] Background information can be given lower priority than object information when determining control instructions. Background information does not need to be classified.

[0037] The plane can be characterized by a predetermined length, width and / or depth, such that the subset of points within this length, width and / or depth can be distributed with a distribution greater than or greater than the distribution threshold.

[0038] The subset of points that fulfill the first and second conditions can be excluded from determining a warning status for a zone in the recording and / or its surroundings. However, critical objects in the object information can be considered for determining the warning status of a zone in the recording and / or its surroundings. The warning status can be communicated to the vehicle's driver. Furthermore, the control instruction can be determined based on the warning status. The procedure can further include:

[0039] - Determining a type and / or position of the background relative to the vehicle, wherein the type of background is characterized by a vertical and a horizontal plane.

[0040] The procedure may include further:

[0041] - Emitting an initial acoustic, haptic and / or visual piece of information to the driver characterizing the type and / or position of the background.

[0042] The vehicle may have a speaker and / or screen for displaying information, warning status, and the like.

[0043] Determining the control instruction can include ignoring or excluding the background information if the background has been defined as a horizontal plane. Such horizontal planes can be assigned to a traveled path, a busy road, or the like. For example, the predetermined depth can determine that, in the case of uneven fielding, minor elevations still belong to the field surface and therefore should not trigger a warning and / or be defined as a critical object.

[0044] A vertical plane can be characterized by being tilted at up to 45° to the direction or axis of gravity. A horizontal plane can be characterized by being tilted between 46° and 135° to the direction or axis of gravity.

[0045] The vertical and / or the horizontal plane can be planar.

[0046] Determining the steering instruction can include reducing the maximum speed and / or preventing the vehicle from accelerating in order to steer the vehicle, if the background is defined as a vertical plane. With a horizontal plane, changing the maximum speed may be omitted. Likewise, the vehicle may continue to accelerate unhindered.

[0047] The control instruction, based on object information, can include a braking operation, an emergency braking operation, a reduction in maximum speed, and / or a steering operation. The control instruction can be configured to prevent a collision with the critical object. Reducing the maximum speed based on object information can be a more drastic reduction compared to reducing the maximum speed based on background information. The control instruction based on object information can be a safety-critical control instruction.

[0048] The procedure may include further:

[0049] - Outputting an acoustic and / or optical second piece of information to the driver characterizing the object information and / or the control instruction.

[0050] The first and / or second piece of information may include a warning and / or the warning status.

[0051] The method was described with respect to one object or one critical object, but the invention is not limited thereto. Multiple objects can be identified as critical objects in the recording, and the object information can consequently characterize multiple critical objects.

[0052] According to a second aspect, the task is solved by a vehicle. The vehicle can be at least partially, and in particular fully, autonomously operated. The vehicle can be a work machine.

[0053] The vehicle includes at least one sensor for capturing a three-dimensional image of the vehicle's surroundings. The sensor can be a lidar sensor, a radar sensor, or a camera. The sensor(s) can be configured to capture two 2D images to create a 3D image. The creation of the 3D image can be performed using the sensor(s) and / or the processor described below. The vehicle further includes the processor, which is configured to:

[0054] - Grouping objects in the recording;

[0055] - Determining one of the grouped objects as a critical object if an object size of the object is smaller than a size threshold.

[0056] The size threshold is determined based on an area of ​​interest of the vehicle.

[0057] The processor is further trained to:

[0058] - Outputting object information that characterizes the critical object;

[0059] - Determining a control instruction to control the vehicle based on the object information;

[0060] - Controlling the vehicle based on the steering instructions.

[0061] Features described in relation to the procedure according to the first procedure can be described as features of the vehicle according to the second aspect and vice versa.

[0062] The task is solved, according to a third aspect, by a computer program product comprising instructions that cause a vehicle, according to the second aspect, to execute the procedure according to the first aspect. The computer program product may be stored in memory. The vehicle may include such memory. The processor may be configured to load the computer program product and execute it.

[0063] Preferred embodiments are explained by way of example with reference to the accompanying figures. Figure 1 shows a schematic representation of a computer-implemented method for controlling a vehicle;

[0064] Fig. 2 is a schematic representation of a recording; and

[0065] Fig. 3 shows a schematic representation of a vehicle for carrying out the

[0066] Procedure.

[0067] Fig. 1 shows a schematic representation of a computer-implemented method 100 for controlling a vehicle 200. The method 100 can be stored in the form of a computer program product, for example on a memory of the vehicle 200.

[0068] Method 100 comprises providing 110 a three-dimensional recording 111 of the vehicle 200's environment. For example, the vehicle 200 can record an environment in the direction of travel of the vehicle 200 by means of one or more sensors 210, see Fig. 2. If the recordings from the sensor(s) are provided as two-dimensional recordings, method 100 can comprise generating the three-dimensional recording based on at least two two-dimensional recordings.

[0069] Procedure 100 further comprises grouping 120 objects 121 in the recording. Subsequently, procedure 100 further comprises determining 130 one of the grouped objects 121 as a critical object 121 if the size of the object 121 is less than a size threshold. The size threshold is determined based on the size of an area of ​​interest 122 of the vehicle 200. The area of ​​interest 122 can be referred to as the region of interest.

[0070] Fig. 2 shows a front view of the vehicle 200 using the sensor 210.

[0071] Vehicle 200 is traveling on a road with a meadow alongside it. An object 121, for example a person or an animal, is located in the right lane of the road. The area of ​​interest 122 is also shown. The area of ​​interest 122 varies and, according to Fig. 2, depends on the current speed of vehicle 200. The area of ​​interest 122 can also be determined based on other parameters. Consequently, the area of ​​interest 122 can have a variable size. The area of ​​interest 122 can also depend on other parameters. According to Fig. 2, the area of ​​interest 122 is shown as a cuboid, but is not limited to this shape.

[0072] As can be seen in Fig. 2, object 121 is smaller than the area of ​​interest 122. The size threshold can, for example, be 50% of the area of ​​interest. Since object 121 is smaller than 50% of the area of ​​interest, it is determined to be a critical object. Furthermore, part of the road fills the remainder of the area of ​​interest 122. This part of the road is larger than the size threshold, and it is concluded that it is not a critical object. A critical object can be a safety-critical object.

[0073] Procedure 100 further includes the output 140 of object information characterizing the critical object 121. The street section, however, is not included in the object information.

[0074] Procedure 100 further includes determining 150 a control instruction for controlling the vehicle 200 based on the object information. Procedure 100 can optionally include controlling the vehicle 200 based on the control instruction.

[0075] By providing object information, the vehicle's control system (200) can be informed whether there are safety-critical objects. Other objects, such as the road segment, are of minor relevance. Consequently, the critical object can be classified based on the object information and not on the entire recording (111), resulting in faster classification due to the reduced data volume. Furthermore, this reduces the error rate for supposedly critical objects. If critical objects are present, a strict vehicle deceleration can be initiated for safety reasons.

[0076] Procedure 100 can further include determining a subset of points 161 in the recording 111 as background if the subset of points 161 simultaneously satisfies a first and a second condition. The first condition defines that the subset of points 161 is approximable to a plane. The second condition defines that the subset of points 161 is distributable within the plane with a distribution greater than or equal to a distribution threshold.

[0077] In Fig. 2, the left meadow is shown as an example of such a subset of points 161. Based on the steps mentioned above, the method 100 can identify further planes in the image 111. According to Fig. 2, the road and the right meadow would also be characterized as planes.

[0078] Procedure 100 can further include the output of background information characterizing a specific subset of points 161. Determining the control instruction can further be based on this background information.

[0079] Consequently, differentiating between object information and background information allows for further processing, such as prioritizing the object information. Background information can be used to exclude areas of recording 111 that should not be classified. This further simplifies subsequent data processing.

[0080] The control instruction, background information, and / or object information can be output to a driver of vehicle 200. For this purpose, vehicle 200 can be equipped with a speaker and / or screen to inform the driver, for example, about the right and left fields and the road ahead. Furthermore, the control instruction can be executed automatically by vehicle 200, particularly with regard to object information, to avoid an accident or collision with the critical object 121. To this end, it can be helpful to inform the driver about this in advance or during the control process. This is especially advantageous if the field of view of sensor 210 and the driver differ.

[0081] The control instruction, based on the object information, can include braking, emergency braking, steering, and / or reducing the vehicle's maximum speed (up to 200 km / h). The control instruction, based on background information, can only result in the relevant information being displayed to the driver for informational purposes.

[0082] Fig. 3 shows a schematic representation of a vehicle 200 for carrying out the method 100. The vehicle 200 comprises the sensor 210 and a processor 220. The processor 220 can be configured to execute the computer program.

[0083] Reference sign

[0084] Computer-implemented method for controlling a vehicle

[0085] Providing a three-dimensional image

[0086] Recording

[0087] Grouping objects in the recording

[0088] object

[0089] Area of ​​interest

[0090] Identifying one of the grouped objects as a critical object

[0091] Outputting object information characterizing the critical object

[0092] Determining a control instruction to control the vehicle based on the object information

[0093] subset of points

[0094] vehicle

[0095] sensor

[0096] processor

Claims

Patent claims 1. Computer-implemented method (100) for controlling a vehicle (200), comprising the steps: Providing (110) a three-dimensional recording (111) of the vehicle's environment (200); Grouping (120) of objects (121) in the recording (111); Determine (130) an object (121) of the grouped objects as a critical object if an object size of the object (121) is less than a size threshold, wherein the size threshold is determined based on an interest size of an area of ​​interest (122) of the vehicle; Output (140) of object information characterizing the critical object (121); Determine (150) a control instruction to control the vehicle (200) based on the object information.

2. Method (100) according to claim 1, wherein the size threshold is dynamically adjusted based on the size of interest, and / or wherein the size threshold is smaller than the size of interest.

3. Method (100) according to claim 1 or 2, wherein the output (140) of the object information further comprises classifying the critical object (232), wherein the object information characterizes the classified critical object (121).

4. Method (100) according to one of the preceding claims, wherein the quantity of interest is determined based on one or more vehicle parameters, weather parameters, subsurface information of a subsurface traversed by the vehicle (200), sensor information and / or a planned movement trajectory of the vehicle (200).

5. Method (100) according to any one of the preceding claims, further comprising: Determining a subset of points (161) in the recording as background if the subset of points (161) simultaneously satisfies a first and a second condition, wherein the first condition defines that the subset of points (161) is approximable to a plane, wherein the second condition defines that the subset of points (161) is distributable within the plane with a distribution greater than or greater than an equal distribution threshold; Outputting background information characterizing the specific subset of points (161), wherein determining the control instruction is further based on the background information.

6. Method (100) according to claim 5, wherein the plane is characterized by a predetermined length, width and / or depth such that the subset of points (161) within this length, width and / or depth can be distributed with a distribution greater than or greater than the distribution threshold.

7. Method (100) according to claim 5 or 6, further comprising: Determining a type and / or position of the background relative to the vehicle (200), wherein the type of background is characterized by a vertical and a horizontal plane.

8. Method (100) according to claim 7, further comprising: Providing the driver with initial acoustic, haptic and / or visual information characterizing the type and / or position of the background.

9. Method (100) according to claim 7 or 8, wherein determining (150) the control instruction comprises ignoring the background when the background has been determined as a horizontal plane, and / or wherein determining the control instruction (150) comprises reducing a maximum speed and / or preventing acceleration of the vehicle (200) for controlling the vehicle (200) when the background has been determined as a vertical plane.

10. Method (100) according to any of the preceding claims, wherein the control instruction based on the object information comprises a braking operation, an emergency braking operation, a reduction of a maximum speed and / or a steering operation.

11. Method (100) according to any one of the preceding claims, further comprising: outputting an acoustic and / or optical second piece of information to the Driver characterizing the object information and / or the control instruction.

12. Vehicle (200), comprising: at least one sensor (210) for recording and / or providing a three-dimensional image of the vehicle's (200) surroundings; a processor (220) configured to: Grouping objects (121) in the recording (111); Determining an object (121) of the grouped objects as a critical object if an object size of the object (121) is less than a size threshold, wherein the size threshold is determined based on an interest size of an area of ​​interest (122) of the vehicle (200); Outputting object information characterizing the critical object (121); Determining a control instruction to control the vehicle (200) based on the object information; Controlling the vehicle (200) based on the control instruction.

13. Vehicle (200) according to claim 12, wherein the vehicle (200) is a working machine, and / or wherein the vehicle (200) is an at least partially autonomous vehicle.

14. Computer program product comprising commands that cause a vehicle (200) according to claim 12 or 13 to execute the method (100) according to any one of claims 1 to 1.

Images