Method and control device for steering a vehicle off-road

Abstract

Method for controlling a vehicle (100) off-road, comprising the steps: reading (S1) a pair of images containing image information about an object (10) located in an environment (2) of the vehicle (100) off-road, and about the environment (2), determining (S2) disparities in the image information of the read image pair, comparing (S3) the determined disparities with a predetermined disparity threshold, filtering (S4) corresponding image information about the object (10) from the image information depending on the comparison results from the step of comparing (S3) the determined disparities, recognizing (S7) the object (10) based on the determined corresponding image information, and outputting (S8) a control signal to an operating unit (110) of the vehicle (100) to control the vehicle's (100's) off-road driving dynamics depending on the recognized object (10).characterized by the further step of determining (S5) a solid body of the object (10) based on the determined corresponding image information, wherein the step of recognizing (S7) the object (10) is carried out based on the determined solid body.

Description

Technical area

[0001] The present invention relates to a method and a control device for controlling the driving dynamics of an off-road vehicle as a function of an object detected in the off-road surroundings of the vehicle. The present invention also relates to a vehicle with such a control device. State of the art

[0002] It is known from the prior art to detect the surroundings of a vehicle using on-board sensors in order to automate vehicle operation. US 2022 / 0043108 A1 discloses an object detection system with which objects in the surroundings of a vehicle are first detected and classified using radar, before the classified objects are visualized in an environment map.

[0003] It is also known from the prior art to measure the volume of objects using sensors. CN 114119710 A discloses a detection system that measures and calculates the volume of an object. The detection system includes a laser scanner, which captures a point cloud as the basis for calculating the volume.

[0004] It is also known from the state of the art to detect and classify sensor-detected objects using artificial intelligence methods. To implement such methods, vehicle automation requires learning algorithms for a recurring environment and maintaining appropriate resources on the vehicle.

[0005] DE 102008002560 A1 discloses a method for visualizing image data, and DE 102008002567 A1 discloses a method for processing raw image data obtained from stereo video. In these methods, disparity information is assigned to pixels from a vehicle's surroundings in order to classify image data.

[0006] DE 102012209316 A1 discloses a method for processing sensor data from a stereo sensor system for stereoscopically capturing the environment of the stereo sensor system. Based on the sensor data, disparity points of a disparity map are classified according to an object depending on a change in disparity. Description of the invention

[0007] In one aspect, the present invention relates to a method for controlling a vehicle off-road. The vehicle may be an off-road vehicle or an off-highway vehicle. The vehicle may be configured to operate off-road. The vehicle may, for example, be a self-propelled work machine that can be operated off-road. The vehicle may, for example, be a construction machine or an agricultural utility vehicle.

[0008] The method comprises, as one step, reading in an image pair. The image pair may comprise at least two images. The image pair may be captured using an image capture system. The at least two images may be captured simultaneously. The method may therefore comprise, as a further step, capturing the image pair using the image capture system.

[0009] According to one embodiment of the method, the read-in image pair can be a camera image pair that has been captured with an image capture system that has at least one camera. The camera image pair can have at least two camera images. The image capture system can therefore be a camera system. The image pair can be a two-dimensional image pair that can have at least two two-dimensional images. The image pair can be or have a stereo image pair. The at least two two-dimensional images can be stereo images. The stereo image pair can be captured by a stereo camera system, wherein the image capture system can be the stereo camera system. The stereo camera system can have at least two cameras.

[0010] The image pair can also be a three-dimensional image pair, which can comprise at least two three-dimensional images. The image pair can be or comprise a point cloud pair. The at least two three-dimensional images can be point clouds. The point cloud pair can be acquired by a distance-measuring system, whereby the image acquisition system can be the distance-measuring system. The distance-measuring system can comprise at least one of at least one laser scanner, at least one radar device, and at least one 3D camera. Alternatively or in addition to the distance-measuring system, the point cloud pair can also be acquired by the stereo camera system.

[0011] The image pair contains image information about an object located in the vehicle's off-road environment. The object can be any object, such as a living being. The image pair also contains image information about the environment. The environment can contain environmental objects. The image information about the environment can therefore contain image information about environmental objects. The environment can be an off-road environment or an off-highway environment.

[0012] The image information can include color information about the object and its surroundings. The color information can be RGB information, which can be captured, for example, with the stereo camera system. The image information can include depth information about the object and its surroundings. The depth information can be point coordinates, which can be captured, for example, with the distance-measuring system. The point coordinates can be at least one-dimensional point coordinates.

[0013] The method comprises, as a further step, determining disparities of the image information of the read-in image pair. The disparities can be image offsets of the image information in the image pair, wherein the image information depicts the object and the surroundings in the image pair. The disparities can therefore be image offsets of corresponding image information to the object and to the surroundings in the image pair. The disparities can therefore be image disparities. According to one embodiment, a difference image can be derived from the read-in image pair, which difference image has differences in the image information. The disparities can be the differences in the difference image.

[0014] If the image pair is a stereo image pair, the disparities can be stereo image disparities. If the image pair is a point cloud pair, the disparities can also be point offsets of the image information in the image pair, where the image information depicts the object and the surroundings in the image pair. The disparities can therefore be point offsets of corresponding image information to the object and the surroundings in the image pair.

[0015] The method comprises, as a further step, comparing the determined disparities with a predetermined disparity threshold. The predetermined disparity threshold can be a predetermined image offset threshold or a predetermined point offset threshold. The disparity threshold can be a minimum disparity with which the determined disparities are compared. The comparison results from the comparing step can be that the determined disparities are below or above the predetermined disparity threshold.

[0016] According to a further embodiment of the method, in the step of comparing the determined disparities, it can be checked whether the determined disparities exceed the predetermined disparity threshold. The comparison results can thus also show that the determined disparities exceed the predetermined disparity threshold.

[0017] The method comprises, as a further step, filtering corresponding image information about the object from the image information. The filtering step is performed depending on the comparison results resulting from the step of comparing the determined disparities. The filtering can comprise a binary decision or check as to whether the image information is corresponding image information about the object.

[0018] According to a further embodiment, in the filtering step, image information from the read-in image pair can be determined as the corresponding image information of the object if the disparities determined from the image information in the comparing step exceed the predetermined disparity threshold. The method comprises, as a further step, recognizing the object based on the determined corresponding image information. The object can be derived from the determined corresponding image information using image processing or pattern recognition methods. The corresponding image information of the object can thus be advantageously determined in a resource-efficient manner and used as the basis for further steps of the method.

[0019] The method comprises, as a further step, outputting a control signal to an operating device of the vehicle for controlling the vehicle's driving dynamics off-road depending on the detected object. The operating device can be a device of a drive train of the vehicle, for example a drive motor, a transmission, a steering device, or a braking device. The driving dynamics can be at least one of a longitudinal dynamics and a lateral dynamics of the vehicle. The operating device can therefore also be a device for controlling the longitudinal dynamics of the vehicle or a device for controlling the lateral dynamics of the vehicle.

[0020] The method can be used to filter out an object in the surroundings of a vehicle and to control the vehicle's off-road operation based on the filtered object. If the filtered object is a potential collision object, the vehicle's driving dynamics can be controlled in such a way that a collision with the object can be avoided. This is based on the finding that disparities in the corresponding image information about the object can be greater than disparities in corresponding image information about the surroundings, for example, about underground objects or background objects in the surroundings. Disparities in corresponding image information can therefore be inversely proportional to the corresponding object depth values ​​of the corresponding image information. Corresponding image information about an object in the vicinity of the vehicle can therefore be determined.Advantageously, the object in the vehicle's surroundings can be robustly filtered and the corresponding image information about the object can be determined without the need for image-based object classification or machine learning.

[0021] The method can filter and detect an object in the vicinity of a vehicle off-road, even when the surroundings are subject to changing environmental conditions. The underlying insight is that disparities in the corresponding image information about the object can be filtered regardless of changing environmental conditions. In contrast to image-based object classification or machine learning, the method can detect an object in the vicinity of a vehicle off-road, regardless of changing environmental conditions.

[0022] The method comprises, as a further step, determining a volume body of the object based on the determined corresponding image information. The step of recognizing the object is carried out based on the determined volume body. The volume body can be derived from the determined corresponding image information. A point cloud with point information about the object can be determined from the corresponding image information using stereophotogrammetry methods. Alternatively or in addition to the derived point cloud, distance information about the object can be determined from the corresponding image information. The volume body can be determined using surface triangulation of the point cloud. Alternatively or in addition to the surface triangulation of the point cloud, the volume body can be determined using surface triangulation of the distance information.The distance information can be depth information. The surface triangulation can be Delaunay triangulation.

[0023] According to a further embodiment of the method, this can comprise, as a further step, a comparison of the volume of the specific volume body with a predetermined volume threshold. The volume threshold can be a minimum volume value. In the step of comparing the specific volume body, it can be checked whether the volume of the specific volume body exceeds the predetermined volume threshold. In the step of recognizing, the object can be recognized depending on a comparison result resulting from the step of comparing the volume. The object can be recognized if the comparison result from the step of comparing is that the volume of the specific volume body exceeds the predetermined volume threshold.

[0024] According to a further embodiment of the method, this comprises, as a further step, determining position information about the object and the surroundings from the image information of the read-in image pair. The position information can be or include height information about the object and the surroundings. The position information can also be or include depth information about the object and the surroundings. The position information about the object and the surroundings can be determined from position information about the vehicle and the image information of the read-in image pair. The position information about the vehicle can be determined from measurement data from at least one position determination sensor arranged on the vehicle. The position determination sensor can be, for example, a satellite navigation system, an inertial measuring unit, or an odometric measuring unit.The method may comprise, as a further step, comparing the determined position information with at least one predetermined position threshold. The at least one predetermined position threshold may be at least one predetermined height threshold. Alternatively or additionally, the at least one predetermined position threshold may be at least one predetermined depth threshold. The step of filtering the corresponding image information relating to the object may be performed depending on a comparison result resulting from the step of comparing the determined position information.

[0025] According to a further embodiment of the method, in the step of determining the position information, height information about the object and the surroundings can be determined from the image information of the read-in image pair. In the step of comparing the determined position information, the determined height information can be compared with at least one predetermined height threshold. The image information can be determined in the filtering step as corresponding image information about the object if the position information about the object exceeds the predetermined height threshold. The determination of corresponding image information about a terrain or subsurface on which the object may be located can thus be efficiently avoided.

[0026] According to a further embodiment of the method, the object can be recognized as a potential collision object in the detection step. The object can be recognized as the potential collision object if the object is located in front of the vehicle in the direction of travel. The object can be recognized as the potential collision object if the object is located in a travel trajectory of the vehicle. In the output step, the control signal can be output to a braking device of the vehicle for inducing an emergency stop of the vehicle off-road depending on the detected potential collision object. A collision with the object recognized as a potential collision object can thus be efficiently avoided.

[0027] In another aspect, the present invention relates to a control device for controlling an off-road vehicle. The control device is configured to carry out the method according to the preceding aspect.

[0028] The control device has a data interface for reading in an image pair containing image information about an object located in the vehicle's off-road environment and about the environment. The control device also has a computing unit configured to determine disparities in the image information of the read-in image pair, compare the determined disparities with a predetermined disparity threshold, filter corresponding image information about the object from the read-in image information depending on the compared disparities, and recognize the object based on the determined corresponding image information. The control device can further be configured to carry out at least one step of the method according to the preceding aspect.The control device also has a control interface for outputting a control signal to an operating device of the vehicle for controlling driving dynamics of the vehicle off-road depending on the detected object.

[0029] In yet another aspect, the present invention relates to a vehicle designed for off-road operation. The vehicle has a control device according to the preceding aspect for controlling the vehicle's driving dynamics off-road. Short description of the characters Fig. 1 shows a vehicle and a control device for controlling the vehicle off-road according to a respective embodiment of the invention. Fig. 2 shows a schematic flow diagram with method steps of a method for controlling a vehicle off-road according to an embodiment of the invention. Detailed description of embodiments

[0030] Fig. 1 shows a vehicle 100 configured for off-road operation. The vehicle 100 has an image capture system 20. The image capture system 20 is arranged at the front of the vehicle and has a detection area looking forward in the direction of travel F of the vehicle 100. In the embodiment shown, the image capture system 20 has two cameras 22. The image capture system 20 forms a stereo camera system.

[0031] The image capture system 20 is configured to capture an object 10 in the forward-looking detection range in the direction of travel F of the vehicle 100. The image capture system 20 is also configured to capture an environment 2 in the forward-looking detection range in the direction of travel F of the vehicle 100. The environment 2 includes environmental objects that are further away from the vehicle 100 than the object 10.

[0032] The vehicle 100 has a control device 120, which is connected to the image acquisition system 20 via a data interface 122. The control device 120 has a computing unit 124, which is configured to process image information relating to the object 10 and the surroundings 2, which image acquisition system 20 acquires.

[0033] The vehicle 100 also has an operating device 110, which, in the embodiment shown, is configured as a braking device 112 for executing an emergency stop of the vehicle 100. The operating device 110 is connected to the control device 120 via a control interface 126. The computing unit 124 is configured to output a control signal for triggering the emergency stop to the braking device 112 via the control interface 126. The control signal is output depending on the object 10 located in front of the vehicle 100 in the direction of travel F, which is detected as a potential collision object.

[0034] Fig. 2 shows a flowchart with steps S0 to S8 for performing a method for controlling the vehicle 100 off-road in a temporal sequence of the steps S0 to S8 according to one embodiment.

[0035] In a step S0, an image pair is captured with the image acquisition system 20, with each camera 22 of the image acquisition system 20 capturing a camera image of the image pair. The image pair contains image information about the object 10, which is located off-road in the environment 2 of the vehicle 100, and about surrounding objects in the environment 2. In a further step S1, the captured image pair is read into the computing unit 124 of the control device 120 via the data interface 122.

[0036] In a further step S2, the computing unit 124 determines image disparities of the image information from a difference image of the image pair. The image disparities are calculated by the computing unit 124 from differences in the image information. In a further step S3, the determined disparities are compared with a predetermined disparity threshold. In step S3, a check is made to determine whether the disparities exceed the predetermined disparity threshold.

[0037] In a further step S4, corresponding image information relating to object 10 is filtered from the image information. In the further step S4, image information whose disparities exceed the predetermined disparity threshold is determined as corresponding image information relating to object 10. In an optional further step T1, position information relating to object 10 and surrounding objects of environment 2 is determined from the image information of the read-in image pair. In yet another optional step T2, the determined position information is compared with at least one predetermined position threshold. In this step, the corresponding image information is optionally filtered from the image information as corresponding image information relating to object 10 if, in step T2, the determined position information exceeds the at least one predetermined position threshold.

[0038] In a further step S5, a volume of the object 10 is determined from the corresponding image information obtained. The volume is determined from a point cloud derived from the corresponding image information. In a further step S6, the volume of the determined volume is compared with a predetermined volume threshold.

[0039] In a further step S7, the object 10 is identified as a potential collision object in the direction of travel F of the vehicle 100 if, in step S6, the volume of the specific volume body exceeds the predetermined volume threshold. In a further step S8, the control device 120 outputs a control signal to the braking device 112 via the control interface 126 to trigger an emergency stop of the vehicle 100 and thus avoid a collision of the vehicle 100 with the object 10. Reference symbol 2 Surroundings 10 objects 20 image acquisition system 22 Camera 100 vehicles 110 Operating equipment 112 Braking device 120 Control device 122 Data interface 124 computing unit 126 Control interface F Direction of travel S0 Capture image pair S1 Read image pair S2 Determine image disparities S3 Compare image disparities S4 Filter image information S5 Determine solid bodies S6 Compare Volume S7 Detect Object S8 Output control signal T1 Determine location information T2 Compare Location Information

Claims

[1] Method for controlling a vehicle (100) off-road, comprising the steps of: reading in (S1) an image pair which has image information about an object (10) which is located in an environment (2) of the vehicle (100) off-road, and about the environment (2), determining (S2) disparities of the image information of the read-in image pair, comparing (S3) the determined disparities with a predetermined disparity threshold, filtering (S4) corresponding image information about the object (10) from the image information depending on comparison results resulting from the step of comparing (S3) the determined disparities, recognizing (S7) the object (10) based on the determined corresponding image information and outputting (S8) a control signal to an operating device (110) of the vehicle (100) for controlling a driving dynamics of the vehicle (100) off-road depending on the recognized object (10), characterized bythe further step of determining (S5) a volume body of the object (10) based on the determined corresponding image information, wherein the step of recognizing (S7) the object (10) is carried out based on the determined volume body. [2] Method according to claim 1, wherein the read-in image pair is a camera image pair which has been captured with an image capture system (20) which has at least one camera (22). [3] Method according to claim 1 or 2, wherein in the step of comparing (S3) the determined disparities it is checked whether the determined disparities exceed the predetermined disparity threshold, and wherein in the step of filtering (S4) image information of the read-in image pair is determined as the corresponding image information of the object (10) if the disparities of the image information exceed the predetermined disparity threshold. [4] Method according to one of the preceding claims, with the further step of comparing (S6) the volume of the determined volume body with a predetermined volume threshold value, wherein in the step of recognition (S7) the object (10) is recognized as a function of a comparison result resulting from the step of comparing (S6) the volume. [5] Method according to one of the preceding claims, comprising the further steps: Determining (T1) position information about the object (10) and the environment (2) from the image information of the read-in image pair, comparing (T2) the determined position information with at least one predetermined position threshold value, wherein the step of filtering (S4) the corresponding image information about the object (10) is carried out as a function of a comparison result resulting from the step of comparing (T2) the determined position information. [6] Method according to claim 5, wherein in the step of determining (T1) the position information, height information about the object (10) and the environment (2) is determined from the image information of the read-in image pair, and wherein in the step of comparing (T2) the determined position information, the determined height information is compared with at least one predetermined height threshold value. [7] Method according to one of the preceding claims, wherein in the step of detection (S7) the object (10) is detected as a potential collision object, and wherein in the step of outputting (S8) the control signal is output to a braking device (112) of the vehicle (100) for bringing about an emergency stop of the vehicle (100) off the road in dependence on the detected potential collision object. [8] Control device (120) for controlling a vehicle (100) off-road, characterized bythat the control device (120) is configured to carry out the method according to one of the preceding claims, wherein the control device (120) has a data interface (122) for reading in an image pair which comprises image information about an object (10) located in an environment (2) of the vehicle (100) off-road, and about the environment (2), a computing unit (124) which is configured to determine disparities of the image information of the read-in image pair, to compare the determined disparities with a predetermined disparity threshold value, to filter corresponding image information about the object (10) from the read-in image information depending on the compared disparities, and to recognize the object (10) based on the determined corresponding image information,and a control interface (126) for outputting a control signal to an operating device (110) of the vehicle (100) for controlling driving dynamics of the vehicle (100) off-road depending on the detected object (10). [9] Vehicle (100) designed to be operated off-road, characterized by that the vehicle (100) has a control device (120) according to claim 8 for controlling driving dynamics of the vehicle (100) off-road.

Images