Determining the condition of a vehicle to support a parking maneuver

Abstract

The invention relates to a method, in particular a computer-implemented method, for determining the condition of a vehicle (1), a parking assistance system for a vehicle (1), a computer program, and a Computer program product. The process comprises the following steps: - Capturing at least one first and one second image (B1, B2) using a vehicle camera system (2) comprising at least one camera (3,3a-3d), wherein the first (B1) and the second image (B2) have at least a partially overlapping capture area, - Determining an optical flow (OF) at least based on the first (B1) and second image (B2), - Determining the direction of travel (dir) of the vehicle (1) based on the optical flow (OF), and - Determining a state of the vehicle (1) at least on the basis of a position (P) and / or orientation (Ψ) of the vehicle (1) taking into account the direction of travel (dir).

Description

[0001] The present invention relates to a method, in particular a computer-implemented method, for determining the position of a vehicle, a parking assistance system for a vehicle, a computer program for carrying out the method and a computer program product with the computer program according to the invention.

[0002] Today, systems are increasingly being offered to assist vehicle parking processes. These systems provide appropriate support to the driver in searching for a parking space and / or in performing a parking maneuver, particularly when parking or exiting a parking space. Such parking assistance systems or Advanced Driver Assistance Systems (ADAS) with appropriate functionality also allow, for example, fully automated parking maneuvers. For (semi-)automatic parking, but not only for this purpose, it is necessary to precisely determine the position and orientation of the vehicle during the maneuver.

[0003] Knowledge of the vehicle's position and orientation is crucial, for example, for maneuvering the vehicle collision-free along a planned trajectory into or out of a final parking position. To determine the vehicle's position and / or orientation, the sensors of a stability control system (ESC), in particular a wheel speed sensor, a yaw rate sensor, and / or longitudinal and / or lateral acceleration sensors, can be used.

[0004] However, the vehicle's position can also be determined using a variety of odometry methods. These methods utilize various information from the vehicle's propulsion system to determine position and / or orientation. For example, the number of wheel revolutions can be used.

[0005] DE102020201658A1 also discloses a method for determining a vehicle position, in which two distance increments are determined: one based on measurement data from at least one ESC sensor and the other based on measurement data from at least one vehicle camera. A Kalman filter is used to combine the two distance increments to determine a fused distance increment, and from this, the vehicle position is determined. This can improve the robustness and accuracy of the position determination.

[0006] During the parking maneuver, the vehicle's direction of travel frequently changes. Accordingly, this direction must be determined reliably and precisely. Determining the respective direction of travel can be particularly problematic during starting maneuvers, i.e., when the vehicle starts moving from a standstill. This can lead to delayed or even incorrect determination, which in turn can lead to unacceptable errors during the parking maneuver.

[0007] The direction of travel can be determined, for example, using a wheel speed sensor and the direction of rotation of the wheels. While this method is highly reliable and precise when the vehicle is already moving, the direction of travel cannot be determined directly at low speeds and especially during acceleration.

[0008] Another option is to detect the currently engaged gear. However, this can be problematic because it cannot be ruled out that the vehicle might move in the opposite direction to the engaged gear, for example, on an incline or near curbs. For these reasons, determining direction solely based on the engaged gear is often not sufficiently reliable.

[0009] DE102019220118A1 and DE102019220120A1 disclose methods for determining the direction of travel or the state of motion of a vehicle based on information about vehicle acceleration. These methods can determine the state of motion of the vehicle or the direction of travel immediately after starting and at low speeds. However, when the vehicle stops, oscillations occur, which are also incorrectly detected by the acceleration sensors and make determining the direction of travel more difficult. Even at low speeds, such as those typical when parking, the accuracy of such a direction determination is reduced.

[0010] DE10202009863A1 also describes a possibility for determining the direction of travel using a yaw rate sensor and a steering wheel angle sensor.

[0011] Based on this, the present invention is based on the object of being able to determine the direction of travel, position and orientation even more reliably.

[0012] This object is achieved by the method according to claim 1, by the advantageous use of the method according to claim 9, by the parking assistance system for a vehicle according to claim 10, by the computer program according to claim 11 and by the computer program product according to claim 12.

[0013] The method according to the invention, in particular a computer-implemented method, for determining the condition of a vehicle comprises the following method steps: - capturing at least a first and a second image by means of a vehicle camera system comprising at least one camera, wherein the first and the second image have an at least partially overlapping detection range, - Determining an optical flow based at least on the first and second images, - Determining a vehicle's direction of travel based on the optical flow, and - Determining a state of the vehicle based at least on a position and / or orientation of the vehicle, taking into account the direction of travel.

[0014] The state determination according to the invention therefore comprises a determination of at least the position and / or orientation of the vehicle, taking into account the respective direction of travel, which is also determined. This advantageously involves a position determination for the vehicle as a function of time. The determined state is advantageously used to assist in a parking process of the vehicle.

[0015] The first and second images, which are recorded or captured by the at least one camera of the vehicle camera system, have a common image area, i.e. at least some of the same features of the vehicle's surroundings are visible in the first and second images. For example, the first and second images can be two images captured at successive points in time that relate either to the same or to a partially overlapping capture area relating to the vehicle's surroundings. The at least one camera can be arranged at different positions relative to the vehicle; for example, it can point forwards, backwards or, preferably, to the side in the direction of travel. The direction of travel can be determined particularly accurately on the basis of the images from a laterally aligned camera.

[0016] Preferably, the overlapping detection area in the two images is determined, for example, by a direct comparison of both images, and the optical flow is determined based on the image areas that relate to the same partial section of the vehicle's surroundings. It should be noted that the optical flow can be determined in various ways, all of which fall within the scope of the present invention. In this context, both classical methods and machine learning methods can be used.

[0017] Determining the vehicle's direction based on optical flow is particularly advantageous when the vehicle is moving at a lower speed, especially when starting from a standstill. This approach also offers high precision in these cases. Precise knowledge of the direction of travel, in turn, enhances various functionalities of parking assistance or driver assistance systems. Thus, an at least partially automatic parking process based on the method according to the invention can be significantly improved, both in terms of the accuracy of planning the respective trajectory and in terms of the reliability of the process.

[0018] Preferably, the optical flow is determined based on coordinates of corresponding features determined in the first and second images. Corresponding features are, for example, objects detected in the images. In particular, they are immobile or non-moving objects. Assuming that the at least one camera is directed forward in the direction of travel of the vehicle, a positive direction of travel, i.e. that the vehicle is moving forward, can be inferred when the corresponding features in the two images approach the camera. Conversely, movement away from the camera allows a conclusion to be drawn about a negative direction of travel, i.e. a backward movement of the vehicle. Similar considerations can be made directly for other arrangements of the at least one camera relative to the vehicle.

[0019] To determine the optical flow, a single corresponding feature can be used. However, a multitude of corresponding features can also be used, in particular, a multitude of detected features on a section of the road visible in the images. If multiple corresponding features are taken into account, a statistical analysis can be used to determine the direction of travel, i.e., for example, the direction of travel that corresponds to a predominant portion of the detected optical flow can be inferred. In this way, the reliability of the direction of travel determination can be further increased.

[0020] It is advantageous if the position and orientation of the vehicle are determined using an odometry method. However, the use of an odometry method is not mandatory. Rather, other methods known from the state of the art for determining position and / or orientation can also be used.

[0021] According to an advantageous embodiment, the camera system comprises at least two cameras, wherein the optical flow is determined at least on the basis of images captured by a first camera, i.e. a first and second image from the first camera, and on the basis of images captured by a second camera, i.e. a first and second image from the second camera. In this case, the optical flow is therefore advantageously determined redundantly, which likewise leads to a further improvement in the reliability of the state determination for the vehicle. An optical flow determined by means of a camera can be used to determine the direction of travel, wherein suitable criteria can be established for selecting the optical flow used in each case. However, it is also conceivable to suitably merge the optical flows determined by means of different cameras.

[0022] Furthermore, it is also conceivable to use images from two different cameras, which have at least a partially overlapping detection range, to determine the optical flow.

[0023] According to a further advantageous embodiment of the method according to the invention, the direction of travel is also determined using a complementary method for determining the direction of travel. The optical flow is thus determined using different methods, in particular redundantly. The use of different determination methods further increases the reliability of the state determination and, above all, improves its accuracy at very low vehicle speeds and especially during a start-up process.

[0024] It is advantageous if the complementary method includes a determination of the direction of travel based on information about an engaged gear of the vehicle, based on a vehicle acceleration, in particular a longitudinal and / or lateral acceleration, which can preferably be detected using at least one acceleration sensor, and / or based on a wheel speed sensor.

[0025] With regard to the additional determination of the direction of travel of the vehicle using a complementary method, it is also advantageous if, in order to determine the direction of travel, the direction of travel determined using the optical flow and the direction of travel determined using at least one complementary method are merged with one another.

[0026] With regard to fusion, it is again advantageous if, in the event that the direction of travel cannot be determined based on the optical flow, the direction of travel determined based on the engaged gear or, preferably, the direction of travel determined based on the vehicle's acceleration is used to determine the vehicle's state, at least based on the vehicle's position and / or orientation. This also increases reliability, as the failsafe nature of the state determination is improved.

[0027] It is also conceivable to verify or falsify the direction of travel determined using optical flow using one or more complementary methods.

[0028] It is also advantageous if, in the case where the direction of travel can be determined using the wheel speed sensor, the direction of travel determined using the wheel speed sensor is used to determine the state of the vehicle, at least based on the position and / or orientation of the vehicle. Determining the direction of travel using a wheel speed sensor is characterized by particularly high accuracy, but only above a certain minimum speed. Thus, according to this embodiment, the direction of travel would be determined based on the optical flow, particularly for speeds below a predeterminable limit speed, and the direction of travel would be determined based on the wheel speed sensor for speeds above the predeterminable limit speed.

[0029] The object underlying the invention is also achieved by using the method according to the invention according to one of the described embodiments, which can be combined with one another in any desired way, for carrying out a parking process, in particular a parking or reversing process for a vehicle. The method according to the invention is therefore advantageously used in a parking assistance system or in a driver assistance system, which may, for example, include a parking assistance system.

[0030] The object underlying the invention is also achieved by a parking assistance system for a vehicle, which is designed to carry out the method according to the invention according to one of the described embodiments, by a computer program with instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the invention according to one of the described embodiments, and by a computer program product on which the computer program according to the invention is stored.

[0031] The invention and its advantageous embodiments are explained in more detail with reference to the following figures. They show: Fig. 1 a schematic representation of a vehicle with a vehicle camera system comprising several cameras; Fig. 2 determining the direction of travel based on optical flow; and Fig. 3 a flow chart to illustrate the method according to the invention.

[0032] In the figures, identical elements are always provided with the same reference symbol.

[0033] In Fig. Figure 1 shows a vehicle 1 with a vehicle camera system 2 comprising four cameras 3a-3d. A first camera 3a is positioned at the front of the vehicle 1 in the direction of travel, a second camera 3b and a third camera 3c are positioned at the side, and a fourth camera 3d is directed rearward.

[0034] The vehicle 1 further comprises a wheel speed sensor 5a-5d on each of the four wheels 4a-4d.

[0035] A position P=(x;y) and an orientation Ψ of the vehicle 1 can be determined, for example, using an odometry method, in particular by summing small longitudinal movement steps Δs. The position (xk +1 ; y k+1 ) and orientation Ψ k+1at time k+1 then result, for example, from the position (x k ; y k ) and orientation Ψ k at time k to: [xk+1yk+1ψk+1]=[xk+dir⋅Δscos(β)⋅cos(ψk+ψk.⋅Δt / 2+β)yk+dir⋅Δscos(β)⋅sin(ψk+ψk.⋅Δt / 2+β)ψk+ψk.⋅Δt]

[0036] The direction of travel is yours and can, for example, take the values ​​+1 for a positive direction of travel (forward) and -1 for a negative direction of travel (backward), ΨΔ˙t is the temporal change of orientation Ψ and β is the float angle.

[0037] The direction of travel is determined according to the invention using the optical flow OF, as shown in Fig. 2. An exemplary possibility for determining the optical flow OF is the detection of corresponding features in the first B1 and second B2 image of camera 3 [not shown separately here]. In the case of Fig. 2, let "x", "+", and "y" be corresponding features in the two images B1 and B2 of camera 3. If the features "x", "+", and "y" approach camera 3, a positive direction of travel dir=+1 can be inferred. If, on the other hand, the features "x", "+", and "y" move away from camera 3, a negative direction of travel dir=-1 exists.

[0038] Fig. 3 relates to a flowchart illustrating the method according to the invention. In a first step, a first image B1 and a second image B2 are captured using the at least one camera 3 of the vehicle camera system 2. These images B1 and B2 are used to determine the optical flow OF, which in turn is used to determine the direction of travel dir of the vehicle 1. This direction of travel dir is then taken into account when determining the state of the vehicle 1, at least based on a position P=(x;y) and / or orientation Ψ of the vehicle 1.

[0039] The direction of travel you com can also optionally be determined using at least one complementary method (shown in dashed lines). Depending on the case, the determined directions of travel dir and dir can then be merged. com be made. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 102020201658A1

[0005] DE 102019220118A1

[0009] DE 102019220120A1

[0009] DE 10202009863A1

[0010]

Claims

[1] Method, in particular computer-implemented method, for determining the condition of a vehicle (1) comprising the following method steps: - capturing at least a first and a second image (B1, B2) by means of a vehicle camera system (2) comprising at least one camera (3, 3a-3d), wherein the first (B1) and the second image (B2) have an at least partially overlapping detection range, - Determining an optical flow (OF) based at least on the first (B1) and second image (B2), - determining a direction of travel (dir) of the vehicle (1) based on the optical flow (OF), and - Determining a state of the vehicle (1) at least based on a position (P) and / or orientation (Ψ) of the vehicle (1) taking into account the direction of travel (dir). [2] Method according to claim 1, wherein the position (P) and orientation (Ψ) of the vehicle (1) are determined using an odometry method. [3] Method according to claim 1 or 2, wherein the camera system (2) comprises at least two cameras (3a-3d), and wherein the optical flow (OF) is determined at least on the basis of images captured by a first camera (3a) and on the basis of images captured by a second camera (3b). [4] Method according to at least one of the preceding claims, wherein the direction of travel (dir com ) using a complementary method (com) to determine the direction of travel (dir com ) is determined. [5] Method according to claim 4, wherein the complementary method (com) comprises determining the direction of travel (dir com ) based on information about an engaged gear of the vehicle (1), based on a vehicle acceleration, in particular a longitudinal and / or lateral acceleration, which can preferably be detected using at least one acceleration sensor, and / or based on a wheel speed sensor. [6] Method according to one of claims 4 or 5, wherein, to determine the direction of travel (dir), the direction of travel (dir) determined in each case using the optical flow (OF) and the direction of travel (dir) determined by means of at least one complementary method (com) are used. com ) are merged with each other. [7] Method according to claim 6, wherein in the event that the direction of travel (dir) cannot be determined based on the optical flow (OF), the direction of travel (dir) determined based on the engaged gear or preferably the direction of travel (dir) determined based on the vehicle acceleration com ) is used to determine the state of the vehicle (1) at least based on a position (P) and / or orientation (Ψ) of the vehicle (1). [8] Method according to claim 6 or 7, wherein in the event that the direction of travel (dir com ) can be determined using a wheel speed sensor, the direction of travel determined using the wheel speed sensor (dir com) is used to determine the state of the vehicle (1) at least based on a position (P) and / or orientation (Ψ) of the vehicle (1). [9] Use of the method according to one of the preceding claims for carrying out a parking operation, in particular a parking or exiting operation of a vehicle (1). [10] Parking assistance system for a vehicle (1), which is designed to carry out the method according to one of claims 1-8. [11] A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to any one of claims 1-8. [12] A computer program product on which the computer program according to claim 11 is stored.

Images