Method for displaying a target parking position of a coupled trailer to a user of a vehicle, computer program, control device or central computing device, and vehicle

Abstract

The invention relates to a method for displaying a target parking position (201, 251) of a coupled trailer (120) to a user of a vehicle (110), wherein the vehicle (110) and / or the trailer (120) comprises at least one camera (114 to 117) as a sensor, having the following steps: capturing (310) sensor data, the sensor data representing the surroundings (100) of the vehicle (110) with the coupled trailer (120); determining (320) a virtual surroundings model (200) on the basis of the captured sensor data; displaying (330) the surroundings model (200); and determining (350) the location of the target parking position for the trailer on the basis of the captured sensor data and / or the determined surroundings model; the following steps being carried out: determining (360) a section (250) of the sensor data and / or of the virtual surroundings model (200), the section (250) depicting the determined location of the target parking position (201, 251) for the trailer (120); displaying (370), in addition to the displayed surroundings model (200), the determined section (250) of the sensor data or the determined section (250) of the virtual surroundings model (200) with a virtual perspective which is modified compared to the displayed surroundings model (200); and overlaying (380) the determined location of the target parking position (201, 251) in the displayed surroundings model (200) and in the displayed determined section (250) of the sensor data or the virtual surroundings model (200).

Description

[0001] R.409863

[0002] - 1 -

[0003] Description

[0004] Method for displaying a target parking position of a coupled trailer to a user of a vehicle, computer program, control unit or central computing device and vehicle

[0005] The present invention relates to a method for displaying the target parking position of a coupled trailer to a vehicle user. The invention also relates to a computer program comprising commands which, when executed by a computer, cause the computer to perform the steps of the method according to the invention. Furthermore, the invention relates to a control unit or a central computing device with a processing unit configured to perform the steps of the method according to the invention, and to a vehicle comprising at least one control unit or central computing device according to the invention.

[0006] State of the art

[0007] Document US 2012 / 0271514 A1 discloses a rotatable control element to assist in steering a trailer.

[0008] Document US 2015 / 197278 A discloses a method for controlling the movement of a vehicle combination comprising a towing vehicle and a trailer, wherein the towing vehicle has a drivetrain electronically controllable by control means of the towing vehicle, wherein the control means of the towing vehicle are wirelessly connected to a touchscreen device, and wherein autonomous movement of the vehicle combination can be remotely controlled by means of the touchscreen device. An articulation angle between a longitudinal axis of the towing vehicle and a longitudinal axis of the trailer is continuously R.409863

[0009] - 2 - regulated, whereby the combination is graphically displayed on the touchscreen of the touchscreen device.

[0010] Document US 2021 / 0229509 A1 discloses a remote control system for trailer maneuvering, which includes a mobile app. The remote control system detects obstacles, provides information and warnings, and requests information and feedback from the user for control purposes.

[0011] Patent application DE 10 2022 122 847 A1 discloses a system and method for rapidly switching between an augmented reality intervention interface and an orbital motion intervention interface, operated via an application on a mobile device. The application can be used to remotely control functions for the parking assistant of a semi-autonomous vehicle. The HMI rapid switching system evaluates user intervention using a complex gesture and video input using the sensors of the mobile device.

[0012] The present invention has the objective, compared to the known prior art, of improving the display of a target parking position of a coupled trailer for a user of a vehicle.

[0013] Disclosure of the invention

[0014] The above problem is solved according to the invention in accordance with independent claims 1 and 7 to 9.

[0015] The present invention relates to a method for displaying a target parking position of a coupled trailer to a user of a vehicle, wherein the vehicle and / or the trailer comprises at least one camera as a sensor, the camera advantageously being configured to detect the vehicle's surroundings. Preferably, the vehicle and / or the trailer comprise several sensors, one of which, for example, is a vehicle sensor and / or a trailer sensor that includes the camera. It can be provided, for example, that the vehicle and / or the trailer each comprises one or more cameras, one or more ultrasonic sensors, one or more radar sensors, and / or one or more lidar sensors as sensors. Different types of cameras can be provided, for example, narrow-angle cameras, R.409863.

[0016] - 3 -

[0017] Wide-angle cameras and / or stereo cameras. In one step of the process, sensor data is acquired, representing the environment of the vehicle with the attached trailer. The sensor data is acquired, in particular, using one or more of the vehicle's sensors and / or one or more of the trailer's sensors. Subsequently, a virtual environment model is determined based on the acquired sensor data, with the environment model representing the vehicle's surroundings. It is therefore possible for the environment model to be based on sensor data from different sensor types. For example, the environment model is determined or generated based on sensor data from several vehicle cameras, an optional camera at the rear of the trailer, several ultrasonic sensors, and / or at least one radar and / or lidar sensor, in particular by fusing sensor data from different sensors.The determined environmental model can, for example, be a surround view model in which current and optionally stored camera images are projected onto a projection surface, the projection surface preferably having horizontal and vertical areas. The projection surface, or the horizontal and / or vertical areas of the projection surface, are advantageously adapted based on sensor data from the at least one camera and / or the at least one ultrasonic sensor and / or the radar sensors and / or the lidar sensor or the distance sensors. For example, the position of the vertical area of ​​the projection surface is shifted towards the vehicle in the case of nearby objects, thereby reducing disturbing artifacts. The determined environmental model is then displayed, in particular by means of a display device.The environmental model is preferably displayed from a virtual perspective from above, in particular from a virtual perspective directly above or from a bird's-eye view. In a further step of the process, the location of the target parking position for the trailer is determined based on the acquired sensor data and / or the environmental model. For example, the location of the target parking position can be determined using ultrasonic sensor data, camera sensor data, radar sensor data, and / or lidar sensor data. For example, the location of the target parking position is detected in one or more camera images as sensor data by a trained machine learning algorithm, such as a neural network, and / or the location of the target parking position is determined based on detected lines.The advantageous options for determining the target parking position are explained or elaborated upon in more detail below. R.409863.

[0018] - 4 -

[0019] Furthermore, a section of the sensor data or the virtual environment model is selected, representing the determined location of the target parking position for the trailer. In other words, for example, a camera image from a side-facing camera is selected, depicting the location of the target parking position. Alternatively, a section of the virtual environment model is selected, representing the determined location of the target parking position for the trailer. Subsequently, the selected section of the sensor data, in particular a camera image, or the selected section of the virtual environment model is displayed alongside the displayed environment model, with a virtual perspective that differs from that of the displayed environment model.In other words, the virtual environment model, advantageously from a top-to-bottom perspective, and the specific section of sensor data or the specific section of the virtual environment model are displayed simultaneously side by side, with the specific section of the virtual environment model exhibiting a virtual perspective that differs from the displayed virtual environment model. In a further step of the process, the determined location of the target parking position is overlaid on the displayed environment model and the displayed specific section of sensor data or the virtual environment model. The determined location of the target parking position is displayed in the displayed environment model and the specific section, for example, as a frame or area, which may be color-coded and / or semi-transparent.This allows the user or driver of the vehicle to clearly see, within the displayed environment model and the specific section, where the trailer should or can be parked. The invention advantageously enables the location of the target parking position to be determined using a displayed view of the environment model and verified in the view of a specific section, for example, in a camera image. This results in an improved overview for the user or driver of the determined location of the target parking position in the vehicle's surroundings, thus providing the driver with enhanced support when controlling the vehicle for parking a trailer.For example, the driver can better estimate relative distances between the vehicle and / or trailer and objects, as well as the route to the target parking position, etc., by simultaneously displaying the environment model and the specific section according to the invention, so that collisions of the vehicle with objects are reduced or avoided. This is particularly advantageous when driving with a coupled trailer, as this involves a greater risk of collision (R.409863).

[0020] - 5 - exists. Alternatively, in the case of automatic guidance of the vehicle for parking the trailer at the target parking position, the user is better informed so that he can advantageously monitor the automatic parking maneuver better.

[0021] In an advantageous embodiment of the invention, the determination of the target parking position is additionally based on user input. In this embodiment, the user can advantageously move the displayed target parking position within the shown environment model and / or within the specific displayed section using a touchscreen, thus advantageously adapting the suggested target parking position to their specific requirements.

[0022] In a further embodiment, at least one possible parking space for the trailer is detected based on the acquired sensor data and / or the virtual environment model, in particular by a trained machine recognition method, preferably a neural network. Subsequently, the at least one detected possible parking space is displayed in the environment model and / or in the displayed specific area for the user to select by input. Alternatively or additionally, the location of the target parking position for the trailer can be determined depending on the at least one detected possible parking space. For example, three possible parking spaces are identified, and based on predefined decision criteria, such as the distance of the trailer from the potential parking space, the recommended parking space is automatically selected.This design at least partially automates the determination of the target parking position.

[0023] In an alternative embodiment of the invention, a freely positionable parking area is displayed in the shown environment model and / or in the displayed specific section, wherein the parking area particularly represents the at least one detected possible parking space. Subsequently, it can be provided that the user's input for determining the location of the target parking position includes a translation and / or rotation of the displayed parking area in the environment model and / or in the displayed specific section, particularly by means of a touchscreen. In other words, for example, one or more possible parking spaces can first be detected and R.409863

[0024] - 6 - are displayed, whereby the user can select one of the displayed possible parking spaces as the target parking position by entering an input, for example via a touchscreen, as a freely positionable parking area and adjust its position by moving and / or rotating it within the displayed environment model and / or within the displayed specific section. This design enables a particularly flexible determination of the target parking position, which advantageously allows all of a user's wishes regarding the location of the target parking position and at the same time strongly supports the user in positioning the target parking position.

[0025] In a further step of the process, the trailer angle between the longitudinal axis of the vehicle and the longitudinal axis of the trailer is determined. Alternatively or additionally, the current steering angle of the vehicle is recorded. Subsequently, a predicted travel corridor for the trailer in the vicinity of the vehicle is determined for future reversing maneuvers, based on the determined trailer angle and / or the recorded steering angle of the vehicle. Following this, the system displays the potential parking space and / or the freely positionable parking area and / or determines the location of the target parking position for the trailer, each based on the determined predicted travel corridor.Alternatively or additionally, the specific predicted driving corridor is displayed in the environment model and / or in the displayed specific section of the sensor data and / or in the displayed specific section of the virtual environment model, and / or the area of ​​the predicted driving corridor is displayed differently in the environment model and / or in the displayed specific section of the sensor data and / or in the displayed specific section of the virtual environment model. Alternatively or additionally, the color and / or gray value of the displayed location of the target parking position in the virtual environment model and / or in the displayed section of the sensor data and / or in the displayed section of the virtual environment model is adjusted depending on the determined predicted driving corridor. This implementation supports the determination of the location of the target parking position.

[0026] Preferably, in a further embodiment of the invention, a distance between the determined target parking position and the trailer is determined or calculated based on the acquired sensor data. Subsequently, the selected section is displayed depending on the determined distance between the R.409863

[0027] - 7 - determined target parking position and the trailer. The display of the specific section is only carried out when the specified distance falls below a distance threshold. Alternatively or additionally, the displayed specific section is adjusted depending on the determined distance between the determined target parking position and the trailer, whereby the displayed specific section is enlarged as long as the specified distance is greater than or equal to a distance threshold. This prevents, for example, the distorted edge areas from being displayed when a specific section of the sensor data, especially a camera image, is shown. Furthermore, this makes the location of the target parking position easier for the user to interpret.

[0028] The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause it to perform the steps of the method according to the invention.

[0029] The invention further relates to a control unit or a central computing device. The control unit or the central computing unit comprises at least one signal input for providing a sensor signal, which was acquired by means of at least one sensor, wherein the sensor signal represents acquired sensor data of the environment of a vehicle with a coupled trailer. The control unit or the central computing unit also comprises at least one signal output for outputting a control signal to a display device, wherein the control signal represents a virtual environment model of the vehicle's environment with a displayed parking space for the trailer and a specific section of the sensor data or a specific section of the virtual environment model with the displayed parking space for the trailer.Furthermore, the control unit or central computing unit comprises a computing unit, in particular a processor, which is configured to perform the steps of the method according to the invention.

[0030] The invention further relates to a vehicle comprising at least one sensor, wherein the at least one sensor includes at least one camera. The vehicle also comprises a display device and a control unit or a central computing device according to the invention. R.409863

[0031] - 8 -

[0032] Further advantages will become apparent from the following description of exemplary embodiments with reference to the figures.

[0033] Figure 1: Vehicle

[0034] Figure 2: Display of the environment model and the specific section

[0035] Figure 3: Flowchart of the process as a block diagram

[0036] Examples of implementation

[0037] Figure 1 schematically depicts the vehicle 110 with the attached trailer 120 in an environment 100 containing objects 101 and potential parking spaces 102, viewed from above. The vehicle 110 is equipped with ultrasonic sensors 113 and wide-angle cameras 114 to 117. The ultrasonic sensors are located on the front 111 and rear 112 of the vehicle 110. The wide-angle camera 114 is located on the front 111 of the vehicle 110. The wide-angle camera 115 is located on the rear 112 of the vehicle 110. The wide-angle camera 116 is located on the right side of the vehicle 110. The wide-angle camera 117 is located on the left side of the vehicle 110. The trailer 120 also includes a trailer camera 122 on its rear 121, which also incorporates a wide-angle lens. The ultrasonic sensors 113 and the wide-angle cameras 114 to 117 are designed to record sensor data.The acquired sensor data represent the vehicle's environment 100. For example, distances between the vehicle and / or trailer and objects 101 are determined based on the acquired sensor data. Furthermore, potential parking spaces 102 in the vehicle's environment 100 are automatically detected, for example, based on the acquired sensor data, by a trained machine recognition method or determined based on detected parking space boundary lines or object boundaries 101. The vehicle also includes a display device 118 with a touchscreen. The vehicle also includes a central computing device 119.

[0038] Figure 2 shows the display of the environment model 200 and the specific

[0039] Section 250 of the sensor data is shown schematically, with the determined

[0040] Section 250 in this embodiment represents a camera image, which R.409863

[0041] - 9 - depicts the determined location of the target parking position 201, 251 for parking the trailer 120 of the vehicle 110. In other words, the location of the target parking position 201, 251 for parking the trailer 120 is displayed both in the displayed environment model 200 and in the displayed specific section 250. Advantageously, the location of the target parking position 201, 251 is determined depending on the detected possible parking spaces, depending on the distance between the trailer 120 and the detected possible parking spaces 102, and / or depending on the determined predicted driving corridor 210, 260 of the trailer 120. In particular, only possible parking spaces that lie within the determined predicted driving corridor 210, 260 are relevant for the target parking position. The determined predicted driving corridor 210, 260 is optionally displayed.By simultaneously displaying the environment model 200 and the specific section 250 of the sensor data, and by showing the location of the target parking position 201, 251 in the environment model 200 and in the specific section 250 of the sensor data, the user can accurately estimate the location of the target parking position 201, 251 relative to the objects 101 in the environment 100 and, if necessary, move or rotate the location of the target parking position 201, 251 in the displayed environment model 200 and / or in the displayed section 250 using the touchscreen on the display device 118.It may be provided that the display of the specific section 250 of the sensor data is shown based on the distance x determined depending on the sensor data between the location of the target parking position 201, 251 and the trailer 120, in particular the specific section 250 of the sensor data is only displayed when the determined distance x between the target parking position 201, 251 and the trailer 120 falls below a distance threshold value.

[0042] Figure 3 schematically depicts a flowchart of the process as a block diagram. The process begins with the acquisition of sensor data 310, where the acquired sensor data represents the environment 100 of the vehicle 110 with the attached trailer 120. Subsequently, in step 320, the virtual environment model 200 is determined based on the acquired sensor data. The virtual environment model 200 is then displayed in step 330.

[0043] In an optional step 340, the trailer angle between the longitudinal axis of the vehicle 110 and the longitudinal axis of the trailer 120 is determined. In an optional step 341, the current steering angle of the vehicle 100 is recorded. In an optional step 342, a predicted travel corridor of the trailer 120 in the vicinity 100 of the vehicle 100 is calculated for future R.409863.

[0044] - 10 -

[0045] The reversing movement of vehicle 100 is determined depending on the determined trailer angle and / or the detected steering angle of vehicle 100. Subsequently, in step 350, the location of the target parking position 201, 251 for the trailer 120 is determined based on the acquired sensor data and / or the determined environment model 200. The determination 350 of the location of the target parking position 201, 251 for the trailer 120 is optionally carried out depending on the determined predicted driving corridor. This determination 350 can be based on detected possible parking spaces 102. The at least one possible parking space 102 for the trailer 120 is determined in a previous optional step 343 based on the acquired sensor data and / or the virtual environment model 200, in particular by a trained machine recognition method, preferably by a neural network.The determination 350 of the location of the target parking position 201, 251 for the trailer 120 is optionally carried out depending on the distance x between the possible parking spaces 102 and the trailer 120, whereby in particular the possible parking space 102 which has the shortest distance to the trailer 120 is determined as the target parking position 201, 251. Furthermore, optionally in step 344, the at least one detected possible parking space 102 is displayed in the environment model 200. The display 344 of the possible parking space is advantageously carried out depending on the determined predicted driving corridor. Subsequently, the user selects a displayed detected parking space 102 by means of an input, whereby the input is recorded by means of the touchscreen of the display device 118. The determination 350 of the location of the target parking position 201, 251 is optionally also carried out depending on this recorded input from the user.Furthermore, in optional step 345, a freely positionable parking area can be displayed in the shown environment model 200, wherein the freely positionable parking area represents, in particular, the identified potential parking space 102. The display 345 of the freely positionable parking area in the environment model is advantageously dependent on the determined predicted driving corridor. When determining 350 the location of the target parking position, the user input is optionally recorded, wherein the user input in step 350 optionally includes a displacement and / or rotation of the displayed freely positionable parking area in the environment model 200. The user input in step 350 is preferably made via the touchscreen of the display device 118.In a further step 360, a section 250 of the sensor data and / or the virtual environment model is determined, whereby section 250 represents the determined location of the target parking position for the trailer 120. Subsequently, in step 360, the determined section 250 of the sensor data or the determined section 250 of the virtual environment model R.409863 is used.

[0046] - 11 -

[0047] 200 with a modified virtual perspective compared to the displayed environment model 200, in addition to the displayed environment model 200. It may be provided that the display 360 of the specific section is adjusted depending on the determined distance between the determined target parking position and the trailer, and / or that, in an additional optional step 370, the specific section 250 is adjusted depending on the determined distance between the determined target parking position and the trailer. In a further step 380, the determined position of the target parking position 201, 251 is displayed both in the displayed environment model 200 and in the displayed specific section 250 of the sensor data or the virtual environment model 200.In an optional step 390, an adjustment of the color of the displayed location of the target parking position 201, 251 in the virtual environment model 200 and / or in the displayed section 250 of the sensor data and / or in the displayed section 250 of the virtual environment model 200 may be provided depending on the predicted driving corridor. Optionally, the predicted driving corridor can also be displayed by overlaying 395 of the predicted driving corridor in the displayed environment model 200 and / or in the displayed section 250 of the sensor data and / or in the displayed section 250 of the virtual environment model 200 and / or by changing 396 of the displayed environment model 200 and / or the displayed section 250 of the sensor data and / or the displayed section 250 of the virtual environment model in the area of ​​the predicted driving corridor.

Claims

R.409863 - 12 - Claims 1. Method for displaying a target parking position (201, 251) of a coupled trailer (120) to a user of a vehicle (110), wherein the vehicle (110) and / or the trailer (120) comprises at least one camera (114 to 117) as a sensor, comprising the following steps • Acquisition (310) of sensor data, wherein the sensor data represent the environment (100) of the vehicle (110) with the attached trailer (120), • Determination (320) of a virtual environment model (200) based on the acquired sensor data, • Display (330) of the environment model (200), and • Determination (350) of a location of the target parking position for the trailer based on the acquired sensor data and / or the determined environment model, characterized in that the following steps are carried out • Determination (360) of a section (250) of the sensor data and / or the virtual environment model (200), wherein the section (250) represents the determined location of the target parking position (201 , 251) for the trailer (120), • Display (370) of a specific section (250) of the sensor data or of a specific section (250) of the virtual environment model (200) with a virtual perspective modified compared to the displayed environment model (200) in addition to the displayed environment model (200), and • Display (380) of the determined location of the target parking position (201 , 251) in the displayed environment model (200) and in the displayed specific section (250) of the sensor data or the virtual environment model (200).

2. Method according to claim 1, wherein the determination of the location of the target parking position (201, 251) is additionally carried out depending on a detected input from the user.

3. A method according to claim 1 or 2, comprising the following step • Detection (341) of at least one possible parking space (102) for the trailer (120) based on the acquired sensor data and / or the virtual environment model (120), in particular by a trained machine recognition method, preferably by a neural network, and R.409863 - 13 - • Display (342) of at least one detected possible parking space (102) in the environment model (200) and / or in the displayed specific section (250) for selection by the user through input and / or • Determination (350) of the location of the target parking position (201 , 251) for the trailer (120) depending on the at least one detected possible parking space (102).

4. A method according to claim 2 or 3, wherein the following steps are performed • Display (343) a freely positionable parking area in the displayed environment model (100) and / or in the displayed specific section (250), wherein the parking area in particular represents the at least one detected possible parking space (102), • wherein the user input captured to determine the location of the target parking position (201 , 251) includes a displacement and / or rotation of the displayed parking area in the environment model (102) and / or in the displayed specific section (250), in particular by means of a touchscreen.

5. A method according to any of the preceding claims, wherein the following steps are performed • Determination (340) of the trailer angle between the longitudinal axis of the vehicle (110) and the longitudinal axis of the trailer (120), and / or • Detection (341) of the current steering angle of the vehicle (110), • Determination (342) of a predicted travel corridor of the trailer (120) in the vicinity (100) of the vehicle (110) during future reversing of the vehicle (110) depending on the determined trailer angle and / or the recorded steering angle of the vehicle (100), • Display (342) of the possible parking space and / or display (343) of the freely positionable parking area in the environment model (200) and / or in the displayed specific section (250) and / or determination (350) of the location of the target parking position for the trailer (120) in each case depending on the determined predicted travel corridor, and / or • Display of the predicted driving corridor by overlaying (395) the predicted driving corridor in the displayed environment model (200) and / or in the displayed section (250) of the sensor data and / or in the displayed section (250) of the virtual environment model (200) and / or by changing (396) the environment model (200) and / or the displayed R.409863 - 14 - Section (250) of the sensor data and / or the displayed section (250) of the virtual environment model (200) in the area of ​​the predicted driving corridor, and / or • Adjustment (390) of the color of the displayed location of the target parking position (201 , 251) in the virtual environment model (200) and / or in the displayed section (250) of the sensor data and / or in the displayed section (250) of the virtual environment model (200) depending on the predicted driving corridor.

6. A method according to any of the preceding claims, wherein the following steps are carried out • Determination (351) of a distance between the determined target parking position and the trailer (120) depending on the recorded sensor data, • wherein the display (370) of the specified section (250) depends on the determined distance (x) between the determined target parking position (201, 251) and the trailer (120), and / or • Adjustment (352) of the displayed specific section depending on the determined distance (x) between the determined target parking position (201 , 251) and the trailer (120).

7. Computer program comprising instructions which, when the program is executed by a computer, cause it to perform the steps of the method according to any of the preceding claims.

8. Control unit or central computing device, comprising at least the following components • at least one signal input for providing a sensor signal which was detected by means of at least one sensor, wherein the sensor signal represents detected sensor data of the environment (100) of a vehicle (110) with a coupled trailer (120), • a signal output for outputting a control signal for a display device (118), wherein the control signal represents a virtual environment model (200) of the environment (100) of the vehicle (110) with a parking space (102) for the trailer (120) displayed and a specific section (250) of the sensor data, in particular a camera image, or a specific section (250) of the virtual environment model (200) with a virtual perspective changed compared to the displayed environment model (200), and R.409863 - 15 - • a computing unit, in particular a processor, configured to perform the steps of the method according to any one of claims 1 to 6.

9. Vehicle (110), comprising the following components • at least one sensor (113 to 117), wherein the at least one sensor (113 to 117) comprises at least one camera (114 to 117), • a display device (118), and • a control unit or a central computing device (119) according to claim 8.

Images