Method, computer program product, parking assistance system, and vehicle

Abstract

The invention relates to a method for operating a parking assistance system (110) for a vehicle (100). The parking assistance system (110) is designed such that, in a tracking mode, it autonomously tracks a trajectory (TR1-TR4) from a number of trajectories (TR1-TR4) learned in a training mode, wherein the respective trajectory (TR1-TR4) is determined by a sequence of positions (P1-P6) and connects a starting position (P1) to a target position (P6), and wherein the respective trajectory (TR1-TR4) is assigned a number of images (IMG1-IMG6) of an environment (200) of the vehicle (100) captured by the vehicle's camera (120) in the respective position (P1-P6) and stored. The method comprises the steps of: Providing (S1) a number of stored images (IMG1-IMG6); determining (S2) a display element (DSP) comprising the provided images (IMG1-IMG6) to represent the respective associated trajectory (TR1-TR4); outputting (S3) the determined display element (DSP) to a display device of a user interface (105) of the vehicle (100); receiving (S4) a user input (SIG) for selecting at least one image (IMG1-IMG6) contained in the display element (DSP) via the user interface (105); and initiating (S5) an autonomous journey of the vehicle (100) along the trajectory (TR1-TR4) associated with the selected image (IMG1-IMG6).

Description

[0001] METHOD, COMPUTER PROGRAM PRODUCT, PARKING ASSISTANCE SYSTEM AND

[0002] VEHICLE

[0003] The present invention relates to a method for operating a parking assistance system, a computer program product, a parking assistance system and a vehicle with a parking assistance system.

[0004] Parking assistance systems are known that can be trained to follow a specific trajectory. This is particularly useful for frequently recurring situations, such as parking a vehicle in a garage or leaving the vehicle in a designated parking space. The driver then only needs to drive the vehicle close to a starting point of the trajectory, for example, to a driveway. The parking assistance system then autonomously follows the trained trajectory, thus relieving the driver of some of the workload.

[0005] Over an extended period of use of the vehicle and / or when multiple people use the vehicle, a large number of different trained trajectories can accumulate over time. This leads to the problem that a user can lose track of the multitude and no longer knows which trajectory is the one they want in a particular situation. In particular, when different people take turns using the vehicle, a subsequent user does not know the destination of a trajectory saved by a previous user and / or how it runs. Furthermore, when multiple trajectories run in the same area, such as on or near the user's property, it is difficult for the user to distinguish them from one another.

[0006] One possibility is to label the different trajectories with keywords and / or a description so that a user can distinguish them from one another. However, this is complex, unintuitive, and does not solve the problem when there are several different users of a vehicle. If a user cannot clearly assign a trajectory, this can lead to them selecting the wrong trajectory to follow in a certain situation, which prevents the vehicle from driving to the desired parking position. The user then has to either park the vehicle manually or try other trajectories in the hope of selecting the correct one. Ultimately, this situation leads to unnecessary consumption of resources such as fuel and time, as well as to user dissatisfaction.Such a lack of assignability of the trained trajectories can lead to users being discouraged from using the parking assistance system overall, thereby reducing the usefulness of the parking assistance system.

[0007] DE 10 2015 010 746 A1 discloses a method for self-localization of a vehicle.

[0008] Here, images of the ground traveled over are taken along a first trajectory using an image acquisition unit whose detection range includes the ground in the surroundings of the vehicle and compared with position-related stored images. Based on the comparison, a current position and / or a current orientation of the vehicle is determined.

[0009] DE 10 2013 015 349 A1 discloses a method for operating a vehicle to approach a parking space in a parking zone that is not visible / far from the street, in which environmental data of the vehicle are recorded, wherein when approaching a parking space in the parking zone it is identified whether this is a home parking space or the parking zone is a home parking zone, and when the home parking space or home parking zone is identified and the vehicle approaches the identified home parking space or the identified home parking zone, environmental data or driving data recorded are stored or updated.

[0010] Against this background, one object of the present invention is to improve the operation of a parking assistance system. According to a first aspect, a method for operating a parking assistance system for a vehicle is proposed. The parking assistance system is configured in a tracking mode for autonomously following a trajectory from a number of trajectories learned in a training mode, wherein the respective trajectory is defined by a sequence of positions and connects a starting position with a target position, and wherein a number of images of the surroundings of the vehicle at the respective position, captured with an on-board camera, are assigned to the respective trajectory and stored. The method comprises the steps:

[0011] Providing a number of stored images,

[0012] Determining a representation that includes the provided images to represent the respective assigned trajectory,

[0013] Outputting the determined representation to a display device of a user interface of the vehicle,

[0014] Receiving a user input to select at least one image contained in the representation from the user interface, and

[0015] Initiate autonomous movement of the vehicle along the trajectory associated with the selected image.

[0016] This method has the advantage that the user can intuitively identify the trajectory, the target position it leads to, and / or how it proceeds based on the image display. This is particularly true when multiple users take turns using the vehicle and therefore may not have practiced a particular trajectory with the vehicle themselves. The advantage is that users can select the correct trajectory with greater reliability in a given situation, and the autonomous parking maneuver can thus be completed successfully. Selecting the wrong trajectory can be avoided, thus also reducing the effort involved in parking the vehicle in an undesirable position. Furthermore, the method saves the user the effort of adding metadata, such as a title, keywords, and / or a description, to each trajectory.In addition, it is not necessary to store such metadata to identify the respective trajectory, which reduces storage requirements.

[0017] The fact that the parking assistance system is set up to autonomously follow the respective trajectory is understood here to mean that the parking assistance system controls the vehicle autonomously. In this case, this is done in particular using camera images, lidar data and / or radar data of the vehicle's surroundings. Based on these images or data, the parking assistance system can, for example, localize and orient itself (this can also be referred to as SLAM, SLAM: Simultaneous Localization And Mapping) and can be set up to detect obstacles in the surroundings. The vehicle with the parking assistance system can also be referred to as a self-driving vehicle. It can be provided that the autonomous control takes place under the supervision of the user, whereby the user does not necessarily have to be in the vehicle.

[0018] For example, the vehicle's level of automation is classified as Level 4 or 5 according to the SAE classification system. The SAE classification system was published in 2014 by SAE International, an automotive standards organization, as J3016, "Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems." It is based on six different levels of automation and takes into account the degree of system intervention and driver attention required. SAE automation levels range from Level 0, which corresponds to a fully manual system, through driver assistance systems at Levels 1 and 2, to semi-autonomous (Levels 3 and 4) and fully autonomous (Level 5) systems, where a driver is no longer required. An autonomous vehicle is a vehicle capable of sensing its environment and navigating without human input.In particular, it corresponds to SAE automation level 5.

[0019] When learning or training the respective trajectory in training mode, the user preferably steers the vehicle manually. The user can also be assisted by the parking assistance system, but the vehicle does not drive autonomously using the parking assistance system. The vehicle user has full control over the vehicle. The trajectory to be learned is unknown at this point and is only determined during the training drive.

[0020] In preferred embodiments, the parking assistance system is configured to follow the respective trajectory using images of the vehicle's surroundings captured by the vehicle's onboard camera. This specifically means that the parking assistance system executes a VSLAM algorithm (VSLAM: Visual Simultaneous Localization and Mapping) using the captured image for orientation in the surroundings, in particular for localizing the vehicle with respect to the positions of the respective trajectory.

[0021] The respective trajectory is defined in particular by a sequence of positions. In particular, the trajectory comprises one or more reference positions. Reference positions correspond in particular to the positions that the vehicle has in training mode, when an image, lidar data and / or radar data of the environment are received and processed in order to determine features in the environment, such as optical features, which can also be referred to as orientation features. A respective group of determined features is assigned to a respective reference position. This means that the reference positions are defined during the training of the trajectory. A respective reference position is defined, for example, by two coordinates in a two-dimensional coordinate system and the orientation of the vehicle at this position.The respective trajectory begins, for example, at a start position, which is in particular a reference position, and ends at a target position, which is in particular also a reference position.

[0022] The number of trajectories includes one trajectory or multiple trajectories. The multiple trajectories can be located in the same areas or regions, for example, close to each other, or in different areas or regions. For example, a user can train multiple trajectories on private property and train additional trajectories at their workplace, a train station, a shopping center, and the like.

[0023] The number of images assigned to the respective trajectory can comprise one image, two images, or more than two images for the respective trajectory. The images are captured and saved, in particular, during the learning of the trajectory. The respective image shows, in particular, a section of the vehicle's surroundings, whereby the section depends on the position of the vehicle at the time the image was captured, its orientation, and the angle of view of the camera used. For example, each position in the trajectory is assigned an image in which the position itself can be seen. Alternatively, the respective position can be assigned an image that was captured from the respective position.

[0024] In a first step of the method, a number of stored images are provided. The number of provided images can be a subset of all stored images, or it can comprise the entire number of stored images. The entire number of stored images includes all stored images for all trained trajectories. If only a subset of all images is provided, this subset is selected from all images based on predetermined selection criteria. For example, only one image, in particular the image of the respective target position, or two images, in particular the image of the respective start position and target position, can be selected for each trajectory.Furthermore, the subset can be selected, for example, based on the current position information of the vehicle and the position information stored for each of the images, with only those images being selected that are close to the current position. "Close" means that the distance between the current position and that of the image is less than or equal to a predetermined threshold.

[0025] In a second step, a representation is determined which comprises the provided images to represent the respective assigned trajectory. The representation can be viewed as a graphical user interface in which the provided images are arranged in a predetermined manner. For example, the representation comprises the images in a list arrangement or a table arrangement. A schematic representation of the recorded path from a bird's eye view is also possible. In addition to the images, the representation can contain further elements, objects and / or information, in particular a numbering of the trajectories, an identification of individual images as a start position or destination position, graphic elements for subdividing and delimiting images of different trajectories from one another, and the like.

[0026] The representation may include dynamic elements, such as an animation comprising several images for a respective trajectory, which are displayed sequentially in time at a specific position in the representation.

[0027] The representation may be optimized for output on the user interface display device. This means, in particular, that the representation has a width and / or height that can be displayed on the display device without additional scaling.

[0028] The display can include several different partial displays, for example, providing different zoom levels for the images. This allows for a detailed display of the images, which can be particularly advantageous for display devices with low resolution.

[0029] In particular, the display may have a greater height than can be displayed on the display device in a display, in which case, for example, scrolling through the display is possible.

[0030] In a third step, the determined representation is output to the display device of the vehicle's user interface. In particular, the representation is transmitted to the display device in the form of an image signal, preferably a digital image signal. The display device receives the image signal and outputs a corresponding image on its display. In other words, the display device displays the representation. The display device comprises, for example, a screen, preferably a touch-sensitive screen.

[0031] In a fourth step, a user input for selecting at least one image contained in the representation is received by the user interface. Since each of the images is assigned to a trajectory, selecting an image corresponds to selecting the assigned trajectory. By selecting the image, the user can thus select the trajectory they wish to follow. This is particularly intuitive using a touch-sensitive screen, where the user simply needs to touch the corresponding image of the representation on the display device.

[0032] The user input can be received by the user interface, in particular, in the form of coordinates relative to the representation. The representation is, in particular, two-dimensional, so that each position in the representation is uniquely defined by two coordinates, for example, pixel coordinates. The user input then contains, for example, a coordinate tuple or a range of coordinates. From these coordinates, it can be deduced which of the images in the representation the user has selected.

[0033] In a fifth step, the vehicle is initiated to drive autonomously along the trajectory associated with the selected image. This means that the parking assistance system controls the vehicle so that it drives along the selected trajectory to the target position and comes to a stop there.

[0034] In embodiments, the display device is a component of the parking assistance system.

[0035] According to one embodiment of the method, the number of stored images for the respective trajectory includes at least one image of the respective target position. In this embodiment, the representation for each trajectory is determined, in particular, using the image of the target position.

[0036] The target position is particularly the most memorable position of a particular trajectory for users, which is why this embodiment offers particularly high reliability in selecting the trajectory.

[0037] It should be noted that the image of the target position can include both an image taken from the target position and an image taken from a location in front of the target position. In particular, taking a picture in front of the target position can be advantageous if the target position itself is close to a wall or other obstacles, since in this case, an image taken from the target position, for example, does not provide a good overview.

[0038] In further embodiments, the number of stored images for the respective trajectory comprises at least one image of the respective starting position and the respective target position.

[0039] According to a further embodiment of the method, the number of stored images for the respective trajectory comprises an image composed of a plurality of individual images.

[0040] The composite image can, for example, comprise a wide-angle view composed of several individual shots, each with a smaller angle of view. During the composite process, various image processing steps can be performed, in particular to correct the distortion of the individual images, adjust the exposure and contrast of the individual images, and the like.

[0041] According to a further embodiment of the method, the composite image comprises a bird's-eye view of the respective position. The bird's-eye view can be generated, in particular, by perspectively distorting images.

[0042] In embodiments, the composite image comprises a bird's-eye view of the entire trajectory. For example, individual images are first converted into a bird's-eye view at a respective position along the trajectory, and then these individual images are combined into a single image. The combination occurs, in particular, in accordance with the determination of a panoramic image from a plurality of individual images.

[0043] According to a further embodiment of the method, this comprises:

[0044] Providing an object to represent a position of the vehicle in the target position of the respective trajectory, and

[0045] Determining the representation using the provided object such that the object in the representation is overlaid at the target position in an image in which the target position can be seen.

[0046] The object representing the vehicle's position includes, for example, a projection of the vehicle's outline at the target position onto the ground. If the image of the target position shows the target position, the outline of the vehicle on the ground is superimposed into this image. This can be done, for example, in the form of a darkened area or as an outline that is superimposed into the image.

[0047] In this context, "object" is understood to mean a graphic object that can be represented in a graphic representation, in particular an image.

[0048] According to a further embodiment of the method, this comprises:

[0049] Determining a digital environment map for the respective trajectory, and

[0050] Determining the representation using the determined digital environment map, so that the determined digital environment map is displayed together with the provided image for the respective trajectory, wherein the image is arranged in the representation depending on its position in the digital environment map.

[0051] The digital environment map specifically comprises a digital representation of the vehicle's surroundings, with detected objects such as buildings or vegetation being depicted in the map. The digital environment map can be determined based on environmental sensor data collected from various environmental sensors, such as cameras, lidar, radar, and / or ultrasound. This can also be referred to as sensor fusion.

[0052] The determination of the digital environment map for the respective trajectory takes place in particular during the learning of the trajectory in the training mode.

[0053] By the arrangement of the image in the representation depending on its position in the digital environment map, we mean, in particular, that each image in the digital environment map is arranged such that the relative position of objects in the digital environment map and an object visible in the respective image corresponds to the relative position of these objects in reality. Simply put, this means that an image corresponding to a target position next to a house is also arranged next to the house in the representation of the digital environment map.

[0054] One can also say that the position of each image depends on and is derived from the position of objects visible in the image.

[0055] If several images are provided for the trajectory, all provided images can be arranged accordingly in the display with the digital environment map.

[0056] According to a further embodiment of the method, the number of learned trajectories comprises at least two trajectories that run in the same area, wherein the representation is determined such that the respective images of the at least two trajectories are displayed in a group.

[0057] The fact that the two trajectories extend within the same range means, for example, that the distance between a first position of the first trajectory and a second position of the second trajectory is less than a predetermined threshold. The predetermined threshold is, for example, 50 m, 30 m, 10 m, or even 5 m. Preferably, the positions are available in a world coordinate system. For this determination, the two positions that are closest to each other can be used, for example.

[0058] By displaying the trajectories in a group, we mean, in particular, that they are graphically separated from other trajectories in the display. It can also be provided that the trajectories are displayed together, with a placeholder, such as a symbol or icon, being displayed instead of the images. By selecting the placeholder, another display can be called up in which the trajectories are represented by their respective images.

[0059] According to a further embodiment of the method, the number of learned trajectories comprises at least two trajectories which run in the same area and with:

[0060] Determining a common digital environment map for the at least two trajectories, and

[0061] Determining the representation using the determined common digital map of the environment, such that the jointly determined digital map of the environment is displayed together with the respective provided image for the respective trajectory, wherein the respective image is arranged in the representation depending on its position in the digital map of the environment. This embodiment is particularly advantageous when multiple trajectories run in the same area, since the risk of confusion is then particularly high. The spatial representation with the digital map of the environment makes it particularly intuitive for the user to select the correct trajectory in a specific situation, even if they have not learned the trajectory themselves.

[0062] According to a further embodiment of the method, the number of images assigned to the respective trajectory is recorded and stored during the learning of the trajectory in the training mode.

[0063] According to a further embodiment of the method, while following the respective trajectory in the following mode, new images are captured with the vehicle's own camera and assigned to the trajectory and stored.

[0064] This is advantageous for incorporating temporal changes in the environment into the images. The new images can replace the original images in whole or in part. Furthermore, this can be beneficial if the original training ride was conducted in poor visibility conditions, such as darkness, rain, snow, and / or a dirty camera lens, resulting in poor quality saved images.

[0065] According to a further embodiment of the method, the respective number of images is captured by a number of cameras, wherein the number comprises a front camera, a rear camera, a side camera on a left side of the vehicle and / or a side camera on a right side of the vehicle.

[0066] According to a further embodiment of the method, each stored image is assigned position information with respect to a world coordinate system, and the representation is determined such that the respective image is arranged relative to other images depending on its assigned position in the world coordinate system. A world coordinate system is understood here to mean that all coordinates, in particular those of different trajectories, refer to the same reference point (origin). Such a world coordinate system is, for example, the coordinate systems provided by a satellite navigation system, such as NAVSTAR GPS, GALILEO, GLONASS, and / or Beidou.

[0067] In this embodiment, the display may, for example, include a road map with the images for the trajectories superimposed. This allows any user of the vehicle to quickly and intuitively determine where a trained trajectory is located.

[0068] According to a second aspect, a computer program product is proposed which comprises instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the first aspect.

[0069] The computer is particularly a component of a vehicle and forms, for example, an electronic control unit (ECU).

[0070] A computer program product, such as a computer program means, can be provided or delivered, for example, as a storage medium, such as a memory card, USB stick, CD-ROM, DVD, or in the form of a downloadable file from a server in a network. This can be done, for example, in a wireless communications network by transmitting a corresponding file with the computer program product or the computer program means.

[0071] According to a third aspect, a parking assistance system for a vehicle is proposed. The parking assistance system is configured in a tracking mode to autonomously track a trajectory from a number of trajectories learned in a training mode. Each trajectory is defined by a sequence of positions and connects a starting position to a target position. A number of images of the vehicle's surroundings at the respective position, captured by an on-board camera, are associated with the respective trajectory and stored there.The parking assistance system comprises: a provision unit for providing a number of stored images, a determination unit for determining a representation which comprises the provided images for representing the respective associated trajectory, an output unit for outputting the determined representation to a display device of the user interface for displaying the provided images, a reception unit for receiving a user input for selecting at least one image contained in the representation from the user interface, and a control unit for initiating an autonomous journey of the vehicle along the trajectory associated with the selected image.

[0072] This parking assistance system has the same advantages as explained for the method according to the first aspect. The embodiments and features described for the proposed method apply accordingly to the proposed parking assistance system.

[0073] The respective unit of the parking assistance system can be implemented in hardware and / or software. In a hardware implementation, the respective unit can be embodied, for example, as a computer or a microprocessor. In a software implementation, the respective unit can be embodied as a computer program product, as a function, as a routine, as an algorithm, as part of a program code, or as an executable object. Furthermore, each of the units mentioned here can also be embodied as part of a higher-level control system of the vehicle, such as a central electronic control device and / or a control unit (ECU: Electronic Control Unit).

[0074] The parking assistance system is particularly configured to carry out the method according to the first aspect. According to a fourth aspect, a vehicle is proposed with a number of cameras for capturing a respective image of the vehicle's surroundings, with a parking assistance system according to the third aspect, and with a user interface comprising a display device.

[0075] The vehicle is, for example, a passenger car or a truck. The vehicle preferably comprises a number of sensor units configured to detect the driving state of the vehicle and to detect the vehicle's surroundings. Examples of such sensor units of the vehicle are image recording devices such as a camera, a radar (radio detection and ranging) or a lidar (light detection and ranging), ultrasonic sensors, location sensors, wheel angle sensors, and / or wheel speed sensors. The sensor units are each configured to output a sensor signal, for example, to the parking assistance system, which performs semi-autonomous or fully autonomous control of the vehicle depending on the detected sensor signals.

[0076] The number of cameras includes a front camera, a rear camera, a side camera on a left side of the vehicle and / or a side camera on a right side of the vehicle.

[0077] Further possible implementations of the invention also include combinations of features or embodiments described above or below with respect to the exemplary embodiments not explicitly mentioned. In this case, the person skilled in the art will also add individual aspects as improvements or additions to the respective basic form of the invention.

[0078] Further advantageous embodiments and aspects of the invention are the subject of the dependent claims and the exemplary embodiments of the invention described below. The invention will be explained in more detail below using preferred embodiments with reference to the accompanying figures. Fig. 1 shows a schematic view of a vehicle from a bird's eye view;

[0079] Fig. 2 shows a schematic view of a two-image display;

[0080] Fig. 3 shows another schematic view of a two-image display;

[0081] Fig. 4 shows a schematic view of a representation with a digital environment map;

[0082] Fig. 5 shows a further schematic view of a representation with a digital environment map;

[0083] Fig. 6 shows a schematic block diagram of an embodiment of a parking assistance system; and

[0084] Fig. 7 shows a schematic block diagram of an embodiment of a method for operating a parking assistance system.

[0085] In the figures, identical or functionally equivalent elements have been given the same reference numerals unless otherwise stated.

[0086] Fig. 1 shows a schematic view of a vehicle 100 from a bird's eye view. The vehicle 100 is, for example, a car arranged in an environment 200. The car 100 has a parking assistance system 110, which is designed, for example, as a control unit. In addition, a plurality of environmental sensor devices 120, 130 are arranged on the car 100, which are, for example, optical sensors 120 and ultrasonic sensors 130. The optical sensors 120 include, for example, visual

[0087] Cameras, a radar, and / or a lidar. The optical sensors 120 can, in particular, each capture an image of a respective area of ​​the environment 200 of the car 100 and output it as an optical sensor signal. The ultrasonic sensors 130 are configured to detect a distance to objects located in the environment 200 and to output a corresponding sensor signal. Using the sensor signals captured by the sensors 120, 130, the parking assistance system 110 is capable of driving the car 100 semi-autonomously or fully autonomously. In addition to the optical sensors 120 and ultrasonic sensors 130 shown in Fig. 1, the vehicle 100 can be provided with various additional sensor devices 120, 130. Examples of these include a wheel speed sensor, a steering angle sensor, a position sensor, a microphone, an acceleration sensor, an antenna with a coupled receiver for receiving electromagnetically transmittable data signals, and the like.

[0088] The vehicle also has a user interface 105, wherein the parking assistance system 110 is communicatively coupled to the user interface 105. This means that the parking assistance system 110 and the user interface 105 are configured to exchange data, in particular in the form of analog or digital data signals. The user interface 105 comprises a display device (not shown) configured to display images IMG1 - IMG6 (see Figs. 2 - 5) and a DSP display (see Figs. 2 - 5). Furthermore, the user interface 105 comprises input means (not shown) via which a user of the vehicle 100 can make inputs to systems of the vehicle 100, in particular to the parking assistance system 110. The input means can include buttons, switches, rotary controls, touch-sensitive elements, voice recognition, gesture recognition, and the like.A respective input may in particular refer to elements displayed by the display device at a specific point in time.

[0089] The parking assistance system 110 is configured in a tracking mode to autonomously track a trajectory TR1-TR4 (see Fig. 4 or 5) from a number of trajectories TR1-TR4 learned in a training mode. The respective trajectory TR1-TR4 is defined by a sequence of positions P1-P6 (see Fig. 4) and connects a starting position P1 with a target position P6. A number of images IMG1-IMG6 of the surroundings 200 of the vehicle 100, captured by an on-board camera 120, are assigned to each of the trajectories TR1-TR4 and stored there. The respective image IMG1 - IMG6 is recorded when the vehicle 100 is located at a respective position P1 - P6 of the trajectory TR1 - TR4, and therefore shows a corresponding section of the environment 200. In embodiments, a respective image IMG1 - IMG6 can contain image information from several images IMG1 - IMG6, in particular it can be a combined image.The parking assistance system 110 is constructed, for example, as shown in Fig. 6 and is configured to carry out the method explained with reference to Fig. 7.

[0090] Fig. 2 shows a schematic view of a DSP representation with two images IMG1, IMG2. The DSP representation is determined, in particular, by the determination unit 112 (see Fig. 6) based on a number of images IMG1 - IMG6 provided by the provision unit 111 (see Fig. 6).

[0091] The two images IMG1, IMG2 are assigned to a learned trajectory TR1-TR4 (see Fig. 4 or 5) that the user has driven, for example, with the vehicle 100 of Fig. 1, wherein the parking assistance system 110 (see Fig. 1 or 6) has recorded the trajectory TR1-TR4. In this example, the images IMG1, IMG2 show a view of the surroundings 200 (see Fig. 1) of the vehicle 100 as seen from a start position P1 (see Fig. 4) (IMG1 overwritten with "Start:") and another view of the surroundings 200 of the vehicle 100 as seen from a target position P6 (see Fig. 4) (IMG2 overwritten with "Target:"), wherein the images IMG1, IMG2 were captured, for example, with a front camera 120.It should be noted that instead of the image IMG2 seen from the target position P6, an image seen from a position in front of the target position P6 is advantageously used to display the target position P6, since such an image offers a better overview and in particular includes the target position P6 itself (see also Fig. 3).

[0092] In the first image IMG1 is a view of a house with a fence or a wall in the

[0093] The background can be seen, as well as a tree and a bush next to the house. In the second image IMG2, only the bush and sections of a house wall and the fence are visible. In this example, the user has trained the vehicle 100, in particular, on the trajectory TR1 shown in Fig. 4, so that the parking assistance system 110 is now able to drive the vehicle 100 autonomously from the starting position P1 to the target position P6.

[0094] The representation DSP is transmitted from the output unit 114 (see Fig. 6) of the parking assistance system 110 to the user interface 105 (see Fig. 1), and the user interface 105 displays the representation DSP on the display device. This enables the user to intuitively grasp the trajectory TR1 associated with the images IMG1, IMG2 and can select them for tracing. It should be noted that in embodiments, the representation DSP contains only one image for the trajectory TR1, or also contains more than two images for the trajectory TR1 and / or contains composite images for the trajectory TR1 (see also Fig. 4).

[0095] If the parking assistance system 110 comprises several trained and stored trajectories TR1 - TR4, a corresponding representation DSP can be determined and output for each additional trajectory TR1 - TR4, so that the user can select the desired trajectory TR1 - TR4 based on the respective images IMG1, IMG2.

[0096] It should be noted that the DSP representation can contain corresponding images IMG1, IMG2 for several trajectories TR1 - TR4, which are, for example, arranged simultaneously next to each other or one below the other in the DSP representation (not shown).

[0097] If the user selects the trajectory TR1, the user interface 105 sends, for example, a corresponding user input to the parking assistance system 110, which initiates the autonomous following of the selected trajectory TR1.

[0098] Fig. 3 shows another schematic view of a DSP representation with two images IMG1 ,

[0099] IMG2. The representation DSP is in particular generated by the determination unit 112 (see Fig. 6) on the basis of a number of data provided by the provision unit 111 (see Fig. 6).

[0100] Images IMG1 - IMG6 determined.

[0101] The representation in Fig. 3 is almost identical to that in Fig. 2, with one difference being that an object OL is additionally depicted in the images IMG1, IMG2. The object OL is, for example, a geometric figure (a rectangle depicted with a distorted perspective) that is superimposed on the respective image IMG1, IMG2 or superimposed on it. In particular, the object OL is superimposed on the image at a position that corresponds to the target position P6 (see Fig. 4) of the trajectory TR1, i.e., the position of the vehicle 100 when it has traveled along the trajectory TR1.

[0102] In this example, the second image IMG2 is taken from a position before the target position P6, for example the position P5 (see Fig. 4), which is why the target position P6 is included in the image IMG2.

[0103] By additionally superimposing the target position P6 in the displayed images IMG1, IMG2, the user can assign the trajectory TR1 even better and is in particular able to distinguish between trajectories that, for example, end in neighboring parking spaces and for which the displayed images are therefore almost identical.

[0104] It should be noted that, as an alternative to the representation DSP in Fig. 3, the object OL is, for example, superimposed on only one of the images IMG1, IMG2.

[0105] Fig. 4 shows a schematic view of a DSP representation with a digital environment map DMAP and a plurality of images IMG1 - IMG6 that are assigned to the trajectory TR1. The environment represented in the digital environment map DMAP is, for example, the same as that shown in the images IMG1, IMG2 in Figs. 2 and 3. In the digital environment map DMAP, the detected objects OB1 - OB4 of the environment are represented accordingly. In this example, these are the house OB1, the bush OB2, the fence OB3 and the tree OB4. The digital environment map DMAP is preferably recorded and stored during the training of the trajectory TR1.

[0106] In this example, the trajectory TR1 comprises six positions P1 - P6, where position P1 is the starting position and position P6 is the target position. The starting position P1 is, for example, in front of the house (shown here as object OB1) and the target position P6 is to the side of the house OB1. At each position P1 - P6, at least one image IMG1 - IMG6 of the environment was captured during training, assigned to the trajectory TR1, and saved. The images IMG1 - IMG6 are preferably composite images that show the respective position from a bird's eye view. This is particularly possible when the vehicle 100 comprises several cameras 120 that can capture a total image angle of 360° around the vehicle 100. However, even if the vehicle 100 only has one front camera 120, the images from the front camera 120 can be converted into a view from above by appropriately distorting their perspective.It should be noted that a respective image IMG1 - IMG6 was not necessarily captured when the vehicle 100 was at the corresponding position.

[0107] Rather, the image IMG3, for example, was captured with the front camera when the vehicle 100 was at position P2 and accordingly for the other images / positions.

[0108] In this example, the display includes both the digital environment map DMAP and images IMG1 - IMG6, which are assigned to the trajectory TR1 and serve to display the trajectory TR1. The images IMG1 - IMG6 are arranged in the digital environment map of the DSP display, in particular corresponding to reality. By overlaying the images IMG1 - IMG6 together with the digital environment map DMAP, the user can even better grasp the trajectory TR, in particular the target position P6 of the trajectory TR1. If the images IMG1 - IMG6 are close enough to one another and / or overlap, they can also be combined into a single image and saved.

[0109] In embodiments, the trajectory TR1 can also be displayed in the DSP representation in the form of an object OL (see Fig. 3).

[0110] Fig. 5 shows another schematic view of a DSP representation with a digital environment map DMAP. This example depicts the same environment 200 as previously explained in Fig. 4. However, in this example, multiple trajectories TR1-TR4 are present in the depicted area of ​​the environment. For example, the various trajectories TR1-TR4 were trained by different users of the vehicle 100. In this example, all trajectories TR1-TR4 begin at the starting position depicted by image IMG1, although this is not mandatory.

[0111] The respective image IMG2 - IMG5 of the respective target position is arranged in the DSP representation with respect to the digital environment map DMAP, in particular the objects OB1 - OB4, corresponding to reality, which makes it considerably easier for the user to derive a position of the vehicle in reality from the displayed target position.

[0112] For example, the user can select one of the images IMG2 - IMG5 in the DSP display to select the associated trajectory TR1 - TR4.

[0113] In embodiments, a respective object OL (see Fig. 3) for each trajectory TR1 - TR4 can also be displayed in the DSP display.

[0114] Fig. 6 shows a schematic block diagram of an embodiment of a parking assistance system 110 that can be used, for example, with the vehicle 100 of Fig. 1. The parking assistance system 110 is set up in a tracking mode for autonomously following a trajectory TR1 - TR4 (see Fig. 4 or 5) from a number of trajectories TR1 - TR4 learned in a training mode. The respective trajectory TR1 - TR4 is defined by a sequence of positions P1 - P6 (see Fig. 4) and connects a starting position P1 with a target position P6. A number of images IMG1 - IMG6 (see Figs. 2 - 6) of the surroundings 200 (see Fig. 1) of the vehicle 100, captured with an on-board camera 120 (see Fig. 1), are assigned to each of the trajectories TR1 - TR4 and stored therein.The respective image IMG1 - IMG6 is recorded when the vehicle 100 is located at a respective position P1 - P6 of the trajectory TR1 - TR4, and therefore shows a corresponding section of the environment 200. The parking assistance system 110 comprises a provision unit 111 for providing a number of stored images IMG, a determination unit 112 for determining a representation DSP which comprises the provided images IMG for representing the respective assigned trajectory TR1 - TR4 (see Fig. 4 or 5), an output unit 114 for outputting the determined representation DSP to a display device of the user interface 105 (see Fig.1), in particular in the form of a digital image or data signal, a receiving unit 116 for receiving a user input SIG for selecting at least one image IMG contained in the representation DSP from the user interface 105, and a control unit 118 for initiating an autonomous journey of the vehicle 100 along the trajectory TR1-TR4 associated with the selected image IMG. In this example, the control unit 118 outputs a corresponding control signal CTR.

[0115] The respective unit 111 - 118 of the parking assistance system 110 can be implemented in hardware and / or software. In a hardware implementation, the respective unit 111 - 118 can be designed, for example, as a computer or as a microprocessor. In a software implementation, the respective unit 111 - 118 can be designed as a computer program product, as a function, as a routine, as an algorithm, as part of a program code, or as an executable object. Furthermore, each of the units 111 - 118 mentioned here can also be designed as part of a higher-level control system of the vehicle, such as a central electronic control device and / or a control unit (ECU: Electronic Control Unit). In embodiments, the display device and / or the user interface 105 is part of the parking assistance system 110 (not shown).

[0116] Fig. 7 shows a schematic block diagram of an embodiment of a method for operating a parking assistance system 110 for a vehicle 100, in particular the parking assistance system 110 shown in Fig. 6 and the vehicle 100 shown in Fig. 1. The parking assistance system 110 is configured in a tracking mode for autonomously tracking a trajectory TR1 - TR4 (see Fig. 4 or 5) from a number of trajectories TR1 - TR4 learned in a training mode, wherein the respective trajectory TR1 - TR4 is defined by a sequence of positions P1 - P6 (see Fig. 4) and connects a starting position P1 with a target position P6, and wherein the respective trajectory TR1 - TR4 is assigned a number of images IMG1 - IMG6 (see Figs. 2 - 6) of an environment 200 (see Fig. 1) of the vehicle 100 at the respective position P1 - P6 is assigned and stored.In a first step S1, a number of stored images IMG1 - IMG6 are provided. In a second step S2, a representation DSP (see Figs. 2 - 5) is determined which uses the provided images IMG1 - IMG6 to represent the respective assigned trajectory TR1 - TR4. In a third step S3, the determined representation DSP is output to a display device of a user interface 105 (see Fig. 1) of the vehicle 100. In a fourth step S4, a user input SIG (see Fig. 6) for selecting at least one image IMG1 - IMG6 contained in the representation DSP is received from the user interface 105. And in a fifth step S5, an autonomous journey of the vehicle 100 along the trajectory TR1 - TR4 assigned to the selected image IMG1 - IMG6 is initiated.

[0117] Although the present invention has been described using exemplary embodiments, it is capable of being modified in many ways.

[0118] 100 vehicles

[0119] 105 User interface

[0120] 110 Parking assistance system

[0121] 111 Provisioning Unit

[0122] 112 Investigation Unit

[0123] 114 Output unit

[0124] 116 Receiving unit

[0125] 118 Control unit

[0126] 120 Environmental sensor device

[0127] 130 Environmental sensor device

[0128] 200 surroundings

[0129] CTR control signal

[0130] DMAP digital environmental map

[0131] DSP display

[0132] IMG1 Image

[0133] IMG2 Image

[0134] IMG3 image

[0135] IMG4 image

[0136] IMG5 image

[0137] IMG6 Image

[0138] OB1 Object

[0139] OB2 Object

[0140] OB3 object

[0141] OB4 object

[0142] OL object

[0143] P1 Position

[0144] P2 Position P3 Position

[0145] P4 Position

[0146] P5 Position

[0147] P6 Position

[0148] 51 process step

[0149] 52 process steps

[0150] 53 Process step

[0151] 54 Process step

[0152] 55 Process step

[0153] SIG user input

[0154] TR1 trajectory

[0155] TR2 trajectory

[0156] TR3 trajectory

[0157] TR4 trajectory

Claims

PATENT CLAIMS 1. Method for operating a parking assistance system (110) for a vehicle (100), wherein the parking assistance system (110) is configured in a following mode for autonomously following a trajectory (TR1 - TR4) from a number of trajectories (TR1 - TR4) learned in a training mode, wherein the respective trajectory (TR1 - TR4) is defined by a sequence of positions (P1 - P6) and connects a starting position (P1) with a target position (P6), and wherein a number of images (IMG1 - IMG6) of an environment (200) of the vehicle (100) at the respective position (P1 - P6), captured by a vehicle-integrated camera (120), are assigned and stored for the respective trajectory (TR1 - TR4), the method comprising the steps: Providing (S1) a number of stored images (IMG1 - IMG6), determining (S2) a representation (DSP) which includes the provided images (IMG1 - IMG6) to represent the respective assigned trajectory (TR1 - TR4), outputting (S3) the determined representation (DSP) to a display device of a user interface (105) of the vehicle (100), Receiving (S4) a user input (SIG) to select at least one image (IMG1 - IMG6) contained in the display (DSP) from the user interface (105), and Initiating (S5) an autonomous journey of the vehicle (100) along the trajectory (TR1 - TR4) assigned to the selected image (IMG1 - IMG6).

2. Method according to claim 1, characterized in that the number of stored images (IMG1 - IMG6) for the respective trajectory (TR1 - TR6) includes at least one image of the respective target position (P6).

3. Method according to claim 1 or 2, characterized in that the number of stored images (IMG1 - IMG6) for the respective trajectory (TR1 - TR4) comprises an image composed of a plurality of individual images.

4. Method according to claim 3, characterized in that the composite The image includes a bird's-eye view of the respective position (P1 - P6).

5. Method according to any one of the preceding claims, characterized by: Providing an object (OL) to represent a position of the vehicle (100) in the target position (P6) of the respective trajectory (TR1 - TR4), and Determine the representation (DSP) using the provided object (OL) so that the object (OL) in the representation (DSP) is superimposed at the target position (P6) in an image (IMG1 - IMG6) in which the target position (P6) is visible.

6. Method according to one of the preceding claims, characterized by: Determining a digital environment map (DMAP) for the respective trajectory (TR1 - TR4), and Determining the representation (DSP) using the determined digital environment map (DMAP), such that the determined digital environment map (DMAP) is displayed together with the provided image (IMG1 - IMG6) for the respective trajectory (TR1 - Tr4), wherein the image (IMG1 - IMG6) is arranged in the representation (DSP) depending on its position in the digital environment map (DMAP).

7. Method according to one of the preceding claims, characterized in that the number of trained trajectories (TR1 - TR4) comprises at least two trajectories that run in the same area, wherein the representation (DSP) is determined such that the respective images (IMG1 - IMG6) of the at least two trajectories are displayed in a group.

8. Method according to one of the preceding claims, characterized in that the number of learned trajectories (TR1 - TR4) comprises at least two trajectories that run in the same area, and with: Determining a common digital environment map (DMAP) for at least two trajectories, and Determining the representation (DSP) using the determined common digital environment map (DMAP), such that the common determined digital environment map (DMAP) is displayed together with the respective provided image (IMG1 - IMG6) for the respective trajectory (TR1 - TR4), wherein the respective image (IMG1 - IMG6) is arranged in the representation depending on its position in the digital environment map (DMAP).

9. Method according to one of the preceding claims, characterized in that the number of images (IMG1 - IMG6) assigned to the respective trajectory (TR1 - TR4) is recorded and stored during the training mode of the trajectory (TR1 - TR4).

10. Method according to one of the preceding claims, characterized in that during a traverse of the respective trajectory (TR1 - TR4) in the traverse mode, new images are captured with the vehicle's own camera (120) and assigned to the trajectory (TR1 - TR4) and stored.

11. Method according to one of the preceding claims, characterized in that the respective number of images (IMG1 - IMG6) is captured by a number of cameras (120), the number comprising a front camera, a rear camera, a side camera on a left side of the vehicle and / or a side camera on a right side of the vehicle.

12. Method according to one of the preceding claims, characterized in that each stored image (IMG1 - IMG6) is assigned position information with respect to a world coordinate system, and wherein the representation is determined such that the respective image is arranged relative to other images depending on its assigned position in the world coordinate system.

13. Computer program product comprising instructions which, when the program is executed by a computer, cause it to execute the method according to any one of claims 1-12.

14. Parking assistance system for a vehicle (100), wherein the parking assistance system (110) is configured in a following mode for autonomously following a trajectory (TR1 - TR4) from a number of trajectories (TR1 - TR4) learned in a training mode, wherein the respective trajectory (TR1 - TR4) is defined by a sequence of positions (P1 - P6) and connects a starting position (P1) with a target position (P6), and wherein a number of images (IMG1 - IMG6) of an environment (200) of the vehicle (100) at the respective position (P1 - P6), captured by a vehicle-integrated camera (120), are assigned and stored for the respective trajectory (TR1 - TR4), the parking assistance system (110) comprising: a provisioning unit (111) for providing a number of stored images (IMG1 - IMG6), a detection unit (112) for Determining a representation (DSP),which comprises the provided images (IMG1 - IMG6) for representing the respective assigned trajectory (TR1 - TR4), an output unit (1 14) for outputting the determined representation (DSP) to a display device of a user interface (105) of the vehicle (100), a receiver unit (116) for receiving a user input (SIG) for selecting at least one image (IMG1 - IMG6) contained in the representation (DSP) from the user interface (105), and a control unit (1 18) for initiating autonomous driving of the vehicle (100) along the trajectory (TR1 - TR4) assigned to the selected image (IMG1 - IMG6).

15. Vehicle (100) with a number of cameras (120) for capturing a respective image (IMG1 - IMG6) of an environment (200) of the vehicle (100), with a parking assistant tension system (110) according to claim 14 and with a user interface (105) comprising a display device.

Images