Method, computer program product, parking assistance system, and vehicle

Abstract

The present invention relates to a method of operating a parking assistance system (110) for a vehicle (100). The parking assistance system (110) is designed to autonomously track a trajectory (TR1-TR4) from a number of trajectories (TR1-TR4) learned in a training mode in a following mode, and each trajectory (TR1-TR4) is determined at a series of positions (P1-P6), connecting a starting position (P1) and a target position (P6), and each trajectory (TR1-TR4) is assigned to an image (IMG1-IMG6) of the environment (200) of the vehicle (100) captured and stored by a camera (120) of the vehicle at each position (P1-P6). The method includes providing (S1) the stored images (IMG1-IMG6), determining (S2) a display (DSP) including the provided images (IMG1-IMG6) representing each associated trajectory (TR1-TR4), outputting (S3) the determined display (DSP) to a display device of a user interface (105) of the vehicle (100), receiving (S4) a user input (SIG) selecting at least one image (IMG1-IMG6) included in the display (DSP) via the user interface (105), and starting (S5) an autonomous driving of the vehicle (100) along the trajectory (TR1-TR4) associated with the selected image (IMG1-IMG6).

Description

Technical Field

[0001] The present invention relates to an operation method of a parking assistance system, a computer program product, a parking assistance system, and a vehicle having the parking assistance system.

Background Art

[0002] There are known parking assistance systems that can be trained to follow a specific trajectory. This is particularly useful in frequently occurring situations such as parking a vehicle in a garage or parking a vehicle in a predetermined parking space. The driver only needs to drive the vehicle close to the starting point of the trajectory, such as a driveway entrance. Since the parking assistance system automatically follows the trained trajectory, the driver is relieved of the burden.

[0003] When a vehicle is used for a long period of time and / or there are multiple people using the vehicle, a large number of different practice trajectories can be accumulated over time. This causes the user to lose sight of the trajectories considering a large number, and it becomes unclear which trajectory is the required trajectory in a specific situation. In particular, when different people use the vehicle alternately, the later user does not know which trajectory stored by the previous user to target and / or what route the trajectory follows. Further, when there are a plurality of trajectories having paths in the same area, for example, in the user's land or nearby land, it is difficult for the user to distinguish them from each other.

[0004] One option is to assign keywords and descriptions to different trajectories so that the user can distinguish between them. However, this is complex, not very intuitive, and furthermore does not solve the problem when there are multiple different users of the vehicle. If the user is not uniquely associated with a trajectory, first, the user may select the wrong trajectory to follow in a particular situation, resulting in the vehicle not being guided to the desired parking position. Then, it is desirable for the user to park the vehicle manually or try other trajectories to select the correct one. Ultimately, this situation consumes resources such as fuel and time unnecessarily and generates user dissatisfaction. The lack of relevance within such trained trajectories leads to a decrease in the overall willingness of the user to use the parking assistance system, which may thereby reduce the advantages of the parking assistance system.

[0005] German Patent Application Publication No. 102015010746 discloses a method for estimating the self-position of a vehicle. This includes an image capture unit including the ground within the vehicle environment, recording an image along a first trajectory during driving, comparing the image with a recorded image related to the position, and this comparison is adopted as a basis for determining the current position and / or current direction of the vehicle.

[0006] German Patent Application Publication No. 102013015349 discloses a method for operating a vehicle to approach a parking space in a blind / off-road parking lot. This method includes collecting environmental data related to the vehicle, and when approaching a parking space in the parking area, it is identified whether the parking space is the home parking space or whether the parking lot is the home parking lot, and the environmental data or driving data collected when approaching the home parking space or the home parking lot is saved or updated.

Summary of the Invention

Problems to be Solved by the Invention

[0007] In response to this problem, one object of the present invention is to improve the parking assistance system.

Means for Solving the Problem

[0008] According to the first aspect, a parking assistance system for a vehicle is proposed. In the follow - up mode, the vehicle following system autonomously follows a certain trajectory from a number of trajectories taught in the training mode. Each trajectory is defined by a series of positions that connect the starting position and the target position, and each trajectory is stored in association with an image of the vehicle's environment at each position captured by an in - vehicle camera. This method includes distributing the stored images, determining a display including the distributed images to represent the trajectories associated with each of them, outputting the determined display to a display device of the vehicle's user interface, receiving a user input for selecting at least one image included in the display from the user interface, starting the autonomous driving of the vehicle along the trajectory associated with the selected image, and

[0009] The advantage of this method is that each user can intuitively recognize from the image what the trajectory is, what target position the trajectory leads to, and / or what route the trajectory follows. This is particularly applicable when multiple users do not learn each trajectory in the vehicle itself for alternating use of the vehicle. As a result, the user can select the correct trajectory in each situation and has the advantage of successfully completing the automatic parking operation with higher reliability. Selecting an incorrect trajectory is avoided, and the labor associated with parking the vehicle in an undesirable parking position can also be avoided. In addition, this method saves the trouble of the user assigning metadata such as titles, keywords, and / or descriptions to each trajectory. Furthermore, since there is no need to store such metadata to identify each trajectory, the memory requirement is reduced.

[0010] The parking system is configured to autonomously follow each track, and in this specification, it can be understood to mean that the parking assistance system autonomously controls the vehicle. This is achieved in particular by using camera images, LiDAR data and / or radar data related to the vehicle's environment in this specification. Based on these images or data, the parking assistance system can, for example, perform self-positioning and orientation determination (which is also called SLAM: Simultaneous Localization And Mapping), and can be configured to detect obstacles in the environment. A vehicle having a parking assistance system can also be called an autonomous driving vehicle. In some cases, it is stipulated that autonomous control is to be executed under the supervision of the user, and the user does not necessarily have to be inside the vehicle.

[0011] The automation level of the vehicle is, for example, level 4 or 5 of the SAE classification system. The SAE classification system was published in 2014 by SAE International, an automotive standards body, as J3016 "Taxonomy and Definitions for Terms Related to On-Road Motor Vehcle Automated Driving Systems". This is based on six different levels of automation, taking into account the degree of system intervention required and the degree of driver attention required. The SAE automation levels range from level 0 corresponding to a fully manual system to levels 1 to 2 for systems that assist the driver, semi-automatic driving (levels 3 to 4), and a fully automatic system (level 5) where a driver is no longer required. An autonomous driving vehicle is a vehicle that can sense and navigate the environment without human input. In particular, it corresponds to SAE automation level 5.

[0012] When teaching or learning respective trajectories in the training mode, it is more desirable for the user to manually control the vehicle. The user can also be assisted by the parking assistance system, but the vehicle is not autonomously driven by the parking assistance system. The user of the vehicle has full control of the vehicle. The taught trajectory is unknown at this point and is determined only when the training run is executed.

[0013] In a preferred embodiment, the parking assistance system is configured to follow respective trajectories using images of the vehicle's environment captured by an in-vehicle camera. This is understood to mean that the parking assistance system in particular executes a VSLAM (Visual Simultaneous Localization And Mapping) algorithm using the captured images of the orientation in the environment, and in particular determines the position of the vehicle related to the position of each trajectory.

[0014] Each trajectory is defined in particular by a series of positions. In particular, the trajectory includes one or more reference positions. The reference positions correspond in particular to the positions of the vehicle in the training mode in order to determine features in the environment such as images related to the environment, LiDAR data, and / or radar data that are received and processed and also referred to as engineering features that are also reference features for orientation. Each reference position is associated with a respective group of determined features. This means that the reference positions are defined while the trajectory is being taught. For example, each reference position is defined by two coordinates in a two-dimensional coordinate system and the direction of the vehicle at this position. For example, each trajectory starts in particular at a starting position which is a reference position and ends in particular at a target position which is also a reference position here.

[0015] This trajectory includes one trajectory or a plurality of trajectories. The plurality of trajectories may be arranged in the same region or the same area, for example, close to each other or in different regions or areas. As an example, the user can teach a plurality of trajectories on private land and other trajectories such as at workplaces, stations, shopping centers, etc.

[0016] This image associated with each trajectory can include one image, two images, or more than two images for each trajectory. This image is captured and stored especially while the trajectory is being taught. Each image shows details from the vehicle's environment in particular, and this detail depends on the position of the vehicle, the orientation, and the field of view of the camera used at the time the image was captured. As an example, each position in the trajectory has a specific associated image where the position itself can be seen. Alternatively, each position can also have an associated image captured from that position.

[0017] The first step of the method is to distribute the stored images. The distributed images may be a subset of all the stored images, or may include all the stored images. All the stored images include all the stored images of all the learned trajectories. If only a subset of all the images is distributed, this subset is selected from among all the images based on a predetermined selection criterion. As an example, one image, especially an image of each target position, or two images, especially an image of each start position and target position, can be selected for each trajectory. Further, the subset can be selected, for example, based on the current position information related to the vehicle and the position information of each image, and only images of positions near the current position are selected. "Near" means that the distance from the current position to the position of the image is less than or equal to a predetermined threshold value.

[0018] The second step includes determining a display that includes the delivered image and represents each associated trajectory. This display can be regarded as a graphical user interface in which the images delivered in a predetermined manner are arranged. As an example, the display includes images arranged in a list or table. It also includes providing an overview schematic display of the recorded route. In addition to the images, the display can include other elements such as objects and / or information, particularly markings of individual images as track numbers, start positions or target positions, and graphic elements that subdivide and distinguish images of different trajectories from each other.

[0019] The display can include dynamic elements such as, for example, an animation that includes multiple images for each trajectory displayed at a specific position within the display.

[0020] The display may be optimized for output on a display device of the user interface. This means, in particular, that the display has a width and / or height that can be displayed on the display device without additional scaling.

[0021] The display can include a plurality of different sub - displays, for example, providing different zoom levels for the image. This enables a detailed display of the image and is particularly advantageous for display devices that only have low resolution.

[0022] The display may have a height that is greater than can be displayed in a visual display on the display device. In this case, for example, the display can be scrolled through.

[0023] The third step includes outputting the determined display to a display device of the vehicle's user interface. In particular, the display is transmitted to the display device in the form of an image signal, preferably a digital image signal. The display device receives the image signal and outputs an image corresponding to its visual display. In other words, the display device displays the display. The display device includes, for example, a screen, preferably a touch - sensor screen.

[0024] The fourth step includes receiving a user input for selecting at least one image included in the display from a user interface. Since each image is associated with a trajectory, the selection of the image corresponds to selecting the associated trajectory. By selecting an image, the user can select the trajectory to follow. This is particularly intuitive by means of a touch sensor screen that requires the user to simply touch the appropriate image of the display on the display device.

[0025] The user input is received at the user interface, particularly in the form of coordinates based on the display. Since the display is particularly two-dimensional, each position during the display can be clearly defined by two coordinates, for example, the coordinates of pixels. The user input includes, for example, a coordinate tuple or a range of coordinates. From these coordinates, it can be determined whether the user has selected any of the images of the display.

[0026] The fifth step includes starting the autonomous driving of the vehicle along the trajectory associated with the selected image. This means that the parking assistance system controls the vehicle to move along the selected trajectory to the target position and stop there.

[0027] In an embodiment, the display device is a component of the parking assistance system.

[0028] According to an embodiment of the method, the stored image for each trajectory includes at least one image of each target position.

[0029] In this embodiment, the display of each trajectory is determined particularly using the image of the target position.

[0030] The target position is particularly the simplest position of each trajectory that the user remembers, which is the reason why this embodiment has a particularly high level of reliability regarding the selection of the trajectory.

[0031] Note that the image of the target position can include both the image recorded from the target position and the image recorded from a position in front of the target position. Recording in front of the target position can be advantageous, particularly when the target position itself is located close to a wall or other obstacle, for example, when the image from the target position does not provide a good view.

[0032] In other embodiments, the stored images for each trajectory include at least one image of each starting position and each target position.

[0033] In another embodiment of the method, the stored images of each trajectory include images edited from a plurality of individual images.

[0034] For example, the edited image can include a large wide-angle view edited from a plurality of individual images each having a smaller field of view. During editing, various image processing steps can be performed, particularly processing such as equalizing the individual images and aligning the exposure and contrast of the individual images.

[0035] In another embodiment of the method, the edited image can include an aerial view of each position.

[0036] This aerial view may be generated, in particular, by distorting the perspective and perspective of the image.

[0037] In an embodiment, the edited image includes an aerial view of the entire trajectory. As an example, this is achieved by first converting the individual images of each position of the trajectory into aerial views and then editing these individual images into one image. The editing is performed, in particular, by determining a panoramic image from a plurality of individual images.

[0038] According to another embodiment of the method, the method comprises distributing an object representing the position of the vehicle at the target position of each trajectory, and Determine a display using the delivered object, and overlay the object in the display on the target position of the image to visualize the target position, including.

[0039] For example, an object representing the position of a vehicle includes a projection of the contour of the vehicle at the target position on the ground. Therefore, if the image of the target position indicates the target position, the contour of the vehicle on the ground is inserted into this image. This can be realized, for example, in the form of the shape of the darkened area or the shape of the contour inserted into the image.

[0040] In this context, "object" is understood to mean, in particular, a graphical object that can be displayed on a graphical display, in particular an image.

[0041] According to another embodiment of the method, the method comprises determining a digital environment map for each trajectory, determining a display using the determined digital environment map, and displaying the determined digital display map together with the delivered image for each trajectory, and the image is arranged in the display based on the position of the image in this digital environment map, including.

[0042] The digital environment map particularly includes a display or representation of the environment of the vehicle, and for example, detected objects such as buildings or vegetation are displayed on the map. The digital environment map can be determined based on the captured environment sensor data of various environment sensors such as cameras, LiDAR, radar and / or ultrasonic waves. This is also called sensor fusion.

[0043] The digital environment map for each trajectory is particularly determined while the trajectory is being taught in the training mode.

[0044] Images arranged within a display based on their positions in a digital environment map are understood to mean that, in particular, each image in the digital environment map is arranged such that the relative positions of the objects in the digital environment map and the positions of the objects visible in each respective image correspond to the relative positions of the actual objects. Put simply, this means that an image corresponding to a target position next to a house is also arranged next to the house within the display of the digital environment map.

[0045] Also, it can be said that the position of each image depends on and is derived from the position of the object visible in the image.

[0046] When multiple images are distributed for a trajectory, all the distributed images may be arranged within the display in a manner appropriate to the digital environment map.

[0047] According to another embodiment, the method is such that the taught trajectory includes at least two trajectories whose paths are in the same area, and the display is determined such that the respective images of the at least two trajectories are displayed within a group.

[0048] For example, that the paths of two trajectories are in the same area is understood to mean that the distance between a first position on the first trajectory and a second position on the second trajectory is less than a predetermined threshold value. The predetermined threshold value is, for example, 50m, 30m, 10m, or 5m. Preferably, the positions exist particularly in a world coordinate system. For example, two positions that are at the shortest distance from each other can be used for this determination.

[0049] Trajectories displayed in a group are understood to mean that they are included in the display in a manner that is graphically distinguishable from other trajectories. There may be regulations for combining and displaying trajectories. For example, a placeholder such as a symbol or icon may be displayed instead of an image. Selecting a placeholder enables calling other displays to include the trajectories in the way each respective image is represented.

[0050] According to another embodiment of the method, the taught trajectory can include at least two trajectories having a path in the same area, determining a common digital environment map for at least two trajectories, determining a display using the determined common digital environment map, and the determined common digital environment map being displayed together with the respective delivered images for each trajectory, including, and each image is positioned in the display based on the position of the image in the digital environment map.

[0051] This embodiment is particularly advantageous when the paths of multiple trajectories are in the same area, especially since the risk of confusion is also particularly high. The spatial display using the digital environment map allows the user to intuitively select the correct trajectory in a specific situation, even if the user has not taught the trajectory himself.

[0052] According to another embodiment of the method, the images associated with each trajectory are captured and stored while the trajectory is being taught in the training mode.

[0053] According to another embodiment of the method, new images are captured by an in-vehicle camera, associated with each trajectory, and stored while following the trajectory in the follow mode.

[0054] This is effective in taking into account the temporal environmental changes of the images. The new images can replace all or part of the original images. This is advantageous when the original training is performed in poor visibility conditions such as darkness, rain, snow, and / or a dirty camera lens, and the quality of the stored images is poor.

[0055] According to another embodiment of the method, each of the images is captured by a plurality of cameras including a front camera, a rear camera, a side camera on the left side of the vehicle, and / or a side camera on the right side of the vehicle.

[0056] According to another embodiment of the method, each stored image has position information associated therewith with reference to a world coordinate system, and the display is determined such that each image is arranged relative to other images based on the position associated with the world coordinate system.

[0057] In the present disclosure, it is understood that the world coordinate system means that all coordinates, particularly coordinates of different orbits, also refer to the same reference point (origin). One such world coordinate system is, for example, a coordinate system provided by a satellite navigation system such as NAVSTAR GPS, GALILEO, GLONASS, and / or Beidou.

[0058] In this embodiment, the display can include, for example, a street map overlaid with an image of an orbit. This enables any user of the vehicle to quickly and intuitively check where the trained orbit is.

[0059] According to a second aspect, there is provided a computer program product including commands for a computer to execute the method according to the first aspect when the program is executed by the computer.

[0060] The computer is particularly a component of a vehicle and forms, for example, an electronic control unit (ECU).

[0061] The computer program product, for example, computer program means, can be distributed or provided in the form of, for example, a memory card, a USB stick, a CD-ROM, a DVD, or a file downloadable from a server on a network. This can be achieved, for example, by transmitting a corresponding file including the computer program product or computer program means in a wireless communication network.

[0062] According to a third aspect, a parking assistance system for a vehicle is proposed. In the follow - up mode, the parking assistance system is configured to follow a trajectory taught in the training mode. Each trajectory is defined by a series of positions, connecting a starting position and a target position. Each trajectory is associated with an image of the vehicle's environment at each position captured by an in - vehicle camera and stored. The parking assistance system has a distribution unit that distributes the stored images, a determination unit that determines a display including the distributed images and represents the trajectory associated with each of them, an output unit that outputs the determined display including the display to a display device of the vehicle's user interface, a reception unit that receives a user input for selecting at least one image included in the display from the user interface, and a control unit that starts the autonomous driving of the vehicle along the trajectory associated with the selected image. It is provided with.

[0063] The parking assistance system has the same advantages as those described for the method according to the first aspect. This embodiment and features described for the proposed method are applied mutatis mutandis to the proposed parking assistance system.

[0064] Each unit of the parking assistance system may be implemented in hardware and / or software. In the hardware implementation, each unit may be in the form of, for example, a computer or a microprocessor. In the software implementation, each unit may be in the form of a computer program product, a function, a routine, an algorithm, a part of program code, or an execution object. Further, each unit in the present disclosure may be in the form of a part of a vehicle's upper - level control system such as a central electronic control device and / or a control unit (ECU: Electronic Control Unit).

[0065] The parking assistance system is configured in particular to execute the method according to the first aspect.

[0066] According to a fourth aspect, a vehicle is proposed which has a camera for capturing respective images of the environment of the vehicle, a parking assistance system according to the third aspect, and a user interface including a display device.

[0067] This vehicle is, for example, a passenger car or a truck. The vehicle preferably comprises a sensor unit configured to detect the driving state of the vehicle and sense the environment of the vehicle. Examples of such a sensor unit of the vehicle are cameras, RADAR (Radio detection and ranging), LiDAR (Light detection and ranging) for vice in image recording, ultrasonic sensors, position sensors, wheel angle sensors and / or wheel speed sensors. The sensor unit is configured to output sensor signals to, for example, a parking assistance system, and the parking assistance system is configured to execute controlling the vehicle semi-autonomously or fully autonomously based on the acquired sensor signals.

[0068] The camera includes a front camera, a rear camera, a side camera on the left side of the vehicle, and / or a side camera on the right side of the vehicle.

[0069] Further possible implementations of the invention also include combinations not explicitly mentioned with respect to the features or embodiments described above or below of the present disclosure for the exemplary embodiments. A person skilled in the art also adds individual aspects as improvements or supplements to the respective basic forms of the invention.

Brief Description of the Drawings

[0070] Further advantageous configurations and aspects of the invention are the subject of the subclaims of the invention described in the present disclosure and also the subject of the exemplary embodiments described below. The invention will be explained in more detail below based on preferred embodiments with reference to the accompanying drawings.

[0071]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Best Mode for Carrying Out the Invention

[0072] In the figures, unless otherwise indicated, the same reference numerals are provided for identical or functionally identical elements.

[0073] FIG. 1 shows a schematic bird's-eye view of vehicle 100. Vehicle 100 is, for example, an automobile arranged within environment 200. Automobile 100 is equipped with a parking assistance system 110 and is formed, for example, by a control unit. Further, a plurality of environmental sensor devices 120, 130 are arranged on automobile 100, which are, by way of example, an optical sensor 120 and an ultrasonic sensor 130. Optical sensor 120 includes, for example, a visual camera, radar, and / or LiDAR. Optical sensor 120 can, in particular, capture an image of each region from the environment 200 of automobile 100 and output the image as an optical sensor signal. Ultrasonic sensor 130 is configured to measure the distance from an object arranged within environment 200 and output a corresponding sensor signal. Based on the sensor signals obtained from sensors 120, 130, parking assistance system 110 can operate automobile 100 semi-autonomously or completely autonomously. In addition to the optical sensor 120 and ultrasonic sensor 130 shown in FIG. 1, vehicle 100 may have various other sensor devices 120, 130. These examples include a wheel speed sensor, a steering angle sensor, a position sensor, a microphone, an acceleration sensor, an antenna connected to a receiver capable of receiving a data signal transmissible by electromagnetic waves, and the like.

[0074] The vehicle also has a user interface 105, and the parking assistance system 110 is communicably connected to the user interface 105. This means that the parking assistance system 100 and the user interface 105 are configured to exchange data, particularly in the form of analog or digital data signals. The user interface 105 includes a display device (not shown) configured to display images IMG1 to IMG6 (see FIGS. 2 to 5) and a display DSP (see FIGS. 2 to 5). Further, the user interface 105 includes input means (not shown) that enable a user of the vehicle 100 to input to the systems of the vehicle 100, particularly to the parking assistance system 110. The input means can include buttons, switches, rotary controls, touch sensors, voice detection, gesture detection, etc. Each input can be associated, in particular, with an element displayed by the display device at a specific time.

[0075] In the follow - up mode, the parking assistance system 110 is configured to autonomously follow the trajectories TR1 to TR4 (see FIGS. 4 and 5) from the trajectories TR1 to TR4 taught in the training mode. Each of the trajectories TR1 to TR4 is defined by a series of positions P1 to P6 (see FIG. 4) that connect the starting position P1 and the target position P6. Each of the trajectories TR1 to TR4 is stored in association with images IMG1 to IMG6 of the environment 200 of the vehicle 100 captured by the in - vehicle camera 120. Each of the images IMG1 to IMG6 is recorded when the vehicle 100 is at each of the positions P1 to P6 of the trajectories TR1 to TR4. In an embodiment, each of the images IMG1 to IMG6 can include image information from a plurality of images IMG1 to IMG6 and can particularly be a merged image. The vehicle assistance system 110 is designed as shown in FIG. 6 and is configured to execute the method described with reference to FIG. 7.

[0076] FIG. 2 is a diagram showing an overview of a display DSP composed of two images, IMG1 and IMG2. The display DSP is determined by a determination unit 112 (see FIG. 6) based on images IMG1 to IMG6 distributed by a distribution unit 111 (see FIG. 6).

[0077] The two images, IMG1 and IMG2, are associated with, for example, the taught trajectories TR1 to TR4 (see FIGS. 4 or 5), which are the trajectories TR1 to TR4 recorded by the parking support system 110 (see FIGS. 1 to 6) when the user drives the vehicle 100 of FIG. 1. In this example, the images IMG1 and IMG2 are a view of the environment 200 (see FIG. 1) of the vehicle 100 as seen from the start position P1 (see FIG. 4) (IMG1 with the caption "Start"), and a view of the environment 200 (see FIG. 1) of the vehicle 100 as seen from the target position P6 (see FIG. 4) (IMG2 with the caption "End"), and these images IMG1 and IMG2 are, for example, captured by the front camera 120. It should be noted that, in order to provide a better overview, especially an overview including the target position P6 itself, an image seen from a position in front of the target position P6 is advantageously used to display the target position P6 instead of the image IMG2 seen from the display position P6 (see also FIG. 3).

[0078] In the first image IMG1, there is a fence or a wall in the background, and a house with a tree and a grove next to it can be seen. In the second image IMG2, only a part of the grove, a part of the wall, and a part of the fence can be seen. In this example, the user uses the vehicle 100, especially to train the trajectory TR1 shown in FIG. 4, and for this purpose, the parking support system 110 can autonomously drive the vehicle 100 from the start position P1 to the target position P6.

[0079] It is shown that the DSP is transmitted from the output unit 114 (see FIG. 6) of the parking assistance system 110 to the user interface 105 (see FIG. 1), and the user interface 105 displays it on the display DSP on the display device. Thereby, the user can intuitively confirm the trajectory TR1 associated with the images IMG1 and IMG2 and can select to follow this trajectory. In an embodiment, the display DSP includes only one image of the trajectory TR1, or includes two or more images of the trajectory TR1 and / or an edited image of the trajectory TR1 (see also FIG. 4).

[0080] When the parking assistance system 110 is trained a plurality of times and includes the stored trajectories TR1 to TR4, the corresponding display can be determined and output for each of the additional trajectories TR1 to TR4. As a result, the user can select the desired trajectories TR1 to TR4 based on the respective images IMG1 and IMG2.

[0081] It should be noted that the display DSP can include the images IMG1 and IMG2 corresponding to the plurality of trajectories TR1 to TR4, and these are arranged adjacent to each other simultaneously or below (not shown) in the display DSP, for example.

[0082] For example, when the user selects the trajectory TR1, the user interface 105 transmits the corresponding user input to the parking assistance system 110, and the parking assistance system can start autonomous following of the selected trajectory TR1.

[0083] FIG. 3 is a diagram showing another concept of a display DSP including two images IMG1 and IMG2. The display DSP is determined by the determination unit 112 (see FIG. 6) based on a plurality of images IMG1 to IMG6 distributed by the distribution unit 111 (see FIG. 6), in particular.

[0084] The display in FIG. 3, which is substantially corresponding to the display in FIG. 2, is different in that an object OL is additionally displayed in the images IMG2. For example, the object OL is a geometric figure (a rectangle with perspective distortion) that is inserted into or overlaid on each of the images IMG1 and IMG2. The object OL is inserted into the image at a position corresponding to the target position P6 (see FIG. 4) on the trajectory TR1, that is, at the position of the vehicle 100 when traveling along the trajectory TR1.

[0085] In this example, the second image IMG2 is recorded at a position in front of the target position P6, that is, for example, from the position P5 (see FIG. 4), which is why the target position P6 is included in the image IMG2.

[0086] By the additional overlay of the target position P6 in the displayed images IMG1 and IMG2, the user can more closely associate with the trajectory TR1. In particular, it can end in an adjacent parking space and distinguish the displayed images with substantially the same trajectory from each other.

[0087] Note that instead of the display DSP in FIG. 3, for example, the object OL is overlaid only on either of the images IMG1 and IMG2.

[0088] FIG. 4 is a diagram showing an overview of a display DSP including a digital environment map DMAP and a plurality of images IMG1 to IMG6 associated with the trajectory TR1. For example, the environment displayed in the digital environment map DMAP is the same as the environment displayed in the images IMG1 and IMG2 in FIGS. 2 and 3. The objects OB1 to OB4 detected in the digital environment map DMAP are displayed corresponding to the environment. In this example, they are a house OB1, a thicket OB2, a fence OB3, and a tree OB4. The digital environment map DMAP is preferably acquired and stored when the trajectory TR1 is learned.

[0089] In this example, the trajectory TR1 comprises six positions P1-P6, where the position P1 is a starting position and the position P6 is a target position. For example, the starting position P1 is in front of the house (here shown as object OB1), and the target position P6 is on the side, next to the house OB1. At each position P1-P6, the training captures at least one of the images IMG1-IMG6 of the environment, associates it with the trajectory TR1, and stores it. The images IMG1-IMG6 are preferably edited images showing a bird's-eye view of the respective position. This is possible especially when the vehicle 100 comprises multiple cameras 120 that can capture a total of 360 degrees of viewing angle around the vehicle 100. Even if the vehicle 100 has only one front camera 120, the image from the front camera 120 can be converted to a top-down view by appropriately distorting the perspective of the image. Note that each of images IMG1-IMG6 was not necessarily captured when the vehicle 100 was in the corresponding position. Rather, image IMG3 was captured by the front camera when the vehicle 100 was in position P2, and this applies mutatis mutandis to the other images / positions.

[0090] In this example, the display includes both the digital environment map DMAP and the images IMG1 to IMG6 associated with the trajectory TR1 and used as a representation of the trajectory TR1. The images IMG1 to IMG6 are arranged in a way that corresponds to reality, particularly in the digital environment map of the display DSP. The display in which the images IMG1 to IMG6 are overlaid together with the digital environment map DMAP allows the user to ascertain the trajectory TR, and in particular the target position P6 on the trajectory TR1, more accurately.

[0091] If images IMG1 to IMG6 are close enough and / or overlapping, they may be merged to generate and store a single image.

[0092] In an embodiment, the trajectory TR1 can also be inserted into the display DSP in the form of an object OL (see FIG. 3).

[0093] FIG. 5 is another diagram showing an overview of a display DSP with a digital environment map. For example, this includes the same environment 200 already described in FIG. 4. In this example, however, there are multiple trajectories TR1 to TR4 in the display area of the environment. As an example, the different trajectories TR1 to TR4 are trained by different users of the vehicle 100. In this example, all the trajectories TR1 to TR4 at the starting position are displayed by the image IMG1, but this is not essential.

[0094] Since each of the images IMG2 to IMG5 of each target position is arranged in a way that actually corresponds to the display DSP with reference to the digital environment map DMAP, in particular, the objects OB1 to OB4, it is quite easy for the user to derive the position of the actual vehicle from the displayed target position.

[0095] For example, the user can select one of the images IMG2 to IMG5 in the display DSP to select the associated trajectories TR1 to TR4.

[0096] In an embodiment, each object OL (see FIG. 3) for the trajectories TR1 to TR4 may also be inserted into the display DSP.

[0097] FIG. 6 shows a schematic block of an exemplary embodiment of a parking assistance system 110 that can be used with, for example, the vehicle 100 of FIG. 1. In the follow mode, the parking assistance system 110 is configured to autonomously follow the tracks TR1 to TR4 (see FIGS. 4 or 5) from a number of tracks TR1 to TR4 taught in the training mode. Each of the tracks TR1 to TR4 is defined by a series of positions P1 to P6 (see FIG. 4) and connects the start position P1 and the target position P6. Each of the tracks TR1 to TR4 is stored in association with images IMG1 to IMG6 (see FIGS. 2 to 6) of the environment 200 (see FIG. 1) of the vehicle 100 captured by the in-vehicle camera 120 (see FIG. 1). Each of the images IMG1 to IMG6 is recorded when the vehicle 100 is at each of the positions P1 to P6 of the tracks TR1 to TR4, and thus shows the corresponding details from the environment 200. The parking assistance system 110 includes a distribution unit 111 that distributes the stored image IMG, a determination unit 112 that determines a display DSP including the distributed image IMG representing each of the associated tracks TR1 to TR4 (see FIGS. 4 or 5), an output unit 114 that outputs the determined display DSP to the display device of the user interface 105 (see FIG. 1), particularly in the form of a digital image or a data signal, a reception unit 116 that receives a user input SIG that selects at least one image IMG included in the display DSP from the user interface 105, and a control unit 118 that starts autonomous driving of the vehicle 100 along the tracks TR1 to TR4 associated with the selected IMG. In this example, the control unit 118 outputs a corresponding control signal CTR.

[0098] Each unit 111 to 118 of the parking assistance system 110 may be implemented in hardware and / or may be implemented in software. In a hardware implementation, each unit 111 to 118 may be in the form of, for example, a computer or a microprocessor. In a software implementation, each unit 111 to 118 may be in the form of a computer program product, a function form, a routine form, an algorithm form, a program code form, or an executable object form. Further, each unit 111 to 118 mentioned here may be in the form of a part of a vehicle's upper control system such as a central electronic control unit and / or a control unit (ECU: Electronic Control Unit).

[0099] In an embodiment, the display device and / or the user interface 105 is a part of a parking assistance system 110 (not shown).

[0100] FIG. 7 shows a schematic block diagram of an exemplary embodiment of a parking assistance system 110 for a vehicle 100, particularly the parking assistance system 110 shown in FIG. 6 and an exemplary method of operating the vehicle 100 shown in FIG. 1. In the follow mode, the parking assistance system 110 is configured to autonomously follow the tracks TR1 - TR4 (see FIGS. 4 or 5) from a plurality of tracks TR1 - TR4 taught in the training mode, where each of the tracks TR1 - TR4 is defined by a series of positions P1 - P6 (see FIG. 4), connecting the starting position P1 and the target position P6, and each of the tracks TR1 - TR4 is associated with and stored in images IMG1 - IMG6 (see FIGS. 2 - 6) captured by an in-vehicle camera 120 (see FIG. 1) at respective positions P1 - P6 of the environment 200 of the vehicle 100. The first step S1 comprises delivering a large number of stored images IMG1 - IMG6. The second step S2 comprises determining a display DSP (see FIGS. 2 - 5) including the delivered images IMG1 - IMG6 showing the tracks TR1 - TR4 associated therewith respectively. The third step S3 comprises outputting the determined display DSP to a display device of a user interface 105 of the vehicle 100 (see FIG. 1). The fourth step S4 comprises receiving a user input SIG (see FIG. 6) for selecting at least one of the images IMG1 - IMG6 included in the display DSP from the user interface 105. The fifth step S5 comprises starting autonomous driving of the vehicle 100 along the tracks TR1 - TR4 associated with the selected images IMG1 - IMG6.

[0101] The present invention has been described based on exemplary embodiments, but can be modified in many ways.

Explanation of Signs

[0102] 100 Vehicle 105 User Interface 110 Parking Assistance System 111 Distribution Unit 112 Decision Unit 114 Output Unit 116 Receiver 118 Control Unit 120 Environmental Sensor Device 130 Environmental Sensor Device 200 Environment CTR Control Signal DMAP Digital Environment Map DSP Display IMG1 Image IMG2 Image IMG3 Image IMG4 Image IMG5 Image IMG6 Image OB1 Object OB2 Object OB3 Object OB4 Object OL Object P1 Position P2 Position P3 Position P4 Position P5 Position P6 Position S1 Method Step S2 Method Step S3 Method Step S4 Method Step S5 Method Step SIG User Input TR1 Trajectory TR2 Trajectory TR3 Trajectory TR4 Trajectory

Claims

1. A method of operating a parking assistance system (110) of a vehicle (100), comprising: In a follow-up mode, the parking assistance system (110) is configured to autonomously follow a plurality of trajectories (TR1-TR4) taught in a training mode; Each of the trajectories (TR1-TR4) is defined by a series of positions (P1-P6) connecting a starting position (P1) and a target position (P6); Each of the trajectories (TR1-TR4) is stored in association with an image (IMG1-IMG6) of the environment (200) of the vehicle (100) at each of the positions (P1-P6) captured by an in-vehicle camera (120); The method comprises: Distributing the stored images (IMG1-IMG6) (S1); Determining a display (DSP) including the distributed images (IMG1-IMG6) (S2) and representing the associated trajectories (TR1-TR4); Outputting the determined display (DSP) to a display device of a user interface (105) of the vehicle (100) (S3); Receiving a user input (SIG) selecting at least one image (IMG1-IMG6) included in the display (DSP) from the user interface (105) (S4); Starting autonomous driving of the vehicle (100) along the trajectory (TR1-TR4) associated with the selected image (IMG1-IMG6) (S5). A method as described above.

2. The stored images (IMG1-IMG6) for each of the trajectories (TR1-TR6) include an image of at least one respective target position (P6). The method according to claim 1.

3. The stored images (IMG1-IMG6) for each of the trajectories (TR1-TR4) include images edited from a plurality of individual images. The method according to claim 1 or claim 2.

4. The edited images include bird's-eye views of each of the positions (P1-P6). The method according to claim 3.

5. Distributing an object (OL) representing the position of the vehicle (100) at the target position (P6) in each of the trajectories (TR1-TR4). Determine the display (DSP) using the delivered object (OL), and overlay the target position (P6) in the images (IMG1 to IMG6) where the object (OL) in the display (DSP) can see the target position (P6). The method according to any one of claims 1 to 4.

6. Determine a digital environment map (DMAP) for each of the trajectories (TR1 to TR4). Using the determined digital environment map (DMAP), determine the display (DSP), and display the determined digital environment map (DMAP) together with the delivered images (IMG1 to IMG6) for each of the trajectories (TR1 to TR4). The images (IMG1 to IMG6) are arranged within the display (DSP) based on the position of the images in the digital environment map (DMAP). The method according to any one of claims 1 to 5.

7. The taught trajectories (TR1 to TR4) include at least two trajectories having paths in the same area, and the display (DSP) is determined such that the respective images (IMG1 to IMG6) of the at least two trajectories are displayed in a group. The method according to any one of claims 1 to 6.

8. The taught trajectories (TR1 to TR4) include at least two trajectories having paths in the same area. Determining a common digital environment map (DMAP) for the at least two trajectories. Using the determined common digital environment map (DMAP) to determine the display (DSP), and displaying the determined common digital environment map (DMAP) together with the respective delivered images (IMG1 to IMG6) for each of the trajectories (TR1 to TR4). Including Each of the images (IMG1 to IMG6) is arranged in the display based on the position of the images in the digital environment map (DMAP). The method according to any one of claims 1 to 7.

9. In the training mode, the images (IMG1 to IMG6) associated with each of the trajectories (TR1 to TR4) are captured and stored while the trajectories (TR1 to TR4) are being taught. The method according to any one of claims 1 to 8.

10. In the following tracking mode, new images captured by the in-vehicle camera (120), each new image associated with a respective one of the tracks (TR1 to TR4), are stored while the tracks (TR1 to TR4) are being tracked. The method according to any one of claims 1 to 9.

11. Each of the images (IMG1 to IMG6) is captured by a plurality of cameras (120) including at least one of a front camera, a rear camera, a side camera on the left side of the vehicle, and / or a side camera on the right side of the vehicle. The method according to any one of claims 1 to 10.

12. Each of the stored images (IMG1 to IMG2) is associated with position information with reference to a world coordinate system, and the display is determined such that each of the images is arranged relative to other images based on the positions associated with the world coordinate system. The method according to any one of claims 1 to 11.

13. When the program is executed by a computer, the computer includes commands for executing the method according to any one of claims 1 to 12. A computer program product.

14. A parking assistance system for a vehicle (100), In a tracking mode, the parking assistance system (110) is configured to autonomously track tracks (TR1 to TR4) from a number of tracks (TR1 to TR4) taught in a training mode. Each of the tracks (TR1 to TR4) is defined by a series of positions (P1 to P6) connecting a starting position (P1) and a target position (P6). Each of the tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the environment (200) of the vehicle (100) at each of the positions (P1 to P6) captured by the in-vehicle camera (120). A distribution unit (111) that distributes the stored images (IMG1 to IMG6); A determination unit (112) that determines a display (DSP) including the distributed images (IMG1 to IMG6) and represents the tracks (TR1 to TR4) associated therewith; An output unit (114) that outputs the determined display (DSP) to a display device of the user interface (105) of the vehicle (100). A receiving unit (116) that receives a user input (SIG) for selecting at least one image (IMG1 to IMG6) included in the display (DSP) from the user interface (105); A control unit (118) that starts autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected image (IMG1 to IMG6); A parking assistance system for a vehicle (100), comprising the above.

15. A parking assistance system (110) according to claim 14, having a user interface (105) including a display device, and having a plurality of cameras (120) that capture respective images (IMG1 to IMG6) of the environment (200) of the vehicle (100); A vehicle (100).

Images