Method, computer program product, parking assist system and vehicle
By transmitting and displaying environmental images in the vehicle, users are allowed to select the correct parking trajectory and start autonomous driving, the problem of difficulty in distinguishing trajectory in multiple people using vehicles is solved, and the reliability and user experience of the parking assist system are improved.
Patent Information
- Application Number
- CN202380039478.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-11
- Filing Date
- 2023-04-27
- Publication Date
- 2024-12-24
AI Technical Summary
In vehicles used by multiple people, the parking assist system is difficult to distinguish and select the correct parking trajectory, causing users to choose the wrong track, increasing resource consumption and user dissatisfaction, and reducing the system usage rate.
By transmitting and displaying multiple images of the vehicle environment, users can intuitively identify the target position and path of the trajectory. Users can select images to start the vehicle's autonomous driving along the corresponding trajectory, and use vehicle-mounted cameras, lidar and radar data for positioning and obstacle detection to achieve independent control.
It improves the reliability of users to choose the correct trajectory, reduces metadata storage needs, saves users' efforts to allocate and store resources, and enhances the reliability and user experience of parking assistance systems.
Smart Images
Figure CN119183432A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for operating a parking assistance system, a computer program product, a parking assistance system and a vehicle having a parking assistance system. Background Art
[0002] As is known, parking assistance systems can be trained to follow a specific trajectory. This is particularly useful for frequently occurring situations, such as parking a vehicle in a garage or parking a vehicle in a predetermined parking space. The driver then only needs to drive the vehicle to the starting point on the trajectory, such as the entrance to a driveway. The parking assistance system then autonomously follows the practiced trajectory, thus relieving the driver of the burden.
[0003] Over a longer period of vehicle use and / or when multiple persons use the vehicle, a large number of different practiced trajectories can accumulate over time. This leads to the problem that users can lose track due to the large number and then no longer know which trajectory they want in a specific situation. In particular, when different persons use the vehicle alternately, the latter user does not know what target the trajectory stored by the previous user had and / or what path the trajectory took. Furthermore, when there are multiple trajectories whose paths are in the same area, for example on or near a user's property, it is difficult for the user to distinguish them from each other.
[0004] One option is to assign keywords and / or descriptions to the different trajectories, so that the user can distinguish them from each other. However, this is complex, not very intuitive, and does not solve the problem when the vehicle has multiple different users. If the user cannot uniquely associate the trajectories, this may firstly lead to the user selecting the wrong trajectory in a given situation, resulting in the vehicle not being able to drive to the desired parking location. The user then has to park the vehicle manually, or try other trajectories in the hope of selecting the correct one. Ultimately, this situation leads to unnecessary consumption of resources, such as fuel and time, and to user dissatisfaction. This lack of association in the practice trajectories may lead to an overall reluctance of the user to use the parking assistance system, thereby reducing the benefits of the parking assistance system.
[0005] DE 10 2015 010 746 A1 discloses a method for vehicle self-positioning, comprising an image capture unit whose coverage area includes the ground in the vehicle environment, for recording images of the ground being driven over along a first trajectory, and comparing the images with position-related stored images, and the comparison is used as a basis for determining the current position and / or current orientation of the vehicle.
[0006] DE 10 2013 015 349 A1 discloses a method for operating a vehicle so that the vehicle approaches a parking space in a non-visible / off-road parking area, the method comprising collecting environmental data about the vehicle, wherein approaching a parking space in the parking area results in identifying whether the parking space is a home parking space or whether the parking area is a home parking area, and the environmental data or driving data collected when the home parking space or home parking area is identified and the vehicle approaches the identified home parking space or identified home parking area is stored or updated. Summary of the invention
[0007] Against this background, it is an object of the present invention to improve the operation of a parking assistance system.
[0008] According to a first aspect, a method for operating a parking assistance system of a vehicle is provided. In a following mode, the parking assistance system is configured to autonomously follow a trajectory from a plurality of trajectories taught in a training mode, wherein the respective trajectory is defined by a series of positions and connects a starting position to a target position, and wherein the respective trajectory has associated and stored a plurality of images of the environment of the vehicle at the respective position captured by an onboard camera. The method comprises the following steps:
[0009] Transferring large amounts of stored images,
[0010] determining including displaying of the transmitted image to represent the corresponding associated trajectory,
[0011] outputting the determined display to a display device of a user interface of the vehicle,
[0012] receiving user input from a user interface to select at least one image for inclusion in the display, and
[0013] Initiate autonomous driving of the vehicle along the trajectory associated with the selected image.
[0014] The advantage of the method is that the respective user can intuitively recognize from the image display what the trajectory is, to what target position the trajectory leads and / or what path the trajectory takes. This is particularly true even if multiple users use the vehicle alternately and therefore may not have practiced the respective trajectory with the vehicle itself. This brings the advantage that the user can select the correct trajectory in the respective case and thus successfully complete the autonomous parking maneuver with greater reliability. The selection of the wrong trajectory and thus also the effort involved in parking the vehicle in an undesirable parking position can be avoided. In addition, the method saves the user the effort of assigning metadata to the respective trajectory, such as a title, keywords and / or description. In addition, it is not necessary to store such metadata to identify the respective trajectory, which reduces the memory requirements.
[0015] A parking assistance system configured to autonomously follow a corresponding trajectory is understood here to mean that the parking assistance system autonomously controls the vehicle. This is achieved here in particular by using camera images, lidar data and / or radar data related to the vehicle environment. Based on these images or data, the parking assistance system can, for example, localize and orient itself (this can also be referred to as SLAM, SLAM: Simultaneous Localization and Mapping) and can be configured to detect obstacles in the environment. A vehicle with a parking assistance system can also be referred to as an autonomous vehicle. Autonomous control can be provided, which is performed under the supervision of a user, who does not necessarily have to be in the vehicle.
[0016] According to the SAE classification system, the level of automation of a vehicle has, for example, automation level 4 or 5. The SAE classification system was published in 2014 by the motor vehicle standardization organization SAE International as J3016 "Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems". It is based on six different automation levels and takes into account the required degree of system intervention and the required degree of driver attention. The SAE automation levels range from level 0, which corresponds to a fully manual system, through driver assistance systems at levels 1 to 2, to semi-automatic (levels 3 and 4) and fully automated (level 5) systems where the driver is no longer required. An autonomous vehicle is a vehicle that is able to perceive its environment and navigate without human input. It corresponds in particular to SAE automation level 5.
[0017] When teaching or practicing a corresponding trajectory in training mode, the user preferably controls the vehicle manually. The user may also be assisted by the parking assistance system, but the vehicle is not driven autonomously by the parking assistance system. The user of the vehicle has full control of the vehicle. The trajectory to be taught is unknown at this point and can only be determined when a training run is performed.
[0018] In a preferred embodiment, the parking assistance system is configured to follow the respective track by using images of the vehicle environment captured by an on-board camera. This is understood to mean in particular that the parking assistance system performs a VSLAM algorithm (VSLAM: Visual Simultaneous Localization and Mapping) for orienting itself in the environment using the captured images, in particular for localizing the vehicle relative to a position on the respective track.
[0019] The respective trajectory is defined in particular by a series of positions. In particular, the trajectory comprises one or more reference positions. The reference positions correspond in particular to the positions that the vehicle had in the training mode when receiving and processing images, lidar data and / or radar data relating to the environment in order to determine features in the environment (e.g. optical features, which may also be referred to as orientation features). The respective reference positions have a respective associated set of determined features. This means that the reference positions are defined when the trajectory is taught. For example, the respective reference positions are defined by two coordinates in a two-dimensional coordinate system and the orientation of the vehicle at that position. For example, the respective trajectory starts at a starting position, which is in particular a reference position, and ends at a target position, which is in particular also a reference position.
[0020] The number of tracks includes one track or multiple tracks. The multiple tracks can be located in the same zone or area, such as close to each other, or in different zones or areas. For example, a user can teach multiple tracks on private property and other tracks at their workplace, station, shopping center, etc.
[0021] The plurality of images associated with the respective track may include one image, two images, or more than two images for the respective track. The images are captured and stored, in particular when the track is taught. The respective images particularly show details from the vehicle's environment, which details depend on where the vehicle was when the image was captured, how it was oriented, and the viewing angle of the camera used. For example, a respective position on the track has a particular associated image in which the position itself can be seen. Alternatively, the respective position may have an associated image captured from the respective position.
[0022] The first step of the method comprises transmitting a plurality of stored images. The plurality of images transmitted may be a subset of all stored images or may comprise the total number of stored images. The total number of stored images comprises all stored images for all practice tracks. If only a subset of all images is transmitted, the subset is selected from all images based on a predetermined selection criterion. For example, for each track only one image may be selected, in particular an image of the corresponding target position, or two images, in particular images of the corresponding starting position and target position. Furthermore, for example, the subset may be selected based on current position information associated with the vehicle and position information stored for each image, selecting only those images which are located close to the current position. "Close" means that the distance from the current position to the image position is less than or equal to a predetermined threshold.
[0023] The second step comprises determining a display comprising the transmitted images to represent the respective relevant trajectories. The display can be regarded as a graphical user interface, in which the transmitted images are arranged in a predetermined manner. For example, the display comprises images arranged in a list or in a table. It is also possible to schematically display the recorded paths from a bird's eye view. In addition to the images, the display can contain other elements, objects and / or information, in particular the numbering of the trajectories, marking of individual images as starting positions or target positions, graphical elements for subdividing and distinguishing the images of different trajectories, etc.
[0024] The display may include dynamic elements, such as an animation including multiple images sequentially displayed in time corresponding to tracks at particular locations in the display.
[0025] The display may be optimized for output on a display device of the user interface. This means in particular that the display has a width and / or height that can be displayed on the display device without additional scaling.
[0026] The display may include a plurality of different sub-displays, for example providing different zoom levels for an image. This allows a detailed display of the image, which is particularly advantageous for display devices having only a low resolution.
[0027] In particular, the display may have a greater height than can be displayed in a visual display on the display device, in which case the display may be scrolled, for example.
[0028] The third step comprises outputting the determined display to a display device of the user interface of the vehicle. In particular, the display is transmitted to the display device in the form of an image signal, preferably in the form of a digital image signal. The display device receives the image signal and outputs a corresponding image on its visual display. In other words, the display device displays the display. The display device comprises, for example, a screen, preferably a touch-sensitive screen.
[0029] A fourth step comprises receiving user input from the user interface to select at least one image to be included in the display. Since each image is associated with a trajectory, the selection of an image corresponds to the selection of the associated trajectory. By selecting an image, the user can thus select the trajectory they want to follow. This can be achieved in a particularly intuitive manner by means of a touch-sensitive screen which requires only that the user touch the displayed applicable image on the display device.
[0030] The user input may be received from the user interface, in particular in the form of coordinates relative to the display. The display is in particular two-dimensional, so that each position in the display is clearly defined by two coordinates; for example, these are pixel coordinates. The user input then contains, for example, a coordinate tuple or a coordinate range. From these coordinates, it can also be inferred which image of the display the user has selected.
[0031] The fifth step consists in starting the autonomous driving of the vehicle along the trajectory associated with the selected image. This means that the parking assistance system controls the vehicle in such a way that it drives along the selected trajectory to the target location and stops there.
[0032] In an exemplary embodiment, the display device is a component of a parking assistance system.
[0033] According to an embodiment of the method, the plurality of stored images of the respective trajectory comprises at least one image of the respective target position.
[0034] In this embodiment, the display for each trajectory is determined using, inter alia, an image of the target location.
[0035] The target position is in particular the position which is most easily remembered by the user on the corresponding trajectory, which is why this embodiment provides a particularly high level of reliability for the selection of the trajectory.
[0036] It should be noted that the image of the target location may include images recorded from the target location and images recorded from a location in front of the target location. In particular, if the target location itself is close to a wall or other obstacle, recording in front of the target location may be advantageous, because for example then the image from the target location may not provide a good overview.
[0037] In other embodiments, the plurality of stored images of the corresponding trajectory includes at least one image of the corresponding starting position and the corresponding target position.
[0038] According to a further embodiment of the method, the plurality of stored images for the respective trajectory comprises images compiled from a plurality of individual images.
[0039] For example, the compiled image may include a wide angle view compiled from multiple individual lenses, each lens having a smaller viewing angle. During compilation, various image processing steps may be performed, in particular equalizing the individual images, aligning the exposure and contrast of the individual images, etc.
[0040] According to a further embodiment of the method, the compiled image comprises a bird's eye view of the respective location.
[0041] A bird's eye view can be produced, in particular, by distorting the perspective of the image.
[0042] In an embodiment, the compiled image comprises a bird's-eye view of the entire track. This is achieved, for example, by first converting individual images at corresponding positions on the track into a bird's-eye view and then compiling these individual images into one image. In particular, the compilation is performed based on determining a panoramic image from a plurality of individual images.
[0043] According to another embodiment of the method, the method comprises:
[0044] Pass an object to represent the position of the vehicle in the target position on the corresponding trajectory, and
[0045] The display is determined by using the transmitted object so that the object in the display is overlaid at the target position in the image in which the target position is visible.
[0046] For example, an object representing a vehicle position comprises a projection of a vehicle outline at a target position on the ground. Thus, if an image of a target position shows the target position, the outline of the vehicle on the ground is inserted into the image. This can be achieved, for example, in the form of a darkened area or in the form of an outline inserted into the image.
[0047] In this context, an “object” is understood to mean, in particular, a graphical object that can be displayed in a graphical display, in particular an image.
[0048] According to another embodiment of the method, the method comprises:
[0049] A digital map of the environment that determines the corresponding trajectory, and
[0050] The display is determined by using the determined digital environment map such that the determined digital environment map is displayed together with the transmitted images for the respective trajectories, wherein the images are arranged in the display based on positions of the images in the digital environment map.
[0051] The digital environment map comprises in particular a digital display or representation of the vehicle environment, wherein for example detected objects, such as buildings or vegetation, are displayed in the map. The digital environment map can be determined based on captured environment sensor data of various environment sensors, such as cameras, lidar, radar and / or ultrasound. This can also be referred to as sensor fusion.
[0052] In particular, when a trajectory is taught in a training mode, a digital environment map for the corresponding trajectory is determined.
[0053] Arranging the images in the display based on their positions in the digital environment map is understood to mean in particular that each image in the digital environment map is arranged such that the relative positions of objects in the digital environment map and objects visible in the respective image correspond to the relative positions of these objects in reality. In simple terms, this means that in the display of the digital environment map, an image corresponding to a target position next to a house is also arranged next to the house.
[0054] It can also be said that the position of the corresponding image depends on and is derived from the position of the visible object in the image.
[0055] If multiple images are transmitted for a trajectory, all transmitted images can be arranged in the display in a manner that is commensurate with the digital environment map.
[0056] According to a further embodiment of the method, the plurality of taught trajectories comprises at least two trajectories whose paths are in the same area, wherein the display is determined such that the corresponding images of the at least two trajectories are displayed in a group.
[0057] For example, the paths of two trajectories are in the same area to be understood to mean that the distance between a first position on the first trajectory and a second position on the second trajectory is less than a predetermined threshold. The predetermined threshold is, for example, 50m, 30m, 10m or 5m. Preferably, the positions are in particular present in a world coordinate system. For example, the two positions that are shortest distances from each other can be used for the determination.
[0058] Tracks displayed in groups are understood to mean in particular that the tracks are included in the display in a manner that is graphically different from other tracks. It can also be provided that the tracks are displayed in a combined manner, for example displaying placeholders, such as symbols or icons, instead of images. Selecting a placeholder allows calling up another display that contains the tracks in the manner that they are represented by their respective images.
[0059] According to another embodiment of the method, the plurality of taught trajectories comprises at least two trajectories, the paths of which are in the same area, and comprises:
[0060] determining a common digital environment map for at least two trajectories, and
[0061] The display is determined by using the determined common digital environment map such that the determined common digital environment map is displayed together with the respective transmitted images for the respective trajectories, wherein the respective images are arranged in the display based on positions of the images in the digital environment map.
[0062] This embodiment is particularly advantageous if the paths of multiple trajectories are in the same area, since then the risk of confusion is also particularly high. The spatial display using a digital environment map makes it particularly intuitive for the user to select the correct trajectory in a particular situation, even if they have not taught the trajectory themselves.
[0063] According to another embodiment of the method, when teaching a trajectory in a training mode, a plurality of images associated with the respective trajectory are captured and stored.
[0064] According to another embodiment of the method, when following a track in the following mode, new images are captured by the on-board camera and associated with the corresponding track and stored.
[0065] This is advantageous to take into account changes in the environment in the image over time. The new image can replace all or part of the original image. This is also advantageous if the initial training runs were performed in conditions with poor visibility, such as in darkness, rain, snow and / or with dirty camera lenses (which is why the stored images are of poor quality).
[0066] According to another embodiment of the method, the respective plurality of images is captured by a plurality of cameras, the plurality of cameras comprising a front camera, a rear camera, a side camera on a left hand side of the vehicle and / or a side camera on a right hand side of the vehicle.
[0067] According to a further embodiment of the method, each stored image has associated position information relative to a world coordinate system, and the display is determined such that the respective image is arranged relative to the other images based on its associated position in the world coordinate system.
[0068] A world coordinate system is understood here as all coordinates, in particular those of different trajectories, referring to the same reference point (origin). One such world coordinate system is for example a coordinate system provided by a satellite navigation system, such as NAVSTAR GPS, GALILEO, GLONASS and / or BeiDou.
[0069] In this embodiment, the display may include, for example, a street map with an image for the track overlaid. This enables any user of the vehicle to quickly and intuitively determine where the practiced track occurs.
[0070] According to a second aspect, a computer program product is proposed comprising instructions which, when the program is executed by a computer, cause said computer to perform the method according to the first aspect.
[0071] The computer is in particular an integral part of the vehicle and forms, for example, an electronic control unit (ECU).
[0072] The computer program product, e.g. a computer program device, can be delivered or provided, e.g. as a storage medium, e.g. a memory card, a USB stick, a CD-ROM, a DVD, or in the form of a file that can be downloaded from a server in a network. This can be achieved, e.g. in a wireless communication network, by transmitting a corresponding file containing the computer program product or the computer program device.
[0073] According to a third aspect, a parking assistance system for a vehicle is provided. In a following mode, the parking assistance system is configured to autonomously follow a trajectory from a plurality of trajectories taught in a training mode, wherein the respective trajectory is defined by a series of positions and connects a starting position to a target position, and wherein the respective trajectory has associated and stored a plurality of images of the environment of the vehicle at the respective position captured by an onboard camera. The parking assistance system comprises:
[0074] a transmitting unit, for transmitting a plurality of stored images,
[0075] a determining unit for determining a display of the image including the transmission to represent the corresponding associated trajectory,
[0076] an output unit, configured to output the determined display to a display device of the user interface to display the transmitted image,
[0077] a receiving unit for receiving user input from a user interface to select at least one image to be included in the display, and
[0078] A control unit is used to initiate autonomous driving of the vehicle along a trajectory associated with the selected image.
[0079] The parking assistance system has the same advantages as explained for the method according to the first aspect. The embodiments and features described for the proposed method apply mutatis mutandis to the proposed parking assistance system.
[0080] The corresponding units of the parking assistance system can be implemented in hardware and / or software. In a hardware implementation, the corresponding units can be in the form of a computer or a microprocessor, for example. In a software implementation, the corresponding units can be in the form of a computer program product, in the form of a function, in the form of a routine, in the form of an algorithm, in the form of a portion of a program code or in the form of an executable object. In addition, each unit mentioned here can also be in the form of a part of a higher-level control system of the vehicle, such as a central electronic control unit and / or a control unit (ECU: Electronic Control Unit).
[0081] The parking assistance system is particularly configured to carry out the method according to the first aspect.
[0082] According to a fourth aspect, a vehicle is proposed, having a plurality of cameras for capturing respective images of the vehicle environment, having a parking assistance system according to the third aspect, and having a user interface comprising a display device.
[0083] For example, the vehicle is a passenger car or a truck. The vehicle preferably includes a plurality of sensor units configured to detect the driving state of the vehicle and sense the environment of the vehicle. Examples of such sensor units of the vehicle are image recording devices such as cameras, radars (radio detection and ranging) or lidars (light detection and ranging), ultrasonic sensors, position sensors, wheel angle sensors and / or wheel speed sensors. The sensor units are respectively configured to output sensor signals to, for example, a parking assistance system, which performs semi-autonomous or fully autonomous control of the vehicle based on the acquired sensor signals.
[0084] The plurality of cameras include a front camera, a rear camera, a side camera on the left side of the vehicle, and / or a side camera on the right side of the vehicle.
[0085] Further possible implementations of the present invention also include combinations of features or embodiments not explicitly mentioned above or below in relation to the exemplary embodiments. Those skilled in the art will also add various aspects as improvements or supplements to various basic forms of the present invention.
[0086] Further advantageous configurations and aspects of the invention are subject matter of the dependent claims and also of exemplary embodiments of the invention described below.The invention is explained in more detail below based on preferred embodiments with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0087] Figure 1 A schematic aerial view of a vehicle is shown;
[0088] Figure 2 shows a schematic diagram of a display including two images;
[0089] Figure 3 Another schematic diagram showing a display including two images;
[0090] Figure 4 shows a schematic diagram of a display including a digital environment map;
[0091] Figure 5 Another schematic diagram showing a display including a digital environment map;
[0092] Figure 6 A schematic block diagram showing an exemplary embodiment of a parking assist system; and
[0093] Figure 7 A schematic block diagram of an exemplary embodiment of a method for operating a parking assistance system is shown.
[0094] In the figures, identical or functionally identical elements are provided with the same reference symbols unless stated otherwise. DETAILED DESCRIPTION
[0095] Figure 1 A schematic bird's-eye view of a vehicle 100 is shown. The vehicle 100 is, for example, a car arranged in an environment 200. The car 100 has a parking assistance system 110, which is, for example, in the form of a control unit. In addition, a plurality of environmental sensor devices 120, 130 are arranged on the car 100, these are, by way of example, optical sensors 120 and ultrasonic sensors 130. The optical sensor 120 comprises, for example, a visual camera, a radar and / or a lidar. The optical sensor 120 can in particular respectively acquire images of corresponding areas from the environment 200 of the car 100 and output the images as optical sensor signals. The ultrasonic sensor 130 is configured to measure the distance to an object arranged in the environment 200 and output a corresponding sensor signal. With the aid of the sensor signals acquired from the sensors 120, 130, the parking assistance system 110 is able to drive the car 100 semi-autonomously or fully autonomously. In addition Figure 1In addition to the optical sensor 120 and the ultrasonic sensor 130 shown in FIG. 1 , provision may be made for the vehicle 100 to have various other sensor devices 120, 130. Examples of these are wheel speed sensors, steering angle sensors, position sensors, microphones, acceleration sensors, antennas with coupled receivers for receiving electromagnetically transmittable data signals, etc.
[0096] The vehicle further has a user interface 105 to which the parking assistance system 110 is communicatively coupled. This means that the parking assistance system 110 and the user interface 105 are configured to exchange data, in particular in the form of analog or digital data signals. The user interface 105 comprises a display device (not shown) configured to display images IMG1-IMG6 (see Figure 2-5 ) and display DSP (see Figure 2-5 ). In addition, the user interface 105 includes input means (not shown) with which a user of the vehicle 100 can input to the systems of the vehicle 100, in particular to the parking assistance system 110. The input means may include buttons, switches, rotary controllers, touch-sensitive elements, voice detection, gesture detection, etc. The respective input may in particular relate to an element displayed by the display device at a particular time.
[0097] In the following mode, the parking assist system 110 is configured to autonomously follow a trajectory TR1 - TR4 of the plurality of trajectories TR1 - TR4 taught in the training mode (see Figure 4 or 5). The corresponding track TR1-TR4 consists of a series of positions P1-P6 (see Figure 4 ) and connects the starting position P1 to the target position P6. Each trajectory TR1-TR4 has an associated and stored plurality of images IMG1-IMG6 of the environment 200 of the vehicle 100 captured by the onboard camera 120. When the vehicle 100 is at the respective position P1-P6 on the trajectory TR1-TR4, the respective image IMG1-IMG6 is recorded, so that it shows the respective details from the environment 200. In an embodiment, the respective image IMG1-IMG6 may contain image information from the plurality of images IMG1-IMG6, in particular, it may be a merged image. The parking assistance system 110 is designed, for example, as Figure 6 , and is configured to perform a reference Figure 7 Method of explanation.
[0098] Figure 2 A schematic diagram of a display DSP comprising two images IMG1, IMG2 is shown. The display DSP is composed in particular of a determination unit 112 (see Figure 6 ) based on the transmission unit 111 (see Figure 6 ) is determined by the multiple images IMG1-IMG6 transmitted.
[0099] The two images IMG1, IMG2 are associated with the taught trajectory TR1-TR4 along which the user has driven (see Figure 4 or Figure 5 ), for example, through Figure 1 The vehicle 100, the parking assistance system 110 (see Figure 1 or Figure 6 ) have recorded the tracks TR1-TR4. In this example, the images IMG1, IMG2 show the tracks TR1-TR4 from the starting position P1 (see Figure 4 ) sees the environment 200 of the vehicle 100 (see Figure 1 ) view (IMG1 with title "Start:"), and from the target position P6 (see Figure 4 ), the images IMG1, IMG2 being captured, for example, by the front camera 120. It should be noted that instead of the image IMG2 seen from the target position P6, an image seen from a position in front of the target position P6 is advantageously used to display the target position P6, since such an image provides a better overview and in particular includes the target position P6 itself (see also in this respect ). Figure 3 ).
[0100] In the first image IMG1, a view of a house with a fence or wall in the background can be seen, as well as trees and bushes next to the house. In the second image IMG2, only the bushes and part of the house wall and fence can be seen. In this example, the user has used the vehicle 100 to practice, in particular Figure 4 , so that parking assistance system 110 is now able to autonomously drive vehicle 100 from starting position P1 to target position P6.
[0101] The display DSP receives the output unit 114 (see FIG. Figure 6 ) is transmitted to the user interface 105 (see Figure 1 ), and the user interface 105 displays the display DSP on the display device. This enables the user to intuitively determine the trajectory TR1 associated with the images IMG1, IMG2, and they can select said trajectory to follow. It should be noted that in embodiments, the display DSP contains only one image for the trajectory TR1, or contains more than two images for the trajectory TR1 and / or contains compiled images for the trajectory TR1 (see also in this respect Figure 4 ).
[0102] If the parking assistance system 110 comprises a plurality of practiced and stored trajectories TR1 - TR4 , a corresponding display DSP may be determined and output for each further trajectory TR1 - TR4 , so that the user can select a desired trajectory TR1 - TR4 based on the respective images IMG1 , IMG2 .
[0103] It should be noted that the display DSP may contain corresponding images IMG1 , IMG2 for a plurality of tracks TR1 - TR4 , which are arranged simultaneously adjacent to or below each other, for example in the display DSP (not shown).
[0104] For example, when the user selects the trajectory TR1 , the user interface 105 sends a corresponding user input to the parking assistance system 110 , which initiates autonomous following of the selected trajectory TR1 .
[0105] Figure 3 Another schematic diagram of a display DSP comprising two images IMG1, IMG2 is shown. The display DSP is composed in particular of a determination unit 112 (see Figure 6 ) based on the transmission unit 111 (see Figure 6 ) is determined by the multiple images IMG1-IMG6 transmitted.
[0106] Figure 3 The display corresponds almost to Figure 2 , with the difference that the images IMG1, IMG2 additionally show an object OL. For example, the object OL is a geometric figure (a rectangle depicted with a perspective deformation) inserted or overlaid on the respective image IMG1, IMG2. The object OL is inserted into the image, in particular at the target position P6 corresponding to the trajectory TR1 (see Figure 4 ), that is, the position when the vehicle 100 has traveled along the trajectory TR1.
[0107] In this example, the second image IMG2 is recorded from a position in front of the target position P6, that is, for example, the position P5 (see Figure 4 ), which is why the target position P6 is included in the image IMG2.
[0108] Due to the additional overlapping of target positions P6 in the displayed images IMG1 , IMG2 , the user can associate the trajectory TR1 even better and in particular can distinguish trajectories which end, for example, in adjacent parking spaces and for which the displayed images are therefore almost identical to one another.
[0109] It should be noted that as Figure 3 Replacement of the display DSP in , for example the object OL is overlaid on only one of the images IMG1 , IMG2 .
[0110] Figure 4 1 shows a schematic diagram of a display DSP, which includes a digital environment map DMAP and a plurality of images IMG1-IMG6 associated with a track TR1. For example, the environment displayed in the digital environment map DMAP is different from that in the Figure 2 and 3The environment shown in the images IMG1, IMG2 is the same. In the digital environment map DMAP, the detected objects OB1-OB4 are displayed according to the environment. In this example, they are the house OB1, the bushes OB2, the fence OB3 and the tree OB4. When practicing the track TR1, the digital environment map DMAP is preferably acquired and stored.
[0111] In this example, the trajectory TR1 includes six positions P1-P6, where position P1 is the starting position and position P6 is the target position. For example, the starting position P1 is located in front of the house (shown here as object OB1), and the target position P6 is located next to the house OB1, on the side. At each position P1-P6, at least one environment image IMG1-IMG6 is captured, associated with the trajectory TR1, and stored during training. The images IMG1-IMG6 are preferably compiled images showing a bird's-eye view of the corresponding position. This is especially possible when the vehicle 100 includes multiple cameras 120, and the camera can capture a total of 360° of viewing angle around the vehicle 100. However, even if the vehicle 100 has only one front camera 120, the image from the front camera 120 can be converted into a top view by appropriately distorting the perspective of the image. It should be noted that when the vehicle 100 is in the corresponding position, the corresponding image IMG1-IMG6 is not necessarily captured. On the contrary, for example, when the vehicle 100 is located at position P2, the image IMG3 is captured by the front camera, and necessary corrections are made to other images / positions.
[0112] In this example, the display comprises a digital environment map DMAP and images IMG1-IMG6 associated with the track TR1 and used to display the track TR1. The images IMG1-IMG6 are arranged in particular in a way corresponding to reality in the digital environment map of the display DSP. The overlapping display of the images IMG1-IMG6 together with the digital environment map DMAP allows the user to determine the track TR, in particular the target position P6 on the track TR1, even better.
[0113] If images IMG1-IMG6 are sufficiently close and / or overlapping, they may also be merged to produce a single image and stored.
[0114] In an embodiment, the track TR1 can also be inserted into the display DSP in the form of an object OL (see Figure 3 ).
[0115] Figure 5 Another schematic diagram of a display DSP including a digital environment map DMAP is shown. For example, this involves the same Figure 41. However, in this example, there are multiple tracks TR1-TR4 in the display area of the environment. For example, different tracks TR1-TR4 are practiced by different users of the vehicle 100. In this example, all tracks TR1-TR4 start from the starting position shown in the image IMG1, but this is not necessary.
[0116] The respective images IMG2-IMG5 of the respective target positions are arranged in the display DSP in a manner corresponding to reality with reference to the digital environment map DMAP, in particular the objects OB1-OB4, which makes it relatively easy for the user to deduce the position of the vehicle in reality from the displayed target positions.
[0117] For example, the user may select one of the images IMG2-IMG5 in the display DSP in order to select the associated track TR1-TR4.
[0118] In the embodiment, a corresponding object OL is provided for each track TR1-TR4 (see Figure 3 ) can also be inserted into the display DSP.
[0119] Figure 6 It is shown that it can be used for example Figure 1 Schematic block diagram of an exemplary embodiment of a parking assistance system 110 for a vehicle 100. In the following mode, the parking assistance system 110 is configured to autonomously follow a trajectory TR1-TR4 of a plurality of trajectories TR1-TR4 taught in the training mode (see Figure 4 or 5). The corresponding track TR1-TR4 consists of a series of positions P1-P6 (see Figure 4 ) and connects the starting position P1 to the target position P6. Each of the tracks TR1-TR4 has a camera 120 (see Figure 1 ) captured environment 200 of vehicle 100 (see Figure 1 ) of the associated and stored multiple images IMG1-IMG6 (see Figure 2-6 ). When the vehicle 100 is at a corresponding position P1-P6 on the trajectory TR1-TR4, the corresponding image IMG1-IMG6 is recorded, so that it shows the corresponding details from the environment 200. The parking assistance system 110 includes a transmission unit 111 for transmitting a plurality of stored images IMG, a determination unit 112 for determining a display DSP, the display DSP including a display representing the corresponding relevant trajectory TR1-TR4 (see Figure 4 or 5) for outputting the determined display DSP to the user interface 105 (see Figure 1), in particular in the form of a digital image or data signal, a receiving unit 116 for receiving a user input SIG from the user interface 105 for selecting at least one image IMG contained in the display DSP, and a control unit 118 for starting the autonomous driving of the vehicle 100 along the trajectory TR1-TR4 associated with the selected image IMG. In this example, the control unit 118 outputs a corresponding control signal CTR.
[0120] The respective units 111-118 of the parking assistance system 110 may be implemented in hardware and / or in software. In a hardware implementation, the respective units 111-118 may be in the form of a computer or a microprocessor, for example. In a software implementation, the respective units 111-118 may be in the form of a computer program product, a function, a routine, an algorithm, a portion of a program code, or an executable object. Furthermore, each of the units 111-118 mentioned here may also be in the form of a part of a higher-level control system of the vehicle, such as a central electronic control unit and / or a control unit (ECU: Electronic Control Unit).
[0121] In an embodiment, the display device and / or user interface 105 is part of a parking assist system 110 (not shown).
[0122] Figure 7 A schematic block diagram shows an exemplary embodiment of a method for operating a parking assistance system 110 of a vehicle 100, in particular Figure 6 The parking assistance system 110 and Figure 1 In the following mode, the parking assistance system 110 is configured to autonomously follow the trajectory TR1-TR4 of the plurality of trajectories TR1-TR4 taught in the training mode (see Figure 4 or 5), wherein the corresponding track TR1-TR4 consists of a series of positions P1-P6 (see Figure 4 ) and connects the starting position P1 to the target position P6, and wherein the corresponding tracks TR1-TR4 have a vehicle-mounted camera 120 (see Figure 1 ) The environment 200 of the vehicle 100 captured at the corresponding positions P1-P6 (see Figure 1 ) of the associated and stored multiple images IMG1-IMG6 (see Figure 2-6 ). The first step S1 comprises transmitting a plurality of stored images IMG1-IMG6. The second step S2 comprises determining a display DSP (see Figure 2-5 ), which includes the transmitted images IMG1-IMG6 to represent the corresponding related tracks TR1-TR4, and the third step S3 includes outputting the determined display DSP to the user interface 105 of the vehicle 100 (see Figure 1) of the display device, the fourth step S4 comprises receiving a user input SIG (see Figure 6 ) to select at least one image IMG1-IMG6 contained in the display DSP, and a fifth step S5 comprises starting autonomous driving of the vehicle 100 along the trajectory TR1-TR4 associated with the selected image IMG1-IMG6.
[0123] Although the present invention has been described based on exemplary embodiments, it can be modified in many ways.
[0124] Reference numerals list
[0125] 100 Vehicles
[0126] 105 User Interface
[0127] 110 Parking assistance system
[0128] 111 Transmission Unit
[0129] 112 Determine unit
[0130] 114 Output Units
[0131] 116 Receiving Unit
[0132] 118 Control Unit
[0133] 120 Environmental sensor devices
[0134] 130 Environmental sensor devices
[0135] 200 Environment
[0136] CTR control signal
[0137] DMAP Digital Environmental Map
[0138] DSP Display
[0139] IMG1 Image
[0140] IMG2 Image
[0141] IMG3 Image
[0142] IMG4 Image
[0143] IMG5 Image
[0144] IMG6 Image
[0145] OB1 Object
[0146] OB2 Object
[0147] OB3 Object
[0148] OB4 Object
[0149] OL Object
[0150] P1 Position
[0151] P2 Position
[0152] P3 Location
[0153] P4 Position
[0154] P5 Position
[0155] P6 Position
[0156] S1 Method steps
[0157] S2 Method Steps
[0158] S3 Method Steps
[0159] S4 Method steps
[0160] S5 Method steps
[0161] SIG User Input
[0162] TR1 track
[0163] TR2 Track
[0164] TR3 Track
[0165] TR4 Track
Claims
1. A method for operating a parking assistance system (110) of a vehicle (100), wherein: In a following mode, the parking assistance system (110) is configured to autonomously follow a trajectory (TR1-TR4) of a plurality of trajectories (TR1-TR4) taught in a training mode, wherein the respective trajectory (TR1-TR4) is defined by a series of positions (P1-P6) and connects a starting position (P1) to a target position (P6), and wherein the respective trajectory (TR1-TR4) has associated and stored a plurality of images (IMG1-IMG6) of an environment (200) of the vehicle (100) at the respective positions (P1-P6) captured by an onboard camera (120), the method comprising the following steps: Transfer (S1) a large number of stored images (IMG1-IMG6), determining (S2) a display (DSP) comprising the transmitted images (IMG1-IMG6) to represent the corresponding associated tracks (TR1-TR4), outputting (S3) the determined display (DSP) to a display device of a user interface (105) of a vehicle (100), receiving (S4) a user input (SIG) from a user interface (105) to select at least one image (IMG1-IMG6) to be included in a display (DSP), and Autonomous driving of the vehicle (100) along the trajectory (TR1-TR4) associated with the selected image (IMG1-IMG6) is initiated (S5).
2. The method according to claim 1, characterized in that The plurality of stored images (IMG1-IMG6) for the respective trajectories (TR1-TR6) include at least one image of the respective target position (P6).
3. The method according to claim 1 or 2, characterized in that The plurality of stored images (IMG1-IMG6) for respective tracks (TR1-TR4) comprise images compiled from a plurality of individual images.
4. The method according to claim 3, characterized in that The compiled images include a bird's eye view of the corresponding locations (P1-P6).
5. The method according to any one of the preceding claims, characterized in that: transmitting an object (OL) to represent the position of the vehicle (100) in a target position (P6) on the corresponding trajectory (TR1-TR4), and The display (DSP) is determined by using the transmitted object (OL) such that the object (OL) in the display (DSP) is overlaid at the target position (P6) in the image (IMG1-IMG6) in which the target position (P6) is visible.
6. A method according to any one of the preceding claims, characterised in that: determining a digital environment map (DMAP) for the corresponding track (TR1-TR4), and The display (DSP) is determined by using a determined digital environment map (DMAP) so that the determined digital environment map (DMAP) is displayed together with the transmitted images (IMG1-IMG6) for the corresponding tracks (TR1-TR4), wherein the images (IMG1-IMG6) are arranged in the display (DSP) based on the positions of the images in the digital environment map (DMAP).
7. A method according to any one of the preceding claims, characterised in that The plurality of taught trajectories (TR1-TR4) comprises at least two trajectories whose paths are in the same area, wherein the display (DSP) is determined such that the corresponding images (IMG1-IMG6) of the at least two trajectories are displayed in a group.
8. A method according to any one of the preceding claims, characterised in that A plurality of taught trajectories (TR1-TR4) includes at least two trajectories whose paths are in the same area, and includes: determining a common digital environment map (DMAP) for the at least two trajectories, and The display (DSP) is determined by using a determined common digital environment map (DMAP) such that the determined common digital environment map (DMAP) is displayed together with the respective transmitted images (IMG1-IMG6) for the respective trajectories (TR1-TR4), wherein the respective images (IMG1-IMG6) are arranged in the display based on the positions of the images in the digital environment map (DMAP).
9. A method according to any one of the preceding claims, characterised in that When a trajectory (TR1-TR4) is taught in the training mode, a plurality of images (IMG1-IMG6) associated with the corresponding trajectory (TR1-TR4) are captured and stored.
10. The method according to any one of the preceding claims, characterized in that When following a track (TR1-TR4) in the following mode, new images are captured by the onboard camera (120) and associated with the corresponding track (TR1-TR4) and stored.
11. The method according to any one of the preceding claims, characterized in that The respective plurality of images (IMG1-IMG6) are captured by a plurality of cameras (120), including a front camera, a rear camera, a side camera on the left side of the vehicle, and / or a side camera on the right side of the vehicle.
12. The method according to any one of the preceding claims, characterized in that Each stored image (IMG1-IMG6) has associated position information relative to a world coordinate system, and wherein the display is determined such that the respective image is arranged relative to the other images based on its associated position in the world coordinate system.
13. A computer program product comprising instructions, which, when executed by a computer, causes the computer to perform the method according to any one of claims 1 to 12.
14. A parking assistance system for a vehicle (100), wherein: In a following mode, the parking assistance system (110) is configured to autonomously follow a trajectory (TR1-TR4) of a plurality of trajectories (TR1-TR4) taught in a training mode, wherein the respective trajectory (TR1-TR4) is defined by a series of positions (P1-P6) and connects a starting position (P1) to a target position (P6), and wherein the respective trajectory (TR1-TR4) has associated and stored a plurality of images (IMG1-IMG6) of an environment (200) of the vehicle (100) at the respective positions (P1-P6) captured by an onboard camera (120), the parking assistance system (110) comprising: A transmission unit (111) for transmitting a plurality of stored images (IMG1-IMG6), a determination unit (112) for determining a display (DSP) comprising the transmitted images (IMG1-IMG6) to represent the corresponding associated tracks (TR1-TR4), an output unit (114) for outputting the determined display (DSP) to a display device of a user interface (105) of the vehicle (100), a receiving unit (116) for receiving a user input (SIG) from a user interface (105) for selecting at least one image (IMG1-IMG6) to be included in a display (DSP), and A control unit (118) is provided for initiating autonomous driving of the vehicle (100) along a trajectory (TR1-TR4) associated with the selected image (IMG1-IMG6).
15. A vehicle (100) having a plurality of cameras (120) for capturing respective images (IMG1-IMG6) of an environment (200) of the vehicle (100), having a parking assistance system (110) as claimed in claim 14, and having a user interface (105) including a display device.