Methods, computer program products, parking assistance systems and vehicles
The method addresses the challenge of multiple users selecting incorrect trajectories in parking assistance systems by associating vehicle environments with images for intuitive selection, enhancing reliability and reducing resource consumption.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- VALEO SCHALTER & SENSOREN GMBH
- Filing Date
- 2023-04-27
- Publication Date
- 2026-06-08
AI Technical Summary
Existing parking assistance systems struggle when multiple users use a vehicle, as they fail to intuitively distinguish between stored trajectories, leading to incorrect trajectory selection and increased resource consumption.
A method where each trajectory is defined by a series of positions and associated with images of the vehicle's environment, displayed to the user for intuitive selection, using onboard cameras and sensors for self-localization and obstacle detection, allowing autonomous driving along the selected trajectory.
Enables users to reliably select the correct trajectory without metadata, reducing resource consumption and user frustration by providing an intuitive interface for trajectory selection.
Smart Images
Figure 0007871414000001 
Figure 0007871414000002 
Figure 0007871414000003
Abstract
Description
Technical Field
[0001] The present invention relates to a method of operating a parking assistance system, a computer program product, a parking assistance system, and a vehicle having the parking assistance system.
Background Art
[0002] Parking assistance systems that can be trained to follow a specific trajectory are known. This is particularly useful in situations that occur frequently, such as parking a vehicle in a garage or parking a vehicle in a predetermined parking space. The driver only needs to drive the vehicle close to the starting point of the trajectory, such as a driveway entrance. Since the parking assistance system automatically follows the trained trajectory, the driver is relieved of the burden.
[0003] When a vehicle is used for a long period of time and / or there are multiple people using the vehicle, a large number of different practice trajectories can be accumulated over time. This causes the user to lose track of the trajectories considering a large number, and makes it unclear which trajectory is the required trajectory in a specific situation. In particular, when different people use the vehicle alternately, the later user does not know which trajectory stored by the previous user to target and / or what route the trajectory follows. Furthermore, when there are multiple trajectories having paths in the same area, for example, in the user's land or nearby land, it is difficult for the user to distinguish them from each other.
[0004] One option is to assign keywords or descriptions to different trajectories so that users can distinguish them from one another. However, this is complex, not very intuitive, and does not solve the problem when there are multiple different users of the vehicle. If users cannot uniquely associate with trajectories, firstly, users will select the wrong trajectory to follow in certain situations, resulting in the vehicle not being guided to the desired parking position. In such cases, users will be expected to park the vehicle manually or try other trajectories to select the correct one. Ultimately, this situation leads to the unnecessary consumption of resources such as fuel and time, and generates user frustration. Such a lack of relevance within trained trajectories can lead to a decrease in the user's overall willingness to use the parking assist system, thereby diminishing the benefits of the parking assist system.
[0005] German Patent Application Publication No. 102015010746 discloses a method for estimating the self-position of a vehicle. This method includes an image capture unit that includes the ground within the vehicle environment, records images along a first trajectory while the vehicle is traveling, compares these images with position-related recorded images, and this comparison is used as a basis for determining the vehicle's current position and / or current orientation.
[0006] German Patent Application Publication No. 102013015349 discloses a method for maneuvering a vehicle to approach a parking space in an invisible / off-road parking lot, the method comprising collecting environmental data relating to the vehicle, and as the vehicle approaches a parking space in the parking area, it is identified whether the parking space is a home parking space or the parking lot is a home parking lot, and the environmental data or driving data collected as the vehicle approaches the home parking space or home parking lot is stored or updated. [Overview of the project] [Problems that the invention aims to solve]
[0007] One objective of the present invention is to improve the parking assistance system in response to this problem. [Means for solving the problem]
[0008] According to the first embodiment, a vehicle parking assist system is proposed. In follow mode, the vehicle follow system autonomously follows a trajectory from a number of trajectories taught in training mode, each trajectory being defined by a series of positions connecting a starting position and a target position, and each trajectory is stored in association with images of the vehicle's environment at each position captured by an onboard camera. This delicious, Distributing the stored images, This involves determining the display, including the distributed images, and representing the associated trajectory for each. The determined display is output to the vehicle's user interface display device, The user interface receives user input to select at least one image to be included in the display, The vehicle will begin autonomous driving along a trajectory associated with the selected image, Includes.
[0009] The advantage of this method is that each user can intuitively recognize from the image what the trajectory is, what target location the trajectory leads to, and / or what path the trajectory follows. This is especially applicable when multiple users take turns using the vehicle and the vehicle itself has not learned each user's trajectory. As a result, users can select the correct trajectory in each situation and successfully complete the automated parking maneuver with greater reliability. Selecting the wrong trajectory is avoided, and the effort associated with parking the vehicle in an undesirable parking position is also avoided. In addition, this method eliminates the need for users to assign metadata such as titles, keywords, and / or descriptions to each trajectory. Furthermore, it reduces memory requirements because there is no need to store such metadata to identify each trajectory.
[0010] The parking system is configured to autonomously follow its respective trajectory, and in this specification, this can be understood as the parking assistance system autonomously controlling the vehicle. This is achieved, in particular, using camera images, LiDAR data, and / or radar data relating to the vehicle's environment. Based on these images or data, the parking assistance system can be configured to perform, for example, self-localization and orientation (also known as SLAM: Simultaneous Localization and Mapping) and to detect obstacles in the environment. A vehicle having a parking assistance system can also be called an autonomous vehicle. Autonomous control may be stipulated to be performed under the supervision of the user, and the user does not necessarily need to be inside the vehicle.
[0011] The level of automation of a vehicle is, for example, level 4 or 5 on the SAE classification system. The SAE classification system was published in 2014 by SAE International, an automotive standardization body, as J3016 "Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems." It is based on six different levels of automation, taking into account the degree of system intervention required and the degree of driver attention required. The SAE automation levels range from level 0, which corresponds to a fully manual system, to levels 1 and 2, which are driver-assistant systems, semi-autonomous driving (levels 3 and 4), and fully automated systems (level 5), where a driver is no longer required. An autonomous vehicle is a vehicle that can sense its environment and navigate without human input. It corresponds specifically to SAE automation level 5.
[0012] In training mode, when teaching or learning each trajectory, it is preferable for the user to manually control the vehicle. The user can also be assisted by the parking assist system, but the vehicle will not be driven autonomously by the parking assist system. The vehicle user retains complete control of the vehicle. The taught trajectory is unknown at this point and will only be determined after the training run has been performed.
[0013] In a preferred embodiment, the parking assist system is configured to track each trajectory using images of the vehicle's environment captured by an onboard camera. This is understood to mean that the parking assist system, in particular, uses the captured images of orientation in the environment to execute a VSLAM (Visual Simultaneous Localization And Mapping) algorithm, specifically to determine the vehicle's position relative to the position of each trajectory.
[0014] Each trajectory is defined in particular by a set of positions. Specifically, a trajectory includes one or more reference positions. The reference positions correspond to the positions the vehicle has in training mode in order to determine environmental features, such as engineering features that are received and processed and also referenced as directional features, in particular images, LiDAR data, and / or radar data related to the environment. Each reference position is associated with a group of determined features. This means that the reference positions are defined while the trajectory is being taught. For example, each reference position is defined by two coordinates in a two-dimensional coordinate system and the direction of the vehicle at this position. For example, each trajectory starts at a starting position, which is specifically a reference position, and ends at a target position, which is also a reference position.
[0015] This track may include one or more tracks. Multiple tracks may be located within the same region or area, for example, in adjacent or different regions or areas. For example, a user may specify multiple tracks on private property, as well as other tracks such as those at a workplace, station, or shopping center.
[0016] Each associated image can include one, two, or more images for each track. These images are captured and stored, particularly while the track is being taught. Each image provides details, especially from the vehicle's environment, and these details depend on the vehicle's position, orientation, and the field of view of the camera used at the time the image was captured. For example, each position on the track may have a specific associated image that shows the position itself. Alternatively, each position may have associated images captured from that position.
[0017] The first step of the method is to distribute the stored images. The distributed images may be a subset of all stored images, or they may include all stored images. All stored images include all stored images for all learned trajectories. If only a subset of all images is distributed, this subset is selected from all images based on predetermined selection criteria. For example, one image, in particular the image for each target position, or two images, in particular the images for each starting position and target position, may be selected for each trajectory. Furthermore, the subset may be selected, for example, based on the current position information related to the vehicle and the position information of each image, and only images of positions close to the current position are selected. "Close" means that the distance from the current position to the position of the image is less than or equal to a predetermined threshold.
[0018] The second step involves determining a display that includes the delivered images and representing each associated trajectory. This display can be considered a graphical user interface in which the delivered images are arranged in a predetermined manner. For example, the display may include images arranged in a list or table. It may also include an overview of the recorded trajectory. In addition to images, the display may include other elements such as objects and / or information, particularly trajectory numbers, markings on individual images as start or target positions, and graphic elements that subdivide and distinguish images of different trajectories from one another.
[0019] The display can include dynamic elements, such as animations that contain multiple images for each trajectory displayed at a specific location within the display.
[0020] The display may be optimized for output on a user interface display device. This means, in particular, that the display has a width and / or height that can be displayed on the display device without additional scaling.
[0021] The display can include multiple different sub-displays, such as providing different zoom levels for an image. This allows for a more detailed view of the image and is particularly advantageous for displays with low resolution.
[0022] The display may have a height greater than what can be displayed in the visual display on the display device, in which case, for example, it may be scrollable through the display.
[0023] The third step includes outputting a determined display to a display device of the vehicle's user interface. In particular, the display is transmitted to the display device in the form of an image signal, preferably a digital image signal. The display device receives the image signal and outputs an image corresponding to its visual display. In other words, the display device displays the display. The display device includes, for example, a screen, preferably a touch sensor screen.
[0024] The fourth step includes receiving a user input for selecting at least one image included in the display from a user interface. Since each image is associated with a trajectory, the selection of an image corresponds to selecting the associated trajectory. By selecting an image, the user can select the trajectory to follow. This is particularly intuitive by means of a touch sensor screen that requires the user to simply touch the appropriate image of the display on the display device.
[0025] The user input is received in the form of coordinates based on the user interface, particularly the display. Since the display is particularly two-dimensional, each position during the display can be clearly defined by two coordinates, for example, the coordinates of pixels. The user input includes, for example, a coordinate tuple or a range of coordinates. From these coordinates, it can be determined whether the user has selected any of the images of the display.
[0026] The fifth step includes starting the autonomous driving of the vehicle along the trajectory associated with the selected image. This means that the parking assistance system controls the vehicle to move along the selected trajectory to the target position and stop there.
[0027] In an embodiment, the display device is a component of the parking assistance system.
[0028] According to one embodiment of the method, the stored image for each trajectory includes at least one image of each target position.
[0029] In this embodiment, the display of each trajectory is determined particularly using the image of the target position.
[0030] The target position is particularly the easiest position of each trajectory for the user to remember, which is the reason why this embodiment has a particularly high level of reliability for the selection of the trajectory.
[0031] Furthermore, the image of the target position may include both an image recorded from the target position and an image recorded from a position in front of the target position. Recording from in front of the target position may be advantageous, especially if the target position itself is located close to a wall or other obstacle, for example, if the image from the target position does not provide a good view.
[0032] In other embodiments, the stored images for each trajectory include at least one image of the respective starting position and the respective target position.
[0033] In another embodiment of the method, the stored image for each trajectory includes an image edited from a plurality of individual images.
[0034] For example, an edited image may include a large wide-angle view edited from multiple individual images, each with a smaller field of view. During editing, various image processing steps may be performed, particularly to equalize the individual images and to match the exposure and contrast of each image.
[0035] In another embodiment of the method, the edited images may include bird's-eye views of each location.
[0036] This bird's-eye view may be generated, in particular, by distorting the perspective and depth of the image.
[0037] In the embodiment, the edited image includes a bird's-eye view of the entire orbit. For example, this is achieved by first converting individual images of each position in the orbit into bird's-eye views, and then editing these individual images into a single image. The editing is performed, in particular, by determining a panoramic image from multiple individual images.
[0038] According to another embodiment of the method, the method is To deliver an object representing the vehicle's position at the target location for each track, The display is determined using the delivered objects, and the objects within the display are overlaid on the target position of the image to visualize the target position. Includes.
[0039] For example, an object representing the position of a vehicle includes a projection of the vehicle's outline at the target location on the ground. Therefore, if an image of the target location indicates the target location, the vehicle's outline on the ground is inserted into this image. This can be achieved, for example, by the shape of a darkened area or by the outline being inserted into the image.
[0040] In this context, "object" is understood to mean a graphic object, particularly an image, that can be displayed on a graphical display.
[0041] According to another embodiment of the method, this method is To determine a digital environment map for each orbit, The display is determined using the determined digital environment map, the determined digital display map is displayed along with the distributed images for each orbit, and the images are positioned in the display based on their positions within this digital environment map. Includes.
[0042] A digital environmental map, in particular, includes a display or representation of the vehicle's environment, where detected objects such as buildings or vegetation are shown on the map. The digital environmental map can be determined based on captured environmental sensor data from various environmental sensors, such as cameras, LiDAR, radar, and / or ultrasound. This is also known as sensor fusion.
[0043] The digital environment map for each trajectory is determined, in particular, while the trajectory is being taught in training mode.
[0044] Images positioned within the display based on their location in the digital environment map are understood to mean that each image in the digital environment map is positioned such that the relative position of the object in the digital environment map and the position of the object visible in each image correspond to the relative position of the real-world object. In short, this means that an image corresponding to a target location next to a house will also be positioned next to the house in the display of the digital environment map.
[0045] Furthermore, it can be said that the position of each image depends on the position of the objects visible in the image, and is derived from that.
[0046] If multiple images are delivered for a trajectory, all delivered images may be placed within the display in a manner consistent with the digital environment map.
[0047] According to another embodiment, the method is such that the taught trajectory includes at least two trajectories whose paths lie in the same area, and the display is determined such that the images of each of the at least two trajectories are displayed within the group.
[0048] For example, the fact that the paths of two orbits lie in the same area is understood to mean that the distance between the first position in the first orbit and the second position in the second orbit is less than a predetermined threshold. The predetermined threshold is, for example, 50m, 30m, 10m, or 5m. Preferably, the positions exist, particularly in a world coordinate system. For example, two positions that are the shortest distance from each other can be used for this determination.
[0049] The grouping of orbits is understood to mean that they are included in the display in a way that is graphically distinct from other orbits. There may also be provisions for displaying orbits in combination; for example, placeholders such as symbols or icons may be displayed instead of images. Selecting a placeholder allows other displays to include the orbits in the way that each image is represented.
[0050] According to another embodiment of the method, the taught trajectory may include at least two trajectories whose paths are in the same area. To determine a common digital environment map for at least two orbits, The display is determined using the determined common digital environment map, and the determined common digital environment map is displayed for each orbit along with the respective distributed images. Each image is positioned for display based on its location within the digital environment map.
[0051] This embodiment is particularly advantageous when multiple trajectory paths lie in the same area, as the risk of confusion is especially high. Spatial representation using a digital environment map allows users to intuitively select the correct trajectory in certain situations, even without the user explicitly teaching the trajectory.
[0052] According to another embodiment of the method, images associated with each trajectory are captured and stored while the trajectory is being taught in training mode.
[0053] According to another embodiment of the method, new images are captured by an onboard camera, associated with their respective trajectories, and stored while the trajectories are being tracked in follow mode.
[0054] This is useful for taking into account changes in the environment of the image over time. The new image can replace all or part of the original image. This is advantageous when the original training was performed in poor visibility conditions such as darkness, rain, snow, and / or a dirty camera lens, which is the reason for the poor quality of the stored image.
[0055] According to another embodiment of the method, each image is captured by a plurality of cameras, which are a front camera, a rear camera, a side camera on the left side of the vehicle, and / or a side camera on the right side of the vehicle.
[0056] According to another embodiment of the method, each stored image is associated with positional information by reference to a world coordinate system, and the display is determined so that each image is positioned relative to other images based on the position associated with that image in the world coordinate system.
[0057] In this disclosure, the world coordinate system is understood to mean that all coordinates, in particular coordinates of different orbits, refer to the same reference point (origin). One such world coordinate system is the coordinate system provided by satellite navigation systems such as NAVSTAR GPS, GALILEO, GLONASS, and / or Beidou.
[0058] In this embodiment, the display may include, for example, a street map with an image of the track overlaid. This allows any user of the vehicle to quickly and intuitively see where the trained track is located.
[0059] According to the second aspect, a computer program product is proposed in which, when the program is executed by a computer, the computer includes a command that causes the computer to perform the method according to the first aspect.
[0060] Computers are components of vehicles, for example, forming electronic control units (ECUs).
[0061] Computer program products, such as computer program means, can be distributed or provided in the form of a file downloadable from a server on a network, for example, on a memory card, USB stick, CD-ROM, DVD, or other similar format. This can be achieved, for example, by transmitting a corresponding file containing the computer program product or computer program means over a wireless communication network.
[0062] According to a third aspect, a vehicle parking assist system is proposed. In follow mode, the parking assist system is configured to follow a trajectory taught in training mode, each trajectory being defined by a series of positions, connecting a starting position and a target position, and each trajectory being stored in association with an image of the vehicle's environment at each position, captured by an onboard camera. The parking assist system is The distribution unit distributes the stored images, A determination unit that determines the display including the distributed images and represents the trajectory associated with each, An output unit that outputs a display including the determined display to the vehicle's user interface display device, A receiving unit that receives user input from the user interface to select at least one image to be included in the display, A control unit initiates autonomous driving of the vehicle along the trajectory associated with the selected image, It is equipped with.
[0063] The parking assistance system has the same advantages as those described in the method according to the first embodiment. The embodiments and features described in the proposed method apply mutatis mutandis to the proposed parking assistance system.
[0064] Each unit of the parking assistance system may be implemented in hardware and / or software. In a hardware implementation, each unit may be, for example, in the form of a computer or a microprocessor. In a software implementation, each unit may be in the form of a computer program product, a function, a routine, an algorithm, a part of program code, or an executable object. Furthermore, each unit in this disclosure may be, for example, a part of a higher-level control system of a vehicle, such as a central electronic control unit and / or an electronic control unit (ECU).
[0065] The parking assistance system is configured, in particular, to perform the method according to the first embodiment.
[0066] According to the fourth embodiment, a vehicle is proposed having a user interface that includes a camera for capturing images of the vehicle's environment, a parking assistance system according to the third embodiment, and a display device.
[0067] This vehicle is, for example, a passenger car or a truck. The vehicle preferably includes a sensor unit configured to detect the vehicle's driving state and sense the vehicle's environment. Examples of such vehicle sensor units include cameras, RADAR (Radio detection and ranging), LiDAR (Light detection and ranging) image recording devices, ultrasonic sensors, position sensors, wheel angle sensors, and / or wheel speed sensors. The sensor unit is configured to output sensor signals to, for example, a parking assist system, which then performs semi-autonomous or fully autonomous control of the vehicle based on the acquired sensor signals.
[0068] The cameras include a front camera, a rear camera, a side camera on the left side of the vehicle, and / or a side camera on the right side of the vehicle.
[0069] Further possible implementations of the present invention also include combinations of features or embodiments described above or below in this disclosure that are not expressly referenced to exemplary embodiments. Those skilled in the art will also add individual embodiments as improvements or supplements to each basic form of the invention. [Brief explanation of the drawing]
[0070] Further advantageous configurations and aspects of the present invention are covered by the subclaims of the present invention described herein and by the exemplary embodiments described below. The present invention is described in more detail below based on preferred embodiments with reference to the accompanying drawings.
[0071] [Figure 1]A diagram showing an overview of the vehicle from a bird's-eye view. [Figure 2] A diagram illustrating the layout of a display that includes two images. [Figure 3] A diagram showing another overview of the display, including two images. [Figure 4] A diagram illustrating the general layout of the display, including the digital environment map. [Figure 5] A diagram showing another overview of the display, including the digital environment map. [Figure 6] A block diagram illustrating an exemplary embodiment of a parking assistance system. [Figure 7] A block diagram illustrating an exemplary embodiment of how to operate a parking assist system. [Modes for carrying out the invention]
[0072] In the diagrams, unless otherwise indicated, the same reference numeral is provided for identical or functionally identical elements.
[0073] Figure 1 shows a schematic bird's-eye view of vehicle 100. Vehicle 100 is, for example, an automobile placed in environment 200. Automobile 100 is equipped with a parking assist system 110, which is formed, for example, by a control unit. Furthermore, a plurality of environmental sensor devices 120, 130 are placed in automobile 100, these of which are, exemplarily, optical sensors 120 and ultrasonic sensors 130. Optical sensors 120 include, for example, a visual camera, radar and / or LiDAR. Optical sensors 120 can, in particular, capture images of each region from environment 200 of automobile 100 and output the images as optical sensor signals. Ultrasonic sensors 130 are configured to measure the distance from objects placed in environment 200 and output corresponding sensor signals. Based on the sensor signals acquired from sensors 120 and 130, the parking assist system 110 can drive the vehicle 100 semi-autonomously or fully autonomously. In addition to the optical sensor 120 and ultrasonic sensor 130 shown in Figure 1, the vehicle 100 may have various other sensor devices 120 and 130. Examples of these include wheel speed sensors, steering angle sensors, position sensors, microphones, acceleration sensors, and antennas connected to receivers capable of receiving electromagnetically transmitted data signals.
[0074] The vehicle also has a user interface 105, and the parking assist system 110 is communicably connected to the user interface 105. This means that the parking assist system 100 and the user interface 105 are configured to exchange data, in particular in the form of analog or digital data signals. The user interface 105 includes a display device (not shown) formed to display images IMG1 to IMG6 (see Figures 2 to 5) and a display DSP (see Figures 2 to 5). Furthermore, the user interface 105 includes input means (not shown) that allow a user of the vehicle 100 to input into the vehicle 100's systems, in particular into the parking assist system 110. The input means may include buttons, switches, rotary controls, touch sensors, voice detection, gesture detection, etc. Each input may be associated, in particular, with an element displayed by the display device at a specific time.
[0075] In follow mode, the parking assist system 110 is configured to autonomously follow trajectories TR1 to TR4 (see Figures 4 and 5) that were taught in training mode. Each of the trajectories TR1 to TR4 is defined by a series of positions P1 to P6 (see Figure 4), connecting a starting position P1 and a target position P6. Each of the trajectories TR1 to TR4 is stored in association with images IMG1 to IMG6 of the vehicle 100's environment 200 captured by the onboard camera 120. Each of the images IMG1 to IMG6 is recorded when the vehicle 100 is at each of the positions P1 to P6 of the trajectory TR1 to TR4. In embodiments, each of the images IMG1 to IMG6 may contain image information from multiple images IMG1 to IMG6, and in particular may be a merged image. The vehicle assistance system 110 is designed as shown in Figure 6 and configured to perform the method described with reference to Figure 7.
[0076] Figure 2 shows a schematic diagram of a display DSP consisting of two images, IMG1 and IMG2. The display DSP is determined by the determination unit 112 (see Figure 6) based on images IMG1 to IMG6 distributed by the distribution unit 111 (see Figure 6).
[0077] The two images IMG1 and IMG2 are associated with taught trajectories TR1 to TR4 (see Figure 4 or Figure 5), which are trajectories TR1 to TR4 recorded by the parking assist system 110 (see Figures 1 to 6) when, for example, a user drives the vehicle 100 in Figure 1. In this example, images IMG1 and IMG2 are views of the environment 200 (see Figure 1) of the vehicle 100 as seen from the starting position P1 (see Figure 4) (IMG1 with the heading "Start") and views of the environment 200 (see Figure 1) of the vehicle 100 as seen from the target position P6 (see Figure 4) (IMG2 with the heading "End"), and these images IMG1 and IMG2 were captured, for example, by the front camera 120. Note that, because it provides a better overview, particularly an overview that includes the target location P6 itself, the image viewed from a position in front of the target location P6 is advantageously used to display the target location P6, rather than the image IMG2 viewed from the display location P6 (see also Figure 3).
[0078] In the first image, IMG1, a house can be seen with a fence or wall in the background and trees and bushes next to it. In the second image, IMG2, only parts of the bushes and wall, and part of the fence can be seen. In this example, the user uses vehicle 100 to train on trajectory TR1 shown in Figure 4, and for this purpose the parking assistance system 110 can autonomously drive vehicle 100 from starting position P1 to target position P6.
[0079] The display DSP is transmitted from the output unit 114 (see Figure 6) of the parking assist system 110 to the user interface 105 (see Figure 1), and the user interface 105 displays the display DSP on the display device. This allows the user to intuitively confirm the trajectory TR1 associated with images IMG1 and IMG2 and select to follow this trajectory. In the embodiment, the display DSP includes only one image of trajectory TR1, or two or more images of trajectory TR1, and / or edited images of trajectory TR1 (see also Figure 4).
[0080] If the parking assistance system 110 has multiple trained and stored trajectories TR1 to TR4, the corresponding display can be determined and output for each additional trajectory TR1 to TR4, allowing the user to select the desired trajectory TR1 to TR4 based on the respective images IMG1 and IMG2.
[0081] The display DSP may include images IMG1 and IMG2 corresponding to multiple orbits TR1 to TR4, and it should be noted that these may be arranged, for example, adjacent to each other or below each other (not shown) simultaneously in the display DSP.
[0082] For example, when the user selects trajectory TR1, the user interface 105 transmits the corresponding user input to the parking assistance system 110, which can then begin autonomously following the selected trajectory TR1.
[0083] Figure 3 shows another concept of a display DSP including two images IMG1 and IMG2. The display DSP is determined by a determination unit 112 (see Figure 6) based on multiple images IMG1 to IMG6 distributed by a distribution unit 111 (see Figure 6).
[0084] The display in Figure 3, which is roughly equivalent to the display in Figure 2, differs in that image IMG2 and object OL are additionally displayed on IMG2. For example, object OL is a geometric shape (a rectangle with perspective distortion) that is inserted into or overlaid on each of the images IMG1 and IMG2. Object OL is inserted into the image at a position corresponding to the target position P6 (see Figure 4) on the track TR1, that is, at the position of vehicle 100 when traveling along track TR1.
[0085] In this example, the second image IMG2 is recorded from a position in front of the target position P6, i.e., position P5 (see Figure 4), which is why the image IMG2 includes the target position P6.
[0086] The additional overlay of the target position P6 in the displayed images IMG1 and IMG2 allows the user to better relate to trajectory TR1, and in particular, to distinguish between trajectories that terminate in adjacent parking spaces and whose displayed images are nearly identical.
[0087] Note that instead of the display DSP in Figure 3, the object OL is overlaid on only one of the images, for example, IMG1 or IMG2.
[0088] Figure 4 shows a schematic of a display DSP containing multiple images IMG1 to IMG6 associated with a digital environment map DMAP and orbit TR1. For example, the environment displayed in the digital environment map DMAP is the same as the environment displayed in images IMG1 and IMG2 in Figures 2 and 3. Objects OB1 to OB4 detected in the digital environment map DMAP are displayed in accordance with the environment. In this example, these are a house OB1, a bush OB2, a fence OB3, and a tree OB4. It is desirable that the digital environment map DMAP be acquired and stored when orbit TR1 is learned.
[0089] In this example, trajectory TR1 has six positions P1 to P6, where position P1 is the starting position and position P6 is the target position. For example, starting position P1 is in front of a house (shown here as object OB1), and target position P6 is on the side next to house OB1. At each of positions P1 to P6, training captures at least one of the environmental images IMG1 to IMG6, which is associated with and stored in trajectory TR1. Images IMG1 to IMG6 are preferably edited images showing a bird's-eye view of each position. This is possible, in particular, if vehicle 100 has multiple cameras 120 that can capture a total of 360 degrees of field of view around vehicle 100. Even if vehicle 100 has only one front camera 120, the image from the front camera 120 can be transformed into a top-down view by appropriately distorting the perspective of the image. Please note that each image, IMG1 to IMG6, was not necessarily captured when vehicle 100 was at the corresponding position. Rather, image IMG3 was captured by the front camera when vehicle 100 was at position P2, and this can be applied by analogy to the other images / positions.
[0090] In this example, the display includes both the digital environment map DMAP and the orbit TR1, and images IMG1 to IMG6 used as a representation of orbit TR1. Images IMG1 to IMG6 are positioned to correspond to reality, particularly within the digital environment map of the display DSP. The display, with images IMG1 to IMG6 overlaid together with the digital environment map DMAP, allows the user to confirm the orbit TR, and in particular, to more accurately confirm the target position P6 in orbit TR1.
[0091] If images IMG1 to IMG6 are sufficiently close together and / or overlap, they may be merged to generate a single image, which can then be stored.
[0092] In this embodiment, the trajectory TR1 can also be inserted into the display DSP in the form of an object OL (see Figure 3).
[0093] Figure 5 is another diagram illustrating the schematic of a display DSP with a digital environment map. For example, this includes the same environment 200 already described in Figure 4. In this example, however, there are multiple tracks TR1 to TR4 within the display area of the environment. For example, different tracks TR1 to TR4 are trained by different users of vehicle 100. In this example, all tracks TR1 to TR4 at the starting position are displayed by image IMG1, but this is not mandatory.
[0094] Since each image IMG2 to IMG5 for each target location is placed in a realistic manner on the display DSP, referencing the digital environment map DMAP, in particular objects OB1 to OB4, it is quite easy for the user to deduce the actual vehicle's position from the displayed target location.
[0095] For example, a user can select one of the images IMG2 to IMG5 in the display DSP and then select the associated trajectories TR1 to TR4.
[0096] In this embodiment, each object OL (see Figure 3) for trajectories TR1 to TR4 may also be inserted into the display DSP.
[0097] Figure 6 shows a schematic block of an exemplary embodiment of a parking assist system 110 that can be used, for example, with the vehicle 100 in Figure 1. In follow mode, the parking assist system 110 is configured to autonomously follow a number of trajectories TR1 to TR4 (see Figure 4 or Figure 5) from a number of trajectories TR1 to TR4 taught in training mode. Each trajectory TR1 to TR4 is defined by a series of positions P1 to P6 (see Figure 4), connecting a starting position P1 and a target position P6. Each of the trajectories TR1 to TR4 is associated with and stored images IMG1 to IMG6 (see Figures 2 to 6) of the environment 200 (see Figure 1) of the vehicle 100 captured by an onboard camera 120 (see Figure 1). Each image IMG1 to IMG6 is recorded when the vehicle 100 is at each position P1 to P6 on the trajectories TR1 to TR4, and thus shows the corresponding details from the environment 200. The parking assist system 110 includes a distribution unit 111 that distributes the stored image IMGs, a determination unit 112 that determines a display DSP containing the distributed image IMGs representing each associated trajectory TR1 to TR4 (see Figure 4 or Figure 5), an output unit 114 that outputs the determined display DSP to the display device of the user interface 105 (see Figure 1), in particular in the form of a digital image or data signal, a receiving unit 116 that receives a user input SIG from the user interface 105 to select at least one image IMG to be included in the display DSP, and a control unit 118 that initiates autonomous driving of the vehicle 100 along the trajectories TR1 to TR4 associated with the selected IMG. In this example, the control unit 118 outputs the corresponding control signal CTR.
[0098] Each unit 111 to 118 of the parking assist system 110 may be implemented in hardware and / or software. In hardware implementation, each unit 111 to 118 may be, for example, in the form of a computer or a microprocessor. In software implementation, each unit 111 to 118 may be in the form of a computer program product, a function, a routine, an algorithm, program code, or an executable object. Furthermore, each unit 111 to 118 mentioned herein may be part of a higher-level control system of the vehicle, such as a central electronic control unit and / or control unit (ECU).
[0099] In this embodiment, the display device and / or user interface 105 is part of a parking assistance system 110 (not shown).
[0100] Figure 7 shows a schematic block diagram of an exemplary embodiment of the parking assist system 110 for vehicle 100, in particular the parking assist system 110 shown in Figure 6 and the method of operating vehicle 100 shown in Figure 1. In follow mode, the parking assist system 110 is configured to autonomously follow a set of trajectories TR1 to TR4 (see Figure 4 or Figure 5) from a set of trajectories TR1 to TR4 taught in training mode, where each trajectory TR1 to TR4 is defined by a series of positions P1 to P6 (see Figure 4), connecting a starting position P1 and a target position P6, and each trajectory TR1 to TR4 is associated with and stored images IMG1 to IMG6 (see Figures 2 to 6) captured by an onboard camera 120 (see Figure 1) at each of the positions P1 to P6 in the vehicle environment 200 of vehicle 100. The first step S1 comprises distributing the stored number of images IMG1 to IMG6. The second step S2 comprises determining a display DSP (see Figures 2 to 5) that includes distributed images IMG1 to IMG6 showing the trajectories TR1 to TR4, each associated with the vehicle; the third step S3 comprises outputting the determined display DSP to the display device of the user interface 105 (see Figure 1) of the vehicle 100; the fourth step S4 comprises receiving a user input SIG (see Figure 6) from the user interface 105 to select at least one image IMG1 to IMG6 included in the display DSP; and the fifth step S5 comprises starting autonomous driving of the vehicle 100 along the trajectories TR1 to TR4 associated with the selected images IMG1 to IMG6.
[0101] The present invention is described based on exemplary embodiments, but can be modified in many ways. [Explanation of Symbols]
[0102] 100 vehicles 105 User Interface 110 Parking Assist System 111 Distribution Department 112 Decision Section Output section of 114 116 Receiving Unit 118 Control Unit 120 Environmental Sensor Devices 130 Environmental Sensor Devices 200 Environment CTR control signal DMAP Digital Environment Map DSP display IMG1 Image IMG2 Image IMG3 Image IMG4 Image IMG5 Image IMG6 Image OB1 object OB2 object OB3 object OB4 object OL object P1 position P2 position P3 position P4 position P5 position P6 position S1 Method Step S2 Method Steps S3 Method Steps S4 Method Steps S5 Method Steps SIG User Input TR1 orbit TR2 orbit TR3 orbit TR4 orbit
Claims
1. A method for operating a parking assist system (110) of a vehicle (100), In follow mode, the parking assist system (110) is configured to autonomously follow a trajectory (TR1 to TR4) from a number of trajectories (TR1 to TR4) taught in training mode. Each of the aforementioned trajectories (TR1 to TR4) is defined by a series of positions (P1 to P6), connecting the starting position (P1) and the target position (P6). Each of the aforementioned tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the vehicle's (100) environment (200) at each of the aforementioned locations (P1 to P6) captured by the onboard camera (120). The aforementioned method, Distributing the stored images (IMG1 to IMG6) (S1), The display (DSP) including the distributed images (IMG1 to IMG6) is determined (S2), and the associated orbits (TR1 to TR4) are represented for each. The determined display (DSP) is output to the display device of the user interface (105) of the vehicle (100) (S3), S4 receives a user input (SIG) from the user interface (105) to select at least one image (IMG1 to IMG6) included in the display (DSP), S5) Starts autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected images (IMG1 to IMG6), A method of providing, A digital environment map (DMAP) is determined for each of the aforementioned orbits (TR1 to TR4), Using the determined digital environment map (DMAP), the display (DSP) is determined, and the determined digital environment map (DMAP) is displayed together with the distributed images (IMG1 to IMG6) for each of the trajectories (TR1 to TR4). The aforementioned images (IMG1 to IMG6) are arranged within the display (DSP) based on their positions in the digital environment map (DMAP). method.
2. A method for operating a parking assist system (110) of a vehicle (100), In follow mode, the parking assist system (110) is configured to autonomously follow a trajectory (TR1 to TR4) from a number of trajectories (TR1 to TR4) taught in training mode. Each of the aforementioned trajectories (TR1 to TR4) is defined by a series of positions (P1 to P6), connecting the starting position (P1) and the target position (P6). Each of the aforementioned tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the vehicle's (100) environment (200) at each of the aforementioned locations (P1 to P6) captured by the onboard camera (120). The aforementioned method, Distributing the stored images (IMG1 to IMG6) (S1), The display (DSP) including the distributed images (IMG1 to IMG6) is determined (S2), and the associated orbits (TR1 to TR4) are represented for each. The determined display (DSP) is output to the display device of the user interface (105) of the vehicle (100) (S3), S4 receives a user input (SIG) from the user interface (105) to select at least one image (IMG1 to IMG6) included in the display (DSP), S5) Starts autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected images (IMG1 to IMG6), A method of providing, The taught trajectories (TR1 to TR4) include at least two trajectories that have a path in the same area. Determining a common digital environment map (DMAP) for at least two of the aforementioned orbits, The display (DSP) is determined using the determined common digital environment map (DMAP), and the determined common digital environment map (DMAP) is displayed together with the respective distributed images (IMG1 to IMG6) for each of the trajectories (TR1 to TR4). This includes, Each of the aforementioned images (IMG1 to IMG6) is positioned on the display based on the position of the image in the Common Digital Environment Map (DMAP). method.
3. A method for operating a parking assist system (110) of a vehicle (100), In follow mode, the parking assist system (110) is configured to autonomously follow a trajectory (TR1 to TR4) from a number of trajectories (TR1 to TR4) taught in training mode. Each of the aforementioned trajectories (TR1 to TR4) is defined by a series of positions (P1 to P6), connecting the starting position (P1) and the target position (P6). Each of the aforementioned tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the vehicle's (100) environment (200) at each of the aforementioned locations (P1 to P6) captured by the onboard camera (120). The aforementioned method, Distributing the stored images (IMG1 to IMG6) (S1), The display (DSP) including the distributed images (IMG1 to IMG6) is determined (S2), and the associated orbits (TR1 to TR4) are represented for each. The determined display (DSP) is output to the display device of the user interface (105) of the vehicle (100) (S3), S4 receives a user input (SIG) from the user interface (105) to select at least one image (IMG1 to IMG6) included in the display (DSP), S5) Starts autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected images (IMG1 to IMG6), A method of providing, In the tracking mode, new images captured by the on-board camera (120) and associated with each of the trajectories (TR1 to TR4) are stored while the trajectories (TR1 to TR4) are being tracked. method.
4. A method for operating a parking assist system (110) of a vehicle (100), In follow mode, the parking assist system (110) is configured to autonomously follow a trajectory (TR1 to TR4) from a number of trajectories (TR1 to TR4) taught in training mode. Each of the aforementioned trajectories (TR1 to TR4) is defined by a series of positions (P1 to P6), connecting the starting position (P1) and the target position (P6). Each of the aforementioned tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the vehicle's (100) environment (200) at each of the aforementioned locations (P1 to P6) captured by the onboard camera (120). The aforementioned method, Distributing the stored images (IMG1 to IMG6) (S1), The display (DSP) including the distributed images (IMG1 to IMG6) is determined (S2), and the associated orbits (TR1 to TR4) are represented for each. The determined display (DSP) is output to the display device of the user interface (105) of the vehicle (100) (S3), S4 receives a user input (SIG) from the user interface (105) to select at least one image (IMG1 to IMG6) included in the display (DSP), S5) Starts autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected images (IMG1 to IMG6), A method of providing, Each of the stored images (IMG1 to IMG2) is associated with positional information by referring to a world coordinate system, and the display is determined so that each of the images is positioned relative to the other images based on the position associated with the world coordinate system. method.
5. Each of the stored images (IMG1 to IMG6) for each of the aforementioned trajectories (TR1 to TR6) includes at least one image of each of the aforementioned target positions (P6). The method according to any one of claims 1 to 4.
6. The stored images (IMG1 to IMG6) for each of the aforementioned trajectories (TR1 to TR4) include images edited from multiple individual images. The method according to any one of claims 1 to 4.
7. The edited images include bird's-eye views of each of the aforementioned locations (P1 to P6). The method according to claim 6.
8. An object (OL) representing the position of the vehicle (100) at the target position (P6) in each of the aforementioned tracks (TR1 to TR4) is distributed. The delivered object (OL) is used to determine the display (DSP), and the object (OL) in the display (DSP) is overlaid on the target position (P6) in the images (IMG1 to IMG6) where the target position (P6) can be seen. The method according to any one of claims 1 to 4.
9. The taught trajectories (TR1 to TR4) include at least two trajectories that have a path in the same area, and the display (DSP) is determined so that the images (IMG1 to IMG6) of each of the at least two trajectories are displayed in groups. The method according to any one of claims 1 to 4.
10. In the training mode, the images (IMG1 to IMG6) associated with each of the trajectories (TR1 to TR4) are captured and stored while the trajectories (TR1 to TR4) are being taught. The method according to any one of claims 1 to 4.
11. Each of the aforementioned images (IMG1 to IMG6) is captured by a plurality of in-vehicle cameras (120), including at least one of a front camera, a rear camera, a side camera on the left side of the vehicle, and / or a side camera on the right side of the vehicle. The method according to any one of claims 1 to 4.
12. When the program is executed on a computer, the computer includes a command to perform the method according to any one of claims 1 to 4, Computer program products.
13. A parking assistance system for a vehicle (100), In follow mode, the parking assist system (110) is configured to autonomously follow a trajectory (TR1 to TR4) from a number of trajectories (TR1 to TR4) taught in training mode. Each of the aforementioned trajectories (TR1 to TR4) is defined by a series of positions (P1 to P6), connecting the starting position (P1) and the target position (P6). Each of the aforementioned tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the vehicle's (100) environment (200) at each of the aforementioned locations (P1 to P6) captured by the onboard camera (120). A distribution unit (111) distributes the stored images (IMG1 to IMG6), A determination unit (112) determines a display (DSP) including the distributed images (IMG1 to IMG6) and represents the associated trajectories (TR1 to TR4) for each, An output unit (114) outputs the determined display (DSP) to the display device of the user interface (105) of the vehicle (100), A receiving unit (116) receives user input (SIG) from the user interface (105) to select at least one image (IMG1 to IMG6) included in the display (DSP), A control unit (118) initiates autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected images (IMG1 to IMG6), Equipped with, A digital environment map (DMAP) is determined for each of the aforementioned orbits (TR1 to TR4), Using the determined digital environment map (DMAP), the display (DSP) is determined, and the determined digital environment map (DMAP) is displayed together with the distributed images (IMG1 to IMG6) for each of the trajectories (TR1 to TR4). The aforementioned images (IMG1 to IMG6) are arranged within the display (DSP) based on their positions in the digital environment map (DMAP). A parking assistance system for a vehicle (100).
14. A parking assistance system for a vehicle (100), In follow mode, the parking assist system (110) is configured to autonomously follow a trajectory (TR1 to TR4) from a number of trajectories (TR1 to TR4) taught in training mode. Each of the aforementioned trajectories (TR1 to TR4) is defined by a series of positions (P1 to P6), connecting the starting position (P1) and the target position (P6). Each of the aforementioned tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the vehicle's (100) environment (200) at each of the aforementioned locations (P1 to P6) captured by the onboard camera (120). A distribution unit (111) distributes the stored images (IMG1 to IMG6), A determination unit (112) determines a display (DSP) including the distributed images (IMG1 to IMG6) and represents the associated trajectories (TR1 to TR4) for each, An output unit (114) outputs the determined display (DSP) to the display device of the user interface (105) of the vehicle (100), A receiving unit (116) receives user input (SIG) from the user interface (105) to select at least one image (IMG1 to IMG6) included in the display (DSP), A control unit (118) initiates autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected images (IMG1 to IMG6), Equipped with, The taught trajectories (TR1 to TR4) include at least two trajectories that have a path in the same area. Determining a common digital environment map (DMAP) for at least two of the aforementioned orbits, The display (DSP) is determined using the determined common digital environment map (DMAP), and the determined common digital environment map (DMAP) is displayed together with the respective distributed images (IMG1 to IMG6) for each of the trajectories (TR1 to TR4). This includes, Each of the aforementioned images (IMG1 to IMG6) is positioned on the display based on the position of the image in the Common Digital Environment Map (DMAP). method.
15. A parking assistance system for a vehicle (100), In follow mode, the parking assist system (110) is configured to autonomously follow a trajectory (TR1 to TR4) from a number of trajectories (TR1 to TR4) taught in training mode. Each of the aforementioned trajectories (TR1 to TR4) is defined by a series of positions (P1 to P6), connecting the starting position (P1) and the target position (P6). Each of the aforementioned tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the vehicle's (100) environment (200) at each of the aforementioned locations (P1 to P6) captured by the onboard camera (120). A distribution unit (111) distributes the stored images (IMG1 to IMG6), A determination unit (112) determines a display (DSP) including the distributed images (IMG1 to IMG6) and represents the associated trajectories (TR1 to TR4) for each, An output unit (114) outputs the determined display (DSP) to the display device of the user interface (105) of the vehicle (100), A receiving unit (116) receives user input (SIG) from the user interface (105) to select at least one image (IMG1 to IMG6) included in the display (DSP), A control unit (118) initiates autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected images (IMG1 to IMG6), Equipped with, In the tracking mode, new images captured by the on-board camera (120) and associated with each of the trajectories (TR1 to TR4) are stored while the trajectories (TR1 to TR4) are being tracked. method.
16. A parking assistance system for a vehicle (100), In follow mode, the parking assist system (110) is configured to autonomously follow a trajectory (TR1 to TR4) from a number of trajectories (TR1 to TR4) taught in training mode. Each of the aforementioned trajectories (TR1 to TR4) is defined by a series of positions (P1 to P6), connecting the starting position (P1) and the target position (P6). Each of the aforementioned tracks (TR1 to TR4) is stored in association with an image (IMG1 to IMG6) of the vehicle's (100) environment (200) at each of the aforementioned locations (P1 to P6) captured by the onboard camera (120). A distribution unit (111) distributes the stored images (IMG1 to IMG6), A determination unit (112) determines a display (DSP) including the distributed images (IMG1 to IMG6) and represents the associated trajectories (TR1 to TR4) for each, An output unit (114) outputs the determined display (DSP) to the display device of the user interface (105) of the vehicle (100), A receiving unit (116) receives user input (SIG) from the user interface (105) to select at least one image (IMG1 to IMG6) included in the display (DSP), A control unit (118) initiates autonomous driving of the vehicle (100) along the trajectory (TR1 to TR4) associated with the selected images (IMG1 to IMG6), Equipped with, Each of the stored images (IMG1 to IMG2) is associated with positional information by referring to a world coordinate system, and the display is determined so that each of the images is positioned relative to the other images based on the position associated with the world coordinate system. method.
17. A parking assistance system (110) according to any one of claims 13 to 16, A user interface (105) including a display device, It has, The vehicle (100) has multiple in-vehicle cameras (120) that capture images (IMG1 to IMG6) of the environment (200). Vehicle (100).