SYSTEM FOR AUGMENTED REALITY-ASSISTED PARKING WITH OPERATOR-ASSISTED SELECTIONS AND ALIGNMENTS OF MOMENTUM ENVIRONMENTS
The AR-assisted parking system addresses sensor limitations by using AR to guide vehicle alignment with real-world features, improving accuracy and operator confidence in parking operations.
Patent Information
- Application Number
- DE102023125973
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2023-09-07
- Filing Date
- 2023-09-26
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2043-09-26
AI Technical Summary
Existing driver assistance systems for autonomous or semi-autonomous vehicle parking face challenges in determining accurate parking paths due to sensor limitations and computational constraints, leading to reduced operator confidence and comfort.
An augmented reality (AR)-assisted parking system that uses feature point environments (FPVs) detected by vehicle sensors, providing enhanced steering guidance through a Head-Up Display (HUD) and Human-Machine Interface (HMI), allowing operators to align the vehicle with real-world features using AR overlays and semantic tags, reducing misalignment and increasing accuracy.
The system improves parking accuracy and operator confidence by overlaying live sensor data with AR, ensuring precise alignment of the vehicle with real-world features, minimizing misalignment and enhancing the overall parking experience.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
INTRODUCTION
[0001] The present invention relates to driver assistance systems in vehicles, in particular to a system for augmented reality (AR)-assisted parking of a vehicle according to the preamble of claim 1, as is essentially known from DE 10 2017 128 294 A1. Further prior art can also be found in the documents DE 10 2021 100 548 A1, DE 10 2021 005 530 A1, and DE 10 2016 108 718 A1.
[0002] In many vehicles, a driver assistance system is capable of parking a vehicle autonomously or semi-autonomously in a parking space nearby. Typical driver assistance systems used for parking vehicles utilize an array of sensors mounted on the vehicle and generate a path to one or more parking spaces. However, such systems often have difficulty determining paths due to the perceptual capabilities of the array of sensors, limited computing power, and other factors.
[0003] While current systems and methods for autonomous or semi-autonomous vehicle parking serve their purpose, there is a need for a new and improved system and method that overcomes the problems with sensor data accuracy without significantly increasing the computational burden, without increasing the complexity of the system and / or components, and that increases the confidence and comfort of the vehicle operator. SUMMARY
[0004] According to the invention, a system for augmented reality (AR)-assisted parking of vehicles is presented, which is characterized by the features of claim 1 or those of claim 4.
[0005] In a further aspect of the present invention, the TS section further comprises a third TS control logic section in which the ARAPA, via the HMI, proposes FPVs and semantic tags corresponding to parking spaces that the system has previously identified at the parking position.
[0006] In another aspect of the present invention, the TS section further comprises a fourth TS control logic section that provides, via the HMI, automatically generated suggested parking space names and opportunities for the operator of the vehicle to change the suggested parking space names, and a fifth TS control logic section that stores the suggested parking space names and semantic labels in memory for use in the LS section of the ARAPA.
[0007] In another aspect of the present invention, the LS section further includes fourth LS control logic that determines that the vehicle is within the FPV preview threshold distance from one or more previously stored FPVs of parking spots at a parking position. The LS section also includes fifth LS control logic that, on a case-by-case basis, displays a preview of the identified, previously stored FPVs on the HMI or a head-up display (HUD) and uses previously identified FPVs as enhanced steering guidance for the vehicle operator to align the vehicle within a parking spot to a distance equal to or less than a predefined FPV alignment threshold distance.
[0008] In another aspect of the present invention, the LS section further includes sixth LS control logic that provides AR assistance to the vehicle operator when the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle to and into the parking spot identified by the stored, previously identified FPVs. The AR assistance overlays the stored, previously identified FPVs with one or more live sensor data displayed on the vehicle's HMI and HUD, such that the FPVs are displayed directly in the vehicle operator's line of sight and at positions that correspond to the actual physical positions of the FPVs from the vehicle operator's visual perspective.A path to the saved, previously identified FPVs will be displayed on the HUD or HMI in a different style, color, or layout than a path to a previously trained parking spot.
[0009] In another aspect of the present invention, the LS section further includes seventh LS control logic that determines that a current distance of the vehicle to a parking space is less than or equal to the FPV alignment threshold distance and activates an FPV alignment mode of the ARAPA. The LS section also includes eighth LS control logic that uses AR to display previously identified or stored FPVs on the HMI or HUD, thereby providing the vehicle operator with references that the vehicle operator uses to align the vehicle within the parking space in real time. The LS section also includes ninth LS control logic that causes the vehicle operator to verify the alignment of the stored FPVs with the corresponding features of a real-world environment for improved parking assistance and to steer or otherwise maneuver the vehicle.The LS section also contains a tenth LS control logic that notifies the vehicle operator that the vehicle has been fully parked when the stored FPVs are aligned with real FPVs or environmental objects displayed on the HMI or HUD, or both.
[0010] In another aspect of the present invention, a method for augmented reality (AR)-assisted vehicle parking with operator-assisted selections and orientations of feature point vicinities (FPVs) comprises acquiring FPVs of a parking position with one or more sensors disposed on a vehicle. The method further comprises executing programmatic control logic stored in a memory of one or more controllers in communication with the one or more sensors and with one or more human-machine interfaces (HMIs) disposed in the vehicle. Each of the one or more controllers includes a processor, the memory, and one or more input / output (I / O) ports. The I / O ports are in communication with one or more sensors and one or more HMIs.The processor executes control logic including an AR-assisted parking application (ARAPA). The method further includes activating the ARAPA through an input from the vehicle operator via the HMI when parking into a parking space at the parking position is complete. The ARAPA is trained in a training session (TS) to detect FPVs of a parking space at the parking position. During a live session (LS) portion of the ARAPA, the vehicle operator is prompted to select the parking space. The method further includes guiding the vehicle operator into the parking space by displaying previously trained FPVs over the live sensor data displayed on the HMI and instructing the vehicle operator to align the previously trained FPVs with the corresponding real FPVs of the parking space by maneuvering the vehicle.
[0011] In another aspect of the present invention, the TS section further comprises indicating that the vehicle has been parked in a preferred parking location via an input from the operator of the vehicle to a first feature of the ARAPA in the HMI.
[0012] In another aspect of the present invention, the TS portion further comprises displaying, via the HMI, a plurality of proposed FPVs overlaid on a real-time display of a view from the one or more sensors. The proposed FPVs define, at least in part, a current or idealized position of the vehicle within the preferred parking location. The method further comprises receiving input from the operator of the vehicle via the HMI, selecting some or all of the plurality of proposed FPVs. Each of the proposed FPVs is spaced appropriately from other FPVs, the appropriately spaced being selected such that all selected FPVs can be displayed simultaneously on the HMI and such that the various FPVs do not overlap within any line of sight of the operator of the vehicle.The appropriate distance reduces any possible misalignment of the vehicle within the parking space from a first level to a second level that is lower than the first level while the vehicle is parked.
[0013] In another aspect of the present invention, the TS portion of the method further comprises proposing, via the HMI, FPVs and semantic labels to the vehicle operator that correspond to parking spaces previously identified by the ARAPA at the parking position. The TS portion of the method further comprises providing automatically generated suggested parking space names and opportunities for the vehicle operator to modify the suggested parking space names, and storing the suggested parking space names and semantic labels in memory for use in the LS portion of the ARAPA.
[0014] In another aspect of the present invention, the LS portion of the method further comprises determining that a current distance between the vehicle and one or more previously trained parking locations is less than or equal to a predefined FPV preview threshold distance. The FPV preview threshold distance is between about six and about ten meters. The LS portion of the method further comprises offering one or more prompts to the operator of the vehicle via the HMI. The one or more prompts request the operator of the vehicle to select one of the one or more previously trained parking locations within the FPV preview threshold distance within which the operator of the vehicle intends to park.
[0015] In another aspect of the present invention, the LS portion of the method further comprises providing augmented reality (AR) assistance to the vehicle operator while the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle to and into the previously trained parking spot. The AR assistance overlays FPVs of the previously trained parking spot with one or more live sensor data displayed on the vehicle's HMI and on a heads-up display (HUD) such that the FPVs are displayed directly in the vehicle operator's line of sight and at positions that correspond to the actual physical positions of the FPVs from the vehicle operator's visual perspective.
[0016] In another aspect of the present invention, the LS portion of the method further comprises determining that the vehicle is within the FPV preview threshold distance from one or more previously stored FPVs of parking spaces at a parking location; and conditionally displaying a preview of the previously stored FPVs on the HMI or a heads-up display (HUD). The LS portion of the method also uses previously stored FPVs as an enhanced guidance aid to the operator of the vehicle to align the vehicle within a parking space to a distance equal to or less than a predefined FPV alignment threshold distance.
[0017] In another aspect of the present invention, the LS portion of the method further comprises providing AR assistance to the vehicle operator while the vehicle operator provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle to and into the parking spot identified by the previously identified FPVs. The AR assistance overlays the previously identified FPVs with one or more live sensor data displayed on the vehicle's HMI and HUD, such that the FPVs are displayed directly in the vehicle operator's line of sight and at positions that correspond to the actual physical positions of the FPVs from the vehicle operator's visual perspective.A path to the identified, previously saved FPVs will be displayed on the HUD or HMI in a different style, color, or layout than a path to a previously trained parking spot.
[0018] In another aspect of the present invention, the LS portion of the method further comprises determining a current distance of the vehicle to a parking space that is less than or equal to the FPV alignment threshold distance, and engaging an FPV alignment mode of the ARAPA, and using AR to display previously identified or stored FPVs on the HMI and / or HUD, thereby providing the vehicle operator with references that the vehicle operator uses to align the vehicle within the parking space in real time.The LS portion of the method further includes prompting the vehicle operator to check the alignments of the stored FPVs with corresponding features of a real-world environment for improved parking assistance and to steer or otherwise maneuver the vehicle, and notifying the vehicle operator that the vehicle has been fully parked when the stored FPVs are aligned with real-world FPVs or environmental objects as viewed on the HMI or HUD, or both.
[0019] In another aspect of the present invention, a method for augmented reality (AR)-assisted vehicle parking with operator-assisted selections and orientations of feature point locations (FPVs) includes acquiring FPVs of a parking position with one or more sensors disposed on a vehicle and executing programmatic control logic stored in a memory by one or more controllers that communicate with the one or more sensors and with one or more human-machine interfaces (HMIs) disposed in the vehicle. Each of the one or more controllers includes a processor, the memory, and one or more input / output (I / O) ports. The I / O ports communicate with one or more sensors and one or more HMIs, and the processor executes control logic that includes an AR-assisted parking application (ARAPA).The method further includes activating the ARAPA via an input from the vehicle operator via the HMI when parking at a parking space at the parking position is complete. The ARAPA has a controller including control logic for training the ARAPA, in a training session (TS) section, to recognize the FPVs of a parking space at the parking position. The TS section further includes control logic for indicating, via an input from the vehicle operator to a first function of the ARAPA in the HMI, that the vehicle has been parked at a preferred parking space and presenting, via the HMI, a plurality of suggested FPVs overlaid on a real-time display of a view from the one or more sensors. The suggested FPVs define, at least in part, a current or idealized position of the vehicle within the preferred parking space.The TS section further includes control logic for receiving inputs from the vehicle operator via the HMI that select some or all of the plurality of suggested FPVs, wherein each of the suggested FPVs has appropriate spacings from other FPVs, wherein the appropriate spacing is selected such that all selected FPVs can be displayed simultaneously on the HMI and that different FPVs do not overlap within any line of sight of the vehicle operator; and wherein the appropriate spacing reduces a potential for misalignment of the vehicle within the parking space from a first level to a second level lower than the first level when the vehicle is parked. The TS section further includes control logic for suggesting to the vehicle operator via the HMI FPVs and semantic labels corresponding to parking spaces that ARAPA has previously identified at the parking location.The TS section further includes control logic to offer, via the HMI, automatically generated parking location name suggestions and opportunities for the vehicle operator to modify the suggested parking location names, and to store the suggested parking location names and semantic labels in memory for use in a Live Session (LS) section of the ARAPA. The ARAPA further includes control logic to prompt the vehicle operator to select a parking location in an LS section of the ARAPA while the vehicle operator is navigating through a parking position with one or more parking locations. The LS section also includes control logic to determine when a current distance between the vehicle and one or more previously trained parking locations is less than or equal to a predefined FPV preview threshold distance.
[0020] The FPV preview threshold distance is between approximately six and approximately ten meters. The LS section further includes control logic to present one or more prompts to the vehicle operator via the HMI. The one or more prompts request the vehicle operator to select one of the one or more previously trained parking spots within the FPV preview threshold distance within which the vehicle operator intends to park. The LS section also includes control logic to assist the vehicle operator through augmented reality (AR) as the vehicle operator provides directional inputs via steering inputs, braking inputs, and acceleration inputs to guide the vehicle to and into the previously trained parking spot.The AR support overlays FPVs of the previously trained parking spot with one or more live sensor data displayed on the vehicle's HMI and on a heads-up display (HUD), so that the FPVs are displayed directly in the vehicle operator's line of sight and at positions that correspond to the actual physical positions of the FPVs from the vehicle operator's visual perspective. The LS section also includes control logic to determine that the vehicle is within the FPV preview threshold distance of one or more previously saved FPVs of parking spots in a parking position.The LS section further includes control logic for conditionally displaying a preview of the previously stored FPVs on the HMI or a heads-up display (HUD) and for using previously identified FPVs as enhanced steering assistance for the vehicle operator to align the vehicle within a parking space at a distance equal to or less than a predefined FPV alignment threshold distance. The LS section further includes control logic for providing AR assistance to the vehicle operator when providing directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle toward and into the parking space identified by the previously identified FPVs.The AR support overlays the previously identified FPVs with one or more live sensor data displayed on the vehicle's HMI and HUD, so that the FPVs are displayed directly in the vehicle operator's line of sight and at positions that correspond to the actual physical positions of the FPVs from the vehicle operator's visual perspective. A path to the identified, previously saved FPVs is displayed on the HUD or HMI in a different style, color, or layout than a path to a previously trained parking spot.The LS section also includes control logic to instruct the vehicle operator to align the previously trained FPVs with the corresponding real FPVs of the parking spot by maneuvering the vehicle by: determining a current distance of the vehicle to a parking spot that is less than or equal to the FPV alignment threshold distance, and engaging an FPV alignment mode of the ARAPA.The LS section further includes control logic for using AR to display previously identified or stored FPVs on the HMI and / or HUD, thereby providing the vehicle operator with references the vehicle operator uses to align the vehicle within the parking space in real time, and for prompting the vehicle operator to verify alignments of the stored FPVs with corresponding features of a real-world environment for improved parking assistance and to steer or otherwise maneuver the vehicle. The LS section also includes control logic that notifies the vehicle operator via the HUD and / or HMI that the vehicle has been fully parked when the stored FPVs align with real-world FPVs or environmental objects visible on the HMI or HUD, or both.
[0021] Further areas of application will become apparent from the description provided here. It is understood that the description and specific examples are for illustrative purposes only. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The drawings described here are for illustrative purposes only. Fig. 1 is a schematic view of a system for augmented reality (AR)-assisted parking with operator-assisted selections and alignments (DSAs) of feature point environments (FPVs) according to an exemplary embodiment; Fig. Figure 2A is a perspective view of the interior of a vehicle equipped with the AR-assisted parking system with DSAs from FPVs of Fig. 1 according to an exemplary embodiment; Fig. Figure 2B is an enlarged view of a human-machine interface (HMI) located inside the vehicle equipped with the system for AR-assisted parking with DSAs from FPVs of Fig. 1 according to an exemplary embodiment of Fig. 2A, according to an exemplary embodiment; Fig. Figure 3 is a flowchart showing the control logic of a training session of the system for AR-assisted parking with DSAs of FPVs of Fig. 1 according to an exemplary embodiment; Fig. Figure 4 is a flowchart showing the control logic of a live session of the AR-assisted parking system with DSAs from FPVs of Fig. 1 according to an exemplary embodiment; and Fig. 5 is a flowchart illustrating a method for AR-assisted parking with DSAs of FPVs according to an exemplary embodiment. DETAILED DESCRIPTION
[0023] The foregoing detailed description is merely exemplary in nature. Furthermore, there is no intention to be bound by the express or implied theories set forth in the foregoing technical field, technical background, brief summary, or the following detailed description. As used herein, the term "module" refers to hardware, software, firmware, electronic control component, processing logic, and / or processor device, individually or in any combination thereof, including, without limitation, an application-specific integrated circuit (ASIC), an electronic circuit, a processor (collectively, dedicated, or as a group), and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality.
[0024] Embodiments of the present invention may be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components may be implemented by a variety of hardware, software, and / or firmware components configured to perform the specified functions. For example, an embodiment of the present invention may utilize various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, lookup tables, or the like, which can perform a variety of functions under the control of one or more microprocessors or other control devices.Furthermore, it will be appreciated by those skilled in the art that embodiments of the present invention may be used in connection with a variety of systems and that the systems described herein are merely exemplary embodiments of the present invention.
[0025] For the sake of brevity, techniques for signal processing, data fusion, signaling, control, and other functional aspects of the systems (and the individual operating components of the systems) are not described in detail here. Furthermore, the connecting lines depicted in the various figures are intended to represent exemplary functional relationships and / or physical connections between the various elements. It should be noted that alternative or additional functional relationships or physical connections may be present in an embodiment of the present invention.
[0026] Fig. 1, Fig. 2A and Fig. 2B illustrate a system 10 for augmented reality (AR)-assisted parking with operator-in-loop selections and alignments (OSAs) of feature point vicinities (FPVs). The system 10 includes a vehicle 12 equipped with one or more sensors 14 and one or more controllers 16. The vehicle 12 may be a variety of vehicles, including cars, trucks, sport utility vehicles (SUVs), buses, semi-trailers, tractors used in agriculture or construction, watercraft, aircraft such as airplanes or helicopters, or the like. In some examples, the vehicle 12 travels on a road 18 having any type of road surface, such as pavement, gravel, dirt, ice, or other surfaces.
[0027] The sensors 14 may be a plurality of sensors 14 connected to and communicating via a communications network 20, such as a vehicle-to-vehicle (V2V) communications network, a vehicle-to-infrastructure (V2I) network, or the like. The sensors 14 collect data about the environment of the vehicle 12, including electromagnetic and / or optical information in a variety of different wavelengths, including those visible to humans, as well as infrared, ultraviolet, and other parts of the light spectrum not visible to humans. The sensors 14 may further include GPS (Global Positioning System) sensors, cameras 26, LiDAR (Light Detection and Ranging) sensors, RADAR (Radio Detection and Ranging) sensors, SONAR (Sound Navigation and Ranging) sensors, ultrasonic sensors, inertial measurement units (IMUs), and so on.inertial measurement units), wheel speed sensors, and a variety of other sensors 14 capable of determining the position and dynamic state of the vehicle 12 with respect to the vehicle's surroundings 12 without departing from the scope or spirit of the present invention. In several aspects, the sensors 14 may be integrated into the vehicle 12 or installed as part of a service program by the manufacturer of the vehicle 12, a dealer, a customer, or other third parties. In some examples, the one or more sensors 14 may be part of a standalone sensor system 14.
[0028] The controllers 16 are non-generalized electronic controllers having a pre-programmed digital computer or processor 28, a non-transitory computer-readable medium or memory 30 used to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and one or more input / output (I / O) ports 32. Computer-readable media or memory 30 includes any type of media accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, compact disc (CD), digital video disc (DVD), semiconductor memory, or any other type of memory. A "non-transitory" computer-readable medium or memory 30 excludes wireless, optical, or other communication links that carry electrical or other signals.A non-transitory computer-readable medium or storage 30 includes media on which data can be permanently stored and media on which data can be stored and later overwritten, such as any type of program code, including source code, object code, and executable code. The processor 28 is configured to execute the code or instructions. In some examples, the controller 16 may be a dedicated wireless or Wi-Fi controller. The I / O ports 32 are configured to communicate via wired or wireless means, such as the communication network 20, including the V2V communication network and / or V2I communication network, using Wi-Fi protocols conforming to IEEE 802.11x, Bluetooth communication protocols, radio frequency protocols, or the like.
[0029] In several aspects, the controllers 16 include one or more applications 34. An application 34 is a software program configured to perform specific functions or groups of functions. The application 34 may include one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or a portion thereof suitable for implementation in suitable computer-readable program code. The applications 34 may be stored within the memory 30 or in an additional or separate memory 30. Examples of the applications 30 include audio or video streaming services, audio or visual processing services, an AR-assisted parking application (ARAPA) 34, and the like.
[0030] In Fig. 2A, it can be seen that the system 10 includes a human-machine interface (HMI) 36 mounted in the vehicle 12 for interacting with the drivers of the vehicle 12. In several aspects, the HMI 36 includes one or more devices capable of interacting with the driver of the vehicle 12, e.g., a screen disposed in the vehicle 12, such as an instrument cluster 38, an infotainment screen 40, a heads-up display (HUD) 42, or the like. However, it should be understood that other HMIs 34 are also contemplated. The HMI 36 may, for example, be a mobile device such as a tablet computer, a cell phone, or the like, and the HMI 36 may be provided by the operator or driver of the vehicle 12 and temporarily attached to or disposed on a component in the passenger compartment of the vehicle 12.In examples where the HMI 36 is a mobile device, as described above, it will be understood that at least part of the ARAPA 34 is executed on board the mobile device.
[0031] In Fig. 2B is further referred to Fig. 1 and Fig. 2A shows an example ARAPA 34 displayed on an example infotainment screen 40 of the vehicle 12. The ARAPA 34 includes control logic portions that cause the HMI 36 to use a display scheme used in the simultaneous localization and mapping (SLAM)-based parking assistance system for vehicles 12. In several aspects, the ARAPA 34 includes a training session (TS) portion 100 and a live session (LS) portion 200. In the TS portion 100, the ARAPA 34 is trained to recognize feature point neighborhoods (FPVs) 44 of a particular parking spot 46. During the TS, the operator of the vehicle 12 parks the vehicle 12 in a parking space 46 at a parking position 48 selected by the operator of the vehicle 12. The operator of the vehicle 12 then activates the ARAPA 34 via the HMI 36.
[0032] In Fig. 3 is further referred to Fig. 1, Fig. 2A and Fig. 2B, the TS portion 100 of the ARAPA 34 is shown in detail. In a first TS control logic portion 102, the operator of the vehicle 12 indicates the completion of parking the vehicle 12 in the ARAPA 34 within the HMI 36 by selecting a first feature located in the ARAPA 34 that notifies or confirms to the system 10 that the vehicle 12 has been parked in a location preferred by the operator of the vehicle 12. The first feature may take a variety of different forms, including an on-screen button on the HMI 36, an audible input, a tactile input via one or more physical buttons inside the vehicle 12 and in communication with the ARAPA 34, or the like.
[0033] Upon receiving information from the operator of the vehicle 12 that the vehicle 12 has been parked, the ARAPA 34 responds by executing a second TS control logic section 104 that presents one or more options to the operator of the vehicle 12 via the HMI 36. In some aspects, the one or more options may be displayed on the HMI 36 and may include one or more prompts superimposed on a real-time display of the view from one or more cameras 26 collecting visual data about the surroundings of the vehicle 12. The options or prompts in the second TS control logic section 104 include presenting the displayed view from the one or more cameras 26 and a plurality of suggested FPVs 44 found in the visual data and within a predefined distance from the vehicle 12.In several aspects, the proposed FPVs 44 at least partially define a current or idealized position of the vehicle 12 within the parking space 46. It should be noted that the predefined distance may vary from application to application, location to location, and vehicle to vehicle without departing from the scope or spirit of the present invention. In response to the second TS control logic section 104, the operator of the vehicle 12 selects some or all of the proposed FPVs 44.In some examples, the operator of the vehicle 12 selects three to five FPVs 44 with appropriate viewing distances so that all selected FPVs 44 can be displayed simultaneously without risk of falling out of the viewing area, and so that the various FPVs 44 do not overlap within a reasonable line of sight of the operator of the vehicle 12, although the number of selected FPVs 44 and the appropriate distances may vary from application to application, location to location, etc. It should be noted, however, that the appropriate distances are selected to ensure the accuracy and precision of the orientation of the vehicle 12 relative to the FPVs 44.That is, the appropriate distances are generally chosen to reduce the likelihood of three-dimensional inaccuracy or misalignment of the position of the vehicle 12 within the parking space 46 from a first level to a second level lower than the first level when the vehicle 12 is parked autonomously or semi-autonomously in a particular parking space 46 in repeated iterations.
[0034] The ARAPA 34 may also execute a third TS control logic section, the third TS control logic section 106, in parallel, simultaneously, or sequentially with the second TS control logic section 104. In the third TS control logic section 106, the ARAPA 34 suggests FPVs 44 that the system 10 has identified as advantageous and provides the operator of the vehicle 12 with corresponding semantic tags with prompts displayed on the HMI 36. The semantic tags typically include the descriptor texts generated by computer vision modules based on the analysis of the selected FPVs 44, e.g., the Fig. 2B, the selected right FPV 44 may be referred to as "a thing on the wall." The suggested FPVs 44 may include previously identified or selected FPVs 44 at a particular parking position 48 or FPVs 44 corresponding to previously identified or selected FPVs 44 stored in memory 30. From the second TS control logic section 104 and third TS control logic section 106, the ARAPA 34 proceeds to a fourth TS control logic section 108. It should be understood that in certain situations where the system 10 has no discernible environmental features within the field of view of the camera 26, the FPVs 44 may be selected as objects salient to the eye of the operator of the vehicle 12.In several examples, the FPVs 44 define at least two visually recognizable, narrow-shaped static objects, such as two or more boundary lines on white walls, and the selection of the FPVs 44 may be made directly by the manual inputs of the operator of the vehicle 12 on the HMI 36 without these FPVs 44 first being suggested by the system 10.
[0035] During execution of the fourth TS control logic section 108, the operator of the vehicle 12 is offered the opportunity, via the HMI 36, to assign a name to the parking space 46 currently occupied by the vehicle 12. A suggested name for the parking space 46 may also be provided automatically by the ARAPA 34, and the operator of the vehicle 12 may therefore be offered the opportunity to change the suggested name of the parking space 46. A fifth TS control logic section 110 is then executed, in which the name of the parking space 46 assigned to the parking space 46, images, and other necessary data, along with semantic labels, are stored in the memory 30 for use in subsequent live session (LS) actions of the ARAPA 34.
[0036] In Fig. 4 is further referred to Fig. 1, Fig. 2A, Fig. 2B and Fig. 3, the LS section 200 of the ARAPA 34 is shown in more detail in flowchart form. During the LS section 200, the system executes a series of LS control logic sections of the ARAPA 34 that assist the operator of the vehicle 12 in parking the vehicle 12 during a live session. Specifically, the operator of the vehicle 12 drives the vehicle 12 to a parking position 48 where one or more parking spaces 46 are available. As the vehicle 12 travels through the area with one or more available parking spaces 46, the ARAPA 34 identifies the parking spaces 46. In a first LS control logic section 202, the ARAPA 34 determines that the current distance to a known parking space 46 is less than or equal to a predefined preview threshold distance of the FPV 44 from the vehicle 12. The predefined preview threshold of the FPV 44 is a predefined distance from the vehicle 12 to the known or desired parking location 46.The predefined preview threshold of the FPV 44 may vary from application to application, from vehicle 12 to vehicle 12, and / or from location to location. However, in some examples, the preview threshold of the FPV 44 is at a variable distance between approximately six meters and approximately ten meters from the vehicle 12.
[0037] After determining that the known parking spot 46 is within the FPV 44 preview threshold distance of a previously trained parking spot 46 from the second LS control logic section 204, the ARAPA 34 presents one or more prompts to the operator of the vehicle 12 indicating that the vehicle 12 is within the FPV 44 preview threshold distance of the previously trained parking spot 46 from the second LS control logic section 204 and offers one or more previously trained parking spots 46 for the operator of the vehicle 12 to select for parking. The operator of the vehicle 12 selects one of the parking spots 46 and then provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle 12 to the previously trained parking spot 46 found by the second LS control logic section 204.While the operator of the vehicle 12 is driving the vehicle 12, the system 10 executes a third LS control logic portion 206 of the ARAPA 34 that provides AR assistance by overlaying the FPVs 44 over the live data from the camera 26 displayed on the HMI 36 or by displaying the FPVs 44 directly on the HUD 42 such that the FPVs 44 are displayed directly in the line of sight of the operator of the vehicle 12 and at positions that correspond to the physical positions of the FPVs 44 from the visual perspective of the operator of the vehicle 12.
[0038] Similarly, the system 10 may execute a fourth LS control logic section 208 when the system 10 determines that the vehicle 12 is within the preview threshold distance of the FPV 44 from one or more previously stored FPVs 44 of parking spots 46 at the same general parking position 48 (i.e., parking spot), and where the previously stored FPVs 44 are of sufficient reliability that they can be used to designate the physical location and three-dimensional parameters of a parking spot 46. Upon identifying previously stored FPVs 44 of parking spots 46 that were not previously selected by the operator of the vehicle 12, the system 10 executes a fifth LS control logic section 210 that, on a case-by-case basis, displays a preview of the identified, previously stored FPVs 44 on the HUD 42 and / or on the HMI 36.More specifically, if the previously stored FPVs 44 can be identified before the vehicle 12 enters a location that is at a distance equal to or less than a predefined alignment threshold distance of the FPV 44 to a previously selected parking spot 46, the system 10 optionally displays a preview of the identified, previously stored FPVs 44 to serve as an enhanced guidance aid to the operator of the vehicle 12. The predefined alignment threshold of the FPV 44 may vary from application to application, from vehicle 12 to vehicle 12, and / or from location to location. In some examples, the alignment threshold of the FPV 44 is generally less than the alignment threshold of the FPV 44 and may range from about one and a half (1.5) to about three (3) meters.
[0039] The operator of the vehicle 12 may select a parking spot 46 within the identified, previously stored FPVs 44 via a selection or prompt displayed by the fifth LS control logic section 210, and then navigate or otherwise guide the vehicle 12 to a parking spot 46 designated by the previously stored FPVs 44. While the operator of the vehicle 12 is driving the vehicle 12, the system 10 executes a sixth LS control logic section 212 of the ARAPA 34, which provides AR assistance by overlaying the FPVs 44 with live data from the camera 26 displayed on the HMI 36 or by directly displaying the FPVs 44 on the HUD 42, such that the FPVs 44 are displayed directly in the line of sight of the operator of the vehicle 12 and at positions that correspond to the physical positions of the FPVs 44 from the visual perspective of the operator of the vehicle 12.In several aspects, a path to the identified, previously stored FPVs 44 may be displayed on the HUD and / or HMI 36 in a different style, color, layout, or the like than a path to a previously trained parking spot 46.
[0040] The system 10 then executes a seventh LS control logic section 214 that determines a current distance to the selected parking location 46 and compares the current distance to the predefined FPV 44 alignment threshold before activating an FPV 44 alignment mode 216 of the ARAPA 34.
[0041] In the FPV 44 alignment mode 216, the system 10 executes a series of LS control logic sections that assist the operator of the vehicle 12 in correctly and completely positioning the vehicle 12 within the selected parking space 46. More specifically, the FPV 44 alignment mode 216 assists the operator of the vehicle 12 by executing an eighth LS control logic section 218 that uses AR to display previously identified or stored FPVs 44 on the HMI 36 and / or the HUD 42, thereby providing the operator of the vehicle 12 with references that the operator of the vehicle 12 uses to align the vehicle 12 within the parking space 46 in real time.The system 10 also executes a ninth LS control logic 220 of the ARAPA 34 in which the operator of the vehicle 12 checks the orientations of the stored FPVs 44 with corresponding features of the real environment for improved parking assistance and steers or otherwise maneuvers the vehicle 12.
[0042] In several aspects, in either the eighth or ninth LS control logic section 218, 220 in alignment mode 216, when using the HMI 36 for alignment, stored images of the FPV 44 are displayed at original pixel positions and overlaid on a real-time view of the area around the vehicle 12 from the camera 26. The arrangements of the foreground, background, and relevant overlays are configured accordingly so that the system 10 can clearly display how accurately the stored images of the FPV 44 are aligned with the respective live FPVs 44 of the same reference points. Likewise, when using AR via the HUD 42 for alignment, the HUD 42 displays the positions and projected shapes of the stored images of the FPV 44 calculated based on a driver-monitoring system (DMS) view in the line of sight of the driver of the vehicle 12.The stored images of the FPV 44 are then displayed with appropriate brightness depending on the ambient light and the real-time settings of the operator of the vehicle 12, so that the HUD 42 clearly indicates how accurately the stored images of the FPV 44 are aligned with the respective live environmental objects that the operator of the vehicle 12 is viewing directly through the windshield of the vehicle 12. The DMS may include a plurality of sensors 14 mounted on or within the vehicle 12 that monitor the occupants of the vehicle 12, particularly the operators of the vehicle 12. The sensors 14 of the DMS may include cameras 26, thermal sensors, sensors capable of registering or detecting haptic inputs, infrared sensors, and the like.The sensors 14 of the DMS can detect the seat position, head position, eye position, hand position, or the like of the operator of the vehicle 12 and use this information to determine how the operator of the vehicle 12 actively controls or maneuvers the vehicle 12 and the vehicle functions.
[0043] In a tenth LS control logic section 222, the system 10 notifies the operator of the vehicle 12 that the vehicle 12 has been fully parked when the stored images of the FPV 44 are sufficiently aligned with real FPVs 44 or surrounding objects, as viewed on the HMI 36 or the HUD 42, or both. The system 10 and / or the operator of the vehicle 12 may determine that the vehicle 12 has been fully parked and aligned in the parking spot 46 when the longitudinal and lateral positions of the stored FPVs 44 are closely aligned with the real FPVs 44 or surrounding objects. The term "sufficiently aligned" is typically an intuitive judgment by the operator or an evaluation by the system as to whether the stored FPVs achieve a sufficiently high intersection-over-union (IOU) metric with their real-world counterparts, e.g.,an IOU not below the threshold of 95% of the operator's visual observation.
[0044] In Fig. 5 is further referred to Fig. 1, Fig. 2A, Fig. 2B, Fig. 3 and Fig.4 illustrates a method 300 for AR-assisted parking with operator-assisted selections and orientations of the FPVs 44 in flowchart form. The method 300 begins in block 302. In block 304, the operator of the vehicle 12 initiates a training session of the ARAPA 34 by maneuvering the vehicle 12 into a new parking space 46 and indicating completion of a parking session via the HMI 36. In block 306, the system 10 displays real-time views of the camera 26 on the HMI 36, which are intended for the selection of the FPV 44, and provides the operator of the vehicle 12 with corresponding instructions or prompts. In block 308, which may be executed concurrently with, in parallel with, or after the processes executed in block 306, the ARAPA 34 suggests FPVs 44 that the system 10 has identified as advantageous and provides the operator of the vehicle 12 with corresponding semantic labels with prompts on the HMI 36.The suggested FPVs 44 may include previously identified or selected FPVs 44 at a particular parking space 46, or FPVs 44 corresponding to previously identified or selected FPVs 44 stored in memory 30. From blocks 306 and 308, the method 300 proceeds to block 310, where the operator of the vehicle 12 is provided, via the HMI 36, with the opportunity to assign a name to the parking space 46 currently occupied by the vehicle 12. A suggested name for the parking space 46 may also be automatically provided by the ARAPA 34, and the operator of the vehicle 12 may therefore be provided with the opportunity to change the suggested name of the parking space 46.From block 310, the method 300 proceeds to block 312 where the parking location name, images, and other necessary data assigned to the parking location 46 are stored in memory 30 along with semantic labels for use in subsequent live session (LS) actions of the ARAPA 34.
[0045] Starting at block 312, the method 300 transitions into a live session of the ARAPA 34, which begins at block 314. Specifically, at block 314, the operator of the vehicle 12 drives the vehicle 12 to a parking position 48 in which one or more parking spaces 46 are available. As the vehicle 12 drives through the parking position 48 with one or more available parking spaces 46, the ARAPA 34 identifies the parking spaces 46 and compares them to a predefined preview threshold distance of the FPV 44. That is, at block 314, the ARAPA 34 determines whether the current distance to a known parking space 46 is less than or equal to a predefined preview threshold distance of the FPV 44 from the vehicle 12. The predefined preview threshold of the FPV 44 is a predefined distance from the vehicle 12 to the known or desired parking space 46.The predefined preview threshold of the FPV 44 may vary from application to application, from vehicle 12 to vehicle 12, and / or from location to location. However, in some examples, the preview threshold of the FPV 44 is at a variable distance between approximately six meters and approximately ten meters from the vehicle 12.
[0046] If it is determined that the known parking spot 46 is greater than the preview threshold distance of the FPV 44, the method 300 proceeds to block 316 where the system 10 continues to search for parking spots 46 before proceeding again to block 314.
[0047] If the known parking spot 46 is determined to be equal to, less than, or equal to the preview threshold of the FPV 44, the system 10 presents one or more prompts to the operator of the vehicle 12 via the HMI 36. More specifically, the ARAPA 34 presents one or more previously trained parking spots 46 to the operator of the vehicle 12 at block 318.
[0048] If the system 10 determines that the vehicle 12 is within the preview threshold distance of the FPV 44 of a previously trained parking spot 46 from block 318, the ARAPA 34 presents one or more prompts to the operator of the vehicle 12 indicating that the vehicle 12 is within the preview threshold distance of the FPV 44 of the previously trained parking spot 46 from block 318. The operator of the vehicle 12 then provides directional commands through steering inputs, braking inputs, and acceleration inputs to guide the vehicle 12 to the previously trained parking spot 46 found in block 318.While the operator of the vehicle 12 is driving the vehicle 12, the method 300 proceeds to block 320 where the ARAPA 34 provides AR assistance by overlaying the FPVs 44 over the live data from the camera 26 displayed on the HMI 36 or by displaying the FPVs 44 directly on the HUD 42 such that the FPVs 44 are displayed directly in the line of sight of the operator of the vehicle 12 and at positions that correspond to the physical positions of the FPVs 44 from the visual perspective of the operator of the vehicle 12.
[0049] The method 300 also proceeds to blocks 322 through 326, which may be performed in parallel, concurrently, or sequentially with the steps of the method 300 of blocks 318 and 320. In block 322, the system 10 determines that the vehicle 12 is within the preview threshold distance of the FPV 44 from one or more previously stored FPVs 44 of the parking spot 46 at the same general parking position 48 (i.e., parking spot), and where the previously stored FPVs 44 are of sufficient reliability that they can be used to designate the physical location and three-dimensional parameters of a parking spot 46.After identifying previously stored FPVs 44 from parking locations 46 that were not previously selected by the operator of the vehicle 12, the method 300 proceeds to block 324, and the system 10 causes the ARAPA 34 to display, on a case-by-case basis, a preview of the identified, previously stored FPVs 44 on the HUD 42 and / or on the HMI 36. More specifically, the previously stored FPVs 44 may be identified before the vehicle 12 enters a position within a predefined alignment threshold of the FPV 44 to a previously selected parking location 46, with the system 10 optionally displaying a preview of the identified, previously stored FPVs 44 to serve as an enhanced guidance aid to the operator of the vehicle 12. The predefined alignment threshold of the FPV 44 may vary from application to application, vehicle 12 to vehicle 12, and / or location to location.In some examples, the alignment threshold of the FPV 44 is generally less than the alignment threshold of the FPV 44 and may range from about one and a half (1.5) to about three (3) meters.
[0050] The operator of the vehicle 12 may select a parking spot 46 within the identified, previously stored FPVs 44 via a selection or prompt displayed at block 324, and then navigate or otherwise guide the vehicle 12 to a parking spot 46 identified by the previously stored FPVs 44. While the operator of the vehicle 12 is driving the vehicle 12, the method 300 proceeds to block 326, and the ARAPA 34 provides AR assistance by overlaying the FPVs 44 over the live data from the camera 26 displayed on the HMI 36 or by displaying the FPVs 44 directly on the HUD 42 such that the FPVs 44 are displayed directly in the line of sight of the operator of the vehicle 12 and at positions that correspond to the physical positions of the FPVs 44 from the visual perspective of the operator of the vehicle 12.In several aspects, a path to the identified, previously stored FPVs 44 may be displayed on the HUD and / or HMI 36 in a different style, color, layout, or the like than a path to a previously trained parking spot 46.
[0051] The method 300 then proceeds to block 328 and causes the system 10 to determine a current distance to the selected parking space 46 and compares the current distance to the predefined FPV 44 alignment threshold before activating the FPV 44 alignment mode 216 of the ARAPA 34. If the current distance of the vehicle 12 to the selected parking space 46 is greater than the FPV 44 alignment threshold distance, the method 300 returns to blocks 320 and / or 326. However, if the current distance of the vehicle 12 to the selected parking space 46 is less than or equal to the predefined FPV 44 alignment threshold distance, the method activates the FPV 44 alignment mode 216.
[0052] In the alignment mode 216 of the FPV 44, the method 300 assists the operator of the vehicle 12 in correctly and completely positioning the vehicle 12 within the selected parking space 46. More specifically, the method 300 utilizes the alignment mode 216 of the FPV 44 to assist the operator of the vehicle 12 in block 330 by using AR to display previously identified or stored FPVs 44 on the HMI 36 and / or the HUD 42, thereby providing the operator of the vehicle 12 with references that the operator of the vehicle 12 uses to align the vehicle 12 within the parking space 46 in real time.
[0053] In block 332, the operator of the vehicle 12 checks the alignments of the stored FPVs 44 with the corresponding features of the real-world environment for improved parking assistance and steers or otherwise maneuvers the vehicle 12. In either of blocks 330 and 332, in alignment mode 216, when using the HMI 36 for alignment, the method 300 causes the ARAPA 34 to display stored images of the FPV 44 at original pixel positions on the HMI 36 overlaid on a real-time view of the area around the vehicle 12 from the camera 26. The arrangements of the foreground, background, and relevant overlays are configured accordingly so that the system 10 can clearly display how accurately the stored images of the FPV 44 are aligned with the respective live FPVs 44 of the same reference points.Likewise, when using AR via the HUD 42 for alignment, the HUD 42 displays the positions and projected shapes of the stored images of the FPV 44, which were calculated based on a view of the driver-monitoring system (DMS) in the line of sight of the driver of the vehicle 12. The stored images of the FPV 44 are then displayed with appropriate brightness depending on the ambient light and the real-time setting of the operator of the vehicle 12, so that the HUD 42 clearly indicates how accurately the stored images of the FPV 44 are aligned with the respective live environmental objects that the operator of the vehicle 12 is viewing directly through the windshield of the vehicle 12. The DMS may include a plurality of sensors 14 mounted on or within the vehicle 12 that monitor the occupants of the vehicle 12, particularly the operators of the vehicle 12.The sensors 14 of the DMS may include cameras 26, thermal sensors, sensors capable of registering or detecting haptic inputs, infrared sensors, and the like. The sensors 14 of the DMS may detect the seating position, head position, eye position, hand position, or the like of the operator of the vehicle 12 and use this information to determine how the operator of the vehicle 12 actively controls or maneuvers the vehicle 12 and the vehicle functions.
[0054] From blocks 330 and / or 332, the method 300 proceeds to block 334, where the operator of the vehicle 12 determines that the vehicle 12 has been fully parked when the stored images of the FPV 44 are sufficiently aligned with real FPVs 44 or surrounding objects as viewed on the HMI 36 or the HUD 42, or both. The operator of the vehicle 12 may determine that the vehicle 12 has been fully parked and aligned in the parking space 46 when the longitudinal and lateral positions of the stored FPVs 44 are closely aligned with the real FPVs 44 or surrounding objects. The method 300 ends at block 336. In several aspects, the method 300 may operate continuously, iteratively, or only upon activation by the operator of the vehicle 12 and / or by an automated parking application or the like.As mentioned above, the term “sufficiently aligned” is the intuitive judgment of the operator or the system’s assessment of whether the stored FPVs achieve a sufficiently high intersection-over-union (IOU) metric with their real-world counterparts, e.g., the IOU is not below the 95% threshold as viewed by the operator’s eyes.
[0055] A system 10 and method 300 for AR-assisted parking with operator selection and orientation of the FPVs 44 in the decision chain offers several advantages. These include the ability to use an AR display scheme for SLAM-based vehicle parking assistance 12 that prevents robustness degradation due to the removal or increase of feature points at a particular geographic location, as well as accounting for changes in feature descriptors due to changes in lighting conditions.Furthermore, the system 10 and method 300 of the present invention require minimal input from the operator of the vehicle 12 in observing and selecting environmental features and reduce the need for a wealth of feature points for the underlying SLAM processes, while ensuring that the parking assistance system effectively and accurately represents or indicates the longitudinal and lateral deviations of the vehicle 12 relative to the FPVs 44, thereby simplifying parking operations and increasing the confidence of the operator of the vehicle 12 while maintaining or reducing the complexity of the system 10.
Claims
[1] A system (10) for augmented reality (AR)-assisted parking of a vehicle (12) with operator-assisted selections and orientations of feature point environments (FPVs) (44), the system (10) comprising: a vehicle (12); one or more human-machine interfaces (HMIs) (36) arranged in the vehicle (12); one or more sensors (14) mounted on the vehicle (12) and detecting FPVs (44) of a parking position (48); one or more controllers (16), each of the one or more controllers (16) having a processor (28), a memory (30), and one or more input / output (I / O) ports (32), the I / O ports (32) communicating with the one or more sensors (14) and the one or more HMIs (36); the memory (30) storing programmatic control logic; the processor (28) executing the programmatic control logic; the programmatic control logic including an AR-assisted parking application (ARAPA) (34), an operator of the vehicle (12) activating the ARAPA (34) via the HMI (36) when completing parking in a parking space (46) at the parking position (48); the ARAPA (34) comprising: a training session (TS) section (100) comprising a first TS control logic section (102) in which the operator of the vehicle (12) indicates that the vehicle (12) has been parked at a preferred parking location (46) by selecting a first feature of the ARAPA (34) via the HMI (36); and a Live Session (LS) section (200), wherein in the TS section (100) the ARAPA (34) is trained to detect FPVs (44) of the parking location (46) at the parking position (48); and wherein in the LS section (200), the ARAPA (34) prompts the operator of the vehicle (12) to select the parking space (46) and guides the operator of the vehicle (12) into the parking space (46) by displaying previously trained FPVs (44) superimposed on the live sensor data displayed on the HMI (36), and instructs the operator of the vehicle (12) to align the previously trained FPVs (44) with corresponding real FPVs (44) of the parking space (46) by maneuvering the vehicle (12);characterized by , that TS section (100) further includes: a second TS control logic section (104), wherein the ARAPA (34) uses the HMI (36) to present a plurality of proposed FPVs (44) superimposed on a real-time display of a view from the one or more sensors (14), the proposed FPVs (44) at least partially defining a current or idealized position of the vehicle (12) within the preferred parking location (46); and Control logic receiving inputs via the HMI (36) from the operator of the vehicle (12) selecting some or all of the plurality of suggested FPVs (44), wherein each of the suggested FPVs (44) has suitable distances from other suggested FPVs, wherein the suitable distance is selected such that all selected FPVs (44) can be displayed simultaneously on the HMI (36) and such that different FPVs (44) do not overlap within any line of sight of the operator of the vehicle (12); and wherein the suitable distance reduces a potential for misalignment of the vehicle (12) within the parking space (46) from a first level to a second level that is lower than the first level when the vehicle (12) is parked. [2] The system (10) of claim 1, wherein the TS section (100) further comprises: a third TS control logic section (106) in which the ARAPA (34) proposes, via the HMI (36), FPVs (44) and semantic labels corresponding to parking locations (46) previously identified by the system (10) at the parking position (48). [3] The system (10) of claim 2, further comprising: a fourth TS control logic section (108) that offers automatically generated suggestions for parking space names and options for the operator of the vehicle (12) to change the suggested parking space names via the HMI (36); and a fifth TS control logic section (110) that stores the proposed parking location names and semantic labels in the memory (30) for use in the LS section (200) of the ARAPA (34). [4] A system (10) for augmented reality (AR)-assisted parking of a vehicle (12) with operator-assisted selections and orientations of feature point environments (FPVs) (44), the system (10) comprising: a vehicle (12); one or more human-machine interfaces (HMIs) (36) arranged in the vehicle (12); one or more sensors (14) mounted on the vehicle (12) and detecting FPVs (44) of a parking position (48); one or more controllers (16), each of the one or more controllers (16) having a processor (28), a memory (30), and one or more input / output (I / O) ports (32), the I / O ports (32) communicating with the one or more sensors (14) and the one or more HMIs (36); the memory (30) storing programmatic control logic; the processor (28) executing the programmatic control logic; the programmatic control logic including an AR-assisted parking application (ARAPA) (34), an operator of the vehicle (12) activating the ARAPA (34) via the HMI (36) when completing parking in a parking space (46) at the parking position (48); the ARAPA (34) comprising: a training session (TS) section (100); and a Live Session (LS) section (200); wherein in the TS section (100) the ARAPA (34) is trained to detect FPVs (44) of the parking location (46) at the parking position (48); wherein in the LS section (200), the ARAPA (34) prompts the operator of the vehicle (12) to select the parking space (46) and guides the operator of the vehicle (12) into the parking space (46) by displaying previously trained FPVs (44) superimposed on the live sensor data displayed on the HMI (36), and instructs the operator of the vehicle (12) to align the previously trained FPVs (44) with corresponding real FPVs (44) of the parking space (46) by maneuvering the vehicle (12); and wherein the LS section (200) further comprises: a first LS control logic section (202) that determines that a current distance between the vehicle (12) and one or more previously trained parking locations (46) is less than or equal to a predefined FPV preview threshold distance, wherein the FPV preview threshold distance is between approximately six meters and approximately ten meters; and a second LS control logic section (204) that offers one or more prompts to the operator of the vehicle (12) via the HMI (36), the one or more prompts requesting the operator of the vehicle (12) to select one of the one or more previously trained parking locations (46) within the FPV preview threshold distance within which the operator of the vehicle (12) intends to park; characterized by , that the LS section (200) further comprises a third LS control logic section (206) that provides augmented reality (AR) assistance to the operator of the vehicle (12) when the operator of the vehicle (12) provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle (12) toward and into the previously trained parking spot, wherein the AR assistance overlays FPVs (44) of the previously trained parking spot with one or more of the live sensor data displayed on the HMI (36) and on a head-up display (HUD) (42) of the vehicle (12) such that the FPVs (44) are displayed directly in a line of sight of the operator of the vehicle (12) and in positions that correspond to the actual physical positions of the FPVs (44) from a visual perspective of the operator of the vehicle (12). [5] The system (10) of claim 4, wherein the LS section (200) further comprises: a fourth LS control logic (208) that determines that the vehicle (12) is within the FPV preview threshold distance from one or more previously stored FPVs (44) of parking locations (46) at a parking position (48); and a fifth LS control logic (210) that displays a preview of the identified, previously stored FPVs (44) on the HMI (36) and / or the head-up display (HUD) (42) on a case-by-case basis and uses previously identified FPVs (44) as enhanced steering guidance for the operator of the vehicle (12) to align the vehicle (12) within a parking space (46) to a distance equal to or less than a predefined FPV alignment threshold distance. [6] The system (10) of claim 5, wherein the LS section (200) further comprises: a sixth LS control logic (212) that provides AR assistance to the operator of the vehicle (12) when the operator of the vehicle (12) provides directional inputs via steering adjustments, braking inputs, and acceleration inputs to guide the vehicle (12) to and into the parking spot (46) identified by stored, previously identified FPVs (44), wherein the AR assistance overlays the stored, previously identified FPVs (44) on one or more of live sensor data displayed on the HMI (36) and on the HUD (42) of the vehicle (12) such that the FPVs (44) are displayed directly in the line of sight of the operator of the vehicle (12) and in positions that correspond to the actual physical positions of the FPVs (44) from the visual perspective of the operator of the vehicle (12), and wherein a path to the stored, previously identified FPVs (44) is displayed on the HUD (42) and / or the HMI (36) in a different style,a different color or layout compared to a route to a previously trained parking space (46). [7] The system (10) of claim 6, wherein the LS section (200) further comprises: a seventh LS control logic (214) that determines that a current distance of the vehicle (12) to a parking location (46) is less than or equal to the FPV alignment threshold distance and activates an FPV alignment mode of the ARAPA (34); and an eighth LS control logic (216) that uses AR to display previously identified or stored FPVs (44) on the HMI (36) and / or the HUD (42), thereby providing the operator of the vehicle (12) with references that the operator of the vehicle (12) uses to orient the vehicle (12) within the parking space (46) in real time; a ninth LS control logic (220) that causes the operator of the vehicle (12) to check alignments of the stored FPVs (44) with corresponding features of a real environment for improved parking assistance and to steer or otherwise maneuver the vehicle (12); and a tenth LS control logic (222) that notifies the operator of the vehicle (12) that the vehicle (12) has been fully parked when the stored FPVs are aligned with real FPVs (44) or environmental objects as displayed on the HMI (36) and / or the HUD (42).
Citation Information
Patent Citations
imaging system for locating a moving object in relation to another object
DE102016108718A1
vehicle environment imaging systems AND METHODS
DE102017128294A1
Motor vehicle with a drain arranged on an underbody and method for operating a motor vehicle
DE102021005530A1
INTELLIGENT VEHICLES, CONTROL LOGIC AND ADVANCED PARKING ASSISTANCE SYSTEMS WITH CAMERA-BASED AUTOMATIC VEHICLE ALIGNMENT
DE102021100548A1
