Augmented reality assisted parking system with driver-in-loop selection and alignment of FPV

Abstract

An augmented reality (AR) assisted vehicle parking system for operator-in-the-loop selection and alignment with feature point proximity (FPV) includes a vehicle, a human-machine interface (HMI), a sensor to detect a parking position (FPV), and a controller. The controller has a processor, a memory, and an input / output (I / O) port in communication with the HMI and the sensor. The memory stores an AR assisted parking application (ARAPA) executed by the processor. The AR-assisted parking application (ARAPA) has a training phase (TS) and a real-time phase (LS) and is activated in the event of completion of parking in a parking space of a parking position. The TS section trains the ARAPA to identify the FPV of the parking space. The LS portion prompts an operator to select a parking space, guides the operator into the parking space by displaying the trained FPV on real-time sensor data displayed on the HMI, and instructs the operator to align the trained FPV with a corresponding real-world FPV by operating the vehicle.

Description

Technical Field

[0001] The present disclosure relates to driver assistance systems for vehicles, and more particularly to driver assistance systems that assist a vehicle operator during a vehicle parking procedure. Background Art

[0002] In many vehicles, driver assistance systems are able to autonomously or semi-autonomously park the vehicle in a parking space near the vehicle. Typical driver assistance systems used in the vehicle parking process use a set of sensors equipped on the vehicle to generate a path to one or more parking spaces. However, such systems often have difficulty determining the path due to the perception capabilities of the set of sensors, computing power limitations, etc.

[0003] Therefore, while current systems and methods for autonomously or semi-autonomously parking vehicles achieve their intended purposes, a new and improved system and method is needed to overcome the fidelity issues of sensor data and increase vehicle operator confidence and comfort without significantly increasing the computational burden or increasing system and / or component complexity. Summary of the invention

[0004] According to some aspects of the present disclosure, an augmented reality (AR) assisted vehicle parking system for operator in-loop selection and alignment of feature point vicinities (FPV) includes a vehicle and one or more human-machine interfaces (HMI) disposed in the vehicle. The system also includes one or more sensors disposed on the vehicle to detect the FPV of the parking position, and one or more controllers. Each of the one or more controllers has a processor, a memory, and one or more input / output (I / O) ports. The I / O port communicates with the one or more sensors and the one or more HMIs. The memory stores program control logic, and the processor executes the program control logic. The program control logic includes an AR-assisted parking application (AR-assistedParking Application, ARAPA). After the vehicle operator completes parking in a parking space at the parking position, ARAPA is activated through the HMI. ARAPA includes a training session (TS) portion and a live session (LS) portion. In the TS portion, ARAPA is trained to identify the FPV of the parking space. In the LS section, ARAPA prompts the vehicle operator to select a parking space and guides the vehicle operator into the parking space by superimposing the pre-trained FPV on the real-time sensor data displayed on the HMI. In the LS section, ARAPA instructs the vehicle operator to align the pre-trained FPV with the corresponding real-world FPV of the parking space by operating the vehicle.

[0005] In another aspect of the present disclosure, the TS portion further includes a first TS control logic portion, wherein a vehicle operator selects a first feature of ARAPA via the HMI to indicate that the vehicle has been parked in a preferred parking space.

[0006] In another aspect of the present disclosure, the TS portion also includes a second TS control logic portion, wherein ARAPA utilizes an HMI to present multiple suggested FPVs overlaid on a real-time display view of one or more sensors. The suggested FPVs at least partially define the current position or ideal position of the vehicle in a preferred parking space. The second TS control logic portion also includes control logic that receives input from a vehicle operator through the HMI, and the input is used to select some or all of the multiple suggested FPVs. Each suggested FPV has an appropriate interval with other suggested FPVs, wherein the appropriate interval is selected so that all selected FPVs can be displayed on the HMI simultaneously and so that different FPVs do not overlap at any line of sight of the vehicle operator. When the vehicle is parked, the appropriate interval reduces the possibility of the vehicle being misaligned in the parking space from a first level to a second level, and the second level is lower than the first level.

[0007] In yet another aspect of the present disclosure, the TS portion further includes a third TS control logic portion, wherein the ARAPA suggests an FPV and a semantic label through the HMI, the FPV and the semantic label corresponding to a parking space pre-identified by the system at the parking position.

[0008] In another aspect of the present disclosure, the TS portion also includes a fourth TS control logic portion, which provides automatically generated suggested parking space names and an opportunity for the vehicle operator to change the suggested parking space names through the HMI; and a fifth TS control logic portion, which stores the suggested parking space names and semantic labels in the memory for use by the LS portion of ARAPA.

[0009] In another aspect of the present disclosure, the LS portion further includes a first LS control logic portion that determines that the current distance between the vehicle and one or more pre-trained parking spaces is less than or equal to a predefined FPV preview threshold distance. The FPV preview threshold distance is between about 6 meters and about 10 meters. The LS portion further includes a second LS control logic portion that provides one or more prompts to the vehicle operator through the HMI. The one or more prompts request the vehicle operator to select a parking space that the vehicle operator is preparing to park in from one or more pre-trained parking spaces within the FPV preview threshold distance.

[0010] In another aspect of the present disclosure, the LS portion further includes a third LS control logic portion that provides AR assistance to the vehicle operator when the vehicle operator provides direction input through steering adjustments, and provides braking input and acceleration input to guide the vehicle to and into a pre-trained parking space. The AR assistance superimposes the FPV of the pre-trained parking space on one or more real-time sensor data displayed on the vehicle HMI and Heads-Up Display (HUD), so that the FPV is displayed directly in the line of sight of the vehicle operator and, from the optical perspective of the vehicle operator, at a location corresponding to the actual physical location of the FPV.

[0011] In another aspect of the present disclosure, the LS portion further includes a fourth LS control logic that determines that the vehicle is within an FPV preview threshold distance, the FPV preview threshold distance being a threshold distance of one or more pre-saved FPVs, the pre-saved FPVs being the FPVs of a parking spot in the parking location. The LS portion further includes a fifth LS control logic that conditionally displays a preview of the identified pre-saved FPV on one or more of an HMI and a head mounted display (HUD), and uses the pre-identified FPV as an enhanced steering guidance for the vehicle operator to align the vehicle within the parking spot to a distance equal to or less than a predefined FPV alignment threshold distance.

[0012] In another aspect of the present disclosure, the LS portion further includes a sixth LS control logic that provides AR assistance to the vehicle operator when the vehicle operator provides direction input through steering adjustments, and provides braking input and acceleration input to guide the vehicle toward and into a parking space identified by a saved pre-identified FPV. The AR assistance superimposes the saved pre-identified FPV on one or more real-time sensor data displayed on the vehicle HMI and HUD, so that the FPV is displayed directly in the line of sight of the vehicle operator and, from the optical perspective of the vehicle operator, at a location corresponding to the actual physical location of the FPV, and wherein the path to the saved pre-identified FPV is displayed on one or more of the HUD and the HMI in a different style, color, or layout than the path to the pre-trained parking space.

[0013] In another aspect of the present disclosure, the LS portion also includes a seventh LS control logic that determines that the current distance of the vehicle to the parking space is less than or equal to the FPV alignment threshold distance, and uses the FPV alignment mode of ARAPA. The LS portion also includes an eighth LS control logic that uses AR to display a pre-identified or saved FPV on one or more of the HMI and HUD, thereby providing a reference to the vehicle operator, and the vehicle operator uses the reference to align the vehicle in real time within the parking space. The LS portion also includes a ninth LS control logic that enables the vehicle operator to check the alignment of the saved FPV with the corresponding features of the real-world environment to enhance parking assistance and steering, or otherwise operate the vehicle. The LS portion also includes a tenth LS control logic that notifies the vehicle operator that the vehicle is fully parked when the saved FPV is aligned with the real-world FPV or environmental objects seen on the HMI and / or HUD.

[0014] In another aspect of the present disclosure, a method for AR-assisted vehicle parking using operator-in-the-loop selection and alignment of FPV includes using one or more sensors located on the vehicle to detect the FPV of the parking position. The method also includes executing program control logic stored in the memory of one or more controllers, which communicate with the one or more sensors, and communicate with one or more HMIs arranged in the vehicle. Each of the one or more controllers has a processor, a memory, and one or more I / O ports. The I / O port communicates with one or more sensors and one or more HMIs. The processor executes control logic including ARAPA. The method also includes activating ARAPA by the vehicle operator through the input of the HMI after the parking space at the parking position is completed. In the TS part, ARAPA is trained to identify the parking space FPV at the parking position. In the LS part of ARAPA, the vehicle operator is prompted to select a parking space. The method also includes guiding the vehicle operator into the parking space by overlaying the pre-trained FPV on the real-time sensor data displayed by the HMI for display, and instructing the vehicle operator to align the pre-trained FPV with the real-world FPV by operating the vehicle.

[0015] In yet another aspect of the present disclosure, the TS portion further includes indicating, via a vehicle operator's first feature input in the ARAPA of the HMI, that the vehicle has been parked in the preferred parking space.

[0016] In another aspect of the present disclosure, the TS portion also includes presenting a plurality of suggested FPVs overlaid on a real-time display view of one or more sensors through an HMI. The suggested FPVs at least partially define the current position or ideal position of the vehicle within a preferred parking space. The method also includes receiving input from a vehicle operator through the HMI, the input being used to select some or all of the plurality of suggested FPVs. Each suggested FPV has an appropriate interval with the other FPVs, wherein the appropriate interval is selected so that all selected FPVs can be displayed on the HMI simultaneously and so that different FPVs do not overlap at any line of sight of the vehicle operator. When the vehicle is parked, the appropriate interval reduces the likelihood of the vehicle being misaligned within the parking space from a first level to a second level, the second level being lower than the first level.

[0017] In yet another aspect of the present disclosure, the TS portion of the method further includes suggesting to the vehicle operator, via the HMI, an FPV and a semantic label corresponding to a parking space pre-identified by ARAPA at the parking location. The TS portion of the method further includes providing, via the HMI, an automatically generated suggested parking space name and an opportunity for the vehicle operator to change the suggested parking space name, and storing the suggested parking space name and the semantic label in a memory for use by the LS portion of ARAPA.

[0018] In another aspect of the present disclosure, the LS portion of the method further includes determining that the current distance between the vehicle and one or more pre-trained parking spaces is less than or equal to a pre-defined FPV preview threshold distance. The FPV preview threshold distance is between about 6 meters and about 10 meters. The LS portion of the method further includes providing one or more prompts to the vehicle operator through the HMI. The one or more prompts request the vehicle operator to select a parking space that the vehicle operator is preparing to park in from one or more pre-trained parking spaces within the FPV preview threshold distance.

[0019] In yet another aspect of the present disclosure, the LS portion of the method further includes providing AR assistance to the vehicle operator when the vehicle operator provides directional input through steering adjustments, and provides braking input and acceleration input to guide the vehicle toward and into a pre-trained parking space. The AR assistance overlays the FPV of the pre-trained parking space on one or more real-time sensor data displayed on the HMI and HUD, so that the FPV is displayed directly in the line of sight of the vehicle operator and, from the optical perspective of the vehicle operator, at a location corresponding to the actual physical location of the FPV.

[0020] In another aspect of the present disclosure, the LS portion of the method further includes determining that the vehicle is within a FPV preview threshold distance, the FPV preview threshold distance being one or more pre-saved FPV threshold distances, the pre-saved FPV being the FPV of a parking space in the parking location. And conditionally displaying a preview of the pre-saved FPV on one or more of the HMI and the HUD. The LS portion of the method also uses the pre-saved FPV as an enhanced steering guide so that the vehicle operator aligns the vehicle in the parking space to a distance equal to or less than the predefined FPV alignment threshold distance.

[0021] In another aspect of the present disclosure, the LS portion of the method further includes providing AR assistance to the vehicle operator when the vehicle operator provides direction input through steering adjustments, and provides braking input and acceleration input to guide the vehicle to and enter a parking space identified by a pre-identified FPV. The AR assistance superimposes the pre-identified FPV on one or more real-time sensor data displayed on the vehicle HMI and HUD, so that the FPV is displayed directly in the line of sight of the vehicle operator and, from the optical perspective of the vehicle operator, at a location corresponding to the actual physical location of the FPV. On one or more of the HUD and HMI, a path to the saved pre-identified FPV is displayed, and its style, color, or layout is different from the path to the pre-trained parking space.

[0022] In another aspect of the present disclosure, the LS portion of the method further includes determining that the current distance of the vehicle to the parking space is less than or equal to the FPV alignment threshold distance, and using the FPV alignment mode of ARAPA, and using AR, a pre-identified or saved FPV can be displayed on one or more of the HMI and HUD to provide a reference to the vehicle operator, and the vehicle operator uses the reference to align the vehicle in real time within the parking space. The LS portion of the method also includes enabling the vehicle operator to check the alignment of the saved FPV with corresponding features of the real-world environment to enhance parking assistance and steering, or otherwise operate the vehicle, and, when the saved FPV is aligned with the real-world FPV or environmental objects seen on the HMI and / or HUD, notifying the vehicle operator that the vehicle has been fully parked.

[0023] In another aspect of the present disclosure, a method for AR-assisted vehicle parking with operator-in-the-loop selection and alignment of FPV includes detecting the FPV of a parking position using one or more sensors located on the vehicle, and executing program control logic stored in the memory of one or more controllers, the controller communicates with the one or more sensors and communicates with one or more HMIs located in the vehicle. Each of the one or more controllers has a processor, a memory, and one or more I / O ports. The I / O port communicates with one or more sensors and one or more HMIs, and the processor executes control logic including ARAPA. The method also includes activating ARAPA by the vehicle operator through the input of the HMI after the parking space at the parking position is completed. ARAPA has a control including control logic for training ARAPA in the TS portion to identify the FPV of the parking space at the parking position. The TS portion also includes the following control logic, the vehicle operator selects the first feature of ARAPA through the HMI to indicate that the vehicle has been parked in a preferred parking space, and presents multiple recommended FPVs overlaid on the real-time display of the view through the HMI, and the view comes from one or more sensors. The suggested FPV at least partially defines the current position or ideal position of the vehicle within the preferred parking space. The TS portion also includes control logic for receiving input from a vehicle operator through the HMI, the input being used to select some or all of a plurality of suggested FPVs, each suggested FPV having an appropriate spacing from the other FPVs, wherein the appropriate spacing is selected so that all selected FPVs can be displayed on the HMI simultaneously and so that the different FPVs do not overlap at any line of sight of the vehicle operator; and when the vehicle is parked, the appropriate spacing reduces the likelihood of the vehicle being misaligned within the parking space from a first level to a second level, the second level being lower than the first level. The TS portion also includes the following control logic, which is used to suggest an FPV and a semantic label to the vehicle operator through the HMI, which corresponds to a parking space pre-identified by the system at the parking location. The TS portion also includes the following control logic for providing an automatically generated suggested parking space name and an opportunity for the vehicle operator to change the suggested parking space name through the HMI, and for storing the suggested parking space name and the semantic label in a memory for use by the LS portion of ARAPA. ARAPA also includes control logic for, in the LS portion of ARAPA, prompting the vehicle operator to select a parking space when the vehicle operator navigates through a parking location having one or more parking spaces. The LS portion also includes control logic for determining that a current distance between the vehicle and one or more pre-trained parking spaces is less than or equal to a predefined FPV preview threshold distance. The FPV preview threshold distance is between about six meters and about ten meters. The LS portion also includes control logic for providing one or more prompts to the vehicle operator via the HMI.The one or more prompts request the vehicle operator to select a parking space that the vehicle operator is going to park in from one or more pre-trained parking spaces within the FPV preview threshold distance. The LS portion also includes the following control logic for providing augmented reality AR assistance to the vehicle operator when the vehicle operator provides direction input through steering adjustment, and provides braking input and acceleration input to guide the vehicle to and enter the pre-trained parking space. AR assistance superimposes the FPV of the pre-trained parking space on one or more real-time sensor data displayed on the HMI and HUD of the vehicle, so that the FPV is directly displayed in the line of sight of the vehicle operator, and from the optical perspective of the vehicle operator, it is displayed at a position corresponding to the actual physical position of the FPV. The LS portion also includes the following control logic for determining that the vehicle is within the FPV preview threshold distance, the FPV preview threshold distance is one or more pre-saved FPV threshold distances, and the pre-saved FPV is the FPV of the parking space in the parking position. The LS portion also includes the following control logic for conditionally displaying a preview of a pre-saved FPV on one or more of the HMI and the HUD, and using the pre-identified FPV as an enhanced steering guide for the vehicle operator, for causing the vehicle operator to align the vehicle in the parking space to a distance equal to or less than a predefined FPV alignment threshold distance. The LS portion also includes the following control logic for providing AR assistance to the vehicle operator when the vehicle operator provides direction input through steering adjustment, and provides braking input and acceleration input to guide the vehicle to and enter the parking space identified by the pre-identified FPV. AR assistance superimposes the pre-identified FPV on one or more real-time sensor data displayed on the vehicle HMI and HUD, so that the FPV is displayed directly in the line of sight of the vehicle operator, and from the optical perspective of the vehicle operator, it is displayed at a position corresponding to the actual physical position of the FPV. On one or more of the HUD and the HMI, a path to the saved pre-identified FPV is displayed, and its style, color or layout is different from the path to the pre-trained parking space. The LS portion also includes the following control logic for instructing the vehicle operator to align the pre-trained FPV of the parking space with the corresponding real-world FPV by determining that the distance of the vehicle to the current parking space is less than or equal to the FPV alignment threshold distance and using the FPV alignment mode of ARAPA. The LS portion also includes control logic for displaying the pre-identified or saved FPV on one or more of the HMI and the HUD using AR, thereby providing a reference to the vehicle operator, the vehicle operator using the reference to align the vehicle in real time within the parking space, and the vehicle operator checking the alignment of the saved FPV with the corresponding features of the real environment to enhance parking assistance and steering, or otherwise operate the vehicle.The LS portion also includes control logic for notifying the vehicle operator that the vehicle is fully parked when the saved FPV is aligned with the real world FPV or environmental objects seen on the HMI and / or HUD.

[0024] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0026] Figure 1 is a schematic diagram of an AR assisted parking system utilizing FPV operator-in-the-loop selection and alignment (DSA) according to an exemplary embodiment;

[0027] Figure 2A According to an exemplary embodiment, there is a Figure 1 An interior perspective view of a vehicle using an AR assisted parking system utilizing FPV DSA is shown;

[0028] Figure 2B is set according to an exemplary embodiment Figure 2A An enlarged view of a HIM inside a vehicle is shown, the vehicle being provided with Figure 1 The AR assisted parking system using FPV DSA is shown;

[0029] Figure 3 is a depiction of an exemplary embodiment Figure 1 Flow chart of the control logic of the training phase of the AR assisted parking system using FPV DSA;

[0030] Figure 4 is a depiction of an exemplary embodiment Figure 1 A flowchart of the control logic of the real-time phase of the AR assisted parking system using FPV DSA is shown; and

[0031] Figure 5 is a flow chart depicting a method of AR-assisted parking using FPV DSA according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] The following detailed description is merely exemplary in nature and is not intended to limit application and use. In addition, there is no intention to be bound by the express or implied theory presented in the aforementioned technical field, background technology, invention content or the following specific embodiments. As used herein, the term "module" refers to separate hardware, software, firmware, electronic control components, processing logic and / or processor devices, or a combination thereof, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or groups) and memories that execute one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functions.

[0033] Embodiments of the present disclosure may be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components may be implemented by multiple hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of the present disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which may perform various functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that embodiments of the present disclosure may be practiced in conjunction with a variety of systems, and the systems described herein are merely exemplary embodiments of the present disclosure.

[0034] For the sake of brevity, technologies related to signal processing, data fusion, signaling, control and other functional aspects of the system (as well as the various operating components of the system) may not be described in detail herein. In addition, the connecting lines shown in the various figures contained herein are intended to represent example functional relationships and / or physical connections between the various elements. It should be noted that alternative or additional functional relationships or physical connections may exist in the embodiments of the present disclosure.

[0035] refer to Figure 1 , 2A 2B, an AR assisted parking system 10 utilizing FPV operator in-the-loop selection and alignment (OSA) is shown. The system 10 includes a vehicle 12 provided with one or more sensors 14 and one or more controllers 16. The vehicle 12 may be any of a variety of vehicles, including a car, a truck, a sport utility vehicle (SUV), a bus, a semi-tractor, a tractor for agriculture or construction, etc., a boat, an aircraft such as an airplane or a helicopter, etc. In some examples, the vehicle 12 travels along a road 18 having any type of road 18 surface, including pavement, gravel, dirt, ice, or other surfaces.

[0036] The sensor 14 may be any of a variety of sensors 14 that may be connected to and communicate on a communication network 20, such as a vehicle-to-vehicle (V2V) communication network, a vehicle-to-infrastructure (V2I) network, etc. The sensor 14 captures data about the surroundings of the vehicle 12, including electromagnetic and / or optical information at various wavelengths, including wavelengths visible to humans, as well as infrared, ultraviolet, and other portions of the spectrum that are not visible to humans. Without departing from the scope or intent of the present disclosure, the sensor 14 may also include any of a global positioning system (GPS) sensor, a camera 26, a light detection and ranging (LiDAR) sensor, a radio detection and ranging (RADAR) sensor, a sound navigation and ranging (SONAR) sensor, an ultrasonic sensor, an inertial measurement unit (IMU), a wheel speed sensor, and any of a variety of other sensors 14 that are capable of determining the position and dynamic information of the vehicle 12 relative to the surroundings of the vehicle 12. In some aspects, the sensors 14 may be integrated onto or into the vehicle 12, or may accommodate after-sales service performed by the vehicle 12 manufacturer, dealer, customer, or by other third parties without departing from the scope or intent of the present disclosure. In some examples, one or more sensors 14 may be part of a stand-alone sensor 14 system without departing from the scope or intent of the present disclosure.

[0037] The controller 16 is a non-general purpose electronic control device having a pre-programmed digital computer or processor 28, a non-transitory computer readable medium or memory 30 for storing data such as control logic, software applications, instructions, computer code, data, look-up tables, and the like, and one or more I / O ports 32. The computer readable medium or memory 30 includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital compact disk, a video compact disk (DVD), a solid-state memory, or any other type of memory. The "non-transitory" computer readable medium or memory 30 does not include wireless, optical, or other communication links that transmit electrical or other signals. The non-transitory computer readable medium or memory 30 includes media that can store data permanently and media that can store data and then be rewritten, such as any type of program code, including source code, object code, and executable code. The processor 28 is configured to execute the code or instructions. In some examples, the controller 16 can be a dedicated wireless or Wi-Fi controller. The I / O port 32 is configured to communicate via wired or wireless means, such as the communication network 20 , including a V2V communication network and / or a V2I communication network using a Wi-Fi protocol under IEEE802.11x, a Bluetooth communication protocol, a radio frequency (RF) protocol, etc.

[0038] In some aspects, the controller 16 includes one or more applications 34. An application 34 is a software program configured to perform a specific function or group of functions. The application 34 may include one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, or portions thereof suitable for implementation in a suitable computer-readable program code. The application 34 may be stored within the memory 30 or in an additional or separate memory 30. Examples of the application 34 include audio or video streaming services, audio or visual processing services, ARAPA 34, and the like.

[0039] More specifically, if Figure 2A As shown, the system 10 also includes an HMI 36 in the vehicle 12 for interacting with the operator of the vehicle 12. In some aspects, the HMI 36 includes one or more devices capable of interacting with the operator of the vehicle 12, such as a screen arranged in the vehicle 12, such as an instrument cluster 38, an infotainment screen 40, a HUD 42, etc. However, it should be understood that other HMIs 34 are also contemplated herein. For example, the HMI 36 may be a mobile device, such as a tablet computer, a mobile phone, etc., and the HMI 36 may be provided by the operator of the vehicle 12 and temporarily installed or set on an interior passenger compartment component of the vehicle 12. In the example where the HMI 36 is a mobile device as described above, it should be understood that at least a portion of the ARAPA 34 is executed on the mobile device.

[0040] Reference Figure 2B , and still refer to Figure 1 and 2A , an exemplary ARAPA 34 is shown as being displayed on an exemplary infotainment screen 40 of the vehicle 12. The ARAPA 34 includes a control logic portion that causes the HMI 36 to utilize a display scheme used in a vehicle 12 parking assistance based on simultaneous localization and mapping (SLAM). In some aspects, the ARAPA 34 includes a TS portion 100 and an LS portion 200. In the TS portion 100, the ARAPA 34 is trained to recognize the FPV 44 of a particular parking space 46. During the TS, the vehicle 12 operator parks the vehicle 12 in the parking space 46 at the parking position 48 selected by the vehicle 12 operator. The vehicle 12 operator then activates the ARAPA 34 through the HMI 36.

[0041] Reference Figure 3 , and continue to refer to Figure 1 , 2A2B, showing the TS portion 100 of the ARAPA 34 in more detail. In the first TS control logic portion 102, the vehicle 12 operator selects a first feature provided in the ARAPA 34 to indicate in the ARAPA 34 through the HMI 36 that the parking of the vehicle 12 is complete, and the first feature notifies or confirms to the system 10 that the vehicle 12 has been parked in the location preferred by the vehicle 12 operator. The first feature can take any of a variety of different forms, including a screen button on the HMI 36, a voice input, a tactile input through one or more physical buttons inside the vehicle 12 that communicate with the ARAPA 34, and the like.

[0042] In response to the vehicle 12 operator receiving the information that the vehicle 12 is parked, the ARAPA 34 executes the second TS control logic portion 104, which presents one or more options to the vehicle 12 operator through the HMI 36. In some aspects, one or more options may be presented on the HMI 36, and may include one or more prompts overlaid on a real-time display of a view of one or more cameras 26, and the camera 26 is used to capture optical data of the surrounding environment of the vehicle 12. The options or prompts in the second TS control logic portion 104 include presenting a display view of one or more cameras 26, and a plurality of suggested FPVs 44 found within the optical data and within a predefined distance from the vehicle 12. In some aspects, the suggested FPVs 44 at least partially define the current position or ideal position of the vehicle 12 in the parking space 46. It should be understood that the predefined distance may vary according to the application, location, and vehicle changes without departing from the scope or intent of the present disclosure. In response to the second TS control logic portion 104, the vehicle 12 operator selects some or all of the suggested FPVs 44. In some examples, the vehicle 12 operator selects three to five FPVs 44 that are visually appropriately spaced so that all selected FPVs 44 can be displayed simultaneously and the risk of falling outside the boundaries of the display area can be avoided, and although the number and appropriate spacing of the selected FPVs 44 can vary according to the application and location, different FPVs 44 do not overlap in any reasonable line of sight of the vehicle 12 operator. However, it should be understood that appropriate spacing is selected to ensure the accuracy and precision of the alignment of the vehicle 12 relative to the FPVs 44. That is, when the vehicle 12 is repeatedly parked autonomously or semi-autonomously in a specific parking space 46, an appropriate spacing is generally selected from a first level to a second level to reduce the possibility of three-dimensional inaccuracy or misalignment of the vehicle 12 position in the parking space 46, wherein the second level is lower than the first level.

[0043] ARAPA 34 may also execute a third TS control logic portion, which is executed in parallel, simultaneously, or sequentially with the second TS control logic portion 104. In the third TS control logic portion 106, ARAPA 34 recommends the FPV 44 identified by the system 10 as a preferred option and provides a corresponding semantic label to the vehicle 12 operator using a prompt on the HMI 36. The semantic label generally includes a descriptor text generated by the computer vision module based on the analysis of the selected FPV 44, for example, Figure 2B The rightmost FPV44 selected in the image may be labeled "Gizmo on the Wall". The suggested FPV44 may include a pre-identified or selected FPV44 at a particular parking location 48, or an FPV44 corresponding to a pre-identified or selected FPV44 already stored in the memory 30. ARAPA34 proceeds from the second TS control logic portion 104 and the third TS control logic portion 106 to the fourth TS control logic portion 108. It should be understood that in some cases, there is a lack of environmental features discernible by the system 10 in the camera 26 view, and the FPV44 may be selected as an item that is salient to the eyes of the vehicle 12 operator. In some examples, the FPV44 defines at least two narrow-shaped static objects discernible by the eye, such as two or more boundary lines on a white wall, and an FPV44 that was not previously suggested by the system may be directly selected by the vehicle 12 operator by hand-drawing in the HMI36.

[0044] During execution of the fourth TS control logic portion 108, the vehicle 12 operator is provided, via the HMI 36, with the opportunity to assign a parking space 46 name to the parking space 46 currently occupied by the vehicle 12. Suggested parking space 46 names may also be automatically provided by the ARAPA 34, and the vehicle 12 operator may therefore be provided with the opportunity to modify the suggested parking space 46 name. The fifth TS control logic portion 110 is then executed, wherein the parking space 46 name assigned to the parking space 46, the image, and other necessary data are then stored in the memory 30 along with the semantic tag for use by the ARAPA 34 in a later LS phase.

[0045] Reference Figure 4 , and continue to refer to Figure 1 , 2A, 2B and 3, the LS portion 200 of the ARAPA 34 is shown in more detail in flowchart form. During the LS portion 200, the system executes a series of LS control logic portions of the ARAPA 34, which assist the vehicle 12 operator in parking the vehicle 12 in a real-time phase. Specifically, the vehicle 12 operator drives the vehicle 12 in a parking location 48, in which there are one or more parking spaces 46. When the vehicle 12 is driven through an area with one or more available parking spaces 46, the ARAPA 34 identifies the parking space 46. In the first LS control logic portion 202, the ARAPA 34 determines a current distance to a known parking space 46 that is less than or equal to a predefined FPV 44 preview threshold distance from the vehicle 12. The predefined FPV 44 preview threshold is a predefined distance from the vehicle 12 to the known or target parking space 46. It should be appreciated that the predefined FPV 44 preview threshold may vary depending on the application, vehicle 12 , and / or location, however, in some examples, the FPV 44 preview threshold is a variable distance of approximately six meters and approximately ten meters from the vehicle 12 .

[0046] In the event that a known parking space 46 is determined to be within the FPV44 preview threshold distance of the pre-trained parking space 46 of the second LS control logic portion 204, the ARAPA 34 presents one or more prompts to the vehicle 12 operator to indicate that the vehicle 12 is within the FPV44 preview threshold distance of the pre-trained parking space 46 of the second LS control logic portion 204, and provides the vehicle 12 operator with one or more pre-trained parking spaces 46 to select parking. The vehicle 12 operator selects one of the parking spaces 46, and then provides direction inputs through steering adjustments, as well as brake inputs and acceleration inputs to guide the vehicle 12 toward the pre-trained parking space 46 found by the second LS control logic portion 204. As the vehicle 12 operator directs the vehicle 12 , the system 10 executes the third LS control logic portion 206 of ARAPA 34 , which provides AR assistance by overlaying FPV 44 on the real-time camera 26 data displayed on the HMI 36 , or by displaying FPV 44 directly on the HUD 42 , so that the FPV 44 is displayed directly in the line of sight of the vehicle 12 operator and in a position corresponding to the physical position of the FPV 44 from the vehicle 12 operator's optical perspective.

[0047] Similarly, when the system 10 determines that the vehicle 12 is within a FPV44 preview threshold distance of a parking space 46 in the same parking location 48 (i.e., parking lot), the system 10 may execute the fourth LS control logic portion 208, the FPV44 preview threshold distance being a threshold distance of one or more pre-saved FPV44, and the pre-saved FPV44 is sufficient to represent the physical location and three-dimensional parameters of the parking space 46. In the event that a pre-saved FPV44 of a parking space 46 that has not been previously selected by the operator of the vehicle 12 is identified, the system 10 executes the fifth LS control logic portion 210, which conditionally displays a preview of the identified pre-saved FPV44 on the HUD 42 and / or HMI 36. More specifically, if a pre-saved FPV44 can be identified before the vehicle 12 enters a position where the distance to the pre-selected parking space 46 is equal to or less than the predefined FPV44 alignment threshold distance, the system 10 optionally displays a preview of the identified, pre-saved FPV44 as an enhanced steering guidance for the operator of the vehicle 12. Without departing from the scope or intent of the present disclosure, the predefined FPV 44 alignment threshold may vary depending on the application, vehicle 12, and / or location. In some examples, the predefined FPV 44 alignment threshold is typically less than the FPV 44 preview threshold and may be between about 1.5 meters and about 3 meters.

[0048] Through the selection or prompt displayed by the fifth LS control logic portion 210, the vehicle 12 operator can select a parking space 46 within the identified, pre-saved FPV44, and then navigate or otherwise guide the vehicle 12 into the parking space 46 represented by the pre-saved FPV44. When the vehicle 12 operator guides the vehicle 12, the system 10 executes the sixth LS control logic portion 212 of the ARAPA34, which provides AR assistance by overlaying the FPV44 on the real-time camera 26 data displayed on the HMI36 or by directly displaying the FPV44 on the HUD42, so that the FPV44 is directly displayed in the line of sight of the vehicle 12 operator and is displayed in a position corresponding to the physical position of the FPV44 from the optical perspective of the vehicle 12 operator. In some aspects, the path to the identified, pre-saved FPV44 can be displayed on the HUD and / or HMI36 in a different style, color, layout, etc. compared to the path to the pre-trained parking space 46.

[0049] The system 10 then executes a seventh LS control logic portion 214 which determines the current distance to the selected parking space 46 and compares the current distance to a predefined FPV 44 alignment threshold before using the FPV 44 alignment mode 216 of the ARAPA 34 .

[0050] In the FPV44 alignment mode 216, the system 10 executes a series of LS control logic portions that assist the vehicle 12 operator in correctly and completely positioning the vehicle 12 within the selected parking space 46. More specifically, the FPV44 alignment mode 216 assists the vehicle operator 12 by executing an eighth LS control logic portion 218, which utilizes AR to display a pre-identified or saved FPV44 on the HMI 36 and / or HUD 42, thereby providing a reference to the vehicle operator, and the vehicle 12 operator aligns the vehicle 12 in real time within the parking space 46 based on the reference. The system 10 also executes a ninth LS control logic 220 of ARAPA34, in which the vehicle 12 operator checks the alignment of the saved FPV44 with corresponding features of the real-world environment to enhance parking assistance and steering or otherwise operate the vehicle 12.

[0051] In some aspects of the eighth or ninth LS control logic portions 218, 220, in alignment mode 216, when alignment is performed using HMI36, the saved FPV44 image is displayed at the original pixel position and superimposed on the real-time camera 26 view of the area surrounding the vehicle 12. The foreground, background, and related transparency are appropriately configured so that the system 10 can clearly indicate how accurately the saved FPV44 image is aligned with the corresponding live FPV44 of the same reference point. Similarly, when alignment is performed using AR through HUD42, HUD42 displays the position and projected shape of the saved FPV44 image calculated based on the driver monitoring system (DMS) view of the vehicle 12 operator's line of sight. Then, based on the ambient light and the real-time adjustment of the vehicle 12 operator, the saved FPV44 image is displayed with appropriate brightness, so that the HUD42 clearly indicates the extent to which the saved FPV44 image is accurately aligned with the corresponding real-time environmental objects observed by the vehicle 12 operator through the vehicle 12 windshield. It should be understood that the DMS may include various sensors 14 disposed on or in the vehicle 12 for monitoring the occupants of the vehicle 12, and specifically the operator of the vehicle 12. The sensors 14 of the DMS may include cameras 26, thermal sensors, sensors capable of recording or detecting tactile input, infrared sensors, etc. The sensors 14 of the DMS may detect the seating position, head position, eye position, hand position, etc. of the operator of the vehicle 12, and such information may be used to determine how the operator of the vehicle 12 is actively managing or operating the vehicle 12 and vehicle functions.

[0052] In the tenth LS control logic portion 222, when the saved FPV44 image seen on the HMI36 and / or HUD42 is sufficiently aligned with the real-world FPV44 or environmental objects, the system 10 notifies the vehicle 12 operator that the vehicle 12 has been fully parked. It should be understood that when the longitudinal and lateral positions of the saved FPV44 are closely aligned with the real-world FPV44 or environmental objects, the system 10 and / or the vehicle 12 operator can determine that the vehicle 12 is fully parked and aligned in the parking space 46. It should also be understood that the concept of "sufficient alignment" is generally an intuitive judgment of the operator or an assessment by the system that the saved FPV achieves a sufficiently high intersection-over-union (IOU) index with its counterpart in the real world, such as an IOU not less than a 95% threshold observed by the operator's eyes.

[0053] Reference Figure 5 , and continue to refer to Figure 1 , Figure 2A , Figure 2B , Figure 3 and Figure 4 , a method 300 for AR-assisted parking with operator-in-the-loop selection and alignment of FPV44 is shown in the form of a flowchart. The method 300 begins at box 302. At box 304, the vehicle 12 operator initiates the training phase of ARAPA34 by operating the vehicle 12 to a new parking space 46 and indicating the completion of the parking process through the HMI36. At box 306, the system 10 presents a real-time camera 26 view for FPV44 selection on the HMI36 and provides prompts to the vehicle 12 operator. At box 308, which can be performed simultaneously, in parallel, or sequentially with the process performed at box 306, ARAPA34 recommends the system 10 to select the identified FPV44 as preferred and provides the vehicle 12 operator with a corresponding semantic label with the HMI36 prompt. The recommended FPV44 may include a pre-identified or selected FPV44 at a particular parking space 46, or an FPV44 corresponding to a pre-identified or selected FPV44 already stored in the memory 30. From blocks 306 and 308, the method 300 proceeds to block 310, where the vehicle 12 operator is provided an opportunity to assign a parking space 46 name to the parking space 46 currently occupied by the vehicle 12 via the HMI 36. Suggested parking space 46 names may also be automatically provided by the ARAPA 34, and thus the vehicle 12 operator may be provided an opportunity to modify the suggested parking space 46 name. From block 310, the method 300 proceeds to block 312, where the parking location name assigned to the parking space 46, the image, and other necessary data are stored in the memory 30 along with the semantic tag for use by the ARAPA 34 in a later LS.

[0054] Starting from box 312, the method 300 enters the real-time phase of ARAPA34, which begins at box 314. Specifically, at box 314, the vehicle 12 operator drives the vehicle 12 in a parking position 48, in which there are one or more parking spaces 46. When the vehicle 12 is driven through a parking position 48 with one or more available parking spaces 46, the ARAPA34 identifies the parking space 46 and compares them with a predefined FPV44 preview threshold distance. That is, at box 314, the ARAPA34 determines whether the current distance to the known parking space 46 is less than or equal to the predefined FPV44 preview threshold distance from the vehicle 12. The predefined FPV44 preview threshold is a predefined distance from the vehicle 12 to the known or target parking space 46. It should be understood that the predefined FPV44 preview threshold can vary depending on the application, the vehicle 12, and / or the location, however, in some examples, the FPV44 preview threshold is a variable distance of about six meters and about 10 meters from the vehicle 12.

[0055] In the event that a known parking space 46 is determined to be greater than the FPV 44 preview threshold distance, the method 300 proceeds to block 316 where the system 10 continues searching for the parking space 46 before proceeding to block 314 .

[0056] When it is determined that the known parking space 46 is less than or equal to the FPV 44 preview threshold, the system 10 then provides one or more prompts to the vehicle 12 operator via the HMI 36. More specifically, at block 318, the ARAPA 34 presents one or more pre-trained parking spaces 46 to the vehicle 12 operator.

[0057] When the system 10 determines from box 318 that the vehicle 12 is within the FPV 44 preview threshold distance of the pre-trained parking space 46, the ARAPA 34 presents one or more prompts to the vehicle 12 operator to indicate from box 318 that the vehicle 12 is within the FPV 44 preview threshold distance of the pre-trained parking space 46. The vehicle 12 operator then provides direction input via steering adjustments, as well as brake input and acceleration input to guide the vehicle 12 toward the pre-trained parking space 46 found at box 318. As the vehicle 12 operator guides the vehicle 12, the method 300 proceeds to box 320, where the ARAPA 34 provides AR assistance by overlaying the FPV 44 on the real-time camera 26 data displayed on the HMI 36, or by directly displaying the FPV 44 on the HUD 42, so that the FPV 44 is displayed directly in the line of sight of the vehicle 12 operator and at a location corresponding to the physical location of the FPV 44 from the optical perspective of the vehicle 12 operator.

[0058] The method 300 also proceeds to box 322 to box 326, which can be performed in parallel, simultaneously or sequentially with the steps of box 318 and box 320 of the method 300. At box 322, the system 10 determines that the vehicle 12 is within the FPV44 preview threshold distance, which is the threshold distance of one or more pre-saved FPV44 of the parking space 46 at the same general parking location 48 (i.e., parking lot), and the pre-saved FPV4 is sufficient to indicate that they can be used to represent the physical location and three-dimensional parameters of the parking space 46. When a pre-saved FPV44 of a parking space 46 that has not been previously selected by the vehicle 12 operator is identified, the method 300 proceeds to box 324, and the system 10 causes the ARAPA 34 to conditionally display a preview of the identified pre-saved FPV44 on the HUD 42 and / or HMI 36. More specifically, a pre-saved FPV44 can be identified before the vehicle 12 enters a position within a pre-defined FPV44 alignment threshold of a pre-selected parking space 46, and the system 10 optionally displays a preview of the identified, pre-saved FPV44 as an enhanced steering guidance for the operator of the vehicle 12. Without departing from the scope or intent of the present disclosure, the pre-defined FPV44 alignment threshold can vary depending on the application, vehicle 12, and / or location. In some examples, the pre-defined FPV44 alignment threshold is generally less than the FPV44 preview threshold and can be between about one and a half meters (1.5) and about three (3) meters.

[0059] The vehicle 12 operator can select a parking space 46 within the identified, pre-saved FPV 44 via a selection or prompt displayed at box 324, and then navigate or otherwise guide the vehicle 12 into the parking space 46 represented by the pre-saved FPV 44. As the vehicle 12 operator guides the vehicle 12, the method 300 proceeds to box 326, and the ARAPA 34 provides AR assistance by overlaying the FPV 44 on the real-time camera 26 data displayed on the HMI 36 or by directly displaying the FPV 44 on the HUD 42, so that the FPV 44 is displayed directly in the line of sight of the vehicle 12 operator and, from the optical perspective of the vehicle 12 operator, at a location corresponding to the physical location of the FPV 44. In some aspects, the path to the identified, pre-saved FPV 44 can be displayed on the HUD and / or HMI 36 in a different style, color, layout, etc. than the path to the pre-trained parking space 46.

[0060] The method 300 then proceeds to box 328 and causes the system 10 to determine the current distance to the selected parking space 46 and compare the current distance to a predefined FPV44 alignment threshold before enabling the FPV44 alignment mode 216 of the ARAPA 34. When the current distance of the vehicle 12 to the selected parking space 46 is greater than the FPV44 alignment threshold distance, the method 300 returns to box 320 and / or box 326. However, when the current distance of the vehicle 12 to the selected parking space 46 is less than or equal to the predefined FPV44 alignment threshold distance, the method uses the FPV44 alignment mode 216.

[0061] In the FPV44 alignment mode 216, the method 300 assists the vehicle 12 operator to correctly and completely position the vehicle 12 within the selected parking space 46. More specifically, the method 300 uses the FPV44 alignment mode 216 at block 330 to assist the vehicle operator 12 by displaying a pre-identified or saved FPV44 on the HMI 36 and / or HUD 42 using AR, thereby providing a reference for the vehicle 12, according to which the vehicle 12 operator aligns the vehicle 12 in real time within the parking space 46.

[0062] At box 332, the vehicle 12 operator checks the alignment of the saved FPV44 with the corresponding features of the real world environment for enhanced parking assistance and steering or otherwise operating the vehicle 12. At either box 330 and 332, in the alignment mode 216, when the HMI36 is used for alignment, the method 300 causes the ARAPA34 to display the saved FPV44 image at the original pixel position on the HMI36, and superimpose the saved FPV44 image on the real-time camera 26 view of the area around the vehicle 12. The foreground, background, and related transparency are appropriately configured so that the system 10 can clearly indicate how the saved FPV44 image is accurately aligned with the corresponding live FPV44 of the same reference point. Similarly, when AR is used for alignment through the HUD42, the HUD42 displays the location and projected shape of the saved FPV44 image calculated based on the driver monitoring system (DMS) view of the vehicle 12 operator's line of sight. The saved FPV44 image is then displayed at an appropriate brightness based on the ambient light and the real-time adjustment of the vehicle 12 operator, so that the HUD 42 clearly indicates the degree to which the saved FPV44 image is accurately aligned with the corresponding real-time environmental objects observed by the vehicle, which are objects directly observed by the vehicle 12 operator through the 12 windshield. It should be understood that the DMS may include a variety of sensors disposed on or within the vehicle 12 for monitoring the occupants of the vehicle 12, specifically the operator of the vehicle 12. The sensors 14 of the DMS may include a camera 26, a thermal sensor, a sensor capable of recording or detecting tactile input, an infrared sensor, etc. The sensors 14 of the DMS may detect the seating position, head position, eye position, hand position, etc. of the vehicle 12 operator, and such information may be used to determine how the vehicle 12 operator actively manages or operates the vehicle 12 and vehicle functions.

[0063] From box 330 and / or box 332, the method 300 proceeds to box 334, where the vehicle 12 operator determines that the vehicle 12 has been fully parked when the saved FPV44 image is sufficiently aligned with the real-world FPV44 or environmental objects seen on the HMI36 and / or HUD42. It should be understood that when the saved longitudinal and lateral positions of the FPV44 are closely aligned with the real-world FPV44 or environmental objects, the vehicle 12 operator can determine that the vehicle 12 has been fully parked and aligned in the parking space 46. At box 336, the method 300 ends. In some aspects, the method 300 can be run continuously, iteratively, or only when activated by the vehicle 12 operator and / or by an automatic parking application, etc., without departing from the scope or intent of the present disclosure. As mentioned above, the concept of "adequate alignment" is determined by the operator's intuitive judgment, or by the system's assessment of a sufficiently high intersection over union (IOU) metric between the saved FPV and its real-world counterpart, such as an IOU not lower than a 95% threshold observed by the operator's eyes.

[0064] The system 10 and method 300 of the present disclosure for AR-assisted parking with in-loop selection and alignment of FPV44 by the operator have multiple advantages. These include using the AR display scheme for SLAM-based vehicle 12 parking assistance, being able to prevent the degradation of robustness due to the removal or addition of feature points at a given geographic location, and being able to take into account changes in feature descriptors caused by changes in lighting conditions. In addition, the system 10 and method 300 of the present disclosure require only minimal input from the operator of the vehicle 12 when observing and selecting environmental features, and while ensuring that the parking assistance system effectively and accurately represents or indicates the longitudinal and lateral offsets of the vehicle 12 relative to the FPV44, it reduces or reduces the need for a large number of feature points in the underlying SLAM process, thereby simplifying the parking procedure and increasing the confidence of the operator of the vehicle 12 while maintaining or reducing the complexity of the system 10.

[0065] The description of the present disclosure is merely exemplary in nature, and changes that do not depart from the gist of the present disclosure should fall within the scope of the present disclosure. These changes should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. An augmented reality (AR) assisted vehicle parking system using operator-in-the-loop selection and alignment of feature point proximity FPV, the system comprising: vehicle; One or more human-machine interfaces HMIs disposed in the vehicle; one or more sensors of FPV disposed on the vehicle and configured to detect a parking position; One or more controllers, each of the one or more controllers having a processor, a memory and one or more input / output I / O ports, the I / O ports communicating with the one or more sensors and the one or more HMIs; the memory storing program control logic; the processor executing the program control logic; the program control logic including an AR assisted parking application ARAPA, the vehicle operator activating the ARAPA through the HMI after completing parking in the parking space at the parking location; the ARAPA including: The TS part of the training phase; and Real-time stage LS part, wherein, in the TS portion, the ARAPA is trained to recognize the FPV of the parking space at the parking position; and Wherein, in the LS part, the ARAPA prompts the vehicle operator to select the parking space, and displays it by superimposing the pre-trained FPV on the real-time sensor data displayed by the HMI, and instructs the vehicle operator to align the pre-trained FPV with the real-world FPV corresponding to the parking space by operating the vehicle to guide the vehicle operator into the parking space.

2. The system according to claim 1, wherein: The TS part also includes: A first TS control logic portion wherein the vehicle operator selects a first feature of the ARAPA via the HMI to indicate that the vehicle is parked in a preferred parking space.

3. The system according to claim 2, wherein: The TS part also includes: a second TS control logic portion, wherein the ARAPA utilizes the HMI to present a plurality of suggested FPVs superimposed on a real-time display view of the one or more sensors, wherein the suggested FPVs at least partially define a current position or a desired position of the vehicle within the preferred parking space; and control logic that receives input from the vehicle operator through the HMI, the input being used to select some or all of the plurality of suggested FPVs, each of the suggested FPVs being appropriately spaced from the other suggested FPVs, wherein the appropriate spacing is selected so that all selected FPVs can be displayed simultaneously on the HMI and so that different FPVs do not overlap at any line of sight of the vehicle operator; and wherein, when parking the vehicle, the appropriate spacing reduces the likelihood of the vehicle being misaligned within the parking space from a first level to a second level that is lower than the first level.

4. The system according to claim 1, wherein: The TS part also includes: A third TS control logic portion, wherein the ARAPA suggests an FPV and a semantic label through the HMI, the FPV and the semantic label corresponding to a parking space pre-identified by the system at the parking position.

5. The system according to claim 4, further comprising: . a fourth TS control logic portion that provides, via the HMI, an automatically generated suggested parking space name and an opportunity for the vehicle operator to change the suggested parking space name; and A fifth TS control logic portion, the fifth TS control logic portion stores the suggested parking space name and semantic label in a memory for use by the LS portion of the ARAPA.

6. The system according to claim 1, wherein: The LS part also includes: a first LS control logic portion that determines that a current distance between the vehicle and one or more pre-trained parking spaces is less than or equal to a predefined FPV preview threshold distance, wherein the FPV preview threshold distance is between about 6 meters and about 10 meters; and A second LS control logic portion, wherein the second LS control logic portion provides one or more prompts to the vehicle operator through the HMI, wherein the one or more prompts request the vehicle operator to select a parking space that the vehicle operator is ready to park in from the one or more pre-trained parking spaces within the FPV preview threshold distance.

7. The system according to claim 6, wherein: The LS part also includes: and a third LS control logic portion, which provides AR assistance to the vehicle operator to guide the vehicle to and enter the pre-trained parking space when the vehicle operator provides direction input through steering adjustment, and provides braking input and acceleration input, wherein the AR assistance overlays the FPV of the pre-trained parking space on one or more real-time sensor data displayed on the HMI and head-up display (HUD) of the vehicle, so that the FPV is directly displayed in the line of sight of the vehicle operator and, from the optical perspective of the vehicle operator, is displayed at a position corresponding to the actual physical position of the FPV.

8. The system according to claim 7, wherein: The LS part also includes: a fourth LS control logic that determines that the vehicle is within the FPV preview threshold distance, the FPV preview threshold distance being one or more pre-saved FPV threshold distances of a parking space in a parking location; and and a fifth LS control logic, which conditionally displays a preview of an identified pre-saved FPV on one or more of the HMI and the head-mounted display (HUD), and uses the pre-identified FPV as an enhanced steering guidance for the vehicle operator to align the vehicle within a parking spot to a distance equal to or less than a predefined FPV alignment threshold distance.

9. The system according to claim 8, wherein: The LS part also includes: A sixth LS control logic, when the vehicle operator provides direction input through steering adjustment, and provides braking input and acceleration input, the sixth LS control logic provides AR assistance to the vehicle operator to guide the vehicle to and enter a parking space identified by a saved pre-identified FPV, wherein the AR assistance overlays the saved pre-identified FPV on the HMI of the vehicle and one or more real-time sensor data displayed on the HUD, so that the FPV is directly displayed in the line of sight of the vehicle operator and, from the optical perspective of the vehicle operator, is displayed at a position corresponding to the actual physical position of the FPV, and wherein the path to the saved pre-identified FPV is displayed on one or more of the HUD and the HMI in a different style, color, or layout than the path to the pre-trained parking space.

10. The system according to claim 9, wherein: The LS part also includes: a seventh LS control logic that determines that a current distance from the vehicle to a parking space is less than or equal to the FPV alignment threshold distance and uses an FPV alignment mode of the ARAPA; and an eighth LS control logic that utilizes AR to display a pre-identified or saved FPV on one or more of the HMI and the HUD to provide a reference for the vehicle operator to use to align the vehicle in real time within the parking space; a ninth LS control logic that enables the vehicle operator to check the alignment of the saved FPV with corresponding features of the real-world environment to enhance parking assistance and steering, or otherwise operate the vehicle; and A tenth LS control logic notifies the vehicle operator that the vehicle is fully parked when the saved FPV is aligned with the real world FPV or environmental objects seen on the HMI and / or the HUD.

Images