Vehicle driving assistance method and system
By installing positioning modules on the vehicle and determining the HMD position using compensation rules, accurately displaying augmented reality elements, the accuracy and efficiency problems of augmented reality display in the prior art are solved, and driving safety and experience are improved.
Patent Information
- Application Number
- CN202080067235.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-09-27
- Filing Date
- 2020-09-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-09-21
AI Technical Summary
The prior art is difficult to accurately and efficiently display augmented reality elements in the driver's field of view, especially during vehicle driving, resulting in limited driving safety and user experience.
By installing a positioning module on the vehicle, detecting the vehicle position, and determining the position of the HMD in combination with the compensation law, calculating the field of view, comparing the position of the object of interest, and calculating and displaying the corresponding image in the field of view, realizing accurate display on the HMD Augmented reality image.
It realizes high accuracy and high precision augmented reality image display during driving, reducing computing costs and hardware requirements, and improving the reliability and user experience of driving assistance.
Smart Images

Figure CN114503008B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of transport vehicles. In particular, the present invention relates to a method and a system for assisting the driving of a vehicle. Background Art
[0002] Hitherto, driving information, such as information about moving speed, fuel level, navigation directions or the like, has been shown in the dashboard of the vehicle or on any infotainment screen with which the vehicle is equipped. Both the dashboard and the screen are usually located in a position in the vehicle that requires the driver to at least partially take his eye away from the road environment, thus reducing both driving safety and the possibility of using such information.
[0003] In the automotive and aviation fields, 'Head Up Displays', or HUDs for short, have been proposed as a partial solution to this problem. A HUD is a system that allows images to be projected onto the windshield of a vehicle. In particular, a HUD allows information to be projected directly onto the windshield of a car, allowing the driver to stay focused on driving, always keeping his gaze on the road.
[0004] However, the current standard for HUDs, called HUD 1.0, is used only to show redundant information provided by classic on-board instruments. Furthermore, the Applicant has observed that HUD technology does not allow for effective depiction of elements of augmented reality. In fact, the range required to fully cover the driver's field of view by the projection system is much larger than what is technically achievable with the current prior art. In particular, no HUD is able to utilize the entire primary field of view, which is essentially defined by the vehicle's windshield, as well as auxiliary fields of view, such as one or more side windows.
[0005] In addition to HUD systems, wearable screen based systems have recently been proposed, better known as 'head mounted displays', or HMDs for short, which include a transparent or translucent screen on which images can be reproduced, e.g. to provide driving assistance information to a user wearing the HMD while driving a vehicle.
[0006] For example, US patent application No. US 2016 / 084661 describes a system and method that acts as a driving tool and provides feedback to the driver, such as real-time visual feedback provided via an augmented reality device. The navigation system collects vehicle-related information and driver information, such as the direction of the driver's gaze determined by the HMD, and uses this input information to generate real-time visual feedback in the form of virtual guide lines and other driving suggestions. These driving suggestions can be presented to the driver via an augmented reality device (such as a HUD display), where the virtual guide lines are projected onto the windshield of the vehicle so as to be superimposed on the actual road surface seen by the driver, and the driver can be shown the line or route to follow. In addition, other driving suggestions, such as braking, acceleration, steering, and gear shifting suggestions, can be given.
[0007] The Applicant has observed that the method proposed in US 2016 / 084661 for determining the field of view observed by the driver and thus effectively displaying real images is complex to implement. In particular, analyzing the driver's gaze as described in US 2016 / 084661 requires a complex implementation from a hardware and software perspective in order to identify the field of view observed by the driver with sufficient accuracy and determine the size and position of one or more augmented reality images on the HMD o HUD.
[0008] Again, European patent No. EP 2933707 describes a method for dynamically orienting content presented by an HMD. The described method comprises collecting HMD movement data using at least one sensor mounted on an HMD worn by a driver of a vehicle, and collecting vehicle movement data using at least one sensor mounted on the vehicle. The method thus involves performing an analysis of the movement data of the HMD and the vehicle movement data to detect any differences therebetween. Based on the differences found, an orientation of the HMD device relative to the vehicle is calculated, which orientation is calculated as regular data to be presented on a screen of the HMD device based on the newly calculated orientation.
[0009] Although the method proposed in EP 2933707 is able to determine the orientation of the HMD, it does not enable satisfactory accuracy and precision to be obtained. In addition, the method requires high computing resources to calculate and generate data from the consistently presented HMD data based on a comparison of images of the scene visible to the driver through the vehicle windshield. Summary of the invention
[0010] The object of the present invention is to overcome the disadvantages of the prior art.
[0011] In particular, the object of the present invention is to propose a method and a system for assisted driving which are able to provide precise and reliable instructions which assist the user when driving a vehicle.
[0012] The object of the present invention is to propose a method and a system for reproducing elements of augmented reality suitable for improving the driving experience of a user when using a vehicle.
[0013] These and other objects of the invention are achieved by a method and a system incorporating the features of the appended claims, which form an integral part of the present description.
[0014] In one embodiment, the method comprises the following steps:
[0015] - Detect the vehicle position through the positioning module installed on the vehicle,
[0016] - Determine the position of the HMD by applying a compensation law to the vehicle position,
[0017] - based on the position of the HMD, determining a view volume corresponding to the volume of the space included in the field of view of the HMD;
[0018] - comparing a set of locations included in the volume of view with at least one location of interest associated with an object of interest stored in a memory area of the system, and
[0019] - If one or more locations of interest are included in the field of view volume, a visualization position of the HMD is calculated in which an image associated with the object of interest is to be displayed, and the image in the visualization position is displayed on the HMD. Advantageously, the visualization position enables a user wearing the screen to see the image corresponding to the object of interest.
[0020] Thanks to this solution, it is possible to accurately display augmented reality images in the field of view of a user wearing the HMD based solely on positioning data. In particular, it is possible to effectively compensate for display errors due to different positions of the positioning module and the HMD; in fact, even a small distance between these two elements may lead to significant inaccuracies in the display of the augmented reality images, with a consequent reduction in the usefulness of the information associated with the augmented reality images, or even worsen the user's driving situation.
[0021] Advantageously, the method according to the present invention only requires the acquisition and processing of positioning information, such as that provided by a global navigation system or GNSS (e.g., GPS, Galileo, GLONASS, BeiDou, etc.), but does not require the processing of acquired images to identify objects visible through the vehicle windshield in order to correctly display augmented reality images. This allows real-time operation with high accuracy and precision in the display of augmented reality images, wherein computational costs and hardware requirements are significantly reduced.
[0022] In one embodiment, the system further comprises a reference element disposed within the vehicle, and
[0023] The step of detecting the position of the vehicle by means of the positioning module comprises detecting the position of the vehicle relative to a global reference system. Preferably, the method further provides for determining a relative position of the HMD relative to a reference element, the relative position being referred to a relative reference system associated with the reference element. In this case, the step of detecting the position of the vehicle by means of the positioning module provides for detecting the position of the vehicle relative to a global reference system; and the step of determining the position of the HMD by applying a compensation law to the position of the vehicle comprises:
[0024] - applying compensation laws to the detected vehicle position to determine the global position of the reference element, and
[0025] - Based on the global position of the reference element, the relative position of the HMD is converted into a corresponding global position.
[0026] Thanks to this solution, it is possible to identify accurately and in real time the relative position of the HMD within the vehicle in which it is used and then convert it into a global position, i.e. with reference to a three-dimensional reference system originating in the center of the Earth, through operations that can be effectively implemented even by electronic components with limited processing power.
[0027] In one embodiment, the step of determining the relative position of the HMD with respect to the reference element comprises:
[0028] - acquiring at least one image of a reference element located within the vehicle, and
[0029] - Calculate the relative position of the HMD with respect to the reference element by processing the acquired images.
[0030] Preferably, the HMD comprises at least two cameras arranged on opposite sides of a screen of the HMD. In this case, the step of determining the relative position of the HMD with respect to the reference element comprises:
[0031] - using each camera of the HMD for acquiring an image of a reference element located inside the vehicle;
[0032] - Calculate the relative position of each camera with respect to the reference element by processing the corresponding acquired images;
[0033] - Calculate the relative position of the HMD with respect to the reference element by combining the relative positions of the cameras.
[0034] These solutions allow determining the position of the HMD in a simple but at the same time precise and accurate way. Furthermore, the use of reference elements eliminates the need for components (cameras, photo cameras, infrared sensors, pressure sensors, etc.) that are external to the HMD and configured to identify user movements in order to assess the position of the HMD worn by the user.
[0035] In one embodiment, the system comprises a plurality of reference elements, wherein a selected one of the reference elements acts as a primary reference element and the other reference elements act as secondary reference elements. Preferably, the method further comprises the step of: wherein, for each secondary reference element, a reference relationship corresponding to a rotational translational relationship between the secondary reference element relative to the primary reference element is calculated. Even more preferably, the step of determining the relative position of the HMD relative to the reference element comprises:
[0036] - calculating the relative position of the HMD with respect to at least two reference elements;
[0037] - applying the rotation-translation relationship to the relative position of the HMD calculated with respect to each auxiliary reference element, and
[0038] - Calculating a combined relative position of the HMD relative to the primary reference element by combining the relative positions calculated relative to at least two reference elements.
[0039] Thanks to this solution, both the relative position and the global position of the HMD are determined in a precise and robust manner.
[0040] Furthermore, it is possible to configure the system such that it is sufficient to identify any reference element to quickly and reliably determine the position and orientation of the HMD relative to the primary reference element.
[0041] This ensures greater versatility of the system - without significantly increasing its complexity - regardless of the vehicle shape and, at the same time, allows the correct display of augmented reality images when the HMD is pointed at various areas of the vehicle equipped with reference elements (e.g., at one or more of the windshield and side or rear windows of a motor vehicle).
[0042] In one embodiment, the step of determining the field of view volume comprises:
[0043] - calculating the orientation of the HMD relative to the reference element by processing at least one acquired image, and
[0044] - Determining a field of view of the HMD based on a global position of the HMD and an orientation of the HMD relative to a reference element.
[0045] Thanks to this solution, it is possible to accurately identify the field of view of the user wearing the HMD even in case of head movements, such as rotation, tilt, which do not change the position of the HMD. As a result, the field of view volume is also identified more accurately.
[0046] In one embodiment, the position and orientation of the HMD relative to the reference element is determined according to the context, i.e. the pose of the HMD relative to the reference element is determined. Therefore, it is also conceivable that the analysis step of the two images acquired by the two cameras and / or the use of several reference elements related to the position of the HMD as described above, also determines the orientation of the HMD, thereby obtaining the same benefits.
[0047] Advantageously, the above solution allows to accurately determine the pose of the HMD even when the vehicle is in motion. In particular, as opposed to known solutions involving the use of IMUs and other sensors to calculate the position and orientation of the HMD (which, moreover, have a limited accuracy when the vehicle is in motion), the pose of the HMD is determined in a more reliable manner and without the need to implement complex hardware and / or software components.
[0048] In one embodiment, the method further comprises the following steps:
[0049] - Select the boresighting position;
[0050] - displaying the axis alignment image in a visualization position on the HMD, the visualization position being calculated from the axis alignment position and the vehicle position;
[0051] - measure the position difference between the axis calibration position and the visualized position, and
[0052] - determining said compensation law based on said difference.
[0053] In this way, the compensation law can be determined accurately and immediately, regardless of the specific characteristics of the vehicle and / or the implementation options chosen during installation of the system.
[0054] Preferably, the measurement of the difference comprises defining a rotation-translation relationship between a virtual axis calibration position and an axis calibration position, the virtual axis calibration position corresponding to a projection of the visualization position in the three-dimensional reference system. Even more preferably, a compensation law is determined based on the rotation-translation relationship.
[0055] Due to this solution, it is possible to define the compensation law by means of an operation which can also be implemented by systems with limited hardware resources and / or with particularly low computing costs.
[0056] According to an embodiment, the method provides that the axis calibration object is located in the axis calibration position. In this case, defining the rotation-translation relationship preferably comprises:
[0057] - Orienting the HMD so that the axis calibration object is included in the field of view of the HMD;
[0058] - translating the axis calibration image displayed on the HMD until an overlap of the axis calibration image with the axis calibration object in the axis calibration position is obtained, and
[0059] - converting said translation of the axis calibration image in the two-dimensional reference system into a translation and rotation of the virtual axis calibration position in the three-dimensional reference system.
[0060] These calibration steps allow the compensation law to be determined in an extremely simple way. In particular, these system calibration steps can be performed by the user without specific skills and / or training. Furthermore, the solution allows a new calibration to be performed quickly and easily if necessary - for example, if the position of one or more reference elements changes, by moving the system from one vehicle to another, and / or periodically to eliminate deviations that may have occurred during use.
[0061] In one embodiment, the method further comprises the following steps:
[0062] - obtain vehicle movement information, and
[0063] The step of displaying an image associated with the object of interest on the HMD comprises:
[0064] - Modify the image based on the vehicle's movement and time.
[0065] Due to this solution, it is possible to further increase the precision and accuracy in the display of augmented reality images, in particular when the vehicle is in motion.
[0066] Various aspects relate to a system for assisting driving of a vehicle.
[0067] In one embodiment, such a system comprises:
[0068] - HMD;
[0069] - a positioning module mounted on the vehicle, configured to detect the vehicle position;
[0070] - a memory area in which at least one location of interest associated with an object of interest is stored, and
[0071] - A processing unit connected to the positioning module and the HMD and configured to implement the method according to any one of the preceding embodiments.
[0072] Preferably, the system further comprises at least one reference element which can be positioned within the vehicle, and even more preferably, the HMD comprises at least one camera.
[0073] The system is particularly compact and allows providing information to a user driving a vehicle in a precise and reliable manner using limited hardware resources.
[0074] In one embodiment, at least one reference element is backlit to allow for easier identification.
[0075] In one embodiment, the system includes a plurality of reference elements, each reference element including a corresponding identification code to allow the reference elements to be distinguished from one another.
[0076] In one embodiment, the positioning module comprises a GNSS module. Additionally or alternatively, the positioning module may comprise a triangulation module of electromagnetic signals, radar, lidar and / or similar devices.
[0077] In one embodiment, the processing unit stores or may be connected to a positioning data database to retrieve at least one location of interest associated with a corresponding object of interest.
[0078] In one embodiment, the processing unit is operably connected to at least one of:
[0079] - BUS for vehicle communication, and
[0080] - Inertial Measurement Unit,
[0081] To obtain vehicle information.
[0082] Thanks to this solution, the system is able to acquire and display a large amount of useful information to assist in driving the vehicle.
[0083] Further features and advantages of the present invention will become more apparent from the description of the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0084] The present invention will be described below with reference to some examples, which are provided for illustrative and non-limiting purposes and are illustrated in the accompanying drawings. These drawings illustrate different aspects and embodiments of the present invention, and where appropriate, reference numerals illustrating similar structures, components, materials and / or elements in different drawings are indicated by similar reference numerals.
[0085] Figure 1 is a schematic diagram of a system according to an embodiment of the present invention mounted on a vehicle;
[0086] Figure 2is a schematic top view of a moving vehicle in which a system according to an embodiment of the present invention is installed;
[0087] Figure 3 is a flow chart of a method according to an embodiment of the present invention;
[0088] Figure 4a and 4b It is a diagram including Figure 1 and 2 Schematic diagram of changes in posture of the HMD in the system;
[0089] Figure 5a-5c schematically illustrates the field of view visible through the HMD;
[0090] Figure 6 It's a picture. Figure 1 Schematic isometric view of the steps of identifying and determining the orientation and position of the markers of the system;
[0091] Figure 7 is a schematic diagram showing different orientations and positions Figure 1 Axonometric drawings of the three marks of the system;
[0092] Figure 8a and 8b is a schematic diagram illustrating the main steps of a system axis calibration process according to an embodiment of the present invention, and
[0093] Fig. 9 is a schematic diagram illustrating displaying an image associated with a corresponding object of interest on an HMD of the system. DETAILED DESCRIPTION
[0094] Although the present invention is susceptible to various modifications and alternative constructions, certain preferred embodiments are shown in the drawings and described in detail below. In any case, it should be noted that it is not intended to limit the present invention to the specific embodiments illustrated, but on the contrary, the present invention is intended to cover all modifications, alternatives and equivalent constructions falling within the scope of the present invention as defined in the claims.
[0095] Unless otherwise indicated, the use of "such as," "etc," "or" indicates non-exclusive alternatives, without limitation. Unless otherwise indicated, the use of "including" means "including but not limited to."
[0096] With reference to the accompanying drawings, a system 1 according to an embodiment of the invention comprises: a wearable screen, more generally denoted a head-mounted display or HMD 10; a positioning module, for example a GNSS module 20 (Global Navigation Satellite System); a processing unit 30 configured to be connected to the GNSS module 20 and to the HMD 10; and one or more markers 40 of the ArUco type in the example considered.
[0097] The GNSS module 20 is configured to provide, periodically and / or upon request, an indication of a detected position, preferably defined in a three-dimensional reference system originating in the center of the Earth - the three-dimensional reference system being referred to below by the term 'global reference system'. For example, the GNSS module 20 comprises a GPS navigator and is configured to provide a set of geographical coordinates indicating a global position detected by the GNSS module 20 and, therefore, the global position of the vehicle 5.
[0098] The HMD 10 includes a transparent and / or semi-transparent screen 11 to allow a user wearing the HMD 10 to see through the screen 11 (e.g. Figure 5a and 6 In addition, the HMD 10 is configured to - for example, it includes suitable circuitry (not shown) - display an image on the screen 11, which is superimposed on the field of view (FOV) of the user wearing the HMD 10 (hereinafter referred to as the 'field of view FOV of the HMD 10' for the sake of brevity). Figure 2 For this purpose, the HMD 10 may include a local processing unit 13 configured to generate an image to be displayed based on data and / or instructions provided by the processing unit 30.
[0099] Preferably, the HMD 10 includes a pair of cameras 15, which are configured to frame the same area of space from different viewpoints (eg, Figure 5a-5c ). Advantageously, the cameras 15 of the HMD 10 are arranged on opposite sides of the frame of the screen 11 of the HMD. Each of the cameras 15 is configured to acquire one or more images substantially corresponding to the FOV of the HMD 10. In particular, by combining the images provided by the cameras 15 at the same moment, it is possible to determine the field of view FOV of the HMD 10.
[0100] The processing unit 30 includes one or more of a microcontroller, a microprocessor, a general-purpose processor (e.g., a CPU) and / or a graphics processor (e.g., a GPU), a DSP, an FPGA, an ASIC, a memory module, a power module for providing energy to the various components of the processing unit 30, and preferably one or more interface modules for connecting to other equipment and / or connecting to exchange data with other entities (e.g., the HMD 10, the GNSS module 20, a remote server, etc.).
[0101] In particular, the processing unit 30 comprises a memory area 31 and / or is connected to a memory module (not shown) - in which it is possible to store positions PW0-PW3 of objects of interest, which are also denoted by the term world points WP0-WP3 (e.g. Figure 2 As will be clear, in this specification, the term "world point" is used to refer to a physical object - such as a road or a portion thereof (e.g. a curved section of road), a building, a roadblock, a crosswalk, a monument, a billboard, a point of cultural interest, etc. - that is associated with a corresponding location or set of locations (i.e., an area or volume) defined in a global reference system.
[0102] For example, the memory area 31 may be configured to store a database comprising geographical coordinates associated with each of the world points WP0-WP3 and possibly one or more items of information about the same world point WP0-WP3 and / or about one or more images associated therewith.
[0103] Alternatively or additionally, the processing unit 30 may be configured to connect to a remote navigation system 7 (e.g., by accessing a software platform via a connection to the telecommunications network 8) and / or a local navigation system (e.g., a satellite navigator of the vehicle 5) in order to acquire one or more information associated with the detected position of the vehicle 5, the HMD 10 and / or one or more world points WP0-WP3.
[0104] In one embodiment, the processing unit 30 is configured to be connected to an inertial measurement unit or IMU 6, and / or to a data BUS 55 of the vehicle 5 on which the processing unit 30 is installed - such as a CAN bus - to access data provided by on-board sensors (not shown) of the vehicle 5 (e.g.: speed, acceleration, steering angle, etc.), to utilize computing power, user interface and / or connectivity of an on-board computer (not shown) of the vehicle 5.
[0105] In a preferred embodiment, each marker 40 comprises a reference pattern, for example a binary matrix consisting essentially of white or black pixels, which enables it to be easily distinguished from the surroundings. Advantageously, the reference pattern of each marker 40 contains an identification code which makes it possible to uniquely identify said marker 40.
[0106] Preferably, but not in a limiting manner, the marker 40 may include a backlight assembly (not shown) configured to backlight the reference pattern of the marker 40 to simplify image-based identification of the marker 40 and its reference pattern, particularly through processing of images acquired by the camera 15 of the HMD 10.
[0107] The described system 1 may be utilized by a user within a passenger compartment 51 of a vehicle 5 (e.g. Figure 1 ) to achieve accurate and reliable driving assistance method 900 (schematically illustrated in Figure 3 Flowchart illustration of the process), while requiring extremely limited hardware and software resources.
[0108] In the installation step, a marker 40 is positioned inside the passenger compartment 51 of the vehicle 5 to operate as a primary reference element, and preferably a variable number of auxiliary markers 40 (three in the example considered in the figures) can be arranged in the passenger compartment to operate as auxiliary reference elements (box 901).
[0109] In the example considered, the marker 40 is located on or at the windscreen 53 of the vehicle 5. This allows identification of the orientation and position of the HMD 10 relative to the marker 40, and therefore of the field of view FOV of the HMD 10, and possibly of the display region R of the screen 11 on which the image is to be displayed - as described below.
[0110] For example, considering the vehicle 5 as illustrated in which the driving position is on the left side, an exemplary arrangement - which allows the orientation and position of the HMD 10 to be identified in a particularly reliable manner - includes positioning the first mark 40 at one left end of the windshield 53, positioning the second mark 40 in a forward position relative to the driver's position - without obstructing the view of the path - and positioning the third mark 40 in a middle position of the windshield 53 relative to its lateral extension.
[0111] Subsequently, the method 900 comprises steps for calibrating the system 1 , which steps comprise an alignment process and an axis calibration process.
[0112] In the alignment step, the relative positions are first identified in the markers 40 located in the passenger compartment 51. For example, during the alignment process, the HMD 10 is worn by the user maintaining a predetermined driving posture; preferably, with the head - and therefore the HMD 10 - facing the windshield 53 (e.g., as shown in FIG. Figure 4a ).
[0113] Initially, image pairs A+ and A- are acquired by camera 15 at substantially the same time (block 903). Preferably, a series of image pairs A+ and A- are acquired during a time interval in which the HMD 10 remains in the same position or moves slowly (e.g., due to normal posture corrections or changes performed by the user wearing the HMD 10). Given the distance between cameras 15, both images A+ and A- will reproduce substantially the same field of view FOV of the HMD 10, but viewed from different viewing points f1 and f2 (as can be seen in FIG. 1 ). Figure 5a-5c ).
[0114] The images A+ and A- of the camera 15 are processed to identify each marker 40 (block 905). In the example considered, the images A+ and A- are combined together so as to define and identify each marker 40 framed in the images A+ and A- using stereoscopy. For example, the images A+ and A- are identification shapes corresponding to the markers 40, and individual markers 40 are identified by identifying corresponding fiducial patterns.
[0115] By analyzing each acquired image, the translation and orientation of each marker 40 is calculated relative to a reference frame associated with the HMD 10, i.e., a three-dimensional reference frame substantially centered at the viewpoint of the driver wearing the HMD 10 (block 907 and in Figure 6 ). Preferably, the translation and rotation values of the marker 40 relative to each camera 15 are calculated, thereby obtaining two pairs of measurements, which are then combined - for example, by a suitable algorithm implementing an averaging and / or correlation operation - to obtain a corresponding combined rotation and orientation measurement associated with each marker 40. Optionally, a scaling value and / or correction factor may also be determined to compensate for deformations and / or aberrations introduced by the specific characteristics of the camera 15 used. Alternatively or additionally, the position and calculated orientation of each marker 40 relative to the HDM 10 are filtered over time to remove any noise.
[0116] A primary marker 40 is then selected, for example a marker 40 having the best visibility in the acquired images A+ and A−, or a marker 40 having a predefined identification code, and they are preferably calculated by rotational translation—that is, the position of each marker 40 is linked to the position of the primary marker 40 (block 908).
[0117] In a preferred embodiment, for each position of a marker 40 determined by analyzing the image pairs A+ and A− acquired at consecutive moments, a rotational translation is calculated that links the position of each marker 40 to the master marker 40. The rotational translations calculated for each marker 40 are then time-averaged in order to obtain a single rotational translation for each marker 40 relative to the master marker 40.
[0118] In summary, the alignment process allows the identification and advantageously storage of the respective rotational-translational relationships (eg, Figure 7 , schematically represented by dashed arrows, where the vector triplet centered on the marker 40 represents the corresponding reference system centered on each marker 40, and the arrows represent the rotational translation operation linking the auxiliary marker 40 to the main marker 40).
[0119] Otherwise, the axis calibration process of the calibration step establishes a compensation law between the position of the GNSS module 20 and the actual position of the HMD 10 relative to the global reference system and thus allows to calculate an optimal display of the image displayed on the HMD 10 based on the measurements provided by the GNSS module 20. In a preferred embodiment, the compensation law is defined by identifying a rotational-translational relationship between a relative reference system associated with the reference marker 40 and a global reference system associated with the GNSS module 20.
[0120] Special reference Figure 8a and 8b Initially, the vehicle 5, in particular the GNSS module 20, is positioned at a predetermined distance from an alignment object or axis calibration world point WPR (eg, a real physical object) d 909). Therefore, the axis calibration position PWR associated with the axis calibration world point WPR is known. Applicants have determined that straight line segments can be used as axis calibration world points WPR and allow for accurate axis calibration of the system 1. However, Applicants have discovered that polygonal graphics and / or three-dimensional objects allow the user to more easily complete the axis calibration process.
[0121] The axis calibration position PWR and the vehicle position PG measured by the GNSS module 20 are used to determine a corresponding (two-dimensional) axis calibration image ARR to be displayed on the screen 11 of the HMD 10 (block 911 ).
[0122] Preferably, the axis calibration image ARR has a certain shape so as to correspond to the axis calibration world point WPR seen through the HMD 10 .
[0123] The visualized positions PAR on the HMD 10 of the axis calibration images ARR correspond to virtual axis calibration positions PVP associated with corresponding virtual objects or virtual axis calibration points VPR. The virtual axis calibration points VPR are virtual replicas of the axis calibration world points WPR, while the virtual axis calibration positions PVR are replicas of the axis calibration positions PWR calculated based on the vehicle position provided by the GNSS module 20 (in the relative reference frame of the HMD 10).
[0124] Due to the different positions of the GNSS module 20 and the HMD 10, typically the axis calibration image ARR will not be superimposed on the axis calibration world point WPR. Therefore, the axis calibration process provides for the axis calibration image ARR to be translated along the screen 11 of the HMD 10 until the two-dimensional image ARR - in the new visualization position PAR' - overlaps the axis calibration world point WPR - visible through the windshield 53 of the vehicle 5 (block 913). For example, the processing unit 30 may be configured to allow a user to move the axis calibration image ARR, for example via a user interface (not shown) of the processing unit 30 or via a user interface of a device connected to the processing unit (e.g. the HMD 10 itself, or a personal computer, a smartphone, a tablet computer, an onboard computer of the vehicle 5, etc.).
[0125] Therefore, the translation on the screen 11 of the HMD 10 resulting in the superposition of the axis calibration image ARR and the axis calibration world point WPR is processed to determine a compensation law capable of compensating for the difference - or offset - between the axis calibration image ARR and the axis calibration world point WPR (block 915).
[0126] For example, the compensation law may be defined by a compensation matrix based on the rotational-translational relationship between the virtual axis calibration position PVR (which is associated with the virtual axis calibration point VPR to which the axis calibration image ARR corresponds) and the alignment position PWR (which is associated with the reference world point WPR).
[0127] In fact, the axis calibration process allows to simply and effectively determine the rotation-translation relationship between the position of the GNSS module 20 and the position of the HMD 10, which is identifiable thanks to the detection of at least one of the markers 40, i.e. a reference element integrated with the vehicle 5. In other words, the rotation-translation relationship relates the position of the GNSS module 20 to the position of at least one marker 40 located in a static position inside the passenger compartment 51 of the vehicle. This allows to precisely and accurately define the actual position of the HMD 10 in the global coordinate system used by the GNSS module 20.
[0128] In summary, the compensation law allows to correct for errors introduced by the different global positions of the HMD 10 through which the user observes the environment and the global position detected by the GNSS module 20. By applying the compensation law, it is possible to correct the reproduced position of any image on the HMD 10 so that it corresponds to the relative world point WPR, regardless of the movements of the HMD 10 within the passenger compartment 51 due to, for example, movements of the head of the user wearing the HMD 10.
[0129] Once the calibration step has been completed, in the operational steps of method 900, the system 1 is able to display one or more images AR1-3 associated with the corresponding world points WP1-3 on the HMD 10 in real time, thereby positioning them on the screen 11 of the HMD 10 with high accuracy and precision (e.g., Fig. 9 Schematically illustrated in FIG.
[0130] Initially, the pose of the HMD 10 relative to the marker 40 is determined (block 917). In other words, the relative position of the HMD 10 relative to the marker 40, which is mounted within and integrated with the vehicle 5, is determined.
[0131] In a preferred embodiment, a calculation is performed of the pose of each camera 15 relative to each identified marker 40. In other words, a pair of images A+ and A- is acquired by the camera 15 to identify the relative position between the camera 15 and the marker 40.
[0132] For example, one can use the method based on the “ Robust Camera Pose Estimation Using 2D Fiducials Tracking for Real-Time Augmented Reality Systems ”, International conference on Virtual Reality continuum and itsapplications in industry, pp. 431-435, 2004 to identify the pose of each camera 15 relative to the marker 40. In addition or alternatively, the algorithm configured to identify the pose of the camera can be based on the teachings contained in the following document: Madjid Maidi, Jean-Yves Didier, Fakhreddine Ababsa, Malik Mallem: “ A performance study for camera pose estimation using visual marker-based tracking”, and / or Francisco J. Romero-Ramirez, Rafael Munoz-Salinas, Rafael Medina-Carnicer in Image and Vision Computing, Volume 76, 2018: Speeded Up Detection of Squared Fiducial Markers ”.
[0133] The rotation and translation measurements are then combined—eg, by a suitable algorithm implementing averaging and / or correlation operations—to obtain respective measurements of the rotation and orientation of the HMD 10 relative to each of the identified markers 40 .
[0134] Advantageously, the rotational-translational relations between the auxiliary markers 40 and the primary markers 40 determined in the calibration step are applied to the pose of the HMD 10 calculated relative to the auxiliary markers 40 so as to obtain a set of poses of the HMD 10 all with reference to the primary markers 40, said poses then being combined with one another - for example by means of suitable algorithms implementing averaging and / or correlation operations - so as to obtain a combined pose of the HMD 10 relative to the primary markers 40, which is particularly accurate. In other words, the orientation and position of the HMD 10 relative to the primary markers 40, i.e. relative to the relative reference system, is determined.
[0135] In addition, one or more identified marks 40 may be used to define the shape and extent of a display region R of the screen 11 in which the image is to be displayed, for example such that the image is superimposed and displayed on the windshield 53 of the vehicle 5 or a portion thereof (e.g. Figure 4a and 4b Schematically illustrated in FIG.
[0136] Subsequently or in parallel, the vehicle position PG is detected by the GNSS module 20 (block 919 ).
[0137] The vehicle position PG is then modified by applying the compensation law defined during the calibration step in order to determine the position of the HMD 10 relative to the global reference system (blocks 921 and Figure 2 ).
[0138] In a preferred embodiment, the vehicle position PG is modified by the rotational-translational relationship determined during the axis calibration process, thereby allowing the relative position of the HMD 10 determined with respect to the master marker 40 to be converted to a position referenced to a global reference system, such as geographic coordinates.
[0139] In other words, due to the compensation laws, the position and orientation of the HMD 10 relative to the global reference system is determined in real time.
[0140] Based on the orientation defined by the posture of the HMD 10, the field of view volume VOL, ie, the volume of the space included in the field of view FOV of the HMD 10, is determined (block 923). Preferably, the field of view volume VOL (in Figure 2 ) extends within a distance predetermined by the current position of the HMD 10 (ie, the depth of the field of view FOV), which distance may be modified based on parameters acquired by the IMU 6 and / or by the vehicle sensors 5 (such as the speed and / or acceleration of the vehicle 5).
[0141] Subsequently, it is verified whether one or more of the positions of interest PW0 - 3 of the world points WP0 - 3 stored in the memory area 31 are included in the volume of view VOL (block 925 ).
[0142] For each position of interest PW1-3 included in the volume of view VOL, a corresponding visualization position PA1-3 is calculated so that the user wearing the screen sees each image AR1-3 at the corresponding world point WP1-3 (block 927). Advantageously, the shape and other characteristics of the image AR1-3 can be based on information related to the corresponding world point WP0-3 - for example geometric information - preferably contained in the memory area 31 associated with the position of interest PW0-3.
[0143] Then, the images AR1-3 are reproduced on the HMD 10, each in a corresponding visualization position PA1-3. Preferably, each image AR1-3 is displayed if it is included in the display area R of the screen 11 superimposed on the windshield 53 of the vehicle.
[0144] For example, by implementing MATLAB ® and in MathWorks' March 2019 version 9.0 (R2019a) of the " Computer Vision Toolbox ™ Reference "Described in the revised version Computer Vision Toolbox ™' worldToImage A similar algorithm of the ' function can generate images AR1-3 so as to be displayed in respective visualization positions PA1-3 corresponding to as many positions of interest PW1-3 as possible.
[0145] Furthermore, the method 900 provides for modifying the two-dimensional images AR associated with the world points WP1-3 (e.g., by changes in scale, perspective, etc.) as a function of time and / or distance between the position of the HMD 10 and such world points WP1-3 (block 929). In other words, tracking or tracing of each world point WP1-3 is provided as long as it is included in the field of view volume VOL as a function of the movement of the vehicle 5 (e.g., estimated based on changes in the position of the vehicle 5). Furthermore, there is provided for dynamically modifying the shape and / or position of the images AR1-3 displayed on the HMD 10 so that each of the images AR1-3 is correctly associated with the corresponding world point WP1-3.
[0146] In other words, during the operating steps, the method 900 allows the display of two-dimensional images (such as driving trajectories, speed limits, information about road conditions, atmospheric conditions and / or relative to points of interest included in the FOV, such as cities, buildings, monuments, commercial establishments, etc.) on the HMD 10, which are accurately and reliably integrated with the content visible in the field of view FOV of the user wearing the HMD 10. Advantageously, the method 900 is configured to modify the shape and visualization position of the displayed images AR1-3 in real time to adapt to changes in the position of both the vehicle 5 and the HMD 10.
[0147] The invention thus conceived is susceptible of several modifications and variations, all of which are within the scope of the inventive concept.
[0148] For example, in one embodiment, the processing unit 30 is configured to utilize measurements acquired by the IMU and / or sensors of the vehicle 5 in order to increase the positioning accuracy of the image on the HMD 10 and / or provide an image containing more detailed and / or additional information items.
[0149] Finally, during the axis calibration process, the possibility of scaling the axis calibration image ARR may also be provided in order to guarantee an optimal overlap between the latter and the reference world points. In this case, the axis calibration image ARR scaling operation may also be taken into account when evaluating the difference between the axis calibration image ARR and the reference world points WPR.
[0150] Furthermore, during the axis calibration process, there is nothing prohibiting automating the overlay step between the axis calibration image ARR and the axis calibration world point WPR. For example, the processing unit 30 may be configured to identify the axis calibration world point WPR when framed in the field of view FOV of the HMD 10, and then automatically overlay the axis calibration image ARR on the axis calibration world point WPR, or automatically directly determine the difference between the axis calibration images ARR at the axis calibration world point WPR by applying one or more suitable algorithms.
[0151] In one embodiment, the method 900 provides for periodic access to the GNSS data database 7 in order to verify the presence of new world points in a geographical area of interest, for example in a field of view volume.
[0152] As will be apparent, after the alignment process described above, the system 1 can be configured to operate using any number of markers 40. For example, during the operating steps of the method 900, a pair of markers 40 or a single marker 40 can be used to determine the pose of the HMD 10. This allows the computational load required by the system 1 to provide driving assistance in real time to be adjusted, wherein the system 1 has better overall responsiveness to changes caused by the movement of the vehicle 5 and / or the world points WP0-3. In addition, this allows the relationship between the accuracy of the identification of the pose of the HMD 10 and the computational load required by the processing unit 30 to be adjusted.
[0153] In one embodiment, for each world point WP1-3 to be displayed, the method 900 provides for defining a relative virtual point relative to at least one identified marker 40. If one or more auxiliary markers are identified, a rotational translation relationship is applied to the relative virtual points calculated relative to the auxiliary markers so as to redefine these relative virtual points relative to the primary marker. The final virtual point is determined by combining all the relevant virtual points with reference to the primary marker - preferably, by a suitable algorithm including, for example, averaging and / or correlation operations. The final virtual point is then converted into a corresponding image to be displayed by applying a compensation law so as to correct the position of the virtual point in the image defined in the two-dimensional reference system of the surface of the screen 11 of the HMD 10.
[0154] In an embodiment not shown, when a world point (e.g. Figure 2 When a world point WP0 outside the display area R is not included in the field of view volume VOL, it is possible to provide a corresponding virtual indicator, such as an arrow, to be displayed on the HMD 10, for example reproduced at the edge of the display area R, with the tip pointing to the position of the corresponding world point WP0. In addition to the arrow, other information about the world point WP0 outside the display area R, such as the name of the world point WP0, the distance, etc., may be displayed.
[0155] In alternative embodiments, the image AR may be rendered in false color based on distance from the vehicle 5 , driving hazards associated with a relative world point, and / or to convey other information.
[0156] Nothing prohibits implementing and / or omitting one or more optional steps of method 900, just as nothing prohibits performing two or more steps in parallel or in a different order.
[0157] Furthermore, one or more of the implementation details may be replaced by other technically equivalent elements.
[0158] For example, other reference elements 30 may be used in addition to or instead of ArUco markers 30 , such as one or more data matrices, QR codes, and / or other types of reference elements.
[0159] Naturally, it is possible to provide alternative arrangements of markings 40 also consisting of a different (greater or lesser) number of markings 40 ; finally, nothing prohibits having a single marking 40 implement the method 900 described above.
[0160] In addition, the markers 40 can be arranged in additional and / or alternative positions. For example, one or more markers 40 can be positioned at one of the windows of the vehicle 5 or on the rear window to allow the correctly positioned augmented reality image to be reproduced even when the user moves his gaze towards them.
[0161] Preferably, but not limiting, the marking 40 is made based on the teachings contained in the following document: Francisco J. Romero-Ramirez, Rafael Muñoz-Salinas, Rafael Medina-Carnicer, published in Image and Vision Computing, Volume 76, Pages 38-47, 2018: " Speeded up detection of squared fiducial markers ”; S. Garrido-Jurado, R. Muñoz Salinas, FJ Madrid-Cuevas, R. Medina-Carnicer, Pattern Recognition, Vol. 51, pp. 481-491, 2016: Generation of fiducial marker dictionaries using mixed integer linear programming ”; and / or Garrido-Jurado, Sergio et al., Pattern Recognition, Vol. 47, No. 6, pp. 2280-2292, 2014: “ Automatic generation and detection of highly reliable fiducial markers under occlusion ”.
[0162] Furthermore, although in the above exemplary embodiments it has been indicated that compensation laws are applied to the —global—position detected by the GNSS module 20 , nothing prohibits defining corresponding compensation laws applicable to the —relative—position of the HMD 10 within the vehicle 5 determined based on the landmarks 40 .
[0163] Furthermore, nothing prohibits identifying the position and orientation of the HMD 10 by two separate operations, which may be performed sequentially and / or in parallel, rather than by a single operation as described above.
[0164] Although applicants have determined that the use of markings 40 is particularly advantageous, there is nothing prohibiting the implementation of alternative approaches in which the position and orientation of the HMD relative to the windshield and / or other elements of the passenger compartment are identified differently, such as by using a video camera and / or photo camera aimed at the driver and / or one or more motion sensors mounted on the HMD.
[0165] The system 1 may be provided as a kit of parts to be assembled in the passenger compartment of a vehicle. In detail, the kit includes at least a processing unit 30, a dedicated GNSS module 20 - or alternatively, a wired and / or wireless connection element between the processing unit and the vehicle's GNSS module, and an HMD 10, preferably including two cameras 10, and connectable to the processing unit. Alternatively, the processing unit 30 may be configured to operate with one or more commercially available HMDs (e.g., Microsoft HoloLens). Thus, one or more versions of the kit do not necessarily contain an HMD.
[0166] Alternatively, there is nothing prohibiting integrating the processing unit 30 into the vehicle 5 or into a user device (smartphone, tablet, computer, etc.) that can be connected to the vehicle or instantiating a software product configured to implement the method 900 in a processing unit of the vehicle 5 or of the user device.
[0167] Furthermore, the connections between the elements of the system 1, in particular the connection between the processing unit 30 and the HMD 10, can be either wired or wireless, and are preferably wireless. Similarly, the connections with the elements of the system 1 and other elements, such as the connection between the processing unit 30 and the IMU, the ECU (not shown) of the vehicle 5, the infotainment system (not shown) of the vehicle 5, etc., can be wired or wireless.
[0168] In practice, the materials used as well as the possible shapes and sizes may be any according to requirements, without thereby departing from the scope of protection of the following claims.
Claims
1. A method for assisting driving of a vehicle (5), the method being implemented by a system (1) comprising an HMD (10) and a positioning module (20) mounted on the vehicle (5), the method comprising the following steps: - detecting the vehicle position through a positioning module (20); - determining the position of the HMD (10) by applying a compensation law to the vehicle position; - based on the position of the HMD (10), determining a field of view volume (VOL) corresponding to the volume of the space included in the field of view (FOV) of the HMD (10); - comparing a set of positions included in the volume of view (VOL) with at least one position of interest (PW1-3) associated with an object of interest (WP1-3) stored in a memory area (31) of the system (1), and - if one or more locations of interest (PW1-3) are included in the field of view (VOL), calculating a visualization position (PA1-3) of the HMD (10) in which an image (AR1-3) associated with the object of interest (WP1-3) is to be displayed, and displaying the image (AR1-3) in the visualization position (PA1-3) on the HMD (10), the visualization position (PA1-3) causing a user wearing the HMD (10) to see the image corresponding to the object of interest (WP1-3); Further comprising the steps of: -Select the axis calibration position (PWR); - displaying an axis calibration image (ARR) in a visualization position on the HMD (10), the visualization position being calculated from the axis calibration position (PWR) and the vehicle position; - measure the position difference between the axis calibration position (PWR) and the visualized position, and - determining said compensation law based on said position difference.
2. The method according to claim 1, wherein: The system (1) further comprises a reference element (40) arranged in the vehicle (5), and Wherein the method further comprises: - determining a relative position of the HMD (10) with respect to the reference element (40), said relative position being referenced to a relative reference system associated with the reference element (40), and The step of detecting the vehicle position by the positioning module (20) comprises: - detect the vehicle's position relative to a global reference frame, and The step of determining the position of the HMD (10) by applying a compensation rule to the vehicle position comprises: - applying a compensation law to the detected vehicle position to determine a global position of the reference element (40), said global position being referenced to a global reference system, and - Converting the relative position of the HMD (10) into a corresponding global position based on the global position of the reference element (40).
3. The method according to claim 2, wherein the step of determining the relative position of the HMD (10) with respect to the reference element (40) comprises: - acquiring at least one image of a reference element (40) located inside the vehicle (5), and - Calculating the relative position of the HMD (10) with respect to the reference element (40) by processing the acquired images.
4. The method according to claim 3, wherein the HMD (10) comprises at least two cameras arranged on opposite sides of a screen (11) of the HMD (10), and wherein the step of determining the relative position of the HMD (10) with respect to the reference element (40) comprises: - using each camera (15) of the HMD (10) for acquiring an image of a reference element (40) located inside the vehicle (5); - calculating the relative position of each camera (15) with respect to the reference element (40) by processing the corresponding acquired images; - Calculating the relative position of the HMD (10) with respect to the reference element (40) by combining the relative positions of the cameras (15).
5. The method according to any one of claims 2 to 4, wherein: The system (1) comprises a plurality of reference elements (40), wherein a selected reference element serves as a primary reference element and the other reference elements serve as auxiliary reference elements, the method further comprising the steps of: - for each auxiliary reference element, calculating a reference relationship corresponding to the rotational-translational relationship between the auxiliary reference element and the main reference element, and The step of determining the relative position of the HMD (10) relative to the reference element (40) comprises: - calculating the relative position of the HMD (10) relative to at least two reference elements (40); - applying the rotational translational relationship to the relative position of the HMD (10) calculated with respect to each auxiliary reference element, and - calculating a combined relative position of the HMD (10) relative to a primary reference element by combining said relative positions calculated relative to at least two reference elements obtained by said calculating and applying the rotational-translational relationship.
6. The method according to any one of claims 2 to 4, wherein the step of determining the volume of view (VOL) comprises: - calculating the orientation of the HMD (10) relative to the reference element (40) by processing at least one acquired image, and - Determining a field of view (FOV) of the HMD (10) based on the global position of the HMD (10) and the orientation of the HMD (10) relative to the reference element (40).
7. The method of claim 1 , wherein measuring the position difference between the axis calibration position (PWR) and the visualization position comprises: - defining a rotational-translational relationship between a virtual axis calibration position (PVR) and an axis calibration position (PWR), the virtual axis calibration position (PVR) corresponding to the projection of the visualization position in the three-dimensional reference system, and Determining the compensation rule includes: - Using the rotation-translation relationship to determine the compensation law.
8. The method according to claim 7, wherein: The axis calibration object is located in the axis calibration position (PW), and The definition of rotation and translation relationship includes: - orienting the HMD (10) so as to include the axis calibration object in the field of view (FOV) of the HMD (10); - translating the axis calibration image (ARR) displayed on the HMD (10) until an overlap of the axis calibration image (ARR) with the axis calibration object (WPR) in the axis calibration position (PWR) is obtained, and - converting said translation of the axis calibration image (ARR) in the two-dimensional reference system into a translation and rotation of the virtual axis calibration position (PVR) in the three-dimensional reference system.
9. The method according to any one of claims 1 to 4, further comprising the following steps: - obtain vehicle movement information, and The step of displaying an image associated with the object of interest on the HMD (10) comprises: - Modify the image according to the movement of the vehicle (5) and time.
10. A system (1) for assisting driving of a vehicle (5), comprising: -HMD (10); - a positioning module (20) mounted on the vehicle and configured to detect the vehicle position, - a memory area (31) in which at least one location of interest associated with an object of interest is stored, and - a processing unit (30) operatively connected to the positioning module (20), the HMD (10) and configured to implement the method according to any of the preceding claims.
11. The system (1) according to claim 10, further comprising: - at least one reference element (40) positioned inside the vehicle (5).
12. The system (1) according to claim 10 or 11, wherein the processing unit (30) is operatively connected to at least one of the following: - a BUS (55) for vehicle communication, and - Inertial measurement unit (6), To obtain vehicle information.
