Vehicle driving assistance method and system

Abstract

The present invention relates to a method (900) for assisting driving of a vehicle (5). The method (900) can be implemented by a relative system (1) including an HMD (10) and a positioning module (20) mounted on the vehicle (5). In particular, the method (900) comprises the steps of: detecting (919) the vehicle position by the positioning module (20); determining (921) the position of the HMD (10) by applying a compensation law to the vehicle position; determining (923) a view volume (VOL) corresponding to a volume of space configured within the field of view (FOV) of the HMD (10) based on the position of the HMD (10); and comparing (925) a set of positions configured within the view volume (VOL) with at least one position of interest (PW1-3) associated with an object of interest (WP1-3) stored in a memory area (31) of the system (1); and, if the one or more positions of interest (PW1-3) are within the view volume (VOL), calculating (927) a visualization position (PA1-3) of the HMD (10) for displaying an image (AR1-3) associated with the object of interest (WP1-3), and displaying (831) the image (AR1-3) at the visualization position (PA1-3). The visualization positions (PA1-3) are positions where a user wearing the HMD (10) can view images (PA1-3) corresponding to the objects of interest (WP1-3).

Description

[Technical Field]

[0001] The present invention relates to the field of transportation vehicles, and in particular to a method and system for assisting vehicle operation. [Background technology]

[0002] Traditionally, driving information such as travel speed, fuel level, and navigation instructions is displayed on the vehicle's dashboard or on the vehicle's infotainment screen, but both the dashboard and the screen are often located in a way that requires the driver to at least partially look away from the road environment, compromising both driving safety and the potential for information utilization.

[0003] In the fields of automobiles and aircraft, a "head-up display," or "HUD" for short, has been proposed as a partial solution to this problem. HUD is a system that projects images onto the windshield of a car. In particular, HUD can project information directly onto the windshield of a car, allowing the driver to keep their eyes on the road and concentrate on driving.

[0004] However, the current HUD standard, called HUD 1.0, is used only to display redundant information provided by classic automotive instruments. Furthermore, the applicant has realized that HUD technology cannot effectively depict augmented reality elements. In fact, the expansion capabilities required by a projection system to fully cover the driver's field of view far exceed those available with current state-of-the-art technology. In particular, no HUD is capable of utilizing the entire primary field of view, which is essentially defined by the vehicle's windshield, as well as secondary fields of view, such as one or more side windows.

[0005] Alongside HUD systems, more recently, wearable screen-based systems, better known as "head-mounted displays", or HMD for short, have been proposed, which consist of a transparent or semi-transparent screen on which images can be reproduced to provide driving assistance information to a user wearing the HMD while, for example, driving a vehicle.

[0006] For example, U.S. Patent Application Publication No. 2016 / 084661 describes a system and method that functions as a driving tool and provides feedback to a driver, such as real-time visual feedback provided via an augmented reality device. The guidance system collects vehicle-related information and driver information, such as the driver's gaze direction as determined by an HUD, and uses this input information to generate real-time visual feedback in the form of virtual guidelines and other driving recommendations. This driving recommendation information is presented to the driver via an augmented reality device, such as a HUD display, which may project virtual guidance lines onto the vehicle's windshield, superimposed on the actual road surface visible to the driver, to indicate a line or route the driver should follow. Additionally, other driving recommendations, such as braking, accelerating, steering, and shifting, may also be made.

[0007] The applicant has realized that the method proposed in U.S. Patent Application No. 2016 / 084661 for determining the field of view observed by the driver and therefore effectively displaying real images is complex to implement. In particular, analyzing the driver's line of sight as described in U.S. Patent Application No. 2016 / 084661 requires complex implementation in terms of hardware and software in order to identify with sufficient accuracy the field of view observed by the driver and to determine the size and position of one or more augmented reality images on the HMD or HUD.

[0008] European Patent No. 2933707 also describes a method for dynamically orienting what is presented by an HMD. The method includes collecting HMD movement data using at least one sensor installed in the HMD worn by the vehicle driver and collecting vehicle movement data using at least one sensor installed in the vehicle. The method then performs an analysis of the HMD movement data and the vehicle movement data to detect any discrepancies therebetween. Based on the detected discrepancies, the orientation of the HMD device relative to the vehicle is calculated, and based on the newly calculated orientation, normal data is calculated to be presented on the screen of the HMD device.

[0009] The method proposed in EP 2933707 can determine the orientation of the HMD, but does not achieve satisfactory accuracy and precision. Furthermore, this method requires high computational resources to calculate and generate data from consistently presented HMD data based on a comparison of images of the scenario seen by the driver through the vehicle windshield. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] U.S. Patent Application No. 2016 / 084661 [Patent Document 2] European Patent No. 2933707 Summary of the Invention

[0011] SUMMARY OF THE INVENTION An object of the present invention is to overcome the drawbacks of the prior art.

[0012] In particular, it is an object of the present invention to present a method and system for driving assistance that is able to provide accurate and reliable indications to assist a user while driving a vehicle.

[0013] The object of the present invention is to present a method and a system for reproducing elements of augmented reality adapted to improve the user's driving experience while using a vehicle.

[0014] These and other objects of the present invention are achieved by a method and system incorporating the features of the appended claims, which form an integral part of this specification. [Means for solving the problem]

[0015] In one embodiment, the method of the present invention comprises: detecting a vehicle position by a positioning module mounted on the vehicle; determining the position of the HMD by applying a compensation law to the vehicle position; determining a view volume corresponding to a volume of space configured within a field of view of the HMD based on a position of the HMD; comparing the set of locations configured within the view volume with at least one location of interest associated with the object of interest stored in a memory area of ​​the system; If there are one or more points of interest within the view volume, a visualization position of the HMD at which an image associated with the object of interest is displayed is calculated, and the image is displayed on the HMD at the visualization position. Advantageously, the visualization position is such that a user wearing the screen can see an image corresponding to the object of interest.

[0016] This solution allows the augmented reality image to be accurately displayed within the field of view of the user wearing the HMD based solely on position information. In particular, it can effectively correct display errors caused by different positions of the positioning module and the HMD. In fact, even a small distance between these two elements can cause significant inaccuracies in the display of the augmented reality image, which can reduce the usefulness of the information related to the augmented reality image or even worsen the user's driving condition.

[0017] Advantageously, the method according to the present invention only needs to acquire and process location information such as that provided by a global navigation system or GNSS (e.g., GPS, Galileo, GLONASS, Beidou, etc.), but does not need to process the acquired images to recognize objects visible through the vehicle windshield in order to correctly display the augmented reality images, thereby enabling it to operate in real time with high precision and accuracy in displaying augmented reality images, at substantially low computational cost and hardware requirements. In one embodiment, the system further comprises a reference element located within the vehicle, and detecting a vehicle position with the positioning module comprises detecting the vehicle position relative to a global reference frame.

[0018] Preferably, the method of the present invention further comprises determining a relative position of the HMD with respect to the reference element, the relative position being referenced to a relative reference system associated with the reference element. In this case, the step of detecting a vehicle position by the positioning module provides detecting the vehicle position with respect to a global reference frame; Meanwhile, the step of determining the HMD position by applying the compensation law to the vehicle position includes: applying a compensation law to the detected vehicle position to determine the global position of the reference element; and converting the relative position of the HMD into a corresponding global position based on the global position of the reference element.

[0019] This solution allows the relative position within the vehicle in which the HMD is used to be determined with precision in real time and converted into a global position referenced to a 3D reference system with the center of the Earth as its origin, in an operation that can be carried out efficiently even with electronic components with low processing power.

[0020] In one embodiment, the step of determining the relative position of the HMD with respect to the reference element comprises: acquiring at least one image of the reference element located inside the vehicle; calculating the relative position of the HMD with respect to the reference element by processing the acquired images; Includes.

[0021] Preferably, the HMD comprises at least two cameras arranged on opposite sides of the screen of the HMD. In this case, the step of determining the relative position of the HMD with respect to the reference element comprises: acquiring images of the reference elements located inside the vehicle using each camera of the HMD; calculating a relative position of each camera with respect to the reference element by processing each acquired image; combining the relative positions of each camera to calculate the relative position of the HMD with respect to the reference element; Includes.

[0022] These solutions allow the position of the HMD to be determined in a simple manner, while being accurate and precise. Furthermore, the use of reference elements obviates the need for components (video cameras, photo cameras, infrared sensors, pressure sensors, etc.) external to the HMD and configured to identify user movements in order to assess the position of the HMD worn by the user.

[0023] In one embodiment, the system of the present invention comprises a plurality of reference elements, one selected of which acts as a primary reference element and the other reference elements act as secondary reference elements. Preferably, the method of the present invention further comprises the step of calculating, for each secondary reference element, a reference relationship corresponding to a rotational-translational relationship between the secondary reference element and the primary reference element. More preferably, the step of determining the relative position of the HMD with respect to the reference element comprises: calculating the relative position of the HMD with respect to at least two reference elements; applying the rotation-translation relationship to the relative position of the HMD calculated with respect to each sub-reference element; calculating a combined relative position of the HMD with respect to the primary reference element by combining the relative positions calculated for the at least two reference elements; Includes.

[0024] Thanks to this solution, both the relative position and the global position of the HMD are determined in an accurate and robust manner.

[0025] Furthermore, the system may be configured such that it is sufficient to identify any reference element in order to quickly and reliably determine the position and orientation of the HMD relative to the primary reference element.

[0026] This ensures great versatility of the system regardless of the shape of the vehicle without substantially increasing its complexity, and at the same time allows the augmented reality image to be displayed correctly when the HMD is pointed at various areas of the vehicle with reference elements (e.g., one or more of the windshield and side windows or rear window of a car).

[0027] In one embodiment, the step of determining the view volume of the present invention comprises: calculating an orientation of the HMD relative to the reference element by processing at least one acquired image; determining a field of view of the HMD based on the global position of the HMD and an orientation of the HMD relative to the reference element; Includes.

[0028] This solution allows accurate determination of the field of view of a user wearing the HMD, even with head movements (rotation, tilt, etc.) that do not change the position of the HMD, and as a result, the view volume can be determined more accurately.

[0029] In one embodiment, the position and orientation of the HMD relative to the reference element is contextually determined, i.e. the pose of the HMD relative to the reference element is determined. Thus, the analysis step of two images acquired by two cameras and / or the use of multiple reference elements as described above in relation to the position of the HMD is also envisaged, with the same advantages being obtained for determining the pose of the HMD.

[0030] Advantageously, the above-described solution allows for accurate determination of the pose of the HMD even when the vehicle is moving. In particular, the pose of the HMD is determined in a more reliable manner and without the need to implement complex hardware and / or software components, in contrast to known solutions that use IMUs and other sensors to calculate the position and orientation of the HMD with even limited accuracy when the vehicle is moving.

[0031] In one embodiment, the method of the present invention comprises: selecting a boresight location; displaying a boresight image at a visualization position on the HMD, the visualization position being calculated according to the boresight position and the vehicle position; measuring a positional deviation between the boresight position and the visualization position; determining the compensation law based on the discrepancy; Equipped with.

[0032] In this way, the compensation law can be accurately and immediately determined, regardless of the particular characteristics of the vehicle and / or the implementation options selected during installation of the system.

[0033] Preferably, measuring the mismatch includes defining a rotational-translational relationship between a virtual boresight position and said boresight position, said virtual boresight position corresponding to a projection of said visualization position in a three-dimensional reference system. More preferably, said compensation law is determined based on said rotational-translational relationship.

[0034] Thanks to this solution, it is possible to define the compensation law in systems with limited hardware resources and / or with operations that can also be implemented at particularly low computational cost.

[0035] According to one embodiment, the method of the present invention provides that a boresight object is located at said boresight location. In this case, defining the rotation-translation relationship preferably involves: orienting the HMD to include the boresight object within a field of view of the HMD; translating the boresight image displayed on the HMD until it overlaps with the boresight object at the boresight location; transforming a translation of the boresight image in a two-dimensional frame of reference into a translation and rotation of the virtual boresight position in the three-dimensional frame of reference; Includes.

[0036] These calibration steps allow the compensation law to be determined in a very simple way. In particular, these system calibration steps can be performed by a user without special skills and / or training. Furthermore, this solution allows new calibrations to be performed quickly and easily, if necessary, when the position of one or more reference elements is changed, for example by moving the system from one vehicle to another, and / or periodically to cancel deviations that may occur during use.

[0037] In one embodiment, the method of the present invention further comprises: acquiring vehicle movement information; The step of displaying an image associated with the subject of interest on the HMD includes: modifying the image as a function of the vehicle's motion and time; Equipped with.

[0038] This solution allows for greater precision and accuracy in displaying augmented reality images, especially when the vehicle is moving.

[0039] Another embodiment of the present invention relates to a system for assisting driving of a vehicle.

[0040] In one embodiment, such a system comprises: HMD and a positioning module mounted on the vehicle configured to detect the vehicle's position; a memory area in which at least one location of interest associated with the object of interest is stored; a processing unit connected to the positioning module and the HMD and configured to perform a method according to any one of the above-mentioned embodiments; Equipped with.

[0041] Preferably, the system further comprises at least one reference element that can be placed inside the vehicle, and more preferably, the HMD comprises at least one camera.

[0042] The system is particularly compact and is able to provide information to the user operating the vehicle in an accurate and reliable manner using limited hardware resources.

[0043] In one embodiment, at least one reference element is backlit to make it easier to identify.

[0044] In one embodiment, the system comprises a plurality of reference elements, each including a respective identification code, such that the reference elements can be distinguished from one another.

[0045] In one embodiment, the positioning module comprises a GNSS module. Additionally or alternatively, the positioning module may comprise an electromagnetic signal triangulation module, radar, lidar, and / or similar devices.

[0046] In an embodiment, the processing unit stores or is connectable to a positioning data database for obtaining at least one position of interest associated with a corresponding object of interest.

[0047] In one embodiment, the processing unit is operatively connected to at least one of a vehicle communication bus and an inertial measurement unit for obtaining vehicle information.

[0048] Thanks to this solution, the system is able to obtain and display a considerable amount of useful information to assist the driving of the vehicle.

[0049] Further features and advantages of the present invention will become more apparent from the description of the accompanying drawings.

[0050] The present invention will now be described with reference to several examples shown in the accompanying drawings, which are provided for illustrative and non-limiting purposes, and which illustrate different aspects and embodiments of the invention, and where appropriate, reference numerals indicating like structures, components, materials and / or elements in different figures are designated with like reference numerals. [Brief explanation of the drawings]

[0051] [Figure 1] 1 is a schematic diagram of a system according to an embodiment of the present invention installed in a vehicle; [Figure 2] 1 is a schematic top view of a moving vehicle in which a system according to an embodiment of the present invention is installed; [Figure 3] 1 is a flowchart of a method according to an embodiment of the present invention. [Figure 4a] 3 is a schematic diagram showing variations in the posture of an HMD configured in the system of FIGS. 1 and 2. FIG. [Figure 4b] 3 is a schematic diagram showing variations in the posture of an HMD configured in the system of FIGS. 1 and 2. FIG. [Figure 5a]FIG. 1 is a diagram schematically illustrating the field of view seen through an HMD. [Figure 5b] FIG. 1 is a diagram schematically illustrating the field of view seen through an HMD. [Figure 5c] FIG. 1 is a diagram schematically illustrating the field of view seen through an HMD. [Figure 6] FIG. 2 is a schematic isometric view illustrating the marker orientation and position identification and determination steps of the system of FIG. 1. [Figure 7] 2A-2C are axial views schematically illustrating three markers of the system of FIG. 1 with different orientations and positions. [Figure 8a] 1 is a schematic diagram illustrating the salient steps of a system boresighting procedure according to an embodiment of the present invention. [Figure 8b] 1 is a schematic diagram illustrating the salient steps of a system boresighting procedure according to an embodiment of the present invention. [Figure 9] 10 is a schematic diagram illustrating the display of an image associated with a corresponding object of interest in the HMD of the system of the present invention. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0052] While the invention is susceptible to various modifications and alternative constructions, certain preferred embodiments are shown in the drawings and will be described in detail below. It should be noted that in all cases there is no intention to limit the invention to the specific embodiments illustrated, but rather the invention is intended to cover all modifications, alternatives, and equivalent constructions falling within the scope of the invention as defined by the claims.

[0053] The use of "for example," "e.g.," and "or" indicates open-ended, non-exclusive alternatives unless otherwise stated. The use of "including" means "including, but not limited to," unless otherwise stated.

[0054] With reference to the figure, a system 1 according to an embodiment of the invention comprises a wearable screen, more generally indicated as a head-mounted display, or HMD 10, a positioning module, for example a GNSS (Global Navigation Satellite System) module 20, a processing unit 30 configured to connect to the GNSS module 20 and to the HMD 10, and one or more markers 40 of the ArUco type in the example considered.

[0055] The GNSS module 20 is preferably configured to provide, periodically and / or on demand, an indication of its detected position defined in a three-dimensional reference frame that originates at the center of the Earth (hereinafter referred to as the "global reference frame"). For example, the GNSS module 20 comprises a GPS navigator and is configured to provide a set of geographic coordinates indicative of the global position detected by the GNSS module 20, and hence the position of the vehicle 5.

[0056] The HMD 10 comprises a transparent and / or translucent screen 11 to allow a user wearing the HMD 10 to see through the screen 11, as shown schematically in Figures 5a and 6. Furthermore, the HMD 10 is configured, for example with appropriate circuitry (not shown), to display images on the screen 11 that are superimposed on what is present in the field of view (FOV) of a user wearing the HMD 10 (for brevity, referred to below as the "field of view FOV of the HMD 10" (shown schematically in Figure 2)), thus creating an augmented reality effect. To this end, the HMD 10 may comprise a local processing unit 13 configured to generate the images to be displayed based on data and / or instructions provided by a processing unit 30.

[0057] Preferably, the HMD 10 comprises a pair of cameras 15 configured to frame the same region of space from different viewpoints (as shown schematically in Figures 5a to 5c). Advantageously, the cameras 15 of the HMD 10 are positioned on opposite sides of the frame of the HMD's screen 11. Each of the cameras 15 is configured to acquire one or more images that substantially correspond to the FOV of the HMD 10. In particular, by combining the images provided by the cameras 15 at the same moment, it is possible to determine the field of view FOV of the HMD 10.

[0058] The processing unit 30 comprises a microcontroller, microprocessor, general purpose processor (e.g., CPU) and / or graphics processor (e.g., GPU), DSP, FPGA, ASIC, memory modules, a power supply module for supplying energy to the various components of the processing unit 30, and preferably one or more interface modules for connecting to other devices and / or exchanging data with other entities (e.g., HMD 10, GNSS module 20, remote server, etc.).

[0059] In particular, the processing unit 30 comprises a memory area 31 (as shown schematically in FIG. 2) and / or is connected to a memory module (not shown) capable of storing positions of interest PW0 to PW3, also denoted by the terms world points WP0 to WP3. As will be apparent, the term world point is used herein to denote a physical object such as a road or part thereof (e.g., a curve in a road), a building, a road block, a crosswalk, a monument, a billboard, a cultural landmark, etc., which is associated with a corresponding position or set of positions (i.e., an area or volume) defined in a global reference frame.

[0060] For example, the memory area 31 may be configured to store a database of geographic coordinates associated with each of the world points WP0-WP3 and, possibly, one or more items of information relating to the same world point WP0-WP3 and / or one or more images associated therewith.

[0061] Alternatively or additionally, the processing unit 30 may be configured to connect to a remote navigation system 7 (e.g., by accessing a software platform via a connection to a telecommunications network 8) and / or to a local navigation system (e.g., a satellite navigator in the vehicle 5) in order to obtain one or more items of information related to the detected position of the vehicle 5, of the HMD 10, and / or of one or more world points WP0-WP3.

[0062] In one embodiment, the processing unit 30 is configured to connect to an inertial measurement unit, or IMU 6, and / or a data bus 55 (e.g., CAN bus) of the vehicle 5 on which the processing unit 30 is mounted, in order to access data (e.g., speed, acceleration, steering angle, etc.) provided by on-board sensors (not shown) of the vehicle 5, utilizing the computing power, user interface, and / or connectivity of the on-board computer (not shown) of the vehicle 5.

[0063] In a preferred embodiment, each marker 40 comprises a fiduciary pattern, e.g., a binary matrix of substantially white or black pixels, which allows it to be easily distinguished from its surrounding environment. Advantageously, the fiduciary pattern of each marker 40 includes an identification code that allows that marker 40 to be uniquely identified.

[0064] Preferably, but not by way of limitation, the marker 40 may be provided with a backlight assembly (not shown) configured to backlight the fiduciary pattern of the marker 40 so as to simplify identification of the marker 40 and its fiduciary pattern based on images, particularly through processing of images acquired by the camera 15 of the HMD 10.

[0065] The described system 1 can be utilized by a user in the passenger compartment 51 of a vehicle 5 (as shown schematically in FIG. 1) and can implement a method 900 of driving assistance (shown by the flowchart in FIG. 3) that is precise and reliable, while at the same time requiring particularly limited hardware and software resources.

[0066] In the installation step, a marker 40 is placed inside the passenger compartment 51 of the vehicle 5 to act as a primary reference element, and preferably a variable number of secondary markers 40, three in the example considered in the figure, may be placed inside the passenger compartment to act as secondary reference elements (block 901).

[0067] In the considered embodiment, the marker 40 is placed on or at the windshield 53 of the vehicle 5. This allows the orientation and position of the HMD 10 relative to the marker 40, and thus the field of view FOV of the HMD 10, and possibly the display area R of the screen 11 for displaying images, to be determined, as will be explained below.

[0068] For example, considering a vehicle 5 with a driving position on the left side as shown, an exemplary arrangement, which allows the orientation and position of the HMD 10 to be determined in a particularly reliable manner, includes placing a first marker 40 at one left edge of the windshield 53, placing a second marker 40 in a position directly in front of the driving position that does not obstruct the view of the route, and placing a third marker 40 in a central position relative to the lateral extension of the windshield 53.

[0069] Subsequently, the method 900 includes a procedure for calibrating the system 1, which comprises an alignment procedure and a boresight procedure.

[0070] In the alignment procedure, first, relative positions are identified between the markers 40 arranged in the passenger compartment 51. For example, during the alignment procedure, the HMD 10 is worn by a user who maintains a predetermined driving posture, preferably with the head, and therefore the HMD 10, facing the windshield 53 (e.g., as shown in FIG. 4a).

[0071] Initially, a pair of images A+ and A− are acquired at substantially the same instant via camera 15 (block 903). Preferably, the successive sets of images A+ and A− are acquired during a time interval in which HMD 10 is held in the same position or slowly moved (e.g., by normal posture corrections or changes made by a user wearing HMD 10). Considering the distance between cameras 15, both images A+ and A− substantially reproduce the same field of view FOV of HMD 10, but are observed from different observation points f1 and f2 (as can be seen in FIGS. 5a-5c).

[0072] Images A+, A- of camera 15 are processed (block 905) to recognize each marker 40. In the example considered, images A+ and A- are combined together to utilize stereoscopic vision to define and identify each marker 40 outlined in images A+ and A-. For example, images A+ and A- identify shapes corresponding to markers 40, and single markers 40 are recognized by identifying corresponding fiduciary patterns.

[0073] Each acquired image is analyzed to calculate the translation and orientation of each marker 40 relative to a reference frame associated with the HMD 10, i.e., a three-dimensional reference frame substantially centered on the viewpoint of the driver wearing the HMD 10 (block 907, shown schematically in FIG. 6). Preferably, translation and rotation values ​​of the markers 40 relative to each camera 15 are calculated, thus obtaining two sets of measurements that are then combined, for example by a suitable algorithm implementing averaging and / or correlation operations, to obtain corresponding combined rotation and orientation measurements associated with each marker 40. Optionally, scale values ​​and / or correction factors may be determined to compensate for deformations and / or aberrations introduced by the particular characteristics of the used camera 15. Alternatively or additionally, the position and calculated orientation of each marker 40 relative to the HDM 10 is filtered over time to remove any noise.

[0074] Next, the primary markers 40 are selected, for example, the markers 40 with the best visibility or the markers 40 with predefined identification codes in the acquired images A+ and A-, which are preferably calculated by rotational translation, i.e., a rotational translation that relates the position of each marker 40 to the position of the primary marker 40 (block 908).

[0075] In a preferred embodiment, a rotational translation relating the position of each marker 40 to the primary marker 40 is calculated for each position of the marker 40 determined by analyzing sets of images A+ and A− acquired at successive moments in time. The calculated rotational translations for each marker 40 are then time-averaged to obtain a single rotational translation of each marker 40 relative to the primary marker 40.

[0076] In summary, the alignment procedure makes it possible to identify and advantageously store the respective rotational-translation relationships relating each of the primary and secondary markers 40 (as represented schematically by dashed arrows in Figure 7, a triad of vectors centered at the markers 40 represents the respective reference frames centered at each marker 40, and the arrows represent the rotational-translation operations relating the secondary markers 40 to the primary marker 40).

[0077] Additionally, the boresight procedure of the calibration step establishes a compensation law between the position of the GNSS module 20 and the actual position of the HMD 10, with respect to a global reference frame, so that an optimal display of the image displayed on the HMD 10 can be calculated based on measurements provided by the GNSS module 20. In a preferred embodiment, the compensation law is defined by specifying a rotational-translational relationship between the relative reference frame associated with the fiducial marker 40 and the global reference frame associated with the GNSS module 20.

[0078] 8a and 8b, initially, the vehicle 5, and in particular the GNSS module 20, is positioned at a predetermined distance d and in a known direction from an alignment object, i.e., the boresight world point WPR, e.g., a real physical object (block 909). Therefore, the boresight position PWR associated with the boresight world point WPR is known. Applicant has determined that a straight line segment can be used as the boresight world point WPR and accurate boresighting of the system 1 is possible. However, Applicant has found that polygonal shapes and / or three-dimensional objects allow a user to more conveniently complete the boresight procedure.

[0079] The boresight position PWR and the vehicle position PG measured by the GNSS module 20 are used to determine a corresponding (2D) boresight image ARR to be displayed on the screen 11 of the HMD 10 (block 911).

[0080] Preferably, the boresight image ARR has a shape that corresponds to the boresight world point WPR as seen through the HMD 10.

[0081] The visualized position PAR of the boresight image ARR on the HMD 10 corresponds to the virtual boresight position PVP associated with the corresponding virtual object, i.e., the virtual boresight point VPR. The virtual boresight point VPR is a virtual replica of the boresight world point WPR, and the virtual boresight position PVR is a replica of the boresight position PWR calculated based on the vehicle position provided by the GNSS module 20 in the relative reference frame of the HMD 10.

[0082] Typically, due to different positions of the GNSS module 20 and the HMD 10, the boresight image ARR will not be superimposed on the boresight world point WPR. Therefore, the boresight procedure provides that the boresight image ARR is translated along the screen 11 of the HMD 10, visible through the windshield 53 of the vehicle 5 (block 913), until the two-dimensional image ARR overlaps with the boresight world point WPR at a new visualization position PAR'. For example, the processing unit 30 may be configured to allow a user to move the boresight image ARR, e.g., via a user interface (not shown) of the processing unit 30 or via a user interface of a device connected to the processing unit (e.g., the HMD 10 itself, or a personal computer, smartphone, tablet, on-board computer of the vehicle 5, etc.).

[0083] Therefore, the translation on the screen 11 of the HMD 10 that results in the superposition of the boresight image ARR and the boresight world point WPR is processed to determine a compensation law that can compensate for the discrepancy, or offset, between the boresight image ARR and the boresight world point WPR (block 915).

[0084] For example, the compensation law can be defined by a compensation matrix based on the rotation-translation relationship between the virtual boresight position PVR (the boresight image ARR is associated with the corresponding virtual boresight point VPR) and the alignment position PWR (related to the reference world point WPR).

[0085] Indeed, the boresight procedure makes it possible to simply and effectively determine the rotational-translational relationship between the position of the GNSS module 20 and the position of the HMD 10, which can be determined thanks to the detection of at least one marker 40, i.e. a reference element integrated with the vehicle 5. In other words, the rotational-translational relationship relates the position of the GNSS module 20 to the position of at least one marker 40 located at a stationary position within the vehicle passenger compartment 51. This allows to accurately and precisely define the actual position of the HMD 10 in the global coordinate system used by the GNSS module 20.

[0086] In summary, the compensation law makes it possible to correct errors caused by differences between the global position of the HMD 10, from which the user observes the environment, and the global position detected by the GNSS module 20. By applying the compensation law, it is possible to correct the reproduced position of any image on the HMD 10 to correspond to the relative world point WPR, regardless of the movement of the HMD 10 within the passenger compartment 51, for example, due to movement of the head of the user wearing the HMD 10.

[0087] Once the calibration step is complete, in the operational step of method 900, system 1 can position one or more images AR1-3 associated with corresponding world points WP1-3 on the screen 11 of HMD 10 with high accuracy and precision and display them in real time on HMD 10 (as shown schematically in Figure 9).

[0088] First, the pose of the HMD 10 with respect to the marker 40 is determined (block 917). In other words, the relative position of the HMD 10 with respect to the marker 40 that is mounted inside and integral with the vehicle 5 is determined.

[0089] In a preferred embodiment, a calculation is performed of the pose of each camera 15 with respect to each recognized marker 40. In other words, a pair of images A+ and A− are acquired by the camera 15 to determine the relative position between the camera 15 and the marker 40.

[0090] For example, the pose of each camera 15 relative to the marker 40 can be determined by an algorithm based on that described below. F. Ababsa, M. Mallem, "Robust Camera Pose Estimation Using 2D Fiducials Tracking for Real-Time Augmented Reality Systems" International conference on Virtual Reality continuum and its applications in industry, pp.431-435, 2004 Additionally or alternatively, an algorithm configured to identify the camera pose may be based on the teachings contained below. Madjid Maidi, Jean-Yves Didier, Fakhreddine Ababsa, Malik Mallem: "A performance study for camera pose estimation using visual marker-based tracking", published in Machine Vision and Application, Volume 21, Issue 3, pages 265-376, year 2010, and / or in Francisco J. Romero-Ramirez, Rafael Munoz-Salinas, Rafael Medina-Carnicer, 2010: "Speeded Up Detection of Squared Fiducial Markers" published in Image and Vision Computing, Volume 76, year 2018.

[0091] The rotational and translational measurements are then combined, for example by a suitable algorithm implementing an averaging and / or correlation operation, to obtain corresponding measurements of the rotation and orientation of the HMD 10 relative to each of the identified markers 40.

[0092] Advantageously, the rotational-translational relationships between the secondary markers 40 and the primary marker 40 determined in the calibration step are applied to the postures of the HMD 10 calculated with reference to the secondary markers 40 to obtain a set of poses of the HMD 10 all with reference to the primary marker 40, which are then combined with each other, for example by means of a suitable algorithm performing averaging and / or correlation operations, to obtain a particularly precise combined pose of the HMD 10 with respect to the primary marker 40. In other words, the orientation and position of the HMD 10 with respect to the primary marker 40, i.e. with respect to the relative reference frame, are determined.

[0093] Furthermore, the one or more identified markers 40 can be used to define the shape and extent of the display area R of the screen 11 in which the image is displayed, for example so that the image is displayed superimposed on the windshield 53 of the vehicle 5 or a portion thereof (as shown schematically in Figures 4a and 4b).

[0094] Subsequently or in parallel, the vehicle position PG is detected via the GNSS module 20 (block 919).

[0095] The vehicle position PG is then corrected by applying the compensation law defined during the calibration step to determine the position of the HMD 10 relative to the global reference frame (block 921 and FIG. 2).

[0096] In a preferred embodiment, the vehicle position PG is corrected through the rotational-translational relationship determined during the boresight procedure, allowing the relative position of the HMD 10 determined with respect to the primary marker 40 to be transformed into a position referenced to a global reference system, e.g., geographic coordinates.

[0097] In other words, thanks to the compensation law, the position and orientation of the HMD 10 relative to the global reference frame is determined in real time.

[0098] Based on the orientation defined by the pose of the HMD 10, a view volume VOL is determined (block 923), i.e., a volume of space constituted in the field of view FOV of the HMD 10. Preferably, the view volume VOL (shown schematically in FIG. 2) extends within a distance predetermined by the current position of the HMD 10, i.e., within the depth of the field of view FOV, and is possibly modified based on parameters acquired by the IMU 6 and / or by vehicle sensors 5, such as the speed and / or acceleration of the vehicle 5.

[0099] It is then verified whether one or more of the positions of interest PWO-3 of the world point WPO-3 stored in the memory area 31 are configured within the view volume VOL (block 925).

[0100] For each position of interest PW1-3 configured in the view volume VOL, a corresponding visualization position PA1-3 is calculated (block 927) so that a user wearing the screen would see each image AR1-3 at a respective world point WP1-3. Advantageously, the shape and other properties of the images AR1-3 can be based on information associated with the corresponding world point WP0-3, e.g., geometric information, which is preferably contained in the memory area 31 associated with the position of interest PW0-3.

[0101] The images AR1-3 are then reproduced at corresponding visualization positions PA1-3 on the HMD 10. Preferably, each of the images AR1-3 is displayed in a display region R of the screen 11 superimposed on the windshield 53 of the vehicle.

[0102] For example, by implementing a similar algorithm of the "worldToImage" function of the Computer Vision Toolbox®, which is comprised of the software product MATLAB® and described in the March 2019 revision of "Computer Vision Toolbox® Reference," version 9.0 (Release R2019a) of The MathWorks, Inc., it is possible to generate images AR1-3 to be displayed at their respective visualization positions PA1-3 corresponding to as many of the interest positions PW1-3 as possible.

[0103] Furthermore, the method 900 provides for modifying the two-dimensional images AR associated with the world points WP1-3 (e.g., through variations in scale, viewpoint, etc.) as a function of time and / or distance between the position of the HMD 10 and such world points WP1-3 (block 929). That is, tracking of each world point WP1-3 is provided as a function of movement of the vehicle 5 (e.g., estimated based on variations in the position of the vehicle 5) as long as it is configured within the view volume VOL. Furthermore, dynamically modifying the shape and / or position of the images AR1-3 displayed on the HMD 10 is provided so that each of the images AR1-3 is correctly associated with the corresponding world point WP1-3.

[0104] In other words, during the operation step, the method 900 is capable of displaying on the HMD 10 two-dimensional images (such as driving trajectory, speed limits, information on road conditions, atmospheric conditions, and / or points of interest configured in the FOV, such as cities, buildings, monuments, commercial facilities, etc.) that accurately and reliably integrate with what is visible in the field of view FOV of the user wearing the HMD 10. Advantageously, the method 900 is configured to modify in real time the shape and visualization position of the displayed images AR1-3 to adapt to positional variations of both the vehicle 5 and the HMD 10.

[0105] The invention thus conceived is susceptible to several modifications and variations, all of which are within the scope of the inventive concept.

[0106] For example, in one embodiment, the processing unit 30 is configured to utilize measurements obtained by the IMU and / or sensors of the vehicle 5 to increase the positional accuracy of the image on the HMD 10 and / or to provide an image with more detailed and / or additional information items.

[0107] Finally, during the boresight procedure, the possibility of scaling the boresight image ARR may also be provided to ensure optimal overlap between the boresight image ARR and the reference world point WPR. In this case, the scaling operation of the boresight image ARR may also be taken into account when assessing the discrepancy between the boresight image ARR and the reference world point WPR.

[0108] Furthermore, nothing prohibits automating the step of overlapping the boresight image ARR with the boresight world point WPR during the boresighting procedure. For example, the processing unit 30 could be configured to identify the boresight world point WPR when framed in the field of view FOV of the HMD 10 and then automatically overlay the boresight image ARR on the boresight world point WPR, or to apply one or more suitable algorithms to automatically determine the mismatch between the boresight image ARR at the boresight world point WPR.

[0109] In one embodiment, the method 900 provides periodic access to the GNSS data database 7, for example to check for the presence of new world points in a geographic region of interest within the view volume.

[0110] As will be apparent, after the alignment procedure described above, system 1 can be configured to operate using any number of markers 40. For example, a pair of markers 40 or a single marker 40 can be used to determine the pose of HMD 10 during the operational steps of method 900. This allows for tuning the computational load required by system 1 to provide real-time driving assistance while improving the overall responsiveness of system 1 to variations due to movement of vehicle 5 and / or movement of world points WP0-WP3. Furthermore, this allows for tuning the relationship between the accuracy of determining the pose of HMD 10 and the computational load required of processing unit 30.

[0111] In one embodiment, for each world point WP1-3 to be displayed, the method 900 provides for defining a relative virtual point with respect to at least one identified marker 40. If one or more secondary markers are identified, a rotation-translation relationship is applied to the relative virtual points calculated with respect to the secondary markers to redefine these relative virtual points with respect to the primary marker. A final virtual point is determined by combining all relative virtual points with respect to the primary marker, preferably by a suitable algorithm including, for example, averaging and correlation operations. The final virtual point is then transformed into the corresponding image to be displayed by applying a compensation law to correct the position of the virtual point in the image defined in the two-dimensional reference frame of the surface of the screen 11 of the HMD 10.

[0112] In an embodiment not shown, when a world point, for example, world point WP0 in FIG. 2, is not configured in the view volume VOL, a corresponding virtual indicator, for example, an arrow, can be displayed on the HMD 10, for example, reproduced at the edge of the display region R, with its tip pointing toward the position of the corresponding world point WP0. Note that other information regarding the world point WP0 outside the display region R, such as the name or distance of the world point WP0, can also be displayed in addition to the arrow.

[0113] In alternative embodiments, the image AR may be reproduced in false color based on distance from the vehicle 5, driving hazards associated with the relative world point, and / or other information to convey.

[0114] Nothing prohibits performing and / or omitting one or more optional steps of method 900, just as nothing prohibits performing two or more steps in parallel or in a different order.

[0115] Furthermore, one or more implementation details may be replaced with other technically equivalent elements.

[0116] For example, in addition to or instead of the ArUco marker 30, other reference elements 30 may be used, such as one or more data matrices, QR codes, and / or other types of reference elements.

[0117] Of course, alternative arrangements of markers 40 consisting of different (greater or fewer) numbers of markers 40 may also be provided, and ultimately there is nothing prohibiting having a single marker 40 to perform the above-described method 900.

[0118] Additionally, markers 40 may be positioned in additional and / or alternative locations. For example, one or more markers 40 may be positioned on one of the windows or the rear window of vehicle 5 to enable playback of a correctly positioned augmented reality image even when the user moves their gaze towards them.

[0119] Preferably, but not by way of limitation, the marker 40 is constructed in accordance with the teachings contained herein. Image and Vision Computing, volume 76, pages 38-47, year 2018, Francisco J. Romero-Ramirez, Rafael Munioz-Salinas, Rafael Medina-Carnicer: "Speeded up detection of squared fiducial markers", S. Garrido-Jurado, R. Munioz Salinas, FJ Madrid-Cuevas, R. Medina-Carnicer: "Generation of fiducial marker dictionaries using mixed integer lineARRrogramming", published in Pattern Recognition volume 51, pages 481-491, year 2016, and / or Garrido-Jurado, Sergio, et al., "Automatic generation and detection of highly reliable fiducial markers under occlusion," published in Pattern Recognition, volume 47, number 6, pages 2280-2292, year 2014

[0120] Furthermore, although in the above exemplary embodiment the compensation law is shown to be applied to the global position detected by the GNSS module 20, nothing prohibits defining a corresponding compensation law to be applied to the relative position of the HMD 10 within the vehicle 5 determined based on the marker 40.

[0121] Furthermore, nothing prohibits determining the position and orientation of the HMD 10 through two separate operations that can be performed sequentially and / or in parallel, rather than through a single operation as described above.

[0122] Although the applicant has identified the use of markers 40 as particularly advantageous, nothing prohibits the implementation of alternative methods in which the position and orientation of the HMD relative to the windshield and / or other elements of the passenger compartment are identified in a different way, for example through the use of a video and / or photo camera pointed at the driver and / or one or more motion sensors attached to the HMD.

[0123] System 1 may be provided as a kit of components to be assembled within the passenger compartment of a vehicle. In particular, the kit includes at least a processing unit 30, a dedicated GNSS module 20, or alternatively, wired and / or wireless connection elements between the processing unit and the vehicle's GNSS module, and an HMD 10, preferably including two cameras 10 and connectable to the processing unit. Alternatively, processing unit 30 may be configured to work with one or more commercially available HMDs (e.g., Microsoft HoloLens). Thus, one or more versions of the kit do not necessarily constitute an HMD.

[0124] Alternatively, nothing prohibits the processing unit 30 from being incorporated into the vehicle 5 or a user device connectable to the vehicle (such as a smartphone, tablet, computer, etc.), or from instantiating a software product configured to perform the method 900 on a processing unit of the vehicle 5 or the user device.

[0125] Furthermore, the connections between the elements of the system 1, in particular the connection between the processing unit 30 and the HMD 10, can be both wired and preferably wireless. Similarly, the connections between the elements of the system 1 and other elements, for example the connections between the processing unit 30 and the IMU, the ECU (not shown) of the vehicle 5, the infotainment system (not shown) of the vehicle 5, etc., can be either wired or wireless.

[0126] In fact, the materials used, as well as the content shape and size, can be anything according to requirements, without for this reason departing from the scope of protection of the following claims.

Claims

1. A method (900) for assisting driving of a vehicle (5) implemented by a system (1) including an HMD (10), a reference element (40) arranged inside the vehicle (5), and a positioning module (20) mounted on the vehicle (5), the method (900) comprising: Detecting (919) a position of the vehicle by said positioning module (20), said position being a position defined with respect to a global reference frame; determining a position of the HMD (10) relative to the global reference frame based on the detected position of the vehicle (5); determining (923) a view volume (VOL) corresponding to a volume of space included in a field of view (FOV) of the HMD (10) based on the position of the HMD (10); comparing (925) the set of locations contained in said view volume (VOL) with at least one location of interest (PW1-3) associated with an object of interest (WP1-3) stored in a memory area (31) of said system (1); If one or more points of interest (PW1-3) are configured in the view volume (VOL), Calculating (927) visualization positions (PA1-3) of the HMD (10) that display images (AR1-3) associated with the objects of interest (WP1-3); Displaying the image (AR1-3) at the visualization position (PA1-3) (927); The visualization position (PA1-3) is a position where a user wearing the HMD (10) can visually recognize the image (PA1-3) corresponding to the subject of interest (WP1-3), Determining a position of the HMD (10) relative to the global reference system based on the detected position of the vehicle (5) comprises: Determining (909-915) a rotational-translational relationship between the global reference system and a relative reference system associated with the reference element (40); acquiring, by said HMD (10), at least one image of said reference element (40) located inside said vehicle (5); determining (917) a relative position of the HMD (10) with respect to the reference element (40) by processing the acquired images, the relative position being referenced to the relative reference system associated with the reference element (40); Transforming (921) the relative position of the HMD (10) with respect to the reference element (40) into a corresponding position referenced to a global reference system by applying the rotation-translation relationship; 23. A method comprising:

2. 2. The method (900) of claim 1, comprising: The HMD (10) comprises at least two video cameras (15) arranged on opposite sides of a screen (11) of the HMD (10); The step (917) of determining the relative position of the HMD (10) with respect to the reference element (40) comprises: using each camera (15) of the HMD (10) to acquire an image of the reference element (40) located inside the vehicle (5); - calculating the relative position of each camera (15) with respect to said reference element (40) by processing each acquired image; calculating a relative position of the HMD (10) with respect to the reference element (40) by combining the relative positions of the cameras (15); Equipped with.

3. 3. The method according to claim 1 or 2, the system (1) comprises a plurality of reference elements (40), one selected reference element of which acts as a primary reference element and the other reference elements act as secondary reference elements; The method further comprises: calculating (903-908) for each secondary reference element a reference relationship corresponding to a rotation-translation relationship between said secondary reference element and said primary reference element, The previous step (917) of determining the relative position of the HMD (10) with respect to the reference element (40) comprises: calculating a relative position of the HMD (10) with respect to at least two reference elements (40) selected in a set comprising the primary reference element and the secondary reference element; applying said rotation-translation relationship to said relative position of said HMD (10) calculated with respect to each selected sub-reference element; calculating a composite relative position of the HMD (10) with respect to the primary reference element by applying the rotation-translation relationship to the relative position of the HMD (10) calculated with respect to each selected secondary reference element and then combining the relative positions calculated with respect to the at least two reference elements; Equipped with.

4. 4. The method (900) of any one of claims 1 to 3, comprising: The step of determining a view volume (VOL) (923) comprises: calculating an orientation of the HMD (10) relative to the reference element (40) by processing the at least one acquired image; determining a field of view (FOV) of the HMD (10) based on a global position of the HMD (10) and an orientation of the HMD (10) relative to the reference element (40); Equipped with.

5. 5. The method (900) of any one of claims 1 to 4, comprising: Selecting a boresight position (PWR) (909); displaying (911) a boresight image (ARR) at a visualization position (PAR) on the HMD (10), the visualization position (PAR) being calculated as a function of the boresight position (PWR) and a position of the vehicle; measuring (913) a position discrepancy between the boresight position (PWR) and the visualization position (PAR); Equipped with.

6. 6. The method (900) of claim 5, comprising: Measuring a mismatch between the boresight position (PWR) and the visualization position (PAR) comprises: - defining a rotation-translation relationship between a virtual boresight position (PVR) and said boresight position (PWR), said virtual boresight position (PVR) corresponding to a projection of said visualization position (PAR) in a three-dimensional reference system; Equipped with.

7. 7. The method (900) of claim 6, comprising: A boresight target is located at the boresight position (PW), The step of defining a rotational-translational relationship comprises: orienting the HMD (10) so that a field of view (FOV) of the HMD (10) includes a boresight object; translating a boresight image (ARR) displayed on the HMD (10) until it overlaps with a boresight object (WPR) at the boresight position (PWR); transforming a translation of the boresight image (ARR) in a two-dimensional reference system and a rotation of the virtual boresight position (PVR) in the three-dimensional reference system; Equipped with.

8. 8. The method (900) of any one of claims 1 to 7, further comprising: Acquiring vehicle movement information (5), The step of displaying an image related to a subject of interest on the HMD (10) comprises: modifying the image as a function of the movement of the vehicle (5) and of time, Equipped with.

9. A system (1) for assisting driving of a vehicle (5), comprising: HMD (10), At least one reference element (40) arranged inside the vehicle (5); a positioning module (20) mounted on the vehicle and configured to detect a position of the vehicle; a memory area (31) in which at least one position of interest associated with an object of interest is stored; A processing unit (30) operatively connected to the positioning module (20) and the HMD (10) and configured to perform the method (900) of any one of claims 1 to 8; Equipped with.

10. A system (1) according to claim 9, The processing unit (30) It is operatively connected to at least one of a bus (55) for vehicle communication (5) and an inertial measurement unit (6) for acquiring vehicle information (5).

Images