Method and system for determining helmet position and orientation

Abstract

The present disclosure relates to a method and tracking system for determining the position and orientation of a helmet (120) while in motion on a vehicle (110), inertial data of the helmet (120) while the helmet (120) is in motion is measured by an inertial measurement system (123) of the helmet (120), inertial data of the vehicle (110) while the vehicle (110) is in motion is measured by an inertial measurement system (112) of the vehicle (110), and the position and orientation of the helmet (120) relative to the vehicle (110) is determined by a processing system (115) using the inertial data of the helmet (120) and the inertial data of the vehicle (110).

Description

[Technical Field]

[0001] The present disclosure relates to systems and methods for determining the position and orientation of a helmet and / or eyewear while moving on or within a vehicle. In particular, the present disclosure relates to methods for determining the position and orientation of a helmet and / or eyewear while moving on or within a vehicle, and to a tracking system for determining the position and orientation of a helmet / eyewear while moving on or within a vehicle. Furthermore, the present disclosure relates to a computer program product comprising a non-transitory computer-readable medium having stored thereon computer program code configured to control a processing system of the tracking system. [Background technology]

[0002] Improving road safety, especially reducing road user fatalities and injuries, is a common goal. This goal has been tackled in recent years, especially by car manufacturers. Various systems have been developed to achieve the above goal, such as lane keeping assist systems, forward collision warning systems, or automatic emergency braking systems. The safety of car and truck users has improved significantly in recent years. Despite these technological developments in the automotive industry, the safety of motorcycles has not improved to the same extent. For example, Ciro Antilock Braking Systems (ABS) have only been introduced to the motorcycle industry in the last decade. Other safety systems from the automotive industry have yet to be introduced to modern motorcycles.

[0003] This discrepancy is difficult to understand, and here's why: Non-motorized two-wheeled vehicles such as bicycles and powered two-wheeled vehicles such as motorcycles, e-bikes, and e-scooters pose a higher risk of injury to riders compared to other modes of transportation, particularly automobiles. For example, in the United States in 2017, motorcycle fatalities and injuries were approximately 27 times higher per mile traveled than automobile fatalities and injuries.

[0004] Some safety features of automobiles are not so easy to implement on motorcycles. For example, automobile head-up displays always project information onto the same part of the automobile's windshield, or in front of the windshield, or onto an additional dedicated display. Because automobile users always look at the same part of the windshield, they can easily read the information presented via the head-up display in almost all driving situations. This improves driving safety because automobile users do not need to lower their eyes. On the other hand, motorcycles, such as electric scooters and high-performance motorcycles, often do not even have a windshield. Even if a motorcycle has a windshield, motorcycle users do not always look through it while driving. Furthermore, because motorcycle users move their heads more frequently than automobile users, if a motorcycle's windshield has a head-up display, it becomes very difficult to present driving information to motorcycle users in an easy-to-read manner while using the motorcycle.

[0005] Conventional systems on motorcycles for displaying driving information, such as speed and navigation information, use fixed displays or tachometers, which require motorcycle riders to look down to read relevant driving information. When motorcycle riders look down, this increases the risk and likelihood of an accident. Summary of the Invention [Problem to be solved by the invention]

[0006] It is an object of the present disclosure to provide a system and method for determining the position and orientation of a helmet or eyewear while in motion on or within a vehicle. In particular, it is an object of the present disclosure to provide a system, method, and computer program product for determining the position and orientation of a helmet or eyewear while in motion on or within a vehicle, which do not have at least some of the disadvantages of the prior art.

[0007] According to the present disclosure, these objects can be achieved by means of the features of the independent claims. Further advantageous embodiments result from the dependent claims and the description.

[0008] According to the present disclosure, the above-mentioned objects are particularly achieved by a method for determining the position and orientation of a helmet while in motion on a vehicle, the method comprising measuring, by an inertial measurement system of the helmet, inertial data of the helmet while the helmet is in motion, the method also comprising measuring, by an inertial measurement system of the vehicle, inertial data of the vehicle while the vehicle is in motion, the method further comprising determining, by a processing system, the position and orientation of the helmet relative to the vehicle using the inertial data of the helmet and the inertial data of the vehicle.

[0009] In one embodiment, the method also includes capturing at least one digital image of the vehicle using a camera disposed on the helmet and / or capturing at least one digital image of the helmet using a camera disposed on the vehicle, and further using the one or more digital images of the vehicle and / or the helmet to determine, by the processing system, a position and orientation of the helmet relative to the vehicle.

[0010] The cameras mentioned above and below may be any sensing device capable of generating data (e.g., digital images) indicative of distance and space. This includes any type of distance sensor, such as a radar sensor or a LIDAR (Light Detection and Ranging) sensor. Furthermore, the sensing device may include a laser-based scanning system that determines distance, for example, by measuring the time-of-flight (TOF) of a laser beam. This data can then generate a two- or three-dimensional digital image of the space, which can be processed by a processing system to determine the helmet's position and orientation. In the following description, when reference is made to a digital image or simply an image, the image may have been captured by such a camera or distance sensor.

[0011] In one embodiment, the at least one captured digital image taken using the camera located on the helmet includes at least one particular feature of the vehicle and / or the at least one digital image taken using the camera located on the vehicle includes at least one particular feature of the helmet, and the processing system further uses one or more of the digital images including the at least one particular feature of the vehicle and / or the helmet to determine the position and orientation of the helmet relative to the vehicle.

[0012] In one embodiment, the method further includes the processing system identifying an alternative feature of the vehicle and / or the helmet by performing the following steps: capturing a digital image using the camera located on the helmet or the vehicle, the digital image including the at least one distinctive feature and an additional feature of the vehicle and / or the helmet; identifying the alternative feature from the image by the processing system; and using the alternative feature as the distinctive feature to identify a position and orientation of the helmet relative to the vehicle.

[0013] In one embodiment, the alternative features are identified by a machine learning model or algorithm using a captured digital image containing one specific feature and / or multiple additional features. In one embodiment, the digital image is processed by a processing system using a neural network as a machine learning model to calculate a preferred alternative feature from multiple possible additional features. In another embodiment, the alternative features are determined by the processing system using feature descriptors such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or Oriented FAST and rotated BRIEF (ORB).

[0014] In one embodiment, the method further comprises determining the position and orientation of the helmet using at least one captured digital image taken by a camera, wherein the at least one digital image is processed by a position and orientation estimator on a processing system, and / or the position and orientation estimator may be a machine learning model such as a neural network.

[0015] In one embodiment, the method further comprises predicting the position and orientation of the helmet using at least one captured digital image taken by a camera, wherein the at least one digital image is processed by a position and orientation estimator on a processing system, the position and orientation estimator may be a machine learning model such as a neural network.

[0016] In the above and following descriptions, when a machine learning model / method is mentioned, it should be understood that the method comprises any model that generates an output for a given input. This also includes any training and use of a pre-trained model that generates an output for a given input, primarily, but not limited to, optimizing the model depending on the input and expected output by minimizing a defined cost function, where the pre-training is performed using past data. The input may be inertial data, and the output may be a determined position and orientation of an object or a prediction of the object's future position and orientation. The input may be a digital image depicting at least a portion of the object, and the output may be a determined position and orientation of the object or a prediction of the object's future position and orientation. The input may also be a weighted combination of the inertial data and the digital image. The input may also be a combination of the inertial data and the digital image by the model itself using an estimator-based data integration. The object may be a helmet, eyewear, a vehicle, an eye, a specific feature, or the surrounding environment that is part of a world reference frame, such as a road. In other cases, for a digital image input, the output may be a specified / defined alternative feature at least partially depicted by the image by selecting an alternative feature from a plurality of features. The machine learning model used may consist of, but is not limited to, direct regression, general deep neural networks, convolutional neural networks, transformer-based neural networks, and recurrent neural networks, or a weighted combination of these and further models.

[0017] In one embodiment, the at least one distinguishing feature of the vehicle and / or helmet is a fiducial marker located on the vehicle or helmet. A fiducial marker or fiducial is an object located in the field of view of the imaging system that is displayed in the generated image for use as a reference point or measure.

[0018] In one embodiment, the method further comprises measuring global position data of the vehicle and / or the helmet relative to a world reference coordinate system by a global positioning system of the vehicle and / or the helmet. The method according to this embodiment further comprises determining, by the processing system, a position and orientation of the helmet relative to the world reference coordinate system, further using the measured global position data of the vehicle and / or the helmet. In one embodiment, the global positioning system uses positioning information obtained via satellites. In another embodiment, the global positioning system uses image-based global position data obtained, for example, from a vehicle controller area network bus (CAN-BUS) unit.

[0019] In one embodiment, the initial position of the helmet is determined by the steps of: taking at least one digital image of the vehicle using the camera located on the helmet and / or taking at least one digital image of the helmet using the camera located on the vehicle; Using one or more of the digital images of the vehicle and / or the helmet, determining, by the processing system, the initial position of the helmet relative to the vehicle. In one embodiment, the digital image may include specific features of the helmet or vehicle.

[0020] In one embodiment, the determination of the position and orientation of the helmet relative to the vehicle and / or world reference coordinate system is performed by the processing system using a data integration method, such as captured digital images, acquired data, measured inertial data of the helmet and / or vehicle, and additional data.

[0021] In one embodiment, the determination of the helmet's position and orientation relative to the vehicle and / or the world reference frame is performed by the processing system using estimator-based data synthesis. In one embodiment, the estimator-based data synthesis is performed using a linear filter, such as a Kalman filter, a non-linear filter, or a combination thereof. The estimator-Kalman filter-based data synthesis allows for smooth, fast, and accurate determination of the helmet's position and orientation relative to the vehicle and / or the world reference frame.

[0022] In one embodiment, the method also includes using the position and orientation of the helmet relative to the vehicle and / or the world reference coordinate system to identify, by the processing system, a display location for data to be displayed on an augmented reality helmet-mounted display located on the helmet. The method further includes using the identified display location to display, by the processing system, the data to a wearer of the helmet on the augmented reality helmet-mounted display. According to this embodiment, the helmet includes an augmented reality helmet-mounted display for displaying data to a wearer of the helmet and a user of the vehicle. In one embodiment, the augmented reality display is fixedly located on the helmet, and in another embodiment, the augmented reality display is removable from the helmet.

[0023] In one embodiment, the method further comprises the steps of: determining, by an eye-tracking system disposed in the helmet, a gaze vector of at least one eye, preferably both eyes, of a wearer of the helmet; and using the determined gaze vector, determining, by the processing system, a display position of the data and / or a type of the data on the augmented reality helmet-mounted display viewed by the wearer. The eye-tracking system is a system for determining the gaze vector of a user of the eye-tracking system.

[0024] In one embodiment, the processing system comprises a processing system in the helmet, a processing system in the vehicle, and / or a processing system in a remote server. In other words, the processing system can be located in the helmet, the vehicle, a remote server, or a combination thereof. Different calculations can be performed on different processing systems. The processing system can further comprise a processing system in a mobile device, such as a user's smartphone, used in combination with the processing system in the helmet and / or vehicle to, for example, determine the position and orientation of the helmet or to update a neural network used to calculate a correction signal for drift of the inertial measurement system.

[0025] In some embodiments, the processing system includes a system on a chip (SoC), a central processing unit (CPU), and / or other more specific processing units such as a graphics processing unit (GPU) or an application specific integrated circuit (ASIC), a reprogrammable processing unit such as a field programmable gate array (FPGA), or even a processing unit specifically configured to determine the position and orientation of the helmet relative to the vehicle.

[0026] In a further aspect, in addition to a method for determining the position and orientation of a helmet, the present disclosure relates to a tracking system for determining the position and orientation of a helmet while in motion on a vehicle, the tracking system comprising an inertial measurement system disposed on the helmet, an inertial measurement system disposed on the vehicle, and a processing system configured to perform the following steps: measuring inertial data of the helmet while the helmet is in motion using the inertial measurement system of the helmet; measuring inertial data of the vehicle while the vehicle is in motion using the inertial measurement system of the vehicle; and Using the inertial data of the helmet and the inertial data of the vehicle to determine the position and orientation of the helmet relative to the vehicle.

[0027] In one embodiment, the tracking system further comprises a camera disposed on the helmet and / or a camera disposed on the vehicle, and the processing system is further configured to perform the steps of: capturing at least one digital image of the vehicle using the camera located on the helmet and / or capturing at least one digital image of the helmet using the camera located on the vehicle; and Further using the digital image(s) of the vehicle and / or the helmet to determine the position and orientation of the helmet relative to the vehicle.

[0028] In one embodiment, at least one captured digital image taken using the camera located on the helmet includes at least one particular feature of the vehicle and / or the at least one captured digital image taken by the camera located on the vehicle includes at least one particular feature of the helmet, and the at least one particular feature of the vehicle and / or the helmet is further used by the processing system to determine the position and orientation of the helmet relative to the vehicle.

[0029] In one embodiment, the tracking system further comprises a global positioning system, and the processing system is further configured to perform the steps of: measuring global position data of the vehicle and / or the helmet relative to a world reference coordinate system using the global positioning system; and Further using the measured global position data of the vehicle and / or the helmet to determine the position and orientation of the vehicle and / or the helmet relative to the world reference coordinate system.

[0030] In one embodiment, the tracking system is configured to use data integration techniques to determine the position and orientation of the helmet relative to the vehicle and / or the world reference coordinate system.

[0031] In one embodiment, the tracking system is configured to determine the position and orientation of the helmet relative to the vehicle and / or the world reference coordinate system using estimator-based data synthesis.

[0032] In one embodiment, the tracking system further comprises an augmented reality display disposed on the helmet, and the processing system is further configured to perform the steps of: using the helmet's position and orientation relative to the vehicle and / or the world reference coordinate system to determine the display location of data to be displayed on the augmented reality helmet-mounted display; and Using the display location, displaying the data to a wearer of the helmet on the augmented reality display.

[0033] In one embodiment, the tracking system further comprises an eye-tracking system disposed on the helmet, and the processing system is further configured to perform the steps of: using the eye-tracking system to determine the gaze vector of at least one eye, preferably both eyes, of the helmet wearer; and Using the identified gaze vector, determining the display position and / or type of the data on the augmented reality display viewed by the wearer.

[0034] In addition to systems and methods for determining the position and orientation of a helmet while in motion on a vehicle, the present disclosure relates to a computer program product comprising a non-transitory computer readable medium having stored thereon computer program code configured to control a processing system of the tracking system such that the tracking system performs the steps of the above-described methods.

[0035] According to another aspect of the present disclosure, the above object is particularly achieved by a method for determining the position and orientation of eyewear while in motion on or within a vehicle. The features and advantages of the aspect for determining the position and orientation of a helmet described above or hereinafter also apply to the aspect for determining the position and orientation of eyewear while in motion on or within a vehicle, as described below. The method includes measuring, by an inertial measurement system of the eyewear, inertial data of the eyewear while the eyewear is in motion. The method also includes measuring, by an inertial measurement system of the vehicle, inertial data of the vehicle while the vehicle is in motion. The method of the present disclosure further includes determining, by a processing system, the position and orientation of the eyewear relative to the vehicle using the inertial data of the eyewear and the inertial data of the vehicle.

[0036] Eyewear, as understood above and below, may consist of any device worn by a human user, i.e., the user of the eyewear, that at least partially covers the user's field of vision, such as glasses, goggles, or a monocle.

[0037] In one embodiment, the method also includes capturing at least one digital image of the vehicle using a camera disposed on the eyewear and / or capturing at least one digital image of the eyewear using a camera disposed on the vehicle, and further using the one or more digital images of the vehicle and / or the eyewear to determine, by the processing system, a position and orientation of the eyewear relative to the vehicle.

[0038] In one embodiment, the method further includes determining a position and orientation of the eyewear relative to the vehicle using at least one captured digital image taken by the camera, the at least one digital image being processed by a position and orientation estimator of the processing system, which may be, for example, a machine learning model such as a neural network.

[0039] In one embodiment, the method further includes predicting a position and orientation of the eyewear relative to the vehicle using at least one captured digital image taken by the camera, the at least one digital image being processed by a position and orientation estimator of the processing system, which may be, for example, a machine learning model such as a neural network.

[0040] In one embodiment, the at least one captured digital image taken using the camera positioned on the eyewear includes at least one specific feature of the vehicle, and / or the at least one captured digital image taken by the camera positioned on the vehicle includes at least one specific feature of the eyewear, and the processing system further uses one or more of the digital images including the at least one specific feature of the vehicle and / or the eyewear to determine the position and orientation of the eyewear relative to the vehicle.

[0041] In one embodiment, the method further includes identifying alternative characteristics of the vehicle and / or the eyewear with the processing system by performing the steps of: The method further includes capturing a digital image of the eyewear or the vehicle using the camera located on the eyewear or the vehicle, the digital image including the at least one identification feature and an additional feature of the vehicle and / or the eyewear, and identifying the alternative feature from the image using the processing system. The method further includes using the alternative feature as the identification feature to identify the position and orientation of the eyewear relative to the vehicle.

[0042] In one embodiment, the alternative features are identified by a machine learning model or algorithm using a digital image containing one specific feature and multiple additional features. In one embodiment, the digital image is processed by the processing system using a neural network to calculate a preferred alternative feature from multiple possible additional features. In another embodiment, the alternative features are determined by the processing system using feature descriptors such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or Oriented FAST and rotated BRIEF (ORB).

[0043] In one embodiment, the at least one distinguishing feature of the vehicle and / or eyewear is a fiducial marker located on the vehicle or eyewear. A fiducial marker or fiducial is an object located within the field of view of the imaging system that is displayed in the generated image for use as a reference point or measure.

[0044] In one embodiment, the method further comprises measuring global position data of the vehicle and / or the eyewear relative to a world reference coordinate system by a global positioning system of the vehicle and / or the eyewear. The method according to this embodiment further comprises determining, by the processing system, a position and orientation of the eyewear relative to the world reference coordinate system using the measured global position data of the vehicle and / or the eyewear. In one embodiment, the global positioning system uses positioning information obtained via satellites. In another embodiment, the global positioning system uses global position data obtained, for example, from a vehicle controller area network bus (CAN-BUS) unit.

[0045] In one embodiment, the initial position of the eyewear is determined by the following steps. capturing at least one digital image of the vehicle using the camera located on the eyewear and / or capturing at least one digital image of the eyewear using the camera located on the vehicle; and Determining, by the processing system, the initial position of the eyewear relative to the vehicle using one or more of the digital images of the vehicle and / or the eyewear, which in one embodiment may include specific features of the eyewear or the vehicle.

[0046] In one embodiment, determining the position and orientation of the eyewear relative to the vehicle and / or world reference coordinate system is performed by the processing system using a data integration method, such as captured digital images, measured inertial data of the eyewear and / or vehicle, and / or additional data.

[0047] In one embodiment, the determination of the position and orientation of the eye relative to the vehicle and / or the world reference frame is performed by the processing system using estimator-based data synthesis. In one embodiment, the estimator-based data synthesis is performed using a linear filter, such as a Kalman filter, a nonlinear filter, or a combination thereof. The estimator-Kalman filter-based data synthesis allows for smooth, fast, and accurate determination of the position and orientation of the eye relative to the vehicle and / or the world reference frame.

[0048] In one embodiment, the method also includes using the position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system to identify, by the processing system, a display location for data to be displayed on an augmented reality eyewear-mounted display disposed on the eyewear. The method further includes using the identified display location to display the data on the augmented reality eyewear-mounted display to a wearer of the eyewear. According to this embodiment, the eyewear includes an augmented reality eyewear-mounted display for displaying data to a wearer of the eyewear and a user of the vehicle. In one embodiment, the augmented reality display is fixedly disposed on the eyewear, while in another embodiment, the augmented reality display is detachable from the eyewear.

[0049] In one embodiment, the method further comprises the steps of: determining, by an eye-gaze tracking system disposed in the eyewear, a gaze vector of at least one eye, preferably both eyes, of a wearer of the eyewear; and determining, by the processing system using the determined gaze vector, a display position and / or type of the data on the augmented reality eyewear display viewed by the wearer. The eye-gaze tracking system is a system for determining the gaze vector of a user of the eye-gaze tracking system.

[0050] In one embodiment, the processing system includes a processing system of the eyewear, a processing system of the vehicle, and / or a processing system of a remote server. In other words, the processing system can be located in the eyewear, the vehicle, a remote server, or a combination thereof. Different calculations can be performed on different processing systems. The processing system can further include a processing system of a mobile device, such as a user's smartphone, used in combination with the processing system of the eyewear and / or the vehicle to, for example, determine the position and orientation of the eyewear or to update a neural network used to calculate a correction signal for drift of an inertial measurement system.

[0051] In a further aspect, the present disclosure relates to a tracking system for determining the position and orientation of eyewear while in motion on or within a vehicle, the tracking system comprising an inertial measurement system disposed on the eyewear, an inertial measurement system disposed on the vehicle, and a processing system configured to perform the following steps: measuring inertial data of the eyewear using the inertial measurement system of the eyewear while the eyewear is in motion; measuring inertial data of the vehicle while the vehicle is in motion with the inertial measurement system of the vehicle; and Using the inertial data of the eyewear and the inertial data of the vehicle to determine a position and orientation of the eyewear relative to the vehicle.

[0052] In one embodiment, the tracking system further comprises a camera disposed on the eyewear and / or a camera disposed on the vehicle, and the processing system is further configured to perform the following steps: capturing at least one digital image of the vehicle using the camera located on the eyewear and / or capturing at least one digital image of the eyewear using the camera located on the vehicle; and Further using the digital images of one or more of the vehicle and / or the eyewear to determine the position and orientation of the eyewear relative to the vehicle.

[0053] In one embodiment, at least one captured digital image taken using the camera located on the eyewear includes at least one specific feature of the vehicle and / or the at least one captured digital image taken by the camera located on the vehicle includes at least one specific feature of the eyewear, and the at least one specific feature of the vehicle and / or the eyewear is further used by a processing system to determine the position and orientation of the eyewear relative to the vehicle.

[0054] In one embodiment, the tracking system further comprises a global positioning system, and the processing system is further configured to perform the steps of: measuring global position data of the vehicle and / or the eyewear relative to a world reference coordinate system using the global positioning system; and Further using the measured global position data of the vehicle and / or the eyewear to determine the position and orientation of the vehicle and / or the eyewear relative to the world reference coordinate system.

[0055] In one embodiment, the tracking system is configured to use a data integration method to determine the position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system.

[0056] In one embodiment, the tracking system is configured to determine the position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system using estimator-based data synthesis.

[0057] In one embodiment, the tracking system further comprises an augmented reality display disposed on the eyewear, and the processing system is further configured to perform the steps of: using the position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system to determine a display location for data to be displayed on the augmented reality eyewear display; and Using the display location, displaying the data to a wearer of the eyewear on the augmented reality display.

[0058] In one embodiment, the camera and the augmented reality display are configured as one unit.

[0059] In one embodiment, the tracking system further comprises an eye-gaze tracking system disposed on the eyewear, and the processing system is further configured to perform the steps of: determining, using the eye-gaze tracking system, a gaze vector of at least one eye, preferably both eyes, of a wearer of the eyewear; Using the identified gaze vector, determining the display position and / or type of the data on the augmented reality display viewed by the wearer.

[0060] In one embodiment, the camera, the augmented reality display, and the eye tracking system are configured as a single unit. In one embodiment, the camera, the inertial measurement system, the augmented reality display, and the eye tracking system are configured as a single unit.

[0061] In addition to systems and methods for determining the position and orientation of eyewear while in motion on a vehicle, the present disclosure relates to a computer program product comprising a non-transitory computer readable medium having stored thereon computer program code configured to control a processing system of the tracking system such that the tracking system performs the steps of the above-described methods.

[0062] In the following, the present disclosure will be explained in more detail, for example with reference to the following figures: [Brief explanation of the drawings]

[0063] [Figure 1] 1 is a schematic diagram of a vehicle carrying a user and a tracking system according to a first exemplary embodiment; [Figure 2] FIG. 10 is a schematic diagram of a vehicle carrying a user and a tracking system according to a second exemplary embodiment. [Figure 3] FIG. 10 is a schematic diagram of a vehicle carrying a user and a tracking system according to a third exemplary embodiment; [Figure 4] 1 is a flowchart illustrating a series of steps for performing helmet position and orientation determination in a first exemplary embodiment. [Figure 5] 10 is a flowchart illustrating a series of steps for performing helmet position and orientation determination in a second exemplary embodiment. [Figure 6] 10 is a flowchart illustrating a series of steps for performing helmet position and orientation determination in a third exemplary embodiment. [Figure 7] 10 is a flowchart illustrating a series of steps for performing helmet position and orientation determination in a fourth exemplary embodiment. [Figure 8] 10 is a flowchart illustrating a series of steps for performing helmet position and orientation determination in a fifth exemplary embodiment. [Figure 9] 1 is a flowchart illustrating a series of steps for displaying data on an augmented reality helmet-mounted display of a helmet in a first exemplary embodiment. [Figure 10] 10 is a flowchart illustrating a series of steps for displaying data on an augmented reality helmet-mounted display of a helmet in a second exemplary embodiment. [Figure 11]10 is a flowchart illustrating a series of steps for displaying data on an augmented reality helmet-mounted display of a helmet in a third exemplary embodiment. [Figure 12] 1 is a flowchart illustrating a series of steps for identifying an alternative feature for a particular feature in one exemplary embodiment. [Figure 13] FIG. 10 is a schematic diagram of a vehicle carrying a user and a tracking system according to a fourth exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0064] FIG. 1 schematically illustrates a vehicle 110 ridden by a user and a tracking system 100 according to a first exemplary embodiment. In this embodiment, the vehicle 110 is a motorized two-wheeled vehicle. The vehicle 110 may be any different type of vehicle, such as a motorized or non-motorized two-wheeled vehicle, a bicycle, a motorcycle, an electric scooter, an electric motorcycle, a car, a jet ski, a snowmobile, skis, and a snowboard, and the user wears a helmet 120. The vehicle 110 includes a global positioning system 111, an inertial measurement system 112, an engine 113, a specific feature 114, a vehicle processing system 115b, and an alternative feature 114a. The global positioning system 111 is configured to measure global position data of the vehicle 110. The inertial measurement system 112 is configured to measure inertial data of the vehicle 110. The engine 113 is configured to provide power to the vehicle 110. The identifying feature 114 is located at or near the cockpit area of ​​the vehicle 110 and is configured to be within the image frame of a digital image captured by a camera 122 located on a helmet 120. The substitute feature 114a is configured to function as a substitute for the identifying feature 114.

[0065] As shown in FIG. 1 , helmet 120 comprises battery 121, camera 122, inertial measurement system 123, augmented reality helmet-mounted display 124, helmet processing system 115a, and eye-tracking system 126. Battery 121 is disposed in helmet 120 and supplies electrical energy to various electrical systems of helmet 120. Camera 122 disposed in helmet 120 is configured to capture images of specific feature 114 and / or alternative feature 114a of vehicle 110. Inertial measurement system 123 is configured to measure inertial data of helmet 120. Augmented reality helmet-mounted display 124 is configured to display data to a wearer of helmet 120. Augmented reality helmet-mounted display 124 may be, for example, fixedly or detachably disposed on helmet 120. Eye-tracking system 126 is configured to measure the line of sight of the wearer of helmet 120. In one embodiment, world reference coordinate system 250 is a coordinate system that associates a unique coordinate position with each location in the world.

[0066] 1 , processing system 115 comprises processing system 115a of helmet 120 and processing system 115b of vehicle 110. Processing system 115 is configured to determine the position and orientation of helmet 120 relative to vehicle 110 and / or world reference coordinate system 250 and display the data to a wearer of helmet 120 on augmented reality helmet display 124. Processing system 115 may further comprise a mobile device, such as a smartphone, which may additionally be coupled to perform the calculations for determining the position and orientation of helmet 120 and / or display the data to a wearer of helmet 120. That is, different computational steps are performed on different components of processing system 115, such as processing system 115a, processing system 115b, or a processing system of the mobile device.

[0067] In addition to the world reference coordinate system 250, Figure 1 also schematically illustrates a helmet reference system 230 and a vehicle reference system 240. The helmet reference system 230 is centered on the helmet 120 and therefore moves with the helmet 120. Statically positioned visual data in the helmet reference system 230 moves statically with the movement of the helmet 120 from the perspective of the user of the helmet 120. The vehicle reference system 240 is centered on the vehicle 110 and therefore moves with the vehicle 110. Statically positioned visual data in the vehicle reference system 240 moves statically with the movement of the vehicle 110 from the perspective of the user of the helmet 120. Furthermore, statically positioned data in the world reference coordinate system 250 does not move relative to the vehicle 110 or the helmet 120 from the perspective of the user of the helmet 120. Of course, the displayed data may move within its associated reference system 230, 240, 250 or from one reference system 230, 240, 250 to another reference system 230, 240, 250.

[0068] Using a processing system 115, which may include a processing system 115a of the helmet 120 and / or a processing system 115b of the vehicle 110, the tracking system 100 determines the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250. In one embodiment, the helmet 120 Processing system 115a and vehicle 110 Communication between the helmet 120 and processing system 115b is via wireless communication or communication module 150 (shown in FIG. 3). In one embodiment, collected data is transmitted from the helmet 120 to processing system 115b, which is located solely on the vehicle 110. In one embodiment, collected data is transmitted from the vehicle 110 to processing system 115a, which is located solely on the vehicle 110. Combinations of these are also contemplated.

[0069] The inertial measurement systems 112, 123 are configured to measure acceleration forces of the helmet 120 and the vehicle 110. In one embodiment, such forces include linear acceleration forces and / or rotational acceleration forces. Movement of the user's head in the vehicle 110 results in the same movement in the helmet 120 worn by the user. Thus, measuring inertial data at the helmet 120 allows a determination to be made about the relative movement of the user's head with respect to the vehicle 110.

[0070] In one embodiment, to increase measurement accuracy, different acceleration sensors of the inertial measurement system 112 in the vehicle 110 can be placed at different locations on the vehicle 110. In one embodiment, to increase measurement accuracy, different acceleration sensors of the inertial measurement system 123 in the helmet 120 can be placed at different locations on the helmet 120.

[0071] Inertial Measurement System of Vehicle 110 112 For example, the inertial measurement system of the vehicle 110 first measures the acceleration in a particular direction, the rotational acceleration about a particular axis, and the deceleration in a particular direction. 112 Using data from the helmet 120, the relative position and orientation of the vehicle 110 can be determined. At the same time, the inertial measurement system 123 of the helmet 120 measures, for example, a first acceleration in a particular direction, a rotational acceleration about a particular axis, and another acceleration in a further particular direction. Using data from the inertial measurement system 123 of the helmet 120, the relative position and orientation of the helmet 120 can be determined. Furthermore, the measurements by the inertial measurement system 123 of the helmet 120 and the inertial measurement system of the vehicle 110 112 In combination with measurements from the vehicle 110, the position and orientation of the helmet 120 relative to the vehicle 110 can be determined.

[0072] The processing system 115 controls the two inertial measurement systems 112, 123 advantageously makes it possible to determine the position of the helmet 120 relative to the vehicle 110, and in particular the position of the user's head within the helmet 120. This information allows the data to be displayed on a conventional data display system, for example a digital display on the motorcycle, in a convenient position for reading by the user of the motorcycle, depending on the determined position and orientation of the user's head. Thus, the method allows safety-related data to be displayed in an advantageous manner that enhances the safety of the user of the vehicle 110.

[0073] In one embodiment, camera 122 located on helmet 120 is aimed towards the cockpit of vehicle 110 and / or towards the rear of vehicle 110 while vehicle 110 is in use.

[0074] In one embodiment, the distinctive feature 114 of the vehicle 110 is a portion of the cockpit of the vehicle 110, such as an edge, a shape, a circular feature, a screw, multiple screws, or a feature on the cockpit, or a combination of the foregoing. In another embodiment, the distinctive feature 114 of the vehicle 110 is a portion of the rear of the vehicle 110, such as an edge, the shape of a feature on the rear of the vehicle 110, or a combination of the foregoing. In one embodiment, the distinctive feature 114 is a fiducial marker.

[0075] 2 differs from FIG. 1 in that the vehicle 110 further includes a camera 116. The camera 116 is fixedly positioned at the rear of the vehicle 110 and is pointed toward the rear of the helmet 120. In another embodiment, the camera 116 may be positioned at or near the cockpit area of ​​the vehicle 110 and may be pointed toward the front of the helmet 120. The camera 116 is configured to capture images from the helmet and / or images of a particular feature 114 located on the helmet 120 to determine the position and orientation of the helmet 120 relative to the vehicle 110.

[0076] In one embodiment, the fiducial marker as the identifying feature 114 is permanently placed on the vehicle 110 or the helmet 120 and is permanently used to identify the position and orientation of the helmet 120 relative to the vehicle 110. In another embodiment, the fiducial marker is temporarily placed on the vehicle 110 or the helmet 120 and is only temporarily used to identify the position and orientation of the helmet 120 relative to the vehicle 110. During the temporary use of the fiducial marker, a substitute feature 114a is identified in place of the fiducial marker, as described below with reference to FIG. 12 . Placing fiducial markers on the helmet 120 and / or the vehicle 110 within the field of view of one or more cameras 116, 122 is a simple and reliable method for obtaining the position and orientation of the helmet 120 relative to the vehicle 110 with high accuracy and for identifying advantageous substitute features 114a for the identifying feature 114.

[0077] FIG. 3 differs from FIG. 1 or FIG. 2 in the following respects: In FIG. 3, the tracking system 100 110The helmet 120 further comprises a communications module 150 disposed thereon, the communications module 150 configured to transmit and receive data / information to and from a remote server 160. The remote server 160 also has its own processing system 115c and forms part of the processing system 115 of the tracking system 100. In one embodiment, the remote server 160 is a cloud server. In one embodiment, the communications module 150 is connected to the processing system 115b of the vehicle 110 via a hardwire. In another embodiment, the communications module 150 is connected to the processing system 115b of the vehicle 110 and / or the processing system 115a of the helmet 120 via a wireless connection. Combinations are also contemplated. In one embodiment, the communications module 150 is further configured to transmit and receive data to and from the processing system 115a of the helmet 120. That is, the communications module 150 is configured to communicate between the processing system 115a of the helmet 120, the processing system 115b of the vehicle 110, and / or the remote server 160 and / or the mobile device's processing system 115c acting as further processing systems.

[0078] In one embodiment, wireless communication via communication module 150 occurs using mobile data networks, such as Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), and Long Term Evolution (LTE) networks, and / or Wi-Fi networks, Bluetooth, and the like. (registered trademark) , and / or near field communication interfaces using other wireless network types and standards.

[0079] In one embodiment, the calculations of the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250 are performed on the processing system 115a of the helmet 120, the processing system 115b of the vehicle 110, and / or the processing system 115c of the remote server 160. Additionally, the processing systems 115a, 115b, 115c, or a combination of the processing systems 115a, 115b, 115c, perform the operations necessary to initialize the tracking system 100, to determine the display position of data to be displayed on the augmented reality helmet mounted display, and / or to determine correction signals for drift in the inertial measurement systems 112, 123, and any additional necessary steps.

[0080] Further types of data may be exchanged between the processing system 115a of the helmet 120, the processing system 115b of the vehicle, and the processing system 115c of a remote server or possibly of a further mobile device such as a smartphone, tablet, or laptop. For example, data indicating a road route or data indicating local weather conditions may be uploaded to the processing system 115 from a remote device such as a mobile phone and / or a vehicle CAN-BUS unit. Furthermore, a human user may use a mobile device such as a smartphone, tablet, or laptop to add data to be displayed at specific points on the road, such as virtual braking points or warnings of upcoming sharp turns. Additionally, the processing system 115 may calculate an optimal trajectory based on the measurement data and the uploaded further types of data. Such further types of data may enable the augmented reality helmet display to display further types of data to the user.

[0081] 4 is a flow chart illustrating a series of steps for performing a determination of the position and orientation of the helmet 120 relative to the vehicle 110. In the following paragraphs written with reference to FIG. 4, a possible sequence of steps is described, which may be performed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110.

[0082] In step S1a, the inertial measurement system 123 of the helmet 120 measures inertial data of the helmet 120. In one embodiment, the inertial data of the helmet 120 includes linear acceleration data, rotational acceleration data, and / or additional data. The inertial measurement system 123 of the helmet 120 is fixedly disposed on the helmet 120, which means that all movements of the helmet 120 are measured by the inertial measurement system 123 of the helmet 120. In one embodiment, the inertial data measured by the inertial measurement system 123 of the helmet 120, which provides information about the relative movements of the helmet 120, is transmitted to the processing system 115a of the helmet 120.

[0083] In step S1b, which in one embodiment is performed simultaneously with step S1a, the inertial measurement system 112 of the vehicle 110 measures inertial data of the vehicle 110. In one embodiment, the inertial data of the vehicle 110 includes linear acceleration data, rotational acceleration data, and / or additional data. The inertial measurement system 112 of the vehicle 110 is fixedly disposed on the vehicle 110, which means that all movements of the vehicle 110 are recorded by the inertial measurement system 112 of the vehicle 110. 112 The vehicle 110's inertial measurement system 112 The inertial data measured by provides information about the relative motion of the vehicle 110. In one embodiment, this information is transmitted to the processing system 115b of the vehicle 110.

[0084] In step S2, the processing system 115, which may comprise, for example, the processing system 115a of the helmet 120 and / or the processing system 115b of the vehicle 110, performs a determination of the position and orientation of the helmet 120 relative to the vehicle 110 using the inertial data of the helmet 120 and the inertial data of the vehicle 110. The combination of the inertial data of the helmet 120 and the inertial data of the vehicle 110 makes it possible to calculate the position and orientation of the helmet 120 relative to the vehicle 110, in particular the relative motion of the helmet 120 with respect to the vehicle 110, while moving on the vehicle 110. This information advantageously makes it possible, for example, to display relevant driving information to a user of the vehicle 110 in an advantageous manner. In particular, the driving information is positioned on a display depending on the position of the helmet 120 relative to the vehicle 110. This improves the visibility of the information to the user and therefore improves the safety of the user in the vehicle 110.

[0085] Figure 5 is a flow chart illustrating a series of steps for performing a determination of the position and orientation of the helmet 120 relative to the vehicle 110. The following paragraphs, written with reference to Figure 5, describe a possible sequence of steps that may be performed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110. In addition to steps S1a, S1b, and S2, step S1c is shown in Figure 5.

[0086] In step S1c, which in one embodiment is performed simultaneously with steps S1a and S1b, one or more cameras 116, 122 are mounted on the vehicle 110 and / or helmet. 120Digital images of the vehicle 110 and / or helmet 120 are captured. These images may consist of only a portion of the vehicle 110 or helmet 120. The image frames of the one or more cameras 116, 122 remain stationary with the helmet 120 or vehicle 110 while the helmet 120 or vehicle 110 is moving because the one or more cameras 116, 122 are fixedly positioned with the helmet 120 or vehicle 110. The position of the vehicle 110 and / or helmet 120 in the image frames of the one or more cameras 116, 122 changes due to the movement of the helmet 120 or vehicle 110. These images can then be used to determine the position of the helmet 120 relative to the vehicle 110. For example, the captured images can be processed by an end-to-end position and orientation estimator, such as a neural network, that directly regresses the position and orientation of the helmet 120. For example, the captured images can be processed directly by such a neural network, and the position and orientation of the helmet 120 relative to the vehicle 110 are the output of the neural network. This calculation is performed using processing system 115 (115a, 115b and / or 115c).

[0087] The training data for the neural network includes, for example, images of the vehicle 110 and / or helmet 120 with data on the position and orientation of the helmet 120 relative to the vehicle 110 as ground truth. The training of the neural network is performed on the processing system 115, preferably the processing system 115c of a remote server. In one embodiment, the updated neural network or neural network updates are sent back to the processing systems 115a, 115b of the vehicle 110 and / or helmet 120 via the communication module 150. The operation / performance of the neural network is performed on the processing system 115, preferably on the processing system 115a of the helmet 120 and / or on the processing system 115b of the vehicle 110 to reduce latency.

[0088] 5, the processing system 115 further uses the determined position and orientation of the helmet 120 relative to the vehicle 110 from the captured images obtained in step S1c to determine the position and orientation of the helmet 120 relative to the vehicle 110 and the world reference coordinate system 250. Combining data from the inertial measurement systems 112, 123 of the helmet 120 and the vehicle 110 with data acquired from the at least one digital image by the processing system 115 improves the accuracy of determining the position and orientation of the helmet 120 relative to the vehicle 110. The use of one or more cameras 116, 122 can improve accuracy, particularly in determining the movement of the helmet 120, because measurements from the inertial measurement systems 112, 123 are based on an integration of their measurement data to determine position and orientation, and this integration accumulates errors, known as drift, over time. In other words, drift of the inertial measurement systems 112, 123 is the accumulation of small errors in the measurements of the inertial measurement systems 112, 123, resulting in increasingly inaccurate integrated inertial data. Because the one or more cameras 116, 122 directly provide orientation and position information, no integration or drift occurs. Therefore, the one or more cameras 116, 122 can be used as a drift correction for the measurements by the inertial measurement systems 112, 123. In this manner, the one or more cameras 116, 122 advantageously allow for increased accuracy in determining the orientation and position of the helmet 120 relative to the vehicle 110, thereby increasing the overall safety of the user in the vehicle 110. The one or more cameras 116, 122 can provide a "correction signal" to correct for drift in the inertial measurement systems 112, 123. This correction signal allows for the measured inertial data to be corrected, improving the accuracy in determining the position and orientation of the helmet 120 relative to the vehicle 110.

[0089] Figure 6 is a flow chart illustrating a series of steps for performing a determination of the position and orientation of the helmet 120 relative to the vehicle 110. The following paragraphs, written with reference to Figure 6, describe a possible sequence of steps that may be performed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110. In addition to steps S1a, S1b, S2, and S1c, step S2a is shown in Figure 6.

[0090] In step S2a, the processing system 115 performs location determination of the specific feature 114 using images captured by one or more cameras 116, 122. The captured images in step S1c include the specific feature 114 of the helmet 120 and / or vehicle 110. Locating the specific feature 114 is performed, for example, by determining which pixels in one or more images represent the specific feature 114 and locating these pixels in a digital image coordinate system, thereby determining the position and orientation of the helmet 120 relative to the vehicle 110. The specific feature 114 may be, for example, one or more screws in the cockpit of the vehicle 110. For example, if the distance between two screws is known, then locating the pixels representing the screws in the digital image allows the position of the helmet 120 relative to the vehicle 110 to be calculated. This calculation requires only geometric algebra and is preferably performed on the processing system 115a of the helmet 120 and / or the processing system 115b of the vehicle 110.

[0091] In step S2, determination of the position and orientation of the helmet 120 relative to the vehicle 110 is performed by the processing system 115 using the determined positions of the specific features 114 in step S2a. According to this embodiment, it is particularly simple to calculate correction signals for drifts of the inertial measurement systems 112, 123.

[0092] Figure 7 is a flow chart illustrating a series of steps for performing a determination of the position and orientation of the helmet 120 relative to the vehicle 110. The following paragraphs, written with reference to Figure 7, describe a possible sequence of steps that may be performed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110. In addition to steps S1a, S1b, and S2, steps S1c, S2b, and S2c are also shown in Figure 7.

[0093] Step S1c is as described above with reference to FIG.

[0094] In step S2b, alternative features 114a are identified using the digital image captured in step S1c. The digital image includes at least one specific feature 114 of the helmet 120 and / or vehicle 110 and additional features of the vehicle 110 and / or helmet 120 adjacent to the specific feature 114, such as a circular tool, a screw, or an edge. The alternative features 114a are learned / identified, for example, by a neural network in the processing system 115 (115a, 115b, 115c). The digital image is processed by the neural network, which calculates a preferred alternative feature 114a from the possible additional features. During the training process, the neural network requires ground truth to be able to learn the alternative feature 114a. In one embodiment, the ground truth for the training process is an image containing the specific feature 114 and / or associated inertial measurement data of the inertial measurement systems 112, 123 and / or associated position information of the cameras.

[0095] In step S2c, the alternative feature 114a is used in place of the identifying feature 114. Here, the identifying feature 114 may be removed from the vehicle 110 or the helmet 120.

[0096] In step S2, determination of the position and orientation of the helmet 120 relative to the vehicle 110 is performed by the processing system 115 using the alternative features 114a. According to this embodiment, the use of the alternative features 114a makes it possible to provide alternatives or combinations for the calculation of correction signals for drift of the inertial measurement systems 112, 123.

[0097] Figure 8 is a flow chart illustrating a series of steps for performing the determination of the position and orientation of the helmet 120 relative to the vehicle 110. The following paragraphs, written with reference to Figure 8, describe a possible sequence of steps that may be performed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110. In addition to steps S1a, S1b, and S2, step S1d is shown in Figure 8.

[0098] In step S1d, which in one embodiment is performed simultaneously with steps S1a, S1b and / or S1c, global position data of the vehicle 110 is measured by a global positioning system 111 of the vehicle 110. In a further embodiment, the global positioning data of the helmet 120 is measured by a global positioning system of the helmet 120. In one embodiment, the global positioning system 111 uses positioning information obtained via satellites. In another embodiment, the global positioning system 111 uses image-based information obtained, for example, from a vehicle CAN-BUS unit. The measured position data of the vehicle 110 and / or the helmet 120 allows for determining the position of the vehicle 110 and / or the helmet 120 relative to the world reference coordinate system 250.

[0099] In step S2, the processing system 115 further uses measured global position data from the global positioning system 111 of the vehicle 110 and / or helmet 120 to determine the position and orientation of the helmet 120 relative to the vehicle 110 and the world reference frame 250. The combination of the inertial data of the helmet 120, the inertial data of the vehicle 110, and the global position data allows for calculation of the position and orientation of the helmet 120 relative to the vehicle 110 while moving on the vehicle 110, and of the position and orientation of the helmet 120 relative to the world reference frame 250 while the vehicle 110 with the helmet 120 is moving within the world reference frame 250. This allows, for example, navigation data to be displayed in an advantageous manner to the user of the vehicle 110, which can increase the safety of use of the vehicle 110 as well as improve the user experience of the vehicle 110.

[0100] The global position data of the vehicle 110, when combined with digital images from one or more cameras 122, 116, and / or inertial data of the helmet 120 and the vehicle 110, can be used to improve the accuracy of determining the position and orientation of the helmet 120 relative to the vehicle 110, as well as to determine the position of the helmet 120 and the vehicle 110 relative to the world reference coordinate system 250. This allows navigation data to be displayed in an advantageous manner to the user of the vehicle 110, increasing the safety of use of the vehicle 110 and improving the experience of using the vehicle 110.

[0101] Figure 9 is a flow chart showing a series of steps for displaying data on the augmented reality helmet-mounted display 124 of the helmet 120. In the following paragraphs written with reference to Figure 9, a possible sequence of steps is described, which steps are executed by the processing system 115 in order to display data on the augmented reality helmet-mounted display 124. For determining the position and orientation of the helmet 120 (step S2), a combination of steps S1a, S1b, S1d, S1c, S2a, S2b and / or S2c may be used, as described above (steps S2a, S2b, S2c are not shown in Figure 9).

[0102] In step S3, using the position and orientation of the helmet 120 relative to the vehicle 110 and / or world reference coordinate system 250 determined in step S2, the processing system 115 determines a display location for data to be displayed on the augmented reality helmet-mounted display 124 located on the helmet 120. In one embodiment, the augmented reality helmet-mounted display 124 is implemented in or forms the visor of the helmet 120. In one embodiment, the displayed data includes navigation information, driving information such as speed and engaged gear, road information such as ideal line and expected obstacles, or warning information such as going too fast on the next road segment. The display location is the location for displaying the data on the augmented reality helmet-mounted display 124. For example, traditional head-up displays in the automotive industry project data onto the bottom of the windshield of a vehicle. The display location of the data displayed on the augmented reality helmet-mounted display 124 can vary across the visor or across the field of view of the user of the vehicle 110.

[0103] In step S4, the processing system 115 displays the data to the wearer of the helmet 120 on the augmented reality helmet-mounted display 124 using the display location determined in step S3. The location of the displayed data depends on the determined position and orientation of the helmet 120. In other words, different data can be displayed on the augmented reality helmet-mounted display 124 depending on the position and orientation of the helmet 120. For example, data can be placed in the helmet reference system 230, and this data appears to the user to be moving exactly as the helmet 120 does, while remaining stationary within the augmented reality helmet-mounted display 124. Data placed 20 cm in front of the user's head in the helmet reference system 230 will always appear 20 cm in front of the user's head, even if the orientation or position of the helmet 120 changes due to user movement. Additionally, data can be placed in a vehicle reference system 240 that is centered on the vehicle 110. Data centered on the vehicle reference system 240 and displayed on the augmented reality helmet-mounted display 124 remains stationary relative to the vehicle 110. If the position or orientation of the helmet 120 changes, the position of this data displayed on the augmented reality display also changes. For example, if the vehicle's speed is associated with the vehicle reference system 240 and is always positioned 3 meters in front of the vehicle 110, it will always remain 3 meters in front of the vehicle 110 and may disappear from the user's field of view due to a given movement of the helmet 120. Further, data may be located in the world reference coordinate system 250. Data centered in the world reference coordinate system 250 may remain stationary relative to the world. If the position and / or orientation of the helmet 120 and / or vehicle 110 changes, the position of this data displayed on the augmented reality helmet-mounted display 124 will also change. For example, if navigation information such as an arrow is located in the world reference coordinate system 250, it will always remain in a particular position in the world reference coordinate system 250 and may disappear from the user's field of view due to a given movement of the head / helmet 120.

[0104] Information about the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250 allows different data to be displayed on the augmented reality helmet-mounted display 124 according to the associated reference systems 230, 240, 250 and according to the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250.

[0105] In one embodiment, the determination of the position and orientation of the helmet 120 is continuously updated with data from the inertial measurement system 123 of the helmet 120, with data from the inertial measurement system 112 of the vehicle 110, with digital images of at least one identified feature 114 of the helmet 120 and / or vehicle 110, and / or with global position data of the helmet and / or vehicle. The continuous updates allow the position and orientation of the helmet 120 to be determined over time, which allows the position of the displayed data to be changed / adapted to the user of the vehicle 110 in response to changes in the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference frame 250. This improves safety for the user of the vehicle 110.

[0106] Figure 10 is a flow chart showing a series of steps for displaying data on the augmented reality helmet-mounted display 124 of the helmet 120. In the following paragraphs written with reference to Figure 10, a possible sequence of steps is described, which are executed by the processing system 115 in order to display data on the augmented reality helmet-mounted display 124. For determining the position and orientation of the helmet 120 (step S2), a combination of steps S1a, S1b, S1d, S1c, S2a, S2b and / or S2c may be used, as described above (steps S2a, S2b, S2c are not shown in Figure 10).

[0107] In step S3a, the processing system uses the eye tracking system 126 to identify the gaze vector of at least one eye, preferably both eyes, of the wearer of the helmet 120. The eye tracking system 126 is a system for determining the gaze vector of the user of the eye tracking system 126. The gaze vector of the wearer of the helmet 120 identifies the direction in which the wearer is looking. Using both eyes to identify the gaze vector improves its accuracy.

[0108] In step S3, the processing system 115 uses the identified line of sight vector to determine the position of the display data viewed by the wearer on the augmented reality helmet mounted display 124. That is, the display position on the augmented reality display 124 is further determined according to the line of sight vector.

[0109] In step S4, processing system 115 further uses the gaze vector of step S3a to display data to the wearer of helmet 120 on augmented reality helmet-mounted display 124. For example, the displayed data, such as vehicle speed, moves within augmented reality helmet-mounted display 124 as the user of vehicle 110 moves their gaze.

[0110] Additionally, processing system 115 can use the identified gaze vector to determine the type of data to be displayed on augmented reality display 124 for the wearer of helmet 120. For example, if the gaze vector determines that the wearer of helmet 120 has not looked at the road for a certain period of time, processing system 115 can alert the user of vehicle 110 by displaying warning information for the wearer of helmet 120 on the augmented reality display. Eye tracking system 126 can determine the location and / or type of data to be displayed on augmented reality helmet-mounted display 124, which can aid in displaying information necessary for safe use of vehicle 110. Additionally, if the wearer of helmet 120 has not looked at an obstacle on the road, processing system 115 can alert the user of vehicle 110 by displaying a warning on augmented reality helmet-mounted display 124.

[0111] 11 is a flow chart illustrating a sequence of steps for displaying data on the augmented reality helmet-mounted display 124 of the helmet 120. In this embodiment, the determination of the position and orientation of the helmet 120 is performed by the processing system 115 (115a, 115b and / or 115c) using a filter based on data integration with Kalman's equation as an estimator. Inputs to the processing system 115 are inertial data from the inertial measurement system 112 of the vehicle 110, inertial data from the inertial measurement system 123 of the helmet 120, digital images captured by the camera 122 of the helmet 120, and global position data from the global positioning system 111. In the following paragraphs written with reference to FIG. 11, a possible sequence of steps is described which may be performed by the processing system 115 to determine the position and orientation of the helmet 120 and to display data on the augmented reality helmet-mounted display 124.

[0112] In step S11, the processing system 115 uses at least one image captured by the helmet camera 122 to determine the position of the helmet camera 122 relative to the vehicle 110 or the vehicle reference system 240. In another embodiment, the processing system 115 uses at least one image captured by the vehicle camera 116 to determine the position of the helmet 120 relative to the vehicle 110. A combination of both is also contemplated.

[0113] In step S12, the processing system 115 110 The vehicle's 110 inertial data measured by the inertial measurement system 112 and global position data measured by the global positioning system 111 are used to determine the position and orientation of the vehicle 110 relative to the world reference coordinate system 250.

[0114] In step S13, the processing system 115 initializes the tracking system 100 by determining an initial position of the helmet 120 relative to the vehicle 110 and the world reference coordinate system 250. The initial position of the helmet 120 is determined using inertial measurement data from the inertial measurement system 123 of the helmet 120, inertial measurement data from the inertial measurement system 112 of the vehicle 110, the position of the helmet camera 122 determined in step S11, and the position of the vehicle 110 relative to the world reference coordinate system 250 determined in step S12.

[0115] The initial position of the helmet 120 is, for example, the position of the helmet 120 on the vehicle 110 before the vehicle 110 is moved by the user. In another embodiment, the initial position of the helmet 120 is the initial position of the helmet 120 when the helmet 120 enters the field of view of the camera 116 located on the vehicle 110, or the initial position of the helmet 120 when a particular feature 114 of the vehicle 110 enters the field of view of the camera 122 located on the helmet 120. Starting from the initial position of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250, the vehicle 110 and / or the helmet 120 may be moved by continuous movement of the vehicle 110 and / or the world reference coordinate system 250. 110The new position of the helmet 120 relative to the vehicle 110 and / or the world reference frame 250 can be updated with data from the inertial measurement systems 112, 123, images captured by one or more cameras 116, 122, and global position data. Thus, determining the initial position allows for increased accuracy and speed in determining the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference frame 250.

[0116] In step S14a, the processing system 115 uses a Kalman filter-inspired equation to predict the future position of the helmet 120. The input data for this step are the inertial measurement data of the vehicle 110 measured by the inertial measurement system 112 of the vehicle 110, the inertial measurement data of the helmet 120 measured by the inertial measurement system 123 of the helmet 120, and the initial position of the helmet 120 determined in step S13. A further input for this step is the updated position of the helmet 120 in step S14b.

[0117] In step S14b, the processing system 115 uses a Kalman filter-inspired equation to update the position of the helmet 120. The input data for this step are the position of the helmet camera 122 determined in step S11, the position of the vehicle 110 in the world reference frame 250 determined in step S12, and the position of the helmet 120 predicted in step S14a.

[0118] Steps S14a and S14b are performed repeatedly, with the predicted position of helmet 120 being used as input for updating the position of helmet 120 at each new time step, and the updated position of helmet 120 being used as input for predicting the position of helmet 120 at each new time step.

[0119] In step S15, the processing system 115 determines the position of the helmet 120 relative to the vehicle 110 and the world reference coordinate system 250.

[0120] In step S16, the processing system 115 displays data on the augmented reality helmet mounted display 124 to the wearer of the helmet 120, who is the operator of the vehicle 110, based on the position of the helmet 120 determined in step S15.

[0121] Figure 12 is a flow chart illustrating a sequence of steps for identifying, in one embodiment, an alternative feature 114a for a particular feature 114. The following paragraphs, written with reference to Figure 12, describe a possible sequence of steps for identifying an alternative feature 114a for a particular feature 114.

[0122] In step S20, a distinctive feature 114 is placed on the vehicle 110. In one embodiment, the distinctive feature 114 is a fiducial marker. The distinctive feature 114 is placed, for example, during installation of the tracking system 100 on the vehicle 110 and helmet 120.

[0123] In step S21, the camera 122 of the helmet 120 captures a digital image including the identifying feature 114 located in step S20 and additional features of the vehicle 110. The digital image shows, for example, the cockpit of the vehicle 110 with its dedicated identifying feature 114. Additionally, the digital image shows additional features of the cockpit, such as a circular fixture, one or more screws, or an edge adjacent to the identifying feature 114.

[0124] In step S22, processing systems 115a, 115b transmit the captured images to processing system 115c of remote server 160 for use in identifying alternative features 114a using the images. Communication between vehicle 110 and remote server 160 occurs, for example, via communication module 150.

[0125] In step S23, the processing system 115c of the remote server 160 identifies alternative features 114a using the at least one image. In one embodiment, the identification of alternative features 114a of the vehicle 110 is performed at the remote server 160 using a neural network. The images are processed by the neural network and the alternative features 114a are identified thereby. In another embodiment, the alternative features 114a are identified at the remote server 160 using feature descriptors such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or Oriented FAST and rotated BRIEF (ORB). In other embodiments, other machine learning approaches are used.

[0126] In step S24, the processing system 115a of the helmet 120 and / or the processing system 115b of the vehicle 110 receives the identified alternative features 114a from the remote server 160. In other words, information about the alternative features 114a identified by the neural network or another feature descriptor is transmitted from the remote server 160 to the processing system 115a of the helmet 120 or the processing system 115b of the vehicle 110, for example, via the communication module 150.

[0127] In step S25, the processing system 115 uses the alternate feature 114a as the specific feature 114 to identify the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250. In one embodiment, the specific feature 114 is removed from the vehicle 110 after the alternate feature 114a has been identified and used by the processing system 115.

[0128] In one embodiment, the distinctive feature 114 is located on the helmet 120 and the image is captured by the camera 116 of the vehicle 110. Combinations of these are also contemplated. In one embodiment, the distinctive feature 114 is a fiducial marker.

[0129] A user of the vehicle 110 places a dedicated identification feature 114, such as a marker, on the vehicle when the system 100 is installed. The user then uses the vehicle 110 for a predetermined period of time. In one embodiment, the predetermined period of time is between 5 and 100 hours of use, preferably between 5 and 20 hours of use. During this period of time, a digital image including the identification feature 114 and the additional feature is transmitted to the remote server 160. The processing system 115c of the remote server 160 processes the image via a neural network to identify a replacement feature 114a. The information about the replacement feature 114a is then transmitted back to the processing system 115 of the vehicle 110, which uses the replacement feature 114a in place of the old identification feature 114. The old identification feature 114 may be removed from the vehicle 110 by the user because it is no longer needed.

[0130] It should be noted that although the order of steps is set forth in a particular order herein, one skilled in the art will understand that the order of at least some of the steps may be changed without departing from the scope of the present disclosure.

[0131] FIG. 13 schematically illustrates a vehicle 110 carrying a user and a tracking system 100 according to a fourth exemplary embodiment. This embodiment differs from the embodiment illustrated in FIGS. 1-3 in that eyewear 120′ replaces the helmet 120 and an eyewear reference system 230′ replaces the helmet reference system 230. It should be understood that all combinations shown for the tracking system 100 including the helmet 120, as presented and described in FIGS. 1-3, can be similarly applied to the tracking system 100 including the eyewear 120′ instead of the helmet 120. It should also be understood that all flowcharts of the method for determining the position and orientation of the helmet 120, as presented and described in FIGS. 4-12, can be similarly applied to the method for determining the position and orientation of the eyewear 120′ instead of the helmet 120. Furthermore, it should be understood that all advantages presented and described with respect to the helmet 120 can be similarly applied to the eyewear 120′.

[0132] In this embodiment, the vehicle 110 is a motorized two-wheeled vehicle, but in another embodiment, it is a motorized four-wheeled vehicle such as a car, and other vehicles such as a truck or a bicycle are also contemplated. The vehicle 110 includes a global positioning system 111, an inertial measurement system 112, an engine 113, a specific feature 114, a vehicle processing system 115b, and an alternate feature 114a. The global positioning system 111 is configured to measure global position data of the vehicle 110. The inertial measurement system 112 is configured to measure inertial data of the vehicle 110. The engine 113 is configured to provide power to the vehicle 110. The specific feature 114 is located in or near the cockpit area of ​​the vehicle 110 and is configured to be within an image frame of a digital image captured by a camera 122 located in the eyewear 120′. The alternate feature 114a is configured to function as a substitute for the specific feature 114.

[0133] As shown in FIG. 13 , the eyewear 120′ comprises a battery 121, a camera 122, an inertial measurement system 123, an augmented reality eyewear-mounted display 124, an eyewear processing system 115a, and an eye-tracking system 126. The battery 121 is disposed in the eyewear 120′ and supplies electrical energy to various electrical systems of the eyewear 120′. The camera 122 disposed in the eyewear 120′ is configured to capture images of the specific feature 114 and / or the alternative feature 114a of the vehicle 110. The inertial measurement system 123 is configured to measure inertial data of the eyewear 120′. The augmented reality eyewear-mounted display 124 is configured to display data to a wearer of the eyewear 120′. The augmented reality eyewear-mounted display 124 may be, for example, fixedly or detachably disposed in the eyewear 120′. The eye-tracking system 126 is configured, for example, to measure a gaze vector of a wearer of the eyewear 120′.

[0134] In one embodiment, the world reference coordinate system 250 is a coordinate system that associates each location in the world with a unique coordinate position. In addition to the world reference coordinate system 250, Figure 13 also schematically illustrates an eyewear reference system 230' and a vehicle reference system 240.

[0135] In one embodiment, the camera 122, the inertial measurement system 123, the augmented reality display 124, the eye tracking system 126, and the processing system 115a of the eyewear 120', which is part of the processing system 115 (see FIG. 13), are contained in a single eyewear unit, such as a pair of specific glasses.

Claims

1. 1. A method for determining the position and orientation of a helmet (120) while in motion on a vehicle (110), comprising: measuring (S1a) inertial data of the helmet (120) while the helmet (120) is in motion by an inertial measurement system (123) of the helmet (120); measuring (S1b) inertial data of the vehicle (110) while the vehicle (110) is in motion by an inertial measurement system (112) of the vehicle (110); (S2) determining, by a processing system (115), a position and orientation of the helmet (120) relative to the vehicle (110) using the inertial data of the helmet (120) and the inertial data of the vehicle (110); A method comprising:

2. taking (S1c) at least one digital image of the vehicle (110) using a camera (122) located on the helmet (120) and / or taking (S1c) at least one digital image of the helmet (120) using a camera (116) located on the vehicle (110); (S2) further using the one or more digital images of the vehicle (110) and / or the helmet (120) to determine, by the processing system (115), the position and orientation of the helmet (120) relative to the vehicle (110); The method of claim 1 further comprising:

3. 3. The method of claim 2, wherein the at least one captured digital image captured using the camera (122) located on the helmet (120) includes at least one particular feature (114) of the vehicle (110) and / or the at least one digital image captured using the camera (116) located on the vehicle (110) includes at least one particular feature (114) of the helmet (120), and the processing system (115) further uses one or more of the digital images including the at least one particular feature (114) of the vehicle (110) and / or the helmet (120) to determine the position and orientation of the helmet (120) relative to the vehicle (110).

4. 4. The method of claim 3, further comprising identifying, by the processing system (115), alternative features (114a) for the particular feature (114) of the vehicle (110) and / or the helmet (120) by performing the following steps: capturing (S1c) a digital image of the vehicle (110) and / or the helmet (120) including the at least one particular feature (114) and additional features using the camera (122) located on the helmet (120) or the vehicle (110); Identifying (S2b) the alternative features (114a) from the digital image by the processing system (115); and using (S2c) the alternative feature (114a) as the specific feature (114) to identify the position and orientation of the helmet (120) relative to the vehicle (110);

5. measuring (S1d) global position data of said vehicle (110) and / or said helmet (120) relative to a world reference coordinate system (250) by a global positioning system (111) of said vehicle (110) and / or said helmet (120); (S2) further using the measured global position data of the vehicle (110) and / or the helmet (120) to determine, by the processing system (115), the position and orientation of the vehicle (110) and / or the helmet (120) relative to the world reference coordinate system (250); 5. The method of claim 1, further comprising:

6. 5. The method according to claim 2, wherein the initial position of the helmet (120) is determined by the following steps: (S1c) taking at least one digital image of the vehicle (110) using the camera (122) located on the helmet (120) and / or taking at least one digital image of the helmet (120) using the camera (116) located on the vehicle (110); and determining (S2) the initial position of the helmet (120) relative to the vehicle (110) by the processing system (115) using one or more of the digital images of the vehicle (110) and / or the helmet (120);

7. 5. The method according to any one of claims 1 to 4, wherein the determination of the position and orientation of the helmet (120) relative to the vehicle (110) and / or the world reference coordinate system (250) is performed by the processing system (115) by data synthesis based on an estimator.

8. using the position and orientation of the helmet (120) relative to the vehicle (110) and / or the world reference coordinate system (250), to determine (S3) a display position for data to be displayed on an augmented reality display (124) located on the helmet (120); displaying (S4) the data to a wearer of the helmet (120) on the augmented reality display (124) by the processing system (115) using the display location; 5. The method of claim 1, further comprising:

9. determining (S3a) a gaze vector of at least one eye, preferably both eyes, of a wearer of the helmet (120) by means of an eye tracking system (126) arranged in the helmet (120); Using the identified gaze vector, the processing system (115) determines (S3) the display position and / or type of the data on the augmented reality display (124) viewed by the wearer; 9. The method of claim 8, further comprising:

10. 10. The method of any one of claims 1 to 4 or claim 9, wherein the processing system (115) comprises a processing system (115a) of the helmet (120), a processing system (115b) of the vehicle (110), and / or a processing system (115c) of a remote server (160).

11. 10. A computer program product comprising: a non-transitory computer readable medium having stored thereon computer program code configured to control a processing system (115) of a tracking system (100) such that the tracking system (100) performs the steps of the method of any one of claims 1 to 4 or claim 9.

12. 1. A tracking system (100) for determining the position and orientation of a helmet (120) while in motion on a vehicle (110), comprising: an inertial measurement system (123) disposed on the helmet (120); an inertial measurement system (112) disposed on the vehicle (110); and a processing system (115) configured to perform the following steps: measuring (S1a) inertial data of the helmet (120) while the helmet (120) is in motion using the inertial measurement system (123) of the helmet (120); measuring (S1b) inertial data of the vehicle (110) while the vehicle (110) is in motion using the inertial measurement system (112) of the vehicle (110); and (S2) determining a position and orientation of the helmet (120) relative to the vehicle (110) using the inertial data of the helmet (120) and the inertial data of the vehicle (110);

13. 13. The tracking system (100) of claim 12, further comprising a camera (122) disposed on the helmet (120) and / or a camera (116) disposed on the vehicle (110), wherein the processing system (115) is further configured to perform the steps of: (S1c) capturing at least one digital image of the vehicle (110) using the camera (122) located on the helmet (120) and / or capturing at least one digital image of the helmet using the camera (116) located on the vehicle (110); and (S2) further using the digital image(s) of the vehicle (110) and / or the helmet (120) to determine the position and orientation of the helmet (120) relative to the vehicle (110).

14. 14. The tracking system (100) of claim 13, wherein the at least one captured digital image taken using the camera (122) disposed on the helmet (120) includes at least one particular feature (114) of the vehicle (110) and / or the at least one captured digital image taken by the camera (116) disposed on the vehicle (110) includes at least one particular feature (114) of the helmet (120), and the processing system (115) is configured to further use one or more of the digital images including the at least one particular feature (114) of the vehicle (110) and / or the helmet (120) to determine the position and orientation of the helmet (120) relative to the vehicle (110).

15. 15. The tracking system (100) of any one of claims 12 to 14, further comprising a global positioning system (111), wherein the processing system (115) is further configured to perform the steps of: measuring (S1d) global position data of the vehicle (110) and / or the helmet (120) relative to a world reference coordinate system (250) using the global positioning system (111); and (S2) further using the measured global position data of the vehicle (110) and / or the helmet (120) to determine the position and orientation of the vehicle (110) and / or the helmet (120) relative to the world reference coordinate system (250).

16. 15. The tracking system (100) of any one of claims 12 to 14, wherein the processing system (115) is configured to determine the position and orientation of the helmet (120) relative to the vehicle (110) and / or the world reference coordinate system (250) by estimator-based data integration.

17. 15. The tracking system (100) of any one of claims 12 to 14, further comprising an augmented reality display (124) disposed on the helmet (120), wherein the processing system (115) is further configured to perform the steps of: Using the position and orientation of the helmet (120) relative to the vehicle (110) and / or the world reference coordinate system (250), determine (S3) the display location of data to be displayed on the augmented reality display (124); and and displaying (S4) the data to a wearer of the helmet (120) on the augmented reality display (124) using the display location.

18. 20. The tracking system (100) of claim 17, further comprising an eye-tracking system (126) disposed in the helmet (120), wherein the processing system (115) is further configured to perform the steps of: Using the eye tracking system (126), determining (S3a) the gaze vector of at least one eye, preferably both eyes, of a wearer of the helmet (120); and Using the identified gaze vector, a step (S3) is performed to determine a display position of the data and / or a type of the data on the augmented reality display (124) viewed by the wearer.

Images