Method and system for determining helmet position and orientation

Abstract

The present disclosure relates to a method and tracking system for determining the position and orientation of a helmet (120) while in motion on a vehicle (110), inertial data of the helmet (120) while the helmet (120) is in motion is measured by an inertial measurement system (123) of the helmet (120), inertial data of the vehicle (110) while the vehicle (110) is in motion is measured by an inertial measurement system (112) of the vehicle (110), and the position and orientation of the helmet (120) relative to the vehicle (110) is determined by a processing system (115) using the inertial data of the helmet (120) and the inertial data of the vehicle (110).

Description

[Technical field]

[0001] The present disclosure relates to a system and method for determining the position and orientation of a helmet and / or eyewear while moving on or within a vehicle. In particular, the present disclosure relates to a method for determining the position and orientation of a helmet and / or eyewear while moving on or within a vehicle, and also to a tracking system for determining the position and orientation of the helmet / eyewear while moving on or within a vehicle. Furthermore, the present disclosure relates to a computer program product comprising a non-transitory computer readable medium having stored thereon computer program code configured to control a processing system of said tracking system. [Background technology]

[0002] Increasing safety on the roads, especially reducing fatalities and injuries among road users, is a common goal. This goal has been addressed in recent years, especially by car manufacturers. Various systems such as lane keeping assist systems, forward collision warning systems, or automatic emergency braking systems have been developed to achieve the above goal. The safety of car and truck users has improved significantly in recent years. Despite these technological developments in the automobile industry, motorcycle safety has not improved to the same extent. For example, anti-block braking systems (ABS) have only been introduced in the motorcycle industry in the last decade. Other safety systems in the automobile industry have not yet been introduced in modern motorcycles.

[0003] This disparity is difficult to understand, and here's why: Non-motorized two-wheeled vehicles, such as bicycles, and motorized two-wheeled vehicles, such as motorcycles, e-bikes, and e-scooters, pose a higher risk of injury to riders than other forms of transportation, especially automobiles. For example, in the United States in 2017, motorcycle fatalities and injuries were about 27 times higher per mile traveled than automobile fatalities and injuries.

[0004] Some safety measures in automobiles are not so easy to introduce into motorcycles. For example, a head-up display in an automobile always projects information on the same part of the automobile's windshield, or in front of the windshield, or on an additional dedicated display. Since the automobile user always looks at the same part of the windshield, the information presented via the head-up display can be easily read in almost all driving situations, and the user of an automobile equipped with a head-up display does not need to lower his / her gaze, which improves driving safety. On the other hand, motorcycles, such as electric scooters and high-performance motorcycles, often do not even have a windshield. Moreover, even if a motorcycle has a windshield, the motorcycle user does not always look into the windshield while driving. Furthermore, since the motorcycle user moves his / her head more frequently than the automobile user, if the motorcycle windshield has a head-up display, it becomes very difficult to present driving information to the motorcycle user in an easy-to-read manner while using the motorcycle.

[0005] Conventional systems on motorcycles for displaying driving information, such as speed and navigation information, use fixed displays or tachometers, which require motorcycle riders to lower their gaze to read the relevant driving information. For motorcycle riders, lowering their gaze increases the risk and likelihood of an accident. Summary of the Invention [Problem to be solved by the invention]

[0006] It is an object of the present disclosure to provide a system and method for determining the position and orientation of a helmet or eyewear while moving on or within a vehicle. In particular, it is an object of the present disclosure to provide a system, method and computer program product for determining the position and orientation of a helmet or eyewear while moving on or within a vehicle, which do not have at least some of the disadvantages of the prior art.

[0007] According to the present disclosure, these objects can be achieved by means of the features of the independent claims. Further advantageous embodiments result from the dependent claims and the description.

[0008] According to the present disclosure, the above mentioned objects are among others achieved by a method of determining a position and orientation of a helmet while in motion on a vehicle, the method comprising the step of measuring, by an inertial measurement system of the helmet, inertial data of the helmet while the helmet is in motion, the method also comprising the step of measuring, by an inertial measurement system of the vehicle, inertial data of the vehicle while the vehicle is in motion, the method further comprising the step of determining, by a processing system, a position and orientation of the helmet relative to the vehicle using the inertial data of the helmet and the inertial data of the vehicle.

[0009] In one embodiment, the method also includes capturing at least one digital image of the vehicle using a camera disposed on the helmet and / or capturing at least one digital image of the helmet using a camera disposed on the vehicle, the method further including determining, by the processing system, a position and orientation of the helmet relative to the vehicle using the one or more of the digital images of the vehicle and / or the helmet.

[0010] The camera mentioned above and below may be any sensing device capable of generating data (e.g. digital images) indicative of distance and space. Thus, it includes any type of distance sensor, such as a radar sensor or a LIDAR (Light Detection and Ranging) sensor. Furthermore, the sensing device may include a laser-based scanning system that determines distance, for example by measuring the time of flight (TOF) of a laser beam. The data may then generate a two- or three-dimensional digital image of the space, which may be processed by a processing system to determine the position and orientation of the helmet. In the following description, when a digital image or simply an image is mentioned, the image may have been taken by such a camera or distance sensor.

[0011] In one embodiment, the at least one captured digital image taken using the camera positioned on the helmet includes at least one particular feature of the vehicle and / or the at least one digital image taken using the camera positioned on the vehicle includes at least one particular feature of the helmet, and the processing system further uses one or more of the digital images including the at least one particular feature of the vehicle and / or the helmet to determine the position and orientation of the helmet relative to the vehicle.

[0012] In one embodiment, the method further comprises the processing system identifying an alternative feature of the vehicle and / or helmet by performing the steps of: capturing a digital image with the camera located on the helmet or vehicle, the digital image including the at least one distinctive feature and an additional feature of the vehicle and / or helmet; identifying the alternative feature from the image by the processing system; and using the alternative feature as the distinctive feature to identify a position and orientation of the helmet relative to the vehicle.

[0013] In one embodiment, the alternative features are identified by a machine learning model or algorithm using a captured digital image containing one particular feature and / or multiple additional features. In one embodiment, the digital image is processed by a processing system using a neural network as a machine learning model to calculate a preferred alternative feature from multiple possible additional features. In another embodiment, the alternative features are determined by the processing system using feature descriptors such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF) or Oriented FAST and rotated BRIEF (ORB).

[0014] In one embodiment, the method further comprises determining a position and orientation of the helmet using at least one captured digital image captured by a camera, the at least one digital image being processed by a position and orientation estimator on a processing system and / or the position and orientation estimator may be a machine learning model such as a neural network.

[0015] In one embodiment, the method further comprises predicting a position and orientation of the helmet using at least one captured digital image captured by a camera, the at least one digital image being processed by a position and orientation estimator on a processing system, the position and orientation estimator being a machine learning model such as a neural network.

[0016] In the above and below description, when a machine learning model / method is mentioned, it should be understood that the method consists of any model that generates an output for a given input. This also includes any manner of training and using a pre-trained model that generates an output for a given input, mainly but not limited to optimizing the model depending on the input and expected output, pre-trained with past data by minimizing a defined cost function. The input may be inertial data and the output may be a specified position and orientation of the object or a prediction of the object's future position and orientation. The input may be a digital image depicting at least a portion of the object and the output may be a specified position and orientation of the object or a prediction of the object's future position and orientation. The input may also be a combination of the inertial data and the digital image by a weighted combination. The input may further be a combination of the inertial data and the digital image by the model itself using data integration based on an estimator. The object may be a helmet, eyewear, a vehicle, an eye, a specific feature, or the surrounding environment that is part of the world reference frame, such as a road. In other cases, for an input of a digital image, the output may be identified / defined alternative features at least partially depicted by the image by selecting alternative features from a plurality of features. The machine learning models used may consist of, but are not limited to, direct regression, general deep neural networks, convolutional neural networks, transformer-based neural networks, and recurrent neural networks, or weighted combinations of these and further models.

[0017] In one embodiment, the at least one distinguishing feature of the vehicle and / or helmet is a fiducial marker located on the vehicle or helmet. A fiducial marker or fiducial is an object located within the field of view of the imaging system that is displayed in the generated image for use as a reference point or measure.

[0018] In one embodiment, the method further comprises measuring global position data of the vehicle and / or helmet relative to a world reference coordinate system by a global positioning system of the vehicle and / or helmet. The method according to this embodiment further comprises determining, by the processing system, a position and orientation of the helmet relative to the world reference coordinate system, further using the measured global position data of the vehicle and / or helmet. In one embodiment, the global positioning system uses positioning information obtained via satellites. In another embodiment, the global positioning system uses image-based global position data, for example obtained from a vehicle controller area network bus (CAN-BUS) unit.

[0019] In one embodiment, the initial position of the helmet is determined by the steps of: taking at least one digital image of the vehicle with the camera located on the helmet and / or taking at least one digital image of the helmet with the camera located on the vehicle; Determining, by the processing system, the initial position of the helmet relative to the vehicle using one or more of the digital images of the vehicle and / or the helmet. In one embodiment, the digital image may include specific features of the helmet or vehicle.

[0020] In one embodiment, the determination of the position and orientation of the helmet relative to the vehicle and / or the world reference coordinate system is performed by the processing system using a data integration method, such as captured digital images, acquired data, measured inertial data of the helmet and / or the vehicle, and additional data.

[0021] In one embodiment, the determination of the helmet position and orientation relative to the vehicle and / or the world reference frame is performed by the processing system using estimator based data synthesis. In one embodiment, the estimator based data synthesis is performed using a linear filter, such as a Kalman filter, a non-linear filter, or a combination thereof. The estimator Kalman filter based data synthesis allows for smooth, fast and accurate determination of the helmet position and orientation relative to the vehicle and / or the world reference frame.

[0022] In one embodiment, the method also includes using the helmet's position and orientation relative to the vehicle and / or the world reference coordinate system to identify, by the processing system, a display location for data to be displayed on an augmented reality helmet mounted display located on the helmet. The method further includes using the identified display location to display, by the processing system, the data to a wearer of the helmet on the augmented reality helmet mounted display. According to this embodiment, the helmet includes an augmented reality helmet mounted display for displaying data to a wearer of the helmet and a user of the vehicle. In one embodiment, the augmented reality display is fixedly located on the helmet, while in another embodiment, the augmented reality display is removable from the helmet.

[0023] In one embodiment, the method further comprises the steps of: determining, by an eye-tracking system arranged in the helmet, a gaze vector of at least one eye, preferably both eyes, of a wearer of the helmet, and using the determined gaze vector, by the processing system, determining a display position of the data and / or a type of the data on the augmented reality helmet mounted display as viewed by the wearer. The eye-tracking system is a system for determining the gaze vector of a user of the eye-tracking system.

[0024] In one embodiment, the processing system comprises a processing system of the helmet, a processing system of the vehicle, and / or a processing system of a remote server. In other words, the processing system can be located in the helmet, the vehicle, a remote server, or a combination thereof. Different calculations can be performed on different processing systems. The processing system can further comprise a processing system of a mobile device, such as a smartphone of a user, used in combination with the processing system of the helmet and / or the vehicle, for example, to determine the position and orientation of the helmet or to update a neural network used to calculate a correction signal for drift of the inertial measurement system.

[0025] In some embodiments, the processing system may include a system on a chip (SoC), a central processing unit (CPU), and / or other more specific processing units such as a graphics processing unit (GPU) or an application specific integrated circuit (ASIC), a reprogrammable processing unit such as a field programmable gate array (FPGA), or even a processing unit specifically configured to determine the position and orientation of the helmet relative to the vehicle.

[0026] In a further aspect, in addition to a method for determining a position and orientation of a helmet, the present disclosure relates to a tracking system for determining a position and orientation of a helmet while in motion on a vehicle, the tracking system comprising an inertial measurement system disposed on the helmet, an inertial measurement system disposed on the vehicle, and a processing system configured to perform the following steps: measuring inertial data of the helmet while the helmet is in motion using the inertial measurement system of the helmet; measuring inertial data of the vehicle while the vehicle is in motion using the inertial measurement system of the vehicle; and Using the inertial data of the helmet and the inertial data of the vehicle to determine a position and orientation of the helmet relative to the vehicle.

[0027] In one embodiment, the tracking system further comprises a camera disposed on the helmet and / or a camera disposed on the vehicle, and the processing system is further configured to perform the steps of: taking at least one digital image of the vehicle with the camera located on the helmet and / or taking at least one digital image of the helmet with the camera located on the vehicle; and Further using the digital image(s) of the vehicle and / or the helmet to determine a position and orientation of the helmet relative to the vehicle.

[0028] In one embodiment, at least one captured digital image taken with the camera located on the helmet includes at least one particular feature of the vehicle and / or the at least one captured digital image of the camera located on the vehicle includes at least one particular feature of the helmet, and the at least one particular feature of the vehicle and / or the helmet is further used by the processing system to determine a position and orientation of the helmet relative to the vehicle.

[0029] In one embodiment, the tracking system further comprises a global positioning system, and the processing system is further configured to perform the steps of: measuring global position data of the vehicle and / or the helmet relative to a world reference coordinate system using the global positioning system; and Further using the measured global position data of the vehicle and / or the helmet to determine a position and orientation of the vehicle and / or the helmet relative to the world reference coordinate system.

[0030] In one embodiment, the tracking system is configured to use data fusion techniques to determine the position and orientation of the helmet relative to the vehicle and / or the world reference coordinate system.

[0031] In one embodiment, the tracking system is configured to determine the position and orientation of the helmet relative to the vehicle and / or the world reference coordinate system using an estimator-based data synthesis.

[0032] In one embodiment, the tracking system further comprises an augmented reality display disposed on the helmet, and the processing system is further configured to perform the steps of: using the helmet's position and orientation relative to the vehicle and / or the world reference coordinate system to determine a display location for data to be displayed on the augmented reality helmet mounted display; and Displaying the data to a wearer of the helmet on the augmented reality display using the display location.

[0033] In one embodiment, the tracking system further comprises an eye-tracking system disposed on the helmet, and the processing system is further configured to perform the steps of: determining, using the eye-tracking system, the gaze vector of at least one eye, and preferably both eyes, of the wearer of the helmet; and Using the identified gaze vector to determine the display position and / or type of the data on the augmented reality display as viewed by the wearer.

[0034] In addition to systems and methods for determining a helmet's position and orientation while in motion on a vehicle, the present disclosure relates to a computer program product comprising a non-transitory computer readable medium having stored thereon computer program code configured to control a processing system of the tracking system such that the tracking system performs steps in the above-mentioned methods.

[0035] According to another aspect of the present disclosure, the above object is achieved in particular by a method for determining the position and orientation of eyewear while moving on or in a vehicle. The features and advantages of the aspect for determining the position and orientation of a helmet described above or hereinafter also apply to the aspect for determining the position and orientation of eyewear while moving on or in a vehicle, as set forth below. The method comprises measuring, by an inertial measurement system of the eyewear, inertial data of the eyewear while the eyewear is moving. The method also comprises measuring, by an inertial measurement system of the vehicle, inertial data of the vehicle while the vehicle is moving. The method according to the present disclosure further comprises determining, by a processing system, the position and orientation of the eyewear relative to the vehicle using the inertial data of the eyewear and the inertial data of the vehicle.

[0036] Eyewear, as understood above and below, may consist of any device worn by a human user, i.e. the user of the eyewear, and which at least partially covers the user's field of vision. This may consist of devices such as spectacles, goggles or monocles.

[0037] In one embodiment, the method also includes capturing at least one digital image of the vehicle using a camera disposed on the eyewear and / or capturing at least one digital image of the eyewear using a camera disposed on the vehicle, the method further includes determining, by the processing system, a position and orientation of the eyewear relative to the vehicle using the one or more of the digital images of the vehicle and / or the eyewear.

[0038] In one embodiment, the method further comprises determining a position and orientation of the eyewear relative to the vehicle using at least one captured digital image captured by the camera, the at least one digital image being processed by a position and orientation estimator of the processing system, the position and orientation estimator being a machine learning model, such as a neural network.

[0039] In one embodiment, the method further comprises predicting a position and orientation of the eyewear relative to the vehicle using at least one captured digital image captured by the camera, the at least one digital image being processed by a position and orientation estimator of the processing system, the position and orientation estimator being a machine learning model, such as a neural network.

[0040] In one embodiment, the at least one captured digital image taken using the camera positioned on the eyewear includes at least one particular feature of the vehicle and / or the at least one captured digital image taken by the camera positioned on the vehicle includes at least one particular feature of the eyewear, and the processing system further uses one or more of the digital images including the at least one particular feature of the vehicle and / or the eyewear to determine the position and orientation of the eyewear relative to the vehicle.

[0041] In one embodiment, the method further includes identifying alternative characteristics of the vehicle and / or the eyewear with the processing system by performing the steps of: capturing a digital image with the camera located on the eyewear or the vehicle, the digital image including the at least one identifying feature and an additional feature of the vehicle and / or the eyewear, and identifying, by the processing system, the alternative feature from the image, the method according to this embodiment further including using the alternative feature as the identifying feature to identify a position and orientation of the eyewear relative to the vehicle.

[0042] In one embodiment, the alternative features are identified by a machine learning model or algorithm using a digital image that includes one specific feature and multiple additional features. In one embodiment, the digital image is processed by the processing system using a neural network to calculate a preferred alternative feature from multiple possible additional features. In another embodiment, the alternative features are determined by the processing system using feature descriptors such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF) or Oriented FAST and rotated BRIEF (ORB).

[0043] In one embodiment, the at least one identifying feature of the vehicle and / or eyewear is a fiducial marker located on the vehicle or eyewear. A fiducial marker, or fiducial, is an object located within the field of view of the imaging system that is displayed in the generated image for use as a reference point or measure.

[0044] In one embodiment, the method further comprises measuring global position data of the vehicle and / or eyewear relative to a world reference coordinate system by a global positioning system of the vehicle and / or eyewear. The method according to this embodiment further comprises determining, by the processing system, a position and orientation of the eyewear relative to the world reference coordinate system using the measured global position data of the vehicle and / or eyewear. In one embodiment, the global positioning system uses positioning information obtained via artificial satellites. In another embodiment, the global positioning system uses global position data obtained, for example, from a vehicle controller area network bus (CAN-BUS) unit.

[0045] In one embodiment, the initial position of the eyewear is determined by the following steps. capturing at least one digital image of the vehicle with the camera disposed on the eyewear and / or capturing at least one digital image of the eyewear with the camera disposed on the vehicle; and Determining, by the processing system, the initial position of the eyewear relative to the vehicle using one or more of the digital images of the vehicle and / or the eyewear, in one embodiment, a digital image may include specific features of the eyewear or vehicle.

[0046] In one embodiment, determining the position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system is performed by the processing system using data integration techniques, such as captured digital images, measured inertial data of the eyewear and / or the vehicle, and / or additional data.

[0047] In one embodiment, the determination of the position and orientation of the eye relative to the vehicle and / or the world reference frame is performed by the processing system using estimator-based data synthesis. In one embodiment, the estimator-based data synthesis is performed using a linear filter, such as a Kalman filter, a non-linear filter, or a combination thereof. The estimator-Kalman filter-based data synthesis allows for smooth, fast, and accurate determination of the position and orientation of the eye relative to the vehicle and / or the world reference frame.

[0048] In one embodiment, the method also includes using a position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system to identify, by the processing system, a display location for data to be displayed on an augmented reality eyewear-mounted display disposed on the eyewear. The method further includes displaying, by the processing system, the data to a wearer of the eyewear on the augmented reality eyewear-mounted display using the identified display location. According to this embodiment, the eyewear includes an augmented reality eyewear-mounted display for displaying data to a wearer of the eyewear and a user of the vehicle. In one embodiment, the augmented reality display is fixedly disposed on the eyewear, and in another embodiment, the augmented reality display is removable from the eyewear.

[0049] In one embodiment, the method further comprises the steps of: determining, by an eye gaze tracking system disposed in the eyewear, a gaze vector of at least one eye, preferably both eyes, of a wearer of the eyewear; and using the determined gaze vector, determining, by the processing system, a display position of the data and / or a type of the data on the augmented reality eyewear mounted display as viewed by the wearer. The eye gaze tracking system is a system for determining the gaze vector of a user of the eye gaze tracking system.

[0050] In one embodiment, the processing system comprises a processing system of the eyewear, a processing system of the vehicle, and / or a processing system of a remote server. In other words, the processing system can be located in the eyewear, the vehicle, a remote server, or a combination thereof. Different calculations can be performed on different processing systems. The processing system can further comprise a processing system of a mobile device, such as a user's smartphone, used in combination with the processing system of the eyewear and / or the vehicle, for example, to determine the position and orientation of the eyewear or to update a neural network used to calculate a correction signal for drift of an inertial measurement system.

[0051] In a further aspect, the present disclosure relates to a tracking system for determining a position and orientation of eyewear while in motion on or within a vehicle, the tracking system comprising an inertial measurement system disposed on the eyewear, an inertial measurement system disposed on the vehicle, and a processing system configured to perform the following steps: measuring inertial data of the eyewear using the inertial measurement system of the eyewear while the eyewear is in motion; measuring, by the inertial measurement system of the vehicle, inertial data of the vehicle while the vehicle is in motion; and Using the inertial data of the eyewear and the inertial data of the vehicle to determine a position and orientation of the eyewear relative to the vehicle.

[0052] In one embodiment, the tracking system further comprises a camera disposed in the eyewear and / or a camera disposed in the vehicle, and the processing system is further configured to perform the steps of: capturing at least one digital image of the vehicle with the camera disposed on the eyewear and / or capturing at least one digital image of the eyewear with the camera disposed on the vehicle; and Further using the digital images of the vehicle and / or the eyewear to determine a position and orientation of the eyewear relative to the vehicle.

[0053] In one embodiment, at least one captured digital image taken with the camera located on the eyewear includes at least one particular feature of the vehicle and / or the at least one captured digital image of the camera located on the vehicle includes at least one particular feature of the eyewear, and the at least one particular feature of the vehicle and / or the eyewear is further used by a processing system to determine a position and orientation of the eyewear relative to the vehicle.

[0054] In one embodiment, the tracking system further comprises a global positioning system, and the processing system is further configured to perform the steps of: measuring global position data of the vehicle and / or eyewear relative to a world reference coordinate system using the global positioning system; and Further using the measured global position data of the vehicle and / or eyewear to determine a position and orientation of the vehicle and / or eyewear relative to the world reference coordinate system.

[0055] In one embodiment, the tracking system is configured to use a data integration method to determine the position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system.

[0056] In one embodiment, the tracking system is configured to determine the position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system using an estimator-based data synthesis.

[0057] In one embodiment, the tracking system further comprises an augmented reality display disposed on the eyewear, and the processing system is further configured to perform the steps of: determining a display location for data to be displayed on the augmented reality eyewear display using the position and orientation of the eyewear relative to the vehicle and / or the world reference coordinate system; and Displaying the data to a wearer of the eyewear on the augmented reality display using the display location.

[0058] In one embodiment, the camera and the augmented reality display are configured as a single unit.

[0059] In one embodiment, the tracking system further comprises an eye gaze tracking system disposed on the eyewear, and the processing system is further configured to perform the steps of: determining, using the eye gaze tracking system, a gaze vector of at least one eye, preferably both eyes, of a wearer of the eyewear; Using the identified gaze vector to determine the display position and / or type of the data on the augmented reality display as viewed by the wearer.

[0060] In one embodiment, the camera, the augmented reality display, and the eye tracking system are configured in a single unit.In one embodiment, the camera, the inertial measurement system, the augmented reality display, and the eye tracking system are configured in a single unit.

[0061] In addition to systems and methods for determining the position and orientation of eyewear while moving on a vehicle, the present disclosure relates to a computer program product comprising a non-transitory computer readable medium having stored thereon computer program code configured to control a processing system of the tracking system such that the tracking system performs steps in the above-mentioned methods.

[0062] In the following, the present disclosure is explained in more detail, for example with reference to the following figures: [Brief description of the drawings]

[0063] [Figure 1] FIG. 1 is a schematic diagram of a vehicle carrying a user and a tracking system according to a first exemplary embodiment. [Diagram 2] FIG. 13 is a schematic diagram of a vehicle carrying a user and a tracking system according to a second exemplary embodiment. [Diagram 3] FIG. 13 is a schematic diagram of a vehicle carrying a user and a tracking system according to a third exemplary embodiment; [Figure 4] 4 is a flow chart illustrating a series of steps for performing helmet position and orientation determination in a first exemplary embodiment. [Diagram 5] 10 is a flow chart illustrating a series of steps for performing helmet position and orientation determination in a second exemplary embodiment. [Figure 6] 10 is a flow chart illustrating a series of steps for performing helmet position and orientation determination in a third exemplary embodiment. [Figure 7] 13 is a flow chart illustrating a series of steps for performing helmet position and orientation determination in a fourth exemplary embodiment. [Figure 8] 13 is a flow chart illustrating a series of steps for performing helmet position and orientation determination in a fifth exemplary embodiment. [Figure 9] 1 is a flow chart illustrating a series of steps for displaying data on an augmented reality helmet mounted display of a helmet in a first exemplary embodiment. [Figure 10] 11 is a flow chart illustrating a series of steps for displaying data on an augmented reality helmet mounted display of a helmet in a second exemplary embodiment. [Figure 11]13 is a flow chart illustrating a series of steps for displaying data on an augmented reality helmet mounted display of a helmet in a third exemplary embodiment. [Figure 12] 1 is a flowchart illustrating a series of steps for identifying alternative features for a particular feature in one exemplary embodiment. [Figure 13] FIG. 13 is a schematic diagram of a vehicle carrying a user and a tracking system according to a fourth exemplary embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] FIG. 1 illustrates a schematic diagram of a vehicle 110 ridden by a user and a tracking system 100 according to a first exemplary embodiment. In this embodiment, the vehicle 110 is a motorized two-wheeler. The vehicle 110 may be any different type of vehicle, such as, for example, a motorized or non-motorized two-wheeler, a bicycle, a motorcycle, an electric scooter, an electric motorcycle, a car, a jet ski, a snowmobile, skis, and a snowboard, and the user wears a helmet 120. The vehicle 110 comprises a global positioning system 111, an inertial measurement system 112, an engine 113, a specific feature 114, a vehicle processing system 115b, and an alternative feature 114a. The global positioning system 111 is configured to measure global position data of the vehicle 110. The inertial measurement system 112 is configured to measure inertial data of the vehicle 110. The engine 113 is configured to provide power to the vehicle 110. The particular feature 114 is located at or near the cockpit area of ​​the vehicle 110 and is configured to be within an image frame of a digital image captured by a camera 122 located on a helmet 120. The alternate feature 114a is configured to function as a substitute for the particular feature 114.

[0065] As shown in FIG. 1, the helmet 120 comprises a battery 121, a camera 122, an inertial measurement system 123, an augmented reality helmet mounted display 124, a helmet processing system 115a, and an eye tracking system 126. The battery 121 is disposed on the helmet 120 and provides electrical energy to various electrical systems of the helmet 120. The camera 122 disposed on the helmet 120 is configured to capture images of the specific feature 114 and / or the alternative feature 114a of the vehicle 110. The inertial measurement system 123 is configured to measure inertial data of the helmet 120. The augmented reality helmet mounted display 124 is configured to display the data to a wearer of the helmet 120. The augmented reality helmet mounted display 124 is disposed, for example, fixedly or removably on the helmet 120. The eye tracking system 126 is configured to measure the line of sight of the wearer of the helmet 120. In one embodiment, the world reference coordinate system 250 is a coordinate system that associates a dedicated coordinate position to each position in the world.

[0066] In the embodiment shown in Fig. 1, the processing system 115 comprises a processing system 115a of the helmet 120 and a processing system 115b of the vehicle 110. The processing system 115 is configured to determine the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250 and display the data to the wearer of the helmet 120 on the augmented reality helmet-mounted display 124. The processing system 115 may further comprise a mobile device, such as a smartphone, which may additionally be combined to perform the calculations for determining the position and orientation of the helmet 120 and / or display the data to the wearer of the helmet 120. That is, different calculation steps are performed on different systems of the processing system 115, such as the processing system 115a, the processing system 115b, or the processing system of said mobile device.

[0067] In addition to the world reference coordinate system 250, Fig. 1 also shows, in schematic form, a helmet reference system 230 and a vehicle reference system 240. The helmet reference system 230 is centered on the helmet 120 and therefore moves with the helmet 120. Visual data statically positioned in the helmet reference system 230 moves statically with the movement of the helmet 120 from the perspective of the user of the helmet 120. The vehicle reference system 240 is centered on the vehicle 110 and therefore moves with the vehicle 110. Visual data statically positioned in the vehicle reference system 240 moves statically with the movement of the vehicle 110 from the perspective of the user of the helmet 120. Furthermore, data statically positioned in the world reference coordinate system 250 does not move relative to the vehicle 110 or the helmet 120 from the perspective of the user of the helmet 120. Of course, it is also possible for the displayed data to move within its associated reference system 230, 240, 250 or from one reference system 230, 240, 250 to another reference system 230, 240, 250.

[0068] Using the processing system 115, which may include the processing system 115a of the helmet 120 and / or the processing system 115b of the vehicle 110, the tracking system 100 determines the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250. In one embodiment, communication between the processing system 115a of the helmet 110 and the processing system 115b of the vehicle 120 is via wireless communication or communication module 150 (shown in FIG. 3). In one embodiment, collected data is transmitted from the helmet 120 to the processing system 115b, which is located only on the vehicle 110. In one embodiment, collected data is transmitted from the vehicle 110 to the processing system 115a, which is located only on the vehicle 110. Combinations of these are also contemplated.

[0069] The inertial measurement systems 112, 123 are configured to measure acceleration forces of the helmet 120 and the vehicle 110. In one embodiment, such forces include linear acceleration forces and / or rotational acceleration forces. Movement of the user's head in the vehicle 110 results in the same movement in the helmet 120 worn by the user. Thus, measuring inertial data at the helmet 120 allows a determination to be made about the relative movement of the user's head with respect to the vehicle 110.

[0070] In one embodiment, different acceleration sensors of the inertial measurement system 112 in the vehicle 110 can be placed at different locations on the vehicle 110 to increase measurement accuracy. In one embodiment, different acceleration sensors of the inertial measurement system 123 in the helmet 120 can be placed at different locations on the helmet 120 to increase measurement accuracy.

[0071] The inertial measurement system 116 of the vehicle 110, for example, first measures an acceleration in a particular direction, a rotational acceleration about a particular axis, and a deceleration in a particular direction. Thus, the data from the inertial measurement system 116 of the vehicle 110 can be used to determine the relative position and orientation of the vehicle 110. At the same time, the inertial measurement system 123 of the helmet 120 can measure, for example, a first acceleration in a particular direction, a rotational acceleration about a particular axis, and another acceleration in a further particular direction. Using the data from the inertial measurement system 123 of the helmet 120, the relative position and orientation of the helmet 120 can be determined. Furthermore, the combination of the measurements from the inertial measurement system 123 of the helmet 120 and the measurements from the inertial measurement system 116 of the vehicle 110 can determine the position and orientation of the helmet 120 with respect to the vehicle 110.

[0072] The combination of measurement data from the two inertial measurement systems 116, 123 by the processing system 115 advantageously makes it possible to determine the position of the helmet 120 relative to the vehicle 110, and in particular the position of the user's head within the helmet 120. This information makes it possible to display the data on a conventional data display system, for example a digital display on the motorcycle, in a position advantageous for reading by the user of the motorcycle, depending on the determined position and orientation of the user's head. Thus, the method makes it possible to display safety-related data in an advantageous manner that enhances the safety of the user of the vehicle 110.

[0073] In one embodiment, a camera 122 disposed on helmet 120 is aimed towards the cockpit of vehicle 110 and / or towards the rear of vehicle 110 while vehicle 110 is in use.

[0074] In one embodiment, the particular feature 114 of the vehicle 110 is a portion of the cockpit of the vehicle 110, such as an edge, a shape, a circular feature, a screw, multiple screws, or a feature in the cockpit, or a combination of the above. In another embodiment, the particular feature 114 of the vehicle 110 is a portion of the rear of the vehicle 110, such as an edge, a shape of a feature in the rear of the vehicle 110, or a combination of the above. In one embodiment, the particular feature 114 is a fiducial marker.

[0075] 2 differs from FIG. 1 in that the vehicle 110 further includes a camera 116, which is fixedly positioned at the rear of the vehicle 110 and pointed toward the rear of the helmet 120. In another embodiment, the camera 116 may be positioned at or near the cockpit area of ​​the vehicle 110 and pointed toward the front of the helmet 120. The camera 116 is configured to capture images from the helmet and / or images of a particular feature 114 located on the helmet 120 to determine the position and orientation of the helmet 120 relative to the vehicle 110.

[0076] In one embodiment, the fiducial marker as the identifying feature 114 is permanently placed on the vehicle 110 or the helmet 120 and is used permanently to identify the position and orientation of the helmet 120 relative to the vehicle 110. In another embodiment, the fiducial marker is temporarily placed on the vehicle 110 or the helmet 120 and is used only temporarily to identify the position and orientation of the helmet 120 relative to the vehicle 110. During the temporary use of the fiducial marker, a substitute feature 114a is identified in place of the fiducial marker, as described below with reference to FIG. 12. Placing fiducial markers on the helmet 120 and / or on the vehicle 110 within the field of view of one or more cameras 116, 122 is a simple and reliable method for obtaining the position and orientation of the helmet 120 relative to the vehicle 110 with high accuracy and for identifying advantageous substitute features 114a for the identifying feature 114.

[0077] FIG. 3 differs from FIG. 1 or FIG. 2 in the following respects: In FIG. 3, the tracking system 100 further comprises a communication module 150 arranged on the vehicle 120, which is configured to transmit and receive data / information to and from a remote server 160, which also has its own processing system 115c and forms part of the processing system 115 of the tracking system 100. In one embodiment, the remote server 160 is a cloud server. In one embodiment, the communication module 150 is connected to the processing system 115b of the vehicle 110 via a wire. In another embodiment, the communication module 150 is connected to the processing system 115b of the vehicle 110 and / or the processing system 115a of the helmet 120 via a wireless connection. Combinations are also possible. In one embodiment, the communication module 150 is further configured to transmit and receive data to and from the processing system 115a of the helmet 120. That is, the communication module 150 is configured to perform communication between the processing system 115a of the helmet 120, the processing system 115b of the vehicle 110, and / or a remote server 160 acting as a further processing system and / or a processing system 115c of a mobile device.

[0078] In one embodiment, wireless communication via communication module 150 occurs using mobile data networks, including Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA) and Long Term Evolution (LTE) networks, and / or short-range wireless communication interfaces using Wi-Fi networks, Bluetooth, and / or other wireless network types and standards.

[0079] In one embodiment, the calculations of the processing system 115 for determining the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250 are performed on the processing system 115a of the helmet 120, the processing system 115b of the vehicle 110 and / or the processing system 115c of the remote server 160. Additionally, the processing systems 115a, 115b, 115c, or a combination of the processing systems 115a, 115b, 115c, perform the necessary calculations to initialize the tracking system 100, to determine the display position of the data to be displayed on the augmented reality helmet mounted display, and / or to determine correction signals for drift of the inertial measurement systems 112, 123, and for any additional necessary steps.

[0080] Further types of data may be exchanged between the processing system 115a of the helmet 120, the processing system 115b of the vehicle and the processing system 115c of a remote server or possibly further mobile devices such as a smartphone, tablet or laptop. For example, data indicating road routes and data indicating local weather conditions may be uploaded to the processing system 115 from a remote device such as a mobile phone and / or a vehicle CAN-BUS unit. Furthermore, a human user may use a mobile device such as a smartphone, tablet or laptop to add data to be displayed at specific points on the road, for example virtual braking points or caution warnings before a sharp turn. Additionally, the processing system 115 may calculate an optimal trajectory based on the measurement data and the further types of data uploaded. Such further types of data may allow the augmented reality helmet mounted display to display further types of data to the user.

[0081] 4 is a flow chart illustrating a sequence of steps for performing the determination of the position and orientation of the helmet 120 relative to the vehicle 110. In the following paragraphs described with reference to FIG. 4, a possible sequence of steps is described which may be performed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110.

[0082] In step S1a, the inertial measurement system 123 of the helmet 120 measures inertial data of the helmet 120. In one embodiment, the inertial data of the helmet 120 includes linear acceleration data, rotational acceleration data, and / or additional data. The inertial measurement system 123 of the helmet 120 is fixedly arranged on the helmet 120, which means that all movements of the helmet 120 are measured by the inertial measurement system 123 of the helmet 120. In one embodiment, the inertial data measured by the inertial measurement system 123 of the helmet 120, which provides information about the relative movements of the helmet 120, is transmitted to the processing system 115a of the helmet 120.

[0083] In step S1b, which in one embodiment is performed simultaneously with step S1a, the inertial measurement system 112 of the vehicle 110 measures inertial data of the vehicle 110. In one embodiment, the inertial data of the vehicle 110 includes linear acceleration data, rotational acceleration data, and / or additional data. The inertial measurement system 112 of the vehicle 110 is fixedly located on the vehicle 110, which means that all movements of the vehicle 110 are measured by the inertial measurement system 116 of the vehicle 110. The inertial data measured by the inertial measurement system 116 of the vehicle 110 provides information about the relative movements of the vehicle 110. In one embodiment, this information is transmitted to the processing system 115b of the vehicle 110.

[0084] In step S2, the processing system 115, for example comprising the processing system 115a of the helmet 120 and / or the processing system 115b of the vehicle 110, performs a determination of the position and orientation of the helmet 120 with respect to the vehicle 110 using the inertial data of the helmet 120 and the inertial data of the vehicle 110. The combination of the inertial data of the helmet 120 with the inertial data of the vehicle 110 makes it possible to calculate the position and orientation of the helmet 120 with respect to the vehicle 110, in particular the relative movement of the helmet 120 with respect to the vehicle 110 while moving on the vehicle 110. This information advantageously makes it possible, for example, to display relevant driving information to the user of the vehicle 110 in an advantageous form. In particular, said driving information is arranged on the display depending on the position of the helmet 120 with respect to the vehicle 110. This improves the visibility of the information to the user and therefore the safety of the user in the vehicle 110.

[0085] Figure 5 is a flow chart showing a sequence of steps for performing the determination of the position and orientation of the helmet 120 relative to the vehicle 110. In the following paragraphs, written with reference to Figure 5, a possible sequence of steps is described, which are executed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110. In addition to steps S1a, S1b and S2, step S1c is shown in Figure 5.

[0086] In step S1c, which in one embodiment is performed simultaneously with steps S1a and S1b, one or more cameras 116, 122 capture digital images of the vehicle 110 and / or helmet 120. These images may consist of only a portion of the vehicle 110 or helmet 120. The image frames of the one or more cameras 116, 122 remain stationary with the helmet 120 or vehicle 110 while the helmet 120 is moving or while the vehicle 110 is moving because the one or more cameras 116, 122 are fixedly positioned with the helmet 120 or vehicle 110. The position of the vehicle 110 and / or helmet 120 in the image frames of the one or more cameras 116, 122 changes due to the movement of the helmet 120 or the movement of the vehicle 110. These images can then be used to determine the position of the helmet 120 relative to the vehicle 110. For example, the captured images are processed by an end-to-end position and orientation estimator, such as a neural network, that directly regresses the position and orientation of the helmet 120. The captured images may, for example, be directly processed by such a neural network, with the output of the neural network being the position and orientation of helmet 120 relative to vehicle 110. This calculation is performed using processing system 115 (115a, 115b and / or 115c).

[0087] The training data for the neural network includes, for example, images of the vehicle 110 and / or helmet 120 with position and orientation data of the helmet 120 relative to the vehicle 110 as ground truth. The training of the neural network is performed on a processing system 115, preferably a processing system 115c of a remote server. In one embodiment, updated neural networks or updates of the neural network are sent back to the processing systems 115a, 115b of the vehicle 110 and / or helmet 120 via the communication module 150. The running / operation of the neural network is performed on the processing system 115, preferably on the processing system 115a of the helmet 120 and / or on the processing system 115b of the vehicle 110 to reduce latency.

[0088] In step S2 of Fig. 5, the processing system 115 further uses the determined position and orientation of the helmet 120 relative to the vehicle 110 in the captured image obtained in step S1c to determine the position and orientation of the helmet 120 relative to the vehicle 110 and the world reference coordinate system 250. Combining data from the inertial measurement systems 112, 123 of the helmet 120 and the vehicle 110 with data acquired by the processing system 115 from the at least one digital image increases the accuracy of determining the position and orientation of the helmet 120 relative to the vehicle 110. The one or more cameras 116, 122 can improve the accuracy, especially in determining the movement of the helmet 120, because the measurements by the inertial measurement systems 112, 123 are based on the integration of their measurement data to determine the position and orientation, and the integration accumulates errors, known as drift, over time. In other words, drift of the inertial measurement system 112, 123 is the accumulation of small errors in the measurements of the inertial measurement system 112, 123, which makes the integrated inertial data increasingly more inaccurate. Since the one or more cameras 116, 122 directly provide orientation and position information, there is no integration or drift. Therefore, the one or more cameras 116, 122 can be used as a drift correction of the measurements by the inertial measurement system 112, 123. In this way, the one or more cameras 116, 122 advantageously allow for a more accurate determination of the orientation and position of the helmet 120 relative to the vehicle 110, which can increase the overall safety of the user in the vehicle 110. The one or more cameras 116, 122 can provide a "correction signal" to correct the drift of the inertial measurement system 112, 123. This correction signal allows for a correction of the measured inertial data, which increases the accuracy of the determination of the position and orientation of the helmet 120 relative to the vehicle 110.

[0089] Figure 6 is a flow chart illustrating a sequence of steps for performing the determination of the position and orientation of the helmet 120 relative to the vehicle 110. In the following paragraphs, written with reference to Figure 6, a possible sequence of steps is described, which are executed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110. In addition to steps S1a, S1b, S2 and S1c, step S2a is shown in Figure 6.

[0090] In step S2a, the processing system 115 performs the determination of the location of the specific feature 114 using the images captured by the one or more cameras 116, 122. The captured images in step S1c include the specific feature 114 of the helmet 120 and / or the vehicle 110. The determination of the location of the specific feature 114 is performed, for example, by determining which pixels of the one or more images represent the specific feature 114 and locating these pixels in a digital image coordinate system, which allows the location and orientation of the helmet 120 relative to the vehicle 110. The specific feature 114 is, for example, one or more screws in the cockpit of the vehicle 110. For example, if the distance between two screws is known, it is possible to calculate the position of the helmet 120 relative to the vehicle 110 by locating the pixels representing the screws in the digital image. This calculation requires only geometric algebra and is preferably performed on the processing system 115a of the helmet 120 and / or the processing system 115b of the vehicle 110.

[0091] In step S2, the determination of the position and orientation of the helmet 120 relative to the vehicle 110 is carried out by the processing system 115 using the determined positions of the characteristic features 114 in step S2a. According to this embodiment, it is particularly simple to calculate correction signals for drifts of the inertial measurement systems 112, 123.

[0092] Figure 7 is a flow chart showing a sequence of steps for performing the determination of the position and orientation of the helmet 120 relative to the vehicle 110. In the following paragraphs, written with reference to Figure 7, a possible sequence of steps is described, which are executed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110. In addition to steps S1a, S1b and S2, steps S1c, S2b and S2c are shown in Figure 7.

[0093] Step S1c is as described above with reference to FIG.

[0094] In step S2b, the alternative features 114a are identified using the digital image captured in step S1c. The digital image includes at least one specific feature 114 of the helmet 120 and / or vehicle 110 and additional features of the vehicle 110 and / or helmet 120, such as, for example, a circular tool, a screw or an edge, adjacent to the specific feature 114. The alternative features 114a are learned / identified, for example, by a neural network in the processing system 115 (115a, 115b, 115c). The digital image is processed by the neural network, which calculates a preferred alternative feature 114a from the possible additional features. In the training process, the neural network needs a ground truth to be able to learn the alternative features 114a. In one embodiment, the ground truth for the training process is an image that includes the specific feature 114 and / or the associated inertial measurement data of the inertial measurement systems 112, 123 and / or the associated position information of the cameras.

[0095] In step S2c, the alternative feature 114a is used in place of the particular feature 114, where the particular feature 114 may be removed from the vehicle 110 or the helmet 120.

[0096] In step S2, the determination of the position and orientation of the helmet 120 relative to the vehicle 110 is performed by the processing system 115 using the alternative features 114a. According to the present embodiment, the use of the alternative features 114a makes it possible to provide alternatives or combinations for the calculation of correction signals for drifts of the inertial measurement systems 112, 123.

[0097] Figure 8 is a flow chart illustrating a sequence of steps for performing the determination of the position and orientation of the helmet 120 relative to the vehicle 110. In the following paragraphs, written with reference to Figure 8, a possible sequence of steps is described, which are executed by the processing system 115 to determine the position and orientation of the helmet 120 relative to the vehicle 110. In addition to steps S1a, S1b and S2, step S1d is shown in Figure 8.

[0098] In step S1d, which in one embodiment is performed simultaneously with steps S1a, S1b and / or S1c, global position data of the vehicle 110 is measured by the global positioning system 111 of the vehicle 110. In a further embodiment, the global positioning data of the helmet 120 is measured by the global positioning system of the helmet 120. In one embodiment, the global positioning system 111 uses positioning information obtained via satellites. In another embodiment, the global positioning system 111 uses image-based information obtained, for example, from the vehicle CAN-BUS unit. The measured position data of the vehicle 110 and / or the helmet 120 allows to determine the position of the vehicle 110 and / or the helmet 120 relative to the world reference coordinate system 250.

[0099] In step S2, the processing system 115 further uses measured global position data from the global positioning system 111 of the vehicle 110 and / or helmet 120 to perform a determination of the position and orientation of the helmet 120 relative to the vehicle 110 and the world reference frame 250. The combination of the inertial data of the helmet 120, the inertial data of the vehicle 110, and the global position data allows the calculation of the position and orientation of the helmet 120 relative to the vehicle 110 while moving on the vehicle 110, and the calculation of the position and orientation of the helmet 120 relative to the world reference frame 250 while the vehicle 110 with the helmet 120 is moving within the world reference frame 250. This allows, for example, navigation data to be displayed in an advantageous manner to the user of the vehicle 110, increasing the safety of the use of the vehicle 110 as well as improving the user experience of the vehicle 110.

[0100] The global position data of the vehicle 110, in combination with digital images from one or more cameras 122, 116, and / or inertial data of the helmet 120 and the vehicle 110, can be used to improve the accuracy of determining the position and orientation of the helmet 120 relative to the vehicle 110, as well as to determine the position of the helmet 120 and the vehicle 110 relative to the world reference coordinate system 250. This allows navigation data to be displayed in an advantageous manner to a user of the vehicle 110, making use of the vehicle 110 safer and improving the experience of using the vehicle 110.

[0101] Fig. 9 is a flow chart showing a sequence of steps for displaying data on the augmented reality helmet mounted display 124 of the helmet 120. In the following paragraphs, written with reference to Fig. 9, a possible sequence of steps is described, which are executed by the processing system 115 in order to display data on the augmented reality helmet mounted display 124. For determining the position and orientation of the helmet 120 (step S2), a combination of steps S1a, S1b, S1d, S1c, S2a, S2b and / or S2c may be used, as described above (steps S2a, S2b, S2c are not shown in Fig. 9).

[0102] In step S3, using the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250 determined in step S2, the processing system 115 determines a display location of the data to be displayed on the augmented reality helmet mounted display 124 located on the helmet 120. In one embodiment, the augmented reality helmet mounted display 124 is implemented in or forms the visor of the helmet 120. In one embodiment, the displayed data includes navigation information, driving information such as speed and engaged gear, road information such as ideal line and expected obstacles, or warning information such as speed too fast on the next road section. The display location is the position for displaying the data on the augmented reality helmet mounted display 124. For example, conventional heads up displays in the automotive industry project data onto the lower part of the windshield of the car. The display location of the data displayed on the augmented reality helmet mounted display 124 may vary across the visor or across the field of view of the user of the vehicle 110.

[0103] In step S4, the processing system 115 uses the display position determined in step S3 to display the data on the augmented reality helmet mounted display 124 to the wearer of the helmet 120. The position of the displayed data depends on the determined position and orientation of the helmet 120. In other words, different data can be displayed on the augmented reality helmet mounted display 124 depending on the position and orientation of the helmet 120. For example, data can be placed in the helmet reference system 230, which appears to the user to be moving exactly like the helmet 120, and remains stationary in the augmented reality helmet mounted display 124. Data placed 20 cm in front of the user's head in the helmet reference system 230 will always appear 20 cm in front of the user's head, even if the orientation and position of the helmet 120 changes due to the user's movements. Furthermore, data can be placed in the vehicle reference system 240, which is centered on the vehicle 110. Data centered on the vehicle reference system 240 and displayed on the augmented reality helmet mounted display 124 remains stationary with respect to the vehicle 110. If the position or orientation of the helmet 120 changes, the position of this data displayed on the augmented reality display also changes. For example, if the vehicle speed is associated with the vehicle reference system 240 and is always located 3 meters in front of the vehicle 110, it will always remain 3 meters in front of the vehicle 110 and may disappear from the user's field of view due to a given movement of the helmet 120. Furthermore, the data may be located in the world reference coordinate system 250. Data centered in the world reference coordinate system 250 may remain stationary relative to the world. If the position and / or orientation of the helmet 120 and / or the vehicle 110 changes, the position of this data displayed on the augmented reality helmet mounted display 124 also changes. For example, if navigation information such as an arrow is located in the world reference coordinate system 250, it will always remain in a particular position in the world reference coordinate system 250 and may disappear from the user's field of view due to a given movement of the head / helmet 120.

[0104] Information about the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250 enables different data to be displayed on the augmented reality helmet-mounted display 124 according to the associated reference systems 230, 240, 250 and according to the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250.

[0105] In one embodiment, the determination of the position and orientation of the helmet 120 is continuously updated with data from the inertial measurement system 123 of the helmet 120, with data from the inertial measurement system 112 of the vehicle 110, with digital images of at least one particular feature 114 of the helmet 120 and / or the vehicle 110, and / or with global position data of the helmet and / or the vehicle. The continuous updates allow the position and orientation of the helmet 120 to be determined over time, which allows the position / adaptation of the displayed data to the user of the vehicle 110 in response to changes in the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference frame 250. This improves safety for the user of the vehicle 110.

[0106] Fig. 10 is a flow chart showing a sequence of steps for displaying data on the augmented reality on-helmet display 124 of the helmet 120. In the following paragraphs, written with reference to Fig. 10, a possible sequence of steps is described, which are executed by the processing system 115 for displaying data on the augmented reality on-helmet display 124. For determining the position and orientation of the helmet 120 (step S2), a combination of steps S1a, S1b, S1d, S1c, S2a, S2b and / or S2c may be used, as described above (steps S2a, S2b, S2c are not shown in Fig. 10).

[0107] In step S3a, the processing system uses the eye tracking system 126 to determine a gaze vector of at least one eye, and preferably both eyes, of a wearer of the helmet 120. The eye tracking system 126 is a system for determining a gaze vector of a user of the eye tracking system 126. The gaze vector of a wearer of the helmet 120 determines the direction in which the wearer is looking. Using both eyes to determine the gaze vector increases its accuracy.

[0108] In step S3, the processing system 115 uses the identified line of sight vector to determine a position of the display data as viewed by the wearer on the augmented reality helmet mounted display 124. That is, the display position on the augmented reality display 124 is further determined according to the line of sight vector.

[0109] In step S4, processing system 115 further uses the gaze vector of step S3a to display data to the wearer of helmet 120 on augmented reality helmet mounted display 124. For example, the displayed data, such as vehicle speed, moves within augmented reality helmet mounted display 124 with the gaze movement of the user of vehicle 110.

[0110] Additionally, the processing system 115 can use the identified gaze vector to determine the type of data to be displayed on the augmented reality display 124 for the wearer of the helmet 120. For example, if the gaze vector determines that the wearer of the helmet 120 is not looking at the road for a certain period of time, the processing system 115 can warn the user of the vehicle 110 by displaying warning information for the wearer of the helmet 120 on the augmented reality display. The gaze tracking system 126 can determine the location and / or type of data to be displayed on the augmented reality helmet mounted display 124, which can help display information necessary for safe use of the vehicle 110. Additionally, if the wearer of the helmet 120 is not looking at an obstacle on the road, the processing system 115 can warn the user of the vehicle 110 by displaying a warning on the augmented reality helmet mounted display 124.

[0111] Fig. 11 is a flow chart showing a sequence of steps for displaying data on the augmented reality helmet mounted display 124 of the helmet 120. In this embodiment, the determination of the position and orientation of the helmet 120 is performed by the processing system 115 (115a, 115b and / or 115c) using a filter based on data integration according to Kalman's equation as an estimator. The inputs of the processing system 115 are the inertial data of the inertial measurement system 112 of the vehicle 110, the inertial data of the inertial measurement system 123 of the helmet 120, the digital images captured by the camera 122 of the helmet 120 and the global position data of the global positioning system 111. In the following paragraphs described with reference to Fig. 11, a possible sequence of steps is described which are executed by the processing system 115 to determine the position and orientation of the helmet 120 and to display data on the augmented reality helmet mounted display 124.

[0112] In step S11, the processing system 115 uses at least one captured image by the helmet camera 122 to determine the position of the helmet camera 122 relative to the vehicle 110 or the vehicle reference system 240. In another embodiment, the processing system 115 uses at least one captured image by the vehicle camera 116 to determine the position of the helmet 120 relative to the vehicle 110. A combination of both is also contemplated.

[0113] In step S12, the processing system 115 uses inertial data of the vehicle 110 measured by the inertial measurement system 112 of the vehicle 120 and global position data measured by the global positioning system 111 to determine the position and orientation of the vehicle 110 relative to the world reference coordinate system 250.

[0114] In step S13, the processing system 115 initializes the tracking system 100 by determining an initial position of the helmet 120 relative to the vehicle 110 and the world reference coordinate system 250. The initial position of the helmet 120 is determined using inertial measurement data of the inertial measurement system 123 of the helmet 120, inertial measurement data of the inertial measurement system 112 of the vehicle 110, the position of the helmet camera 122 determined in step S11, and the position of the vehicle 110 relative to the world reference coordinate system 250 identified in step S12.

[0115] The initial position of the helmet 120 is, for example, the position of the helmet 120 on the vehicle 110 before the vehicle 110 is moved by the user. In another embodiment, the initial position of the helmet 120 is the initial position of the helmet 120 when the helmet 120 enters the field of view of the camera 116 arranged on the vehicle 110, or the initial position of the helmet 120 when a particular feature 114 of the vehicle 110 enters the field of view of the camera 122 arranged on the helmet 120. Starting from the initial position of the helmet 120 relative to the vehicle 110 and / or the world reference frame 250, the new position of the helmet 120 relative to the vehicle 120 and / or the world reference frame 250 due to the continuous movement of the vehicle 110 and the helmet 120 can be updated by data from the inertial measurement systems 112, 123, images captured by one or more cameras 116, 122, and global position data. Thus, determining the initial position allows for increased accuracy and speed in determining the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference frame 250.

[0116] In step S14a, the processing system 115 predicts the future position of the helmet 120 using a Kalman filter inspired equation. The input data for this step are the inertial measurement data of the vehicle 110 measured by the inertial measurement system 112 of the vehicle 110, the inertial measurement data of the helmet 120 measured by the inertial measurement system 123 of the helmet 120, and the initial position of the helmet 120 determined in step S13. A further input in this step is the updated position of the helmet 120 in step S14b.

[0117] In step S14b, the processing system 115 updates the position of the helmet 120 using a Kalman filter inspired equation. The input data for this step are the position of the helmet camera 122 determined in step S11, the position of the vehicle 110 in the world reference frame 250 determined in step S12, and the predicted position of the helmet 120 in step S14a.

[0118] Steps S14a and S14b are performed repeatedly. The predicted position of helmet 120 is used as an input for updating the position of helmet 120 at each new time step. Furthermore, the updated position of helmet 120 is used as an input for predicting the position of helmet 120 at each new time step.

[0119] In step S15, the processing system 115 determines the position of the helmet 120 relative to the vehicle 110 and the world reference coordinate system 250.

[0120] In step S16, the processing system 115 displays data on the augmented reality helmet mounted display 124 to the wearer of the helmet 120, who is the operator of the vehicle 110, based on the position of the helmet 120 determined in step S15.

[0121] Figure 12 is a flow chart illustrating a sequence of steps for identifying, in one embodiment, an alternative feature 114a for a particular feature 114. In the following paragraphs described with reference to Figure 12, a possible sequence of steps for identifying an alternative feature 114a for a particular feature 114 is described.

[0122] In step S20, a characteristic feature 114 is placed on the vehicle 110. In one embodiment, the characteristic feature 114 is a fiducial marker. The characteristic feature 114 is placed, for example, during installation of the tracking system 100 on the vehicle 110 and helmet 120.

[0123] In step S21, the camera 122 of the helmet 120 captures a digital image including the identifying feature 114 located in step S20 and additional features of the vehicle 110. The digital image shows, for example, the cockpit of the vehicle 110 having a dedicated identifying feature 114. In addition, the digital image shows additional features of the cockpit, for example, a circular fixture, one or more screws or edges adjacent to the identifying feature 114.

[0124] In step S22, the processing systems 115a, 115b transmit the captured images to the processing system 115c of the remote server 160 for use in identifying alternative features 114a using the images. Communication between the vehicle 110 and the remote server 160 is performed, for example, via the communication module 150.

[0125] In step S23, the processing system 115c of the remote server 160 identifies alternative features 114a using the at least one image. In one embodiment, the identification of alternative features 114a of the vehicle 110 is performed at the remote server 160 using a neural network. The images are processed by the neural network and the alternative features 114a are identified thereby. In another embodiment, the alternative features 114a are identified at the remote server 160 using feature descriptors such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF) or Oriented FAST and rotated BRIEF (ORB). In other embodiments, other machine learning approaches are used.

[0126] In step S24, the processing system 115a of the helmet 120 and / or the processing system 115b of the vehicle 110 receives the identified alternative features 114a from the remote server 160. In other words, information of the alternative features 114a identified by the neural network or another feature descriptor is transmitted from the remote server 160 to the processing system 115a of the helmet 120 or the processing system 115b of the vehicle 110, for example via the communication module 150.

[0127] In step S25, the processing system 115 uses the alternative feature 114a as the specific feature 114 to identify the position and orientation of the helmet 120 relative to the vehicle 110 and / or the world reference coordinate system 250. In one embodiment, the specific feature 114 is removed from the vehicle 110 after the alternative feature 114a has been identified and used by the processing system 115.

[0128] In one embodiment, the distinctive feature 114 is placed on the helmet 120 and the image is captured by the camera 116 of the vehicle 110. Combinations of these are also contemplated. In one embodiment, the distinctive feature 114 is a fiducial marker.

[0129] A user of the vehicle 110 places a dedicated identification feature 114, such as a marker, on the vehicle when the system 100 is installed. The user then uses the vehicle 110 for a predetermined time. In one embodiment, the predetermined time is between 5 hours and 100 hours of use, preferably between 5 hours and 20 hours of use. During this specified time, a digital image including the identification feature 114 and the additional feature is sent to the remote server 160. The processing system 115c of the remote server 160 processes the image via a neural network to identify a replacement feature 114a. The information of the replacement feature 114a is then sent back to the processing system 115 of the vehicle 110, which uses the replacement feature 114a in place of the old identification feature 114. The old identification feature 114 may be removed from the vehicle 110 by the user as it is no longer needed.

[0130] It should be noted that although the order of steps is set forth in a particular order herein, one of ordinary skill in the art would understand that the order of at least some of the steps may be varied without departing from the scope of the present disclosure.

[0131] FIG. 13 shows a schematic diagram of a vehicle 110 with a user and a tracking system 100 according to a fourth exemplary embodiment. This embodiment differs from the embodiment shown in FIGS. 1-3 in that eyewear 120' replaces the helmet 120 and eyewear reference system 230' replaces the helmet reference system 230. It should be understood that all combinations shown for the tracking system 100 including the helmet 120 shown and described in FIGS. 1-3 can be applied to the tracking system 100 including the eyewear 120' instead of the helmet 120 as well. It should also be understood that all flow charts of the method for determining the position and orientation of the helmet 120 shown and described in FIGS. 4-12 can be applied to the method for determining the position and orientation of the eyewear 120' instead of the helmet 120 as well. Furthermore, it should be understood that all advantages shown and described with respect to the helmet 120 can be applied to the eyewear 120' as well.

[0132] The vehicle 110 is a two-wheeled motor vehicle in this embodiment, but in another embodiment is a four-wheeled motor vehicle such as a car, and other vehicles such as trucks and bicycles are also contemplated. The vehicle 110 includes a global positioning system 111, an inertial measurement system 112, an engine 113, a specific feature 114, a vehicle processing system 115b, and an alternative feature 114a. The global positioning system 111 is configured to measure global position data of the vehicle 110. The inertial measurement system 112 is configured to measure inertial data of the vehicle 110. The engine 113 is configured to provide power to the vehicle 110. The specific feature 114 is located in or near the cockpit area of ​​the vehicle 110 and is configured to be within an image frame of a digital image captured by a camera 122 located in the eyewear 120'. The alternative feature 114a is configured to function as a substitute for the specific feature 114.

[0133] As shown in FIG. 13, the eyewear 120′ includes a battery 121, a camera 122, an inertial measurement system 123, an augmented reality eyewear-mounted display 124, an eyewear processing system 115a, and an eye-gaze tracking system 126. The battery 121 is disposed in the eyewear 120′ and supplies electrical energy to various electrical systems of the eyewear 120′. The camera 122 disposed in the eyewear 120′ is configured to capture images of the specific feature 114 and / or the alternative feature 114a of the vehicle 110. The inertial measurement system 123 is configured to measure inertial data of the eyewear 120′. The augmented reality eyewear-mounted display 124 is configured to display data to a wearer of the eyewear 120′. The augmented reality eyewear-mounted display 124 is, for example, fixedly or removably disposed in the eyewear 120′. The eye-gaze tracking system 126 is, for example, configured to measure a gaze vector of a wearer of the eyewear 120′.

[0134] In one embodiment, the world reference coordinate system 250 is a coordinate system that associates each location in the world with a unique coordinate position. In addition to the world reference coordinate system 250, Figure 13 also shows, in schematic form, an eyewear reference system 230' and a vehicle reference system 240.

[0135] In one embodiment, the camera 122, the inertial measurement system 123, the augmented reality display 124, the eye tracking system 126, and the processing system 115a of the eyewear 120', which is part of the processing system 115 (see FIG. 13), are provided in a single eyewear unit, such as a particular pair of glasses.

Claims

1. 1. A method for determining a position and orientation of a helmet (120) while in motion on a vehicle (110), comprising: measuring (S1a) inertial data of the helmet (120) while the helmet (120) is in motion by an inertial measurement system (123) of the helmet (120); measuring (S1b) inertial data of the vehicle (110) while the vehicle (110) is in motion by an inertial measurement system (112) of the vehicle (110); determining (S2) a position and orientation of the helmet (120) relative to the vehicle (110) using the inertial data of the helmet (120) and the inertial data of the vehicle (110) by a processing system (115); The method according to claim 1, further comprising:

2. taking (S1c) at least one digital image of the vehicle (110) using a camera (122) arranged on the helmet (120) and / or taking (S1c) at least one digital image of the helmet (120) using a camera (116) arranged on the vehicle (110); determining (S2) a position and orientation of the helmet (120) relative to the vehicle (110) by the processing system (115) further using the digital image(s) of the vehicle (110) and / or the helmet (120); The method of claim 1 further comprising:

3. 3. The method of claim 2, wherein the at least one captured digital image captured using the camera (122) disposed on the helmet (120) includes at least one particular feature (114) of the vehicle (110) and / or the at least one digital image captured using the camera (116) disposed on the vehicle (110) includes at least one particular feature (114) of the helmet (120), and the processing system (115) further uses one or more of the digital images including the at least one particular feature (114) of the vehicle (110) and / or the helmet (120) to determine a position and orientation of the helmet (120) relative to the vehicle (110).

4. 4. The method of claim 3, further comprising identifying, by the processing system (115), alternative features (114a) for the particular feature (114) of the vehicle (110) and / or the helmet (120) by performing the steps of: capturing (S1c) a digital image of the vehicle (110) and / or the helmet (120) including the at least one particular feature (114) and additional features using the camera (122) located on the helmet (120) or the vehicle (110); Identifying (S2b) the alternative features (114a) from the image by the processing system (115); and using (S2c) said alternative feature (114a) as said specific feature (114) to identify the position and orientation of said helmet (120) relative to said vehicle (110);

5. - measuring (S1d) global position data of said vehicle (110) and / or said helmet (120) relative to a world reference coordinate system (250) by a global positioning system (111) of said vehicle (110) and / or said helmet (120); determining (S2) by the processing system (115) a position and orientation of the vehicle (110) and / or the helmet (120) relative to the world reference coordinate system (250) using the measured global position data of the vehicle (110) and / or the helmet (120) further; 2. The method of any one of the preceding claims, further comprising:

6. 6. The method according to claim 2, characterized in that the initial position of the helmet (120) is determined by the following steps: taking (S1c) at least one digital image of the vehicle (110) using the camera (122) located on the helmet (120) and / or taking (S1c) at least one digital image of the helmet (120) using the camera (116) located on the vehicle (110); and determining (S2) the initial position of the helmet (120) relative to the vehicle (110) by the processing system (115) using one or more of the digital images of the vehicle (110) and / or the helmet (120);

7. 10. The method according to any one of the preceding claims, characterized in that the determination of the position and orientation of the helmet (120) relative to the vehicle (110) and / or the world reference coordinate system (250) is performed by the processing system (115) by data synthesis based on an estimator.

8. determining (S3) by the processing system (115) a display position of data to be displayed on an augmented reality display (124) arranged on the helmet (120) using the position and orientation of the helmet (120) relative to the vehicle (110) and / or the world reference coordinate system (250); displaying (S4) said data to a wearer of said helmet (120) on said augmented reality display (124) by said processing system (115) using said display location; 2. The method of any one of the preceding claims, further comprising:

9. determining (S3a) a gaze vector of at least one eye, and preferably both eyes, of a wearer of the helmet (120) by means of an eye-tracking system (126) arranged in the helmet (120); determining (S3) by the processing system (115) a display position of the data and / or a type of the data on the augmented reality display (124) viewed by the wearer using the identified gaze vector; The method of claim 8 further comprising:

10. 10. The method according to any one of the preceding claims, characterized in that the processing system (115) comprises a processing system (115a) of the helmet (120), a processing system (115b) of the vehicle (110), and / or a processing system (115c) of a remote server (160).

11. 11. A computer program product comprising a non-transitory computer readable medium having stored thereon computer program code configured to control a processing system (115) of a tracking system (100) such that the tracking system (100) performs the steps of the method according to any one of claims 1 to 10.

12. A tracking system (100) for determining a position and orientation of a helmet (120) while moving on a vehicle (110), comprising: an inertial measurement system (123) disposed on the helmet (120); an inertial measurement system (112) disposed on the vehicle (110); and a processing system (115) configured to perform the following steps: measuring (S1a) inertial data of the helmet (120) while the helmet (120) is in motion using the inertial measurement system (123) of the helmet (120); measuring (S1b) inertial data of the vehicle (110) while the vehicle (110) is in motion using the inertial measurement system (112) of the vehicle (110); and determining (S2) a position and orientation of the helmet (120) relative to the vehicle (110) using the inertial data of the helmet (120) and the inertial data of the vehicle (110);

13. 13. The tracking system (100) of claim 12, further comprising a camera (122) disposed on the helmet (120) and / or a camera (116) disposed on the vehicle (110), and wherein the processing system (115) is further configured to perform the steps of: (S1c) taking at least one digital image of the vehicle (110) using the camera (122) located on the helmet (120) and / or taking at least one digital image of the helmet using the camera (116) located on the vehicle (110); and (S2) further using the digital image(s) of the vehicle (110) and / or the helmet (120) to determine a position and orientation of the helmet (120) relative to the vehicle (110).

14. 14. The tracking system (100) of claim 13, wherein the at least one captured digital image taken using the camera (122) arranged on the helmet (120) includes at least one particular feature (114) of the vehicle (110) and / or the at least one captured digital image taken by the camera (116) arranged on the vehicle (110) includes at least one particular feature (114) of the helmet (120), and the processing system (115) is configured to further use one or more of the digital images including the at least one particular feature (114) of the vehicle (110) and / or the helmet (120) to determine the position and orientation of the helmet (120) relative to the vehicle (110).

15. 15. The tracking system (100) of any one of claims 12 to 14, further comprising a global positioning system (111), wherein the processing system (115) is further configured to perform the steps of: measuring (S1d) global position data of the vehicle (110) and / or the helmet (120) relative to a world reference coordinate system (250) using the global positioning system (111); and (S2) further using the measured global position data of the vehicle (110) and / or the helmet (120) to determine the position and orientation of the vehicle (110) and / or the helmet (120) relative to the world reference coordinate system (250).

16. 16. The tracking system (100) of any one of claims 12 to 15, characterized in that the processing system (115) is configured to determine the position and orientation of the helmet (120) relative to the vehicle (110) and / or the world reference coordinate system (250) by data integration based on an estimator.

17. 17. The tracking system (100) of any one of claims 12 to 16, further comprising an augmented reality display (124) disposed on the helmet (120), and wherein the processing system (115) is further configured to perform the steps of: Using the position and orientation of the helmet (120) relative to the vehicle (110) and / or the world reference coordinate system (250) to determine (S3) a display position for data to be displayed on the augmented reality display (124); and Displaying (S4) the data to a wearer of the helmet (120) on the augmented reality display (124) using the display location.

18. 20. The tracking system (100) of claim 17, further comprising an eye-tracking system (126) disposed on the helmet (120), the processing system (115) being further configured to perform the steps of: determining (S3a) a gaze vector of at least one eye, and preferably both eyes, of a wearer of the helmet (120) using said gaze tracking system (126); and Using the identified gaze vector, a step (S3) is performed to determine a display position of the data and / or a type of the data on the augmented reality display (124) viewed by the wearer.

Images