Mobile calibration of displays for smart helmets

Abstract

Mobile calibration of a display for a smart helmet is provided. A smart helmet includes a heads-up display (HUD) configured to output a graphical image within a virtual field of view on a face shield of the smart helmet. A transceiver is configured to communicate with a mobile device of a user. A processor is programmed to receive, via the transceiver, calibration data from the mobile device related to one or more captured images from a camera on the mobile device, and alter the virtual field of view of the HUD based on the calibration data. This allows a user to calibrate the HUD of his / her smart helmet based on images received from the user's mobile device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to smart helmets or intelligent helmets, such as those utilized while riding two-wheeled vehicles, such as motorcycles and off-road bicycles, three-wheeled vehicles, or four-wheeled vehicles, such as all-terrain vehicles. BACKGROUND

[0002] Riders of powered two-wheeled vehicles (PTWs) can utilize smart helmets. The smart helmets can utilize a heads-up display to display information on a transparent visor or shield of the helmet. The information is superimposed onto the field of view of the real world and appears in focus at an appropriate distance so that the rider can safely view the digital information on the visor while safely maintaining focus on the road ahead. SUMMARY

[0003] In one embodiment, a smart helmet includes a heads-up display (HUD) configured to output a graphical image within a virtual field of view on a visor of the smart helmet, a transceiver configured to communicate with a mobile device of a user, and a processor in communication with the transceiver and the HUD. The processor is programmed to receive calibration data from the mobile device via the transceiver related to one or more captured images from a camera on the mobile device and alter the virtual field of view of the HUD based on the calibration data.

[0004] In another embodiment, a system for calibrating a heads-up display of a smart helmet includes a mobile device having a camera configured to capture an image of a user’s face, a smart helmet having a heads-up display (HUD) configured to display a virtual image within a virtual field of view on a visor of the smart helmet, and one or more processors. The one or more processors are configured to determine one or more facial characteristics of the captured image of the user’s face, determine an offset value for offsetting the virtual field of view based on the one or more facial characteristics, and calibrate the virtual field of view based on the offset value to adjust a visibility of the virtual image displayed by the HUD.

[0005] In yet another embodiment, one or more non-transitory computer-readable media including executable instructions are provided, where the instructions, in response to execution by one or more processors, cause the one or more processors to capture one or more digital images of a user’s face via a camera of a mobile device, determine a facial feature of the face based on the captured image, transmit a signal from the mobile device to a smart helmet, where the signal includes data related to the facial feature of the face, receive the signal at the smart helmet, and calibrate a virtual field of view of a heads-up display of the smart helmet based on the received signal. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1This is an example of a system design that includes smart helmets and saddle-ride vehicles such as motorcycles.

[0007] Figure 2 The diagram illustrates a block diagram of a calibration system according to one embodiment, which is performed to render virtual content in an optically transparent display.

[0008] Figure 3 The illustration shows a schematic representation of one or more cameras in a mobile device according to one embodiment, which captures images of a user's face for calibration of an optical perspective display.

[0009] Figure 4 The illustration shows a block diagram of a mobile device for calibrating a reference facial structure model and calibrating calibration parameters of an optical perspective display, according to one embodiment.

[0010] Figure 5 The illustration shows a flowchart of a process performed via a communication link between a smart helmet and a mobile device according to one embodiment. Detailed Implementation

[0011] This document describes embodiments of the present disclosure. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various forms and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but only as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any one figure may be combined with features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of illustrated features provides representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desired for a particular application or implementation.

[0012] This disclosure refers to helmets and saddle-mounted vehicles. It should be understood that “saddle-mounted vehicle” generally refers to a motorcycle, but can include any type of motorized vehicle in which the rider is typically seated on a saddle and typically wears a helmet due to the lack of a protective cabin for the rider. Besides motorcycles, this can also include other motorized two-wheeled vehicles (PTWs), such as off-road bicycles, scooters, etc. This can also include motorized three-wheeled vehicles, or motorized four-wheeled vehicles (such as all-terrain vehicles (ATVs), etc.). Unless otherwise stated, any specific reference to a motorcycle also applies to any other saddle-mounted vehicle.

[0013] Smart helmets, or "intelligent helmets," for saddle-mounted vehicles typically include a head-up display (HUD), also known as an optical see-through display, which can be positioned on, for example, the helmet's visor. The HUD can display augmented reality (AR), graphic images including vehicle data, and other information displayed away from the driver, allowing the driver to safely view information while properly maneuvering the vehicle. The visual display source in the HUD needs to be properly positioned for the driver to see correctly. Different drivers have different head sizes and varying distances between their eyes, which can affect the ability to properly view information on the HUD among different drivers. However, due to manufacturing limitations, universal or standard designs suitable for most (but not all) users are designed for production.

[0014] Therefore, according to the embodiments disclosed herein, a system is disclosed that utilizes a camera on a mobile device (e.g., a smartphone) to capture images of a cyclist, which can then be analyzed for calibrating a HUD system in a smart helmet. For example, based on communication with the mobile device capturing the user's images, the pre-programmed and standard generic design of the smart helmet can be calibrated to better accommodate the user's facial features. Although a smart helmet may be equipped with a camera facing the user's face for the purpose of calibrating the HUD system, the helmet camera may be too close to the user's face for proper calibration. Having the camera too close to the user's face can distort the image, for example, elongating the appearance of the user's face. This can give inappropriate measurements of the user's facial size and contours, including the distance between the user's eyes, which may inappropriately affect the calibration process and overall functionality of the HUD system.

[0015] Figure 1 This is an example of a system 100 including a smart helmet 101 and a saddle-mounted riding vehicle 103. The smart helmet 101 and the saddle-mounted riding vehicle 103 may include various components and sensors that interact with each other. The smart helmet 101 may focus on collecting data related to the rider's body and head movements. In one example, the smart helmet 101 may include a camera 102. The camera 102 of the helmet 101 may include a main sensor for position and orientation recognition in the moving vehicle. Therefore, the camera 102 may be oriented towards the outside of the helmet 101 to track other vehicles and objects around the rider. In another example, attached to or replacing the camera 102, the helmet 101 may include radar or LIDAR sensors.

[0016] Helmet 101 may also include a helmet inertial measurement unit (IMU) 104. The helmet IMU 104 can be used to track the high-dynamic movements of the rider's head. Therefore, the helmet IMU 104 can be used to track the direction the rider is facing or the direction the rider is looking. Additionally, the helmet IMU 104 can be used to track sudden movements and other potential issues. The IMU may include one or more motion sensors.

[0017] An IMU (Inertial Measurement Unit) can use a combination of accelerometers and gyroscopes, and sometimes magnetometers, to measure and report specific forces, angular rates, and sometimes magnetic fields of a subject. IMUs are commonly used for control: aircraft, including unmanned aerial vehicles (UAVs) among others; and spacecraft, including satellites and landers. IMUs can be used as components of inertial navigation systems used in various transportation systems. Data collected from the IMU's sensors allows computers to track motor positions.

[0018] An IMU can detect the current rate of acceleration using one or more accelerometers and changes in rotational properties such as pitch, roll, and yaw using one or more gyroscopes. An IMU may also include a magnetometer, which can be used to help calibrate for orientation drift. Inertial navigation systems contain IMUs with angular and linear accelerometers (for position changes); some IMUs include gyroscopes (for maintaining an absolute angular reference). An angular rate meter measures how a vehicle can rotate in space. At least one sensor can be present for each of the three axes: pitch (nose up and down), yaw (nose left and right), and roll (clockwise or counterclockwise from the cockpit). A linear accelerometer measures the vehicle's non-gravitational acceleration. Since it can move along three axes (up & down, left & right, forward & backward), a linear accelerometer can be present for each axis. These three gyroscopes are typically placed in a similar orthogonal pattern to measure rotational position with reference to an arbitrarily chosen coordinate system. A computer can continuously calculate the vehicle's current position. For each of the six degrees of freedom (x, y, z, and Ox, Oy, and Oz), it can integrate the sensed acceleration over time, along with an estimate of gravity, to calculate the current velocity. It can also integrate the velocity to calculate the current position. Some measurements provided by the IMU are as follows:

[0019]

[0020] in These are the raw measurements from the IMU within the main IMU framework. , It is the expected correct acceleration and gyroscope rate measurement. , It is the bias offset of the accelerometer and gyroscope. , It is noise from the accelerometer and gyroscope.

[0021] Helmet 101 may also include an eye tracker 106. The eye tracker 106 can be used to determine the direction the rider of the saddle-mounted vehicle 103 is looking. The eye tracker 106 can also be used to identify drowsy and fatigued riders or those experiencing PTW (post-traumatic stress disorder). The eye tracker 106 can identify different parts of the eye (e.g., the retina, cornea, etc.) to determine where the user is looking. The eye tracker 106 may include a camera or other sensors to help track the rider's eye movements.

[0022] Helmet 101 may also include a helmet processor 108. Helmet processor 108 can be used for sensor fusion of data collected by various cameras and sensors on both the saddle-mounted vehicle 103 and helmet 101. In other embodiments, the helmet may include one or more transceivers for short-range and long-range communication. Short-range communication of the helmet may include communication with other vehicles and objects in or near the saddle-mounted vehicle 103. In another embodiment, long-range communication may include communication with non-airborne servers, the Internet, the “cloud,” cellular communications, etc. Helmet 101 and saddle-mounted vehicle 103 may communicate with each other using wireless protocols implemented by transceivers located on both helmet 101 and saddle-mounted vehicle 103. Such protocols may include Bluetooth, Wi-Fi, etc.

[0023] Helmet 101 also includes a head-up display (HUD) 110, also known as an optical see-through display, for outputting graphic images onto a transparent visor of helmet 101, for example. Various types of HUD systems can be utilized. In one embodiment, HUD 110 is a projection-based system having a projector unit, a combiner, and a video generation computer. The projector unit may be an optical collimator setup with a convex lens or concave mirror, having a cathode ray tube, light-emitting diode (LED) display, or liquid crystal display (LCD) at its focal point. This design produces an image in which the light is collimated and the focal point is perceived as being at infinity. The combiner may be an angled plate of glass positioned directly in front of the viewer, which redirects the projected image onto the transparent display, allowing the user to view the field of view and project an image at infinity.

[0024] In another embodiment, HUD 110 is a waveguide-based system where optical waveguides generate images directly in the combiner, rather than using a projector. This embodiment is better suited to the small package constraints within helmet 101, while also reducing the overall mass of the HUD compared to projection-based systems. In this embodiment, surface gratings are provided on the screen (e.g., visor) of the helmet itself. For example, the screen can be made of glass or plastic. A microprojector can project an image directly onto the screen, with the exit pupil of the microprojector placed on the surface of the screen. Gratings within the screen deflect the light, causing it to be trapped inside the screen due to total internal reflection. One or two additional gratings can then be used to progressively extract the light, creating a displaced copy of the exit pupil. The resulting image is visible to the user, displayed at a focal point of infinite length, allowing the user to simultaneously view the surrounding environment and augmented reality or displayed data.

[0025] Other embodiments of the HUD 110 can be utilized. Among other embodiments, these embodiments include, but are not limited to: generating images on a screen that serves as a fluorescent screen using a cathode ray tube (CRT); displaying images using a solid-state light source (e.g., an LED) modulated by the screen (which is an LCD screen); and displaying images on the screen using a scanning laser. Furthermore, the screen can use liquid crystal on silicon (LCoS), digital micromirror display (DMD), or organic light-emitting diode (OLED).

[0026] The HUD 110 can receive information from the helmet CPU 108. The helmet CPU 108 can connect to the saddle-mounted vehicle 103 (e.g., a transceiver-to-transceiver connection or other short-range communication protocols described herein) so that various vehicle data can be displayed on the HUD. For example, the HUD 110 can display to the user vehicle speed, fuel level, blind spot warnings (via sensors on the vehicle 103), turn-by-turn navigation based on the corresponding system on the vehicle 103, or GPS location, etc. The HUD 110 can also display information transmitted via link 117 from the mobile device 115, such as information about incoming / outgoing calls, direction, GPS and location information, health monitoring data (e.g., heart rate) from the wearable device, etc.

[0027] The saddle-mounted vehicle 103 can communicate with the smart helmet 101 via a short-range communication link, as explained above. The saddle-mounted vehicle 103 may include a forward-facing camera 105. The forward-facing camera 105 may be positioned on the headlight or other similar area of ​​the saddle-mounted vehicle 103. The forward-facing camera 105 can be used to help identify where the PTW is moving. Furthermore, the forward-facing camera 105 can identify various objects or vehicles in front of the saddle-mounted vehicle 103. Therefore, the forward-facing camera 105 can assist various safety systems, such as smart cruise control or collision detection systems.

[0028] The saddle-mounted vehicle 103 may include a bicycle IMU 107. The bicycle IMU 107 may be attached to the PTW's headlight or other similar area. The bicycle IMU 107 can collect inertial data, which can be used to understand the bicycle's movement. The bicycle IMU 107 may be a multi-axis accelerometer, such as a three-axis, four-axis, five-axis, or six-axis accelerometer. The bicycle IMU 107 may also include multiple gyroscopes. The bicycle IMU 107 can work with a processor or controller to determine the bicycle's position relative to a reference point and its orientation.

[0029] The saddle-mounted riding vehicle 103 may include a rider camera 109. The rider camera 109 can be used to track the rider of the saddle-mounted riding vehicle 103. The rider camera 109 can be mounted in various locations along the handlebars of the saddle-mounted riding vehicle, or mounted in other locations facing the rider. The rider camera 109 can be used to capture images or videos of the rider, which are then used for various calculations, such as identifying various body parts or movements of the rider. The rider camera 109 can also be used to focus on the rider's eyes. In this way, eye gaze movement can be determined to determine where the rider is looking.

[0030] The saddle-mounted riding vehicle 103 may include an electronic control unit 111. The ECU 111 can be used to process data collected by sensors on the saddle-mounted riding vehicle and by sensors on the helmet. The ECU 111 can utilize data received from various IMUs and cameras to process and calculate various positions or perform object recognition. The ECU 111 can communicate with the rider camera 109 and the forward-facing camera 105. For example, data from the IMUs can be fed into the ECU 111 to identify position and orientation relative to a reference point. When image data is combined with such calculations, the movement of the bicycle can be used to identify where the rider is facing or focused. Image data from both the forward-facing camera on the bicycle and the camera on the helmet are compared to determine the relative orientation between the bicycle and the rider's head. Image comparison can be performed based on sparse features extracted from the two cameras (e.g., the rider camera 109 and the forward-facing camera 105). In one embodiment, the saddle-mounted riding vehicle 103 includes a bicycle central processing unit 113 that communicates with the ECU 111. Therefore, the system can continuously monitor the cyclist's attention, posture, position, orientation, contact (e.g., grip on the handlebars), cyclist slippage (e.g., contact between the cyclist and the seat), cyclist's relationship with the vehicle, and cyclist's relationship with the world.

[0031] One or both of the smart helmet 101 and the saddle-mounted riding vehicle 103 can communicate with the mobile device 115 via a communication link 117 or a network. The mobile device 115 can be or include a cellular phone, smartphone, tablet, or smart wearable device such as a smartwatch. The wireless communication link 117 can facilitate the exchange of information and / or data. In some embodiments, one or more components of the smart helmet 101 and / or the saddle-mounted riding vehicle 103 (e.g., controllers 108, 111, 113) can send and / or receive information and / or data to the mobile device 115. For example, the helmet CPU 108 or other similar controller can receive information from the mobile device 115 to offset or recalibrate commands sent to the HUD 110 for display on the transparent visor of the helmet 101. To perform the exchange, the smart helmet and / or the saddle-mounted riding vehicle can be equipped with a corresponding transceiver configured to communicate with the transceiver of the mobile device 115. In some embodiments, the wireless communication link 117 can be any type of wired or wireless network, or a combination thereof. By way of example only, wireless communication link 117 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), and short-range communications (such as Bluetooth). TM Network, Purple Bee TMThe wireless communication link 117 may include one or more network access points. For example, the wireless communication link 117 may include wired or wireless network access points (such as base stations and / or internet exchange points), through which one or more components of the smart helmet 101 and / or saddle-mounted riding vehicle 103 can connect to the wireless communication link 117 to exchange data and / or information.

[0032] Various processing units and control units have been described above as part of the smart helmet 101 or saddle-mounted riding vehicle 103. This includes, for example, a helmet CPU 108, a bicycle CPU 113, and an ECU. These processing units and control units may be more generally referred to as processors or controllers, and may be any controller capable of receiving information from various hardware (e.g., from a camera, IMU, etc.), processing information, and outputting instructions to the HUD 110. In this disclosure, the terms "controller" and "system" may refer to or include processor hardware (shared, dedicated, or grouped) that executes code and memory hardware (shared, dedicated, or grouped) that stores the code executed by the processor hardware, or parts thereof. This code is configured to provide the characteristics of the controller and system described herein. In one example, the controller may include a processor, memory, and a non-volatile storage device. The processor may include one or more devices selected from: a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field-programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory. The memory may include a single memory device or multiple memory devices, including, but not limited to, random access memory (“RAM”), volatile memory, non-volatile memory, static random access memory (“SRAM”), dynamic random access memory (“DRAM”), flash memory, cache memory, or any other device capable of storing information. A non-volatile storage device may include one or more persistent data storage devices, such as hard disk drives, optical drives, magnetic tape drives, non-volatile solid-state devices, or any other device capable of persistently storing information. A processor may be configured to read from memory and execute computer-executable instructions embodying one or more software programs residing in the non-volatile storage device. Programs residing in the non-volatile storage device may include, or be part of, an operating system or application, and may be compiled or interpreted from computer programs created using a variety of programming languages ​​and / or technologies, which, without limitation, and individually or in combination, include Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL. The computer-executable instructions of the program can be configured to, upon execution by the processor, cause the HUD system to be altered, offset, or calibrated based on information provided, for example, by the mobile device 115 via communication link 117.

[0033] The implementation of the subject matter and operations described in this specification can be implemented in digital electronic circuits, or in computer software embodied on tangible media, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of these. The implementation of the subject matter described in this specification can be implemented as one or more computer programs embodied on tangible media, i.e., one or more modules of computer program instructions encoded on one or more computer storage media for execution by or control of the operation of a data processing apparatus. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these, or be included therein. The computer storage medium can also be one or more separate components or media (e.g., multiple CDs, disks, or other storage devices), or be included therein. The computer storage medium can be tangible and non-transitory.

[0034] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled languages, interpreted languages, declarative languages, and procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but does not need to, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple coordinating files (e.g., files that store one or more modules, libraries, subroutines, or code portions). A computer program can be deployed to execute on a single computer, located at a single site, or distributed across multiple sites and interconnected on multiple computers.

[0035] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry, such as field-programmable gate arrays (“FPGAs”) or application-specific integrated circuits (“ASICs”), and devices can also be implemented as special-purpose logic circuitry. Such special-purpose circuitry can be referred to as a computer processor, even if it is not a general-purpose processor.

[0036] To render virtual content on the HUD, calibration must be performed. During calibration, spatial transformations between different elements in the system are estimated. The required spatial transformations can be categorized into two types: (1) transformations associated with rigid bodies within the helmet, and (2) transformations associated with the user's face and the helmet. Transformations independent of the user's facial structure can be performed at the helmet's factory or manufacturer before delivery to the user. However, transformations associated with the user's facial structure must be estimated before the user uses the helmet. Although a smart helmet may be equipped with a camera facing the user's face for the purpose of calibrating the HUD system, the helmet camera may be too close to the user's face to be properly calibrated. This disclosure envisions using a mobile device 115 for such calibration, thereby producing improved results by using one or more cameras that are located away from the helmet 101 but still in communication with the helmet 101.

[0037] The disclosed calibration system estimates the transformation between the user's eyes and the HUD screen to accurately render and display virtual content. Because each user's facial structure is different, calibration is performed per user. The calibration results are used to adjust overscan buffers (e.g., waveguides) on the HUD screen and to adjust virtual camera rendering parameters.

[0038] Per-user calibration can be performed by the user at home. In short, an application (“application”) on a mobile device (e.g., a smartphone) is used to collect images of the user’s face via the mobile device’s camera to create a facial structure model of the user’s face. This model is then transmitted to the smart helmet 101 to correct overscan offsets and projection parameters. The user can also use a touch interface on the mobile device to adjust the projection parameters to fine-tune the settings based on user needs.

[0039] Figure 2 This illustration shows an example of the overall overview of a calibration system 200 performed to render virtual content on the HUD of a smart helmet. At 202, factory calibration is provided directly from the manufacturer or supplier of helmet 101. Then, at 204, the user can calibrate settings on a per-user basis. Calibration can correct overscan buffer settings at 206, including corrections at 208 for offsets including horizontal offset (“offset x”) and vertical offset (“offset y”). Specifically, helmet manufacturers provide standard viewing display areas to accommodate various head sizes and interpupillary distances (IPDs) for a wide range of users. A wide viewing area in the HUD allows various head sizes and IPDs to view images at an infinitely far focal point; however, a wide viewing area may also reduce the quality and positional accuracy of the data displayed on the screen.

[0040] Projection devices (e.g., light sources, waveguides, etc.) can create a virtual field of view of virtual images or graphics on a helmet screen (e.g., a face mask). As is typical in HUD systems, the virtual field of view is only visible when the user's eyes are in the correct position. For example, if the user's eyes are too high, too low, too far from either side, or too far from or too close to the assumed location of the eyes in the pre-programmed system, the graphics on the screen will either be invisible to the user, distorted, or not overlaid with a real-life view in the correct position. To accommodate various head sizes, shapes, IPDs, etc., of various users, the projection devices and associated structures are pre-programmed to provide a wider virtual field of view than necessary. However, having a wider virtual field of view than necessary may result in presenting the user with graphics that are inaccurately overlaid with a real-life view, or may render the virtual image invisible to a particular user who may have facial features outside the pre-programmed boundaries of the helmet. The viewing area on the HUD can be reduced by adjusting the vertical and horizontal offsets based on the user's head size, shape, and IPD known from the mobile device. This can improve the quality and accuracy of the data, for example, by allowing data (e.g., colors around vehicles in a view) to be properly positioned on the HUD screen. This can also be very difficult for any cameras or sensing devices mounted on the helmet due to the structural constraints of the helmet. In some embodiments of the optical display unit for eyeglasses, the light source projector and optical coupler can be electronically or mechanically adjusted to change the HUD display area. For example, to adjust the offset, a controller can move the eyeglasses in the light source projector and / or optical coupler via an electrical or mechanical adjustment mechanism.

[0041] The per-user calibration also includes correcting the inherent properties of the virtual camera at point 210, and correcting the projection parameters of the HUD's light source at point 212. The projection parameters are used to transform a reference virtual object into a correctly focused display image on the HUD. To correct the image with the correct focus and size, the projection parameters should take eye position into account. The projection parameters are modeled using a virtual camera placed at the center of the HUD user's eyes. The virtual camera projection parameters are determined based on IPD (Integrated Per Dimension) extracted from the user's face and measurements.

[0042] Figure 3The illustration shows a schematic use of a mobile device application to perform a user's facial scan for calibration purposes, generally shown at 300. The helmet calibration application on the mobile device can be opened by user 302, activating the camera 304 on the mobile device for integration with the application. With camera 304 active, user 302 can hold the mobile device at arm's length, facing the user's face, and move the mobile device along path 306 around the user's face. This allows the camera to capture a sequence of images of the user's face. The screen of the mobile device can be used to provide motion feedback, guiding the user as they perform arc-shaped movements along path 306. The user can move the mobile device along path 306 to capture images from multiple angles, heights, etc.

[0043] Applications on mobile devices can analyze captured images to calculate head shape, size, depth, and features such as IPD or depth to the eyes. The mobile device can calculate calibration offsets (e.g., adjusting the values ​​of offset x and offset y), or it can push this data to the helmet so that the helmet's CPU can perform the calibration offset. Figure 4 The diagram illustrates the adjustment of the offset (e.g., the visible area of ​​a virtual image provided by a helmet-mounted light source). The mobile device can perform an automatic offset initialization sequence 402. Specifically, at 404, the mobile device's application is launched, followed by 406 activating the camera and using it to detect the user's face. This can be accomplished by detecting the user's facial contours corresponding to a database of pre-programmed facial shapes to find a match, thereby confirming that the camera is capturing a face. This step may include using the camera to detect the user's pupils, for example, by comparing the captured image with a pre-programmed database of faces containing pupils. At 408, the mobile device's controller measures the distance between the pupils to estimate the IPD.

[0044] At 410, the determined or estimated IPD can be fed into the adjustment offset feature. As explained above, the adjustment offset feature can modify the offset x and offset y of the HUD virtual field of view. In one embodiment, the controller of the mobile device or helmet provides and accesses a lookup table that maps the offset x and offset y to the IPD. The offset x and offset y can also be adjusted based on other detected features from the camera of the mobile device, such as the distance between the eyes and the mask, the distance between the top of the head and the eyes, etc. This step can be performed by a controller on the mobile device or a controller in the helmet.

[0045] At point 412, an application on the mobile device accessed by the user provides a mobile touch interface. In this step, the application can provide manual adjustment of the offset. If the mobile device's camera and associated software fail to produce an appropriate virtual field of view for the user, the user can access the mobile touch interface to manually adjust the offset until the virtual data is properly visible to the user.

[0046] Figure 5 The diagram illustrates an example flowchart of algorithm 500, which will be executed by one or more of the controllers described herein. The process begins at 502. At 504, the mobile device detects that the user has activated an application for adjusting the HUD display. This can be done, for example, by selecting an application on the mobile device's touchscreen. In response to the application activation, at 506, the camera on the mobile device can be activated or woken up, making the camera ready to capture an image.

[0047] Through the application, the camera captures an image of the user's face, and the controller on the mobile device determines at 508 whether a face has been detected. This can be done according to the method described above, including, for example, comparing the captured facial contour with a contour database. If a face is detected at 508, the controller on the mobile device determines at 510 whether a pupil has been detected. This can be done according to the method described above, including, for example, comparing the facial contour with pupils or the pupil relative to the facial contour with a database of such contours or images. Using the positive identification of the pupil, the controller can measure the IPD at 512. At 514, according to the method described above, an offset adjustment is determined based on the IPD. This can include, for example, accessing a lookup table that associates the IPD with offsets x and y to adjust the virtual field of view. The offset can be pushed to the helmet, whereby the helmet CPU 108 can adjust the HUD 110 to adapt to the offset and change the virtual field of view. The offset adjustment can also be provided manually via a mobile touch interface.

[0048] Steps 502-514 can be performed by the controller onboard the camera and mobile device. However, in other embodiments, communication between the helmet and the mobile device allows data to be shared, and the processing steps are separated between the mobile device and the helmet. In yet another embodiment, images captured by the mobile device can be sent to a remote database via a wireless network (e.g., the cloud), where processing can then occur, and calibration instructions can be sent from the cloud to the helmet.

[0049] The processes, methods, or algorithms disclosed herein may be deliverable to / implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, processes, methods, or algorithms may be stored in various forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media such as ROM devices and information reproducibly stored on writable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media. Processes, methods, or algorithms may also be implemented in a software executable object. Alternatively, processes, methods, or algorithms may be embodied, wholly or partially, using suitable hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0050] While exemplary embodiments have been described above, they are not intended to describe all possible forms covered by the claims. The language used in this specification is descriptive rather than limiting, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form other embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing advantages over or preferred over other embodiments or prior art implementations in one or more desired features, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Accordingly, any embodiment described as less desirable in one or more features than other embodiments or prior art is not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A smart helmet, comprising: A head-up display (HUD) is configured to output graphic images within a virtual field of view on the visor of the smart helmet; The transceiver is configured to communicate with the user's mobile device, which has a helmet calibration application installed. as well as The processor communicates with the transceiver and HUD, and is programmed to: Calibration data associated with the image sequence is received from the mobile device via a transceiver, wherein the helmet calibration application is configured to guide the user to perform arc-shaped movements along a path around the user's face to capture the image sequence from multiple angles and heights using the camera on the mobile device. The virtual field of view of the HUD is changed based on calibration data.

2. The smart helmet according to claim 1, wherein the calibration data includes the user's facial measurements.

3. The smart helmet according to claim 1, wherein the calibration data includes the user's interpupillary distance.

4. The smart helmet according to claim 1, wherein, The mobile device includes a processor coupled to a camera and configured to determine interpupillary distance, and calibration data received by the processor of the smart helmet is based on the interpupillary distance.

5. The smart helmet of claim 1, wherein the processor is programmed to change the horizontal and vertical dimensions of the virtual field of view based on calibration data.

6. The smart helmet of claim 1, wherein the virtual field of view is pre-programmed into the processor.

7. The smart helmet of claim 1, wherein the processor is configured to adjust the light source projector based on calibration data to change the virtual field of view of the HUD.

8. A system for calibrating a head-up display for a smart helmet, the system comprising: A mobile device having a camera configured to capture images of a user’s face and having a helmet calibration application installed, the helmet calibration application being configured to guide the user to perform arcuate movements along a path around the user’s face in order to capture image sequences from multiple angles and heights using the camera on the mobile device; A smart helmet having a head-up display (HUD) configured to display virtual images within a virtual field of view on the helmet's visor; One or more processors are configured as follows: Determine one or more facial features of the image series of the user's face; The offset value used to offset the virtual field of view is determined based on the one or more facial features; and The virtual field of view is calibrated based on the offset value to adjust the visibility of the virtual image displayed by the HUD.

9. The system of claim 8, wherein the one or more facial features include the user's interpupillary distance.

10. The system according to claim 9, wherein, The one or more processors are configured to access a lookup table to determine the offset value based on the interpupillary distance.

11. The system of claim 8, wherein the offset value includes a horizontal offset and a vertical offset.

12. The system of claim 11, wherein the virtual field of view is pre-programmed, and the horizontal and vertical offsets are configured to reduce the pre-programmed virtual field of view after calibration.

13. The system of claim 8, wherein the one or more processors are configured to adjust the light source projector based on an offset value to calibrate the virtual field of view.

14. One or more non-transitory computer-readable media including executable instructions, wherein the instructions, in response to execution by one or more processors, cause the one or more processors to: A sequence of digital images of the user's face is captured via a mobile device with a helmet calibration application installed; Facial features of the face were determined based on the captured images, including the helmet. The calibration application is configured to guide the user to perform arc-shaped movements along a path around the user's face in order to capture the digital image sequence from multiple angles and heights using the camera on the mobile device; Transmitting signals from a mobile device to a smart helmet, wherein the signals include data related to facial features; The signal is received at the smart helmet; as well as The virtual field of view of the head-up display of the smart helmet is calibrated based on the received signals.

15. One or more non-transitory computer-readable media according to claim 14, wherein the facial features include interpupillary distance.

16. One or more non-transitory computer-readable media according to claim 14, wherein, The instruction further causes the one or more processors to apply vertical and horizontal offset values ​​to the virtual field of view to calibrate the virtual field of view.

17. One or more non-transitory computer-readable media according to claim 16, wherein, The instruction further causes the one or more processors to reduce the virtual field of view to apply vertical and horizontal offset values.

18. One or more non-transitory computer-readable media according to claim 14, wherein, The instructions further cause the one or more processors to adjust the light source projector based on the received signals to calibrate the virtual field of view.

19. The one or more non-transitory computer-readable media of claim 14, wherein the virtual field of view is initially pre-programmed onto the one or more non-transitory computer-readable media.

20. One or more non-transitory computer-readable media of claim 14, wherein the calibration of the virtual field of view alters the pre-programmed virtual field of view.

Images