System and method for utilizing augmented reality to display information to vehicle user

Abstract

A system and method for displaying information to a vehicle user using augmented reality, having an eye movement detection device and a user device for activating an application unit connected to the eye movement detection device, a visual operation instruction is input to the eye movement test application unit through the eye movement detection device so as to operate the application unit, and the application unit can perform visual identification training, visual space training, memory training, theme background training and shielding deformation identification training.

Description

Technical Field

[0001] The present invention relates to a system for displaying information to a vehicle user using augmented reality (AR), a head-up display (HUD) system, a method for displaying information using AR, and a computer, a computer program, and a non-transitory computer-readable storage medium. Background Art

[0002] Known heads-up display (HUD) systems used in vehicles such as automobiles and aircraft provide vehicle users with computer-generated virtual graphics that enhance the user's visual sensory perception of real-world features or objects visible during vehicle operation. HUD systems can be configured to generate virtual graphics and project the virtual graphics as images or text through a windshield or other display screen so that users can look up to view information without diverting their attention away from real-world features visible during vehicle operation.

[0003] In known systems, the location of a feature such as another vehicle is determined and an AR graphic is displayed to the user to alert the user to the presence of the other vehicle. The AR graphic is displayed to the user on the HUD in the form of a line in the vicinity of the other vehicle. This simple augmentation can be applied to safety, awareness or general situations.

[0004] However, there are many problems with this approach. First, the AR graphics displayed to the user may be difficult to distinguish from the real-world view and may "blend" into the background to some extent. Second, if the user believes that the AR graphics do not indicate the feature with a high degree of accuracy or if the AR graphics do not provide the user with useful information about the feature, the user may tend to ignore the AR graphics. This ultimately leads to the user not trusting the HUD system. In this case, the AR graphics displayed to the user will unnecessarily distract the vehicle user. This is a problem for general use of the system and comfortable engagement with the system, but it may also pose a significant safety issue by distracting the driver's attention or causing the driver to focus too much on the AR function in an attempt to compensate for any deficiencies in or around the function.

[0005] Many commercial pickup vehicles are now equipped with collision avoidance systems, which may be incorporated into the automatic braking system. The purpose of a collision avoidance system is to avoid or reduce the severity of a collision.

[0006] However, in known collision avoidance systems, the warnings provided to the vehicle user are insufficient. For example, in known systems, the user is only alerted when a collision is imminent and when the automatic braking system is operated, for example to reduce the speed of the vehicle. This does not allow the user to decide whether preventive measures are necessary or whether the user should take preventive measures before engaging the automatic braking system. Furthermore, the warnings provided to the vehicle user are usually displayed on the dashboard or are simply projected towards the lower part of the windshield, so the vehicle user needs to divert their attention from the road.

[0007] It is an object of the present invention to provide an improved system and / or method thereof and / or to solve one or more of the problems discussed above, or elsewhere, or to at least provide an alternative system and / or method. Summary of the invention

[0008] The following summary description is about many features. These aspects are only features of the present invention, wherein the system and method are defined in the scope of the claims set forth below. The reader will be aware that features that are not within the scope of the present invention may still be incorporated into the features of the present invention that are within the scope of the present invention. For this reason, features that lack these features are still retained to provide the reader with useful background information.

[0009] A first feature provides a system for displaying information to a user of a vehicle using augmented reality (AR), the system comprising a processor configured to: interact with a display unit to display information using AR in a field of view associated with the system, and a detector for detecting features in an environment external to the vehicle; process AR graphics for display on the display unit, the AR graphics being three-dimensionally matched to the features detected by the detector; and cause the display unit to display the AR graphics.

[0010] In one example, the processor is configured to cause the display unit to display AR graphics to enhance the feature.

[0011] In one example, the processor is configured to: process the AR graphics so that a change in a three-dimensional attribute of a feature is represented by the three-dimensionally matched AR graphics.

[0012] In one example, the processor is configured to process an AR graphic for display at the display unit, the AR graphic being three-dimensionally matched to the proximity of the feature.

[0013] In one example, the processor is configured to process AR graphics for display at the display unit, the AR graphics being scaled to characteristics in the field of view.

[0014] In one example, the processor is configured to process AR graphics for display at the display unit, the AR graphics being directed toward a feature in the field of view.

[0015] In one example, the processor is configured to process an AR graphic for display at the display unit, the AR graphic having a first graphic portion provided at a fixed apparent depth and a second graphic portion provided at an increasing apparent depth.

[0016] In one example, the second graphical portion indicates a direction of the characteristic.

[0017] In one example, the processor is configured to process the AR graphic for display at the display unit by updating the scale or orientation of the AR graphic in real time based on the proximity or orientation of the feature.

[0018] In one example, the processor is configured to process a first type of AR graphic for display at the display unit if the characteristic is within the field of view.

[0019] In one example, the processor is configured to process a second type of AR graphic for display at the display unit if the feature is outside the field of view.

[0020] In one example, the processor is configured to process AR graphics for display at the display unit, wherein the AR graphics are used to enhance visibility and / or user awareness of features within a field of view.

[0021] In one example, the processor is configured to process an AR graphic for display at the display unit, the AR graphic representing a collision time with a characteristic.

[0022] A second feature provides a head-up display (HUD) system including the system according to the first feature.

[0023] A third feature provides a vehicle comprising a system or a HUD system according to the first feature or the second feature.

[0024] A fourth feature provides a method for displaying information to a vehicle user using augmented reality (AR), the method comprising: interacting with a display unit for displaying information using AR in a field of view and a detector for detecting characteristics in an external environment of the vehicle; processing an AR graphic for display at the display unit, the AR graphic being three-dimensionally matched to the characteristics detected by the detector; and causing the display unit to display the AR graphic.

[0025] A fifth feature provides a computer comprising a processor and a memory configured to execute the method according to the fourth feature.

[0026] A sixth feature provides a computer program comprising instructions, which, when executed by a computer comprising a processor and a memory, can cause the computer to perform the method according to the fourth feature.

[0027] A seventh feature provides a non-transitory computer-readable storage medium comprising instructions, which when executed by a computer comprising a processor and a memory can cause the computer to perform the method according to the fourth feature.

[0028] An eighth feature provides a system for displaying information to a user of a vehicle using augmented reality (AR), the system comprising a processor configured to: interact with a display unit for displaying information using AR in a field of view associated with the system and a detector for detecting a feature in an environment external to the vehicle; process an AR graphic for display at the display unit, the AR graphic representing a time of collision with the feature; and cause the display unit to display the AR graphic.

[0029] In one example, the processor is configured to cause the display unit to display AR graphics to enhance the feature.

[0030] In one example, the processor is configured to process an AR graphic for display at the display unit, the AR graphic being proportional to a collision time with the feature.

[0031] In one example, the processor is configured to process a first type of AR graphic for display at the display unit if the feature is in the field of view and a collision time to the feature is greater than a first threshold collision time.

[0032] In one example, the first type of AR graphic is a three-dimensional AR graphic.

[0033] In one example, the first type of AR graphic represents a low risk factor.

[0034] In one example, the processor is configured to process a second type of AR graphic for display at the display unit if the feature is in the field of view, the collision time for the feature is less than a first threshold collision time, and the collision time for the feature is greater than a second threshold collision time.

[0035] In one example, the second type of AR graphic is a three-dimensional AR graphic.

[0036] In one example, the second type AR graphic represents a moderate risk factor.

[0037] In one example, the processor is configured to process a third type of AR graphic for display at the display unit if the feature is in the field of view and the collision time to the feature is less than a second threshold collision time.

[0038] In one example, the third type of AR graphics is a three-dimensional AR graphics.

[0039] In one example, the third type of AR graphic represents a high risk factor.

[0040] In one example, the processor is configured to process a fourth type of AR graphic for display at the display unit if the characteristic is outside the field of view.

[0041] In one example, the fourth type of AR graphic is a two-dimensional AR graphic.

[0042] In one example, the processor is configured to calculate a time to collision based on the characteristics and relative dynamics of the vehicles.

[0043] In one example, the processor is configured to process an AR graphic for display at the display unit, the AR graphic being three-dimensionally matched to the characteristic detected by the detector.

[0044] A ninth feature provides a head-up display (HUD) system including the system according to the eighth feature.

[0045] The tenth feature provides a vehicle including a system or a HUD system according to the eighth feature or the ninth feature.

[0046] The eleventh feature provides a method for displaying information to a user of a vehicle using augmented reality (AR), the method comprising: interacting with a display unit for displaying information using AR in a field of view and a detector for detecting a feature in an external environment of the vehicle; processing an AR graphic for display at the display unit, the AR graphic representing a time of collision with the feature; and causing the display unit to display the AR graphic.

[0047] A twelfth feature provides a computer comprising a processor and a memory configured to execute the method according to the eleventh feature.

[0048] The thirteenth feature provides a computer program comprising instructions, and when the instructions are executed by a computer comprising a processor and a memory, the computer can execute the method according to the eleventh feature.

[0049] A fourteenth feature provides a non-transitory computer-readable storage medium comprising instructions, and when the instructions are executed by a computer comprising a processor and a memory, the computer can perform the method according to the eleventh feature.

[0050] It will of course be appreciated that features described with respect to one feature may be incorporated into other features. For example, any feature of the method may be incorporated into any feature described with respect to any feature of the apparatus, and vice versa.

[0051] Other preferred and advantageous features of the present invention will become apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1.1 A schematic diagram showing a system for displaying information to vehicle users using augmented reality (AR);

[0053] Figure 1.2 Display the field of view of the user and the display unit;

[0054] Figure 2 A schematic diagram showing a heads-up display (HUD) system;

[0055] Figure 3 displaying a first view in which an AR graphic is displayed;

[0056] Figure 4 displaying a second view in which the AR graphic is displayed;

[0057] Figure 5 displaying a third view in which the AR graphic is displayed;

[0058] Figure 6 displaying a fourth view in which the AR graphic is displayed;

[0059] Figure 7 displaying a fifth view in which the AR graphic is displayed;

[0060] Figure 8 Displays a vehicle including a system or a HUD system;

[0061] Fig. 9 Demonstrate general principles of approach; and

[0062] Fig.10 Shows general methodological principles. DETAILED DESCRIPTION

[0063] The following description describes a number of embodiments. These embodiments are merely embodiments of the present invention, wherein the systems and methods are as defined in the claims set forth below. The reader will appreciate that features of embodiments that are not within the scope of the present invention may still be incorporated into embodiments of the present invention that are within the scope of the present invention. For this reason, descriptions of embodiments lacking these features are retained to provide the reader with useful background information.

[0064] refer to Figure 1.1, a system 10 for displaying information to a vehicle user using augmented reality (AR) is shown. The system 10 includes a processor 100. The processor 100 is configured to interact with a display unit 200 and a detector 300. The processor 100 is configured to interact with the display unit 200 and the detector 300, which means that the processor 100 is configured to receive information from and send information to the display unit 200 and the detector 300.

[0065] Here, the "user of the vehicle" is typically the driver, driver or operator of the vehicle. However, the user may be one or more other users of the vehicle, such as a passenger. In one example, the vehicle may be operated in a fully autonomous mode.

[0066] Includes a mode without any direct control from the vehicle user (e.g., the driver). In this case, the driver may indeed be a passenger in the vehicle. In a fully autonomous mode, the system 10 (and / or the HUD system 50 as described below) may encourage the user to maintain their attention on the road ahead by displaying AR graphics on the display unit 200 to expand the field of view, even though they are not controlling the vehicle themselves. This may be referred to as "gamification" of the display. More importantly, if the vehicle is not driven or controlled, games can be played while using the vehicle. Games may involve the use of AR graphics.

[0067] For the avoidance of doubt, the system 10 can be retrofitted to a vehicle containing the display unit 200 and the detector 300. In this case, the system 10 itself need not include the display unit 200 and the detector 300. However, in an alternative configuration that does not involve modification, the system 10 may include the display unit 200 and the detector 300. This may be a physical modification or a software update or upgrade.

[0068] The display unit 200 is used to display information using AR in the field of view (FoV). The display unit 200 may include a holographic display unit, or be in the form of a holographic display unit. A holographic display is a display that uses light diffraction to create a virtual three-dimensional image. The holographic display unit may include a combiner. The combiner may be a windshield (i.e., all or part of a windshield). In other words, the system 10 of the present invention is an optical see-through system. Optical see-through systems allow users to "directly" view the real world. These AR systems add virtual content in the form of AR graphics by adding additional light to the light from the real world.

[0069] Now refer to Figure 1.2, a plan view of an exemplary FoV of a user and a FoV associated with the system 10 is shown. The FoV of a user (referred to herein as the "user FoV") is labeled at 20, and the FoV associated with the system 10 is labeled at 30.

[0070] In this example, the FoV associated with the system 10 is the FoV of the display unit 200. As shown, the user's FoV is greater than the FoV of the display unit 200. Typically, the user's FoV ranges from 120 to 200 degrees. The FoV of the display unit has a vertical FoV of approximately 4 degrees and a horizontal FoV in the range of 15 to 20 degrees. It can be understood from this figure that the display unit 200 uses AR to display information in the user's FoV.

[0071] In the present description, "the FoV of the display unit" refers to the FoV of the expandable area. This is the portion of the human field of vision where the display unit 200 can display virtual content to enhance the view of the real world. That is, the FoV of the display unit 200 is the area observable by the user, and the display unit 200 can augment the user's view of the real world. In this sense, the FoV of the display unit can be referred to as the "field of view of the user using the system 10" or the "field of view of the user using / viewing the display unit 200". In a HUD system (such as the HUD system 50), this can be referred to as the "field of view of the user using the HUD system 50".

[0072] In this disclosure, the FoV of the display unit 200 will be referred to as "FoV". It should be understood that the display unit 200 is located in an area that can be observed by the user. In a specific example, this means that the display unit 200 is located at or in front of the windshield (i.e., facing the user), and the user can observe the road ahead through the windshield when using the vehicle. In other examples, the display unit 200 can be installed on the side or rear of the vehicle, so the FoV can be for the side or rear of the vehicle.

[0073] The detector 300 is used to detect characteristics in the vehicle's external environment.

[0074] The detector 300 may be in the form of an optical detector, such as a camera. Additionally or alternatively, the detector 300 may be in the form of a radio detection and ranging (radar) system or a light detection and ranging (lidar) system.

[0075] A characteristic may include another vehicle, such as a vehicle on the road or a vehicle on the side of the road. A characteristic may also include a road sign or general road equipment. A characteristic may be a human or an animal. The types of characteristics detected or tracked may vary depending on the purpose or mode or type of system, vehicle or scenario.

[0076] Here, the "environment outside the vehicle" may include detecting characteristics in front of the vehicle, to the side of the vehicle, and to the rear of the vehicle. In other words, the detector 300 may be operable to detect characteristics outside the field of view.

[0077] The processor 100 is configured to process AR graphics for display at the display unit. AR graphics are graphics displayed using augmented reality (AR). The processor 100 is configured to process AR graphics in a variety of different ways, as described in further detail below.

[0078] The processor 100 is configured to cause the display unit 200 to display AR graphics.

[0079] Although the processor 100 may cause the display unit 200 to display AR graphics in actual implementation, this may be considered optional for some embodiments. For example, the core differentiating feature is processing three-dimensional matching graphics, or processing AR graphics indicating collision time. In terms of the key features that distinguish the present invention, causing the display unit 200 to display AR graphics may be optional.

[0080] Reference Figure 2 , a head-up display (HUD) system 50 is shown. The HUD system 50 includes an image generation unit 500, a combiner unit 600 and a corrector unit 700.

[0081] These combiners and correctors may be applicable, for example, to holographic systems, or other systems for presenting to a user a surface that is both viewed or seen through and provided with graphic properties by the system. For example, the combiner may be an at least partially (e.g., half) transparent surface used in the system to superimpose an image presented by a projector (which may be part of the image generating unit 500 or separate from the image generating unit 500) on top of the user's physical world. The combiner is at least partially transparent to allow the user to see through it, while reflecting or otherwise directing light (e.g., AR functionality) to the user. The corrector unit may be used to correct aberrations, filter light, and / or improve light utilization efficiency, or generally to correct some optical property. In some examples, one or more optical devices or lenses may be utilized, including filters.

[0082] The system 10 described herein may be incorporated into the HUD system 50. The system 10 and / or the HUD system 50 may be incorporated into or form part of an advanced driver assistance system (ADAS).

[0083] Although the present disclosure relates to the system 10 and HUD system 50 for installation in a vehicle, the present disclosure can also be applied in the context of a head mounted display (HMD). For example, the system 10 can be incorporated into a pair of smart glasses or other wearable devices.

[0084] Figures 3 to 7 The figures show a front view through a vehicle windshield, wherein one or more AR graphics are displayed within the FoV (i.e., the field of view of the display unit 200 as described above). The operation and functionality of the system 10 will be described with reference to these figures. In the figures, the user FoV and the FoV of the display unit 200 are illustrated as rectangular FoVs. It should be understood that such FoVs are not actually rectangular.

[0085] Figure 3 A first user FoV 3000 is displayed, wherein a first AR graphic 3010 is displayed in the FoV 3200 (ie, the FoV of the display unit 200 ).

[0086] Figure 4 A second user FoV 4000 is displayed, wherein a second AR graphic 4010 is displayed in the FoV 4200 (ie, the FoV of the display unit 200 ).

[0087] Figure 5 A third user FoV 5000 is displayed, wherein a third AR graphic 5010 is displayed in the FoV 5200 (ie, the FoV of the display unit 200 ).

[0088] Figure 6 A fourth user FoV 6000 is displayed, wherein a fourth AR graphic 6010 is displayed in the FoV 6200 (ie, the FoV of the display unit 200 ).

[0089] Figure 7 A fifth user FoV 7000 is displayed, wherein a fifth AR graphic 7010 is displayed in the FoV 7200 (ie, the FoV of the display unit 200 ).

[0090] The processor 100 is configured to cause the display unit 200 to display AR graphics 3010:7010 (increment 1000) to augment the feature. Figures 3 to 7As shown, the AR graphic 3010:7010 provides information to the user about the feature, such as information related to the three-dimensional properties of the feature (such as relative to the system / vehicle), such as proximity, position, direction, and also provides the user with information about the (estimated or calculated) collision time with the feature. In other words, the three-dimensional properties of the feature (such as relative to the system / vehicle) are taken into account when processing and providing the AR graphic.

[0091] By augmenting the feature, the system provides the user with useful information about the feature. This improves the user experience, encourages safe vehicle operation, and alerts vehicle users to the presence of the feature.

[0092] exist Figures 3 to 7 In each of the figures, the feature is another vehicle, generally indicated at 1000. This other vehicle will be referred to herein as "detected vehicle 1000" because it is a feature in the vehicle's external environment that is detected by the detector 300. However, those skilled in the art will appreciate that the present teachings may be applied to other features, including road signs or general road facilities as described above.

[0093] The processor 100 is configured to process AR graphics in a number of different ways, as described in further detail below. Two such ways include three-dimensional matching and indicating collision time.

[0094] In a first example, the processor 100 is configured to process an AR graphic for display at the display unit 200 , the AR graphic being three-dimensionally matched to the characteristic detected by the detector 300 .

[0095] Here, a "3D-matched" AR graphic refers to a graphic that matches the position and / or direction (including direction of travel) of the feature in three dimensions or represents a three-dimensional property of the feature, such as its shape. In this way, the AR graphics displayed on the display unit 200 can be scaled to the detected feature. In other words, the three-dimensional properties of the feature (e.g., relative to the system / vehicle) are taken into account when processing and providing the AR graphics. This is in contrast to a simplistic approach in which a feature is detected and an AR graphic is provided only near the feature without considering the three-dimensional relationship.

[0096] In the examples described in this article, and Figures 3 to 5In the example shown, three-dimensional matching is achieved by an AR graphic having a first graphic portion 3012:5012 (increments of 1000) that is provided at a fixed apparent depth (e.g., a horizontal line), and a second graphic portion 3014:5014 (increments of 1000) that is provided at an increasing apparent depth (e.g., two extended lines that appear to extend away from the user and indicate a dimension of the feature, such as width). The second graphic portion also indicates the direction of the feature. That is, when the feature turns to the side, the second graphic portion will indicate this point by following (e.g., extending along) the side of the vehicle in the FoV.

[0097] In this way, the AR graphics displayed to the user can better represent the three-dimensional nature of the feature in real life. The three-dimensional AR graphics are easily recognized by the user. In addition, the three-dimensional AR graphics provide the user with a higher level of information because the three-dimensional AR graphics can indicate the location or orientation of the feature, not just the location of the feature. The three-dimensional AR graphics also indicate to the user that the system 10 is tracking the feature correctly. This can increase confidence in the system 10, which may be particularly useful in autonomous vehicle implementations. The three-dimensional matching AR graphics indicate the location or orientation of the feature with high accuracy. Ultimately, the user experience of the system 10 is thereby improved. The degree of distraction and concentration is reduced, thereby improving safety.

[0098] In providing a three-dimensionally matched AR graphic, the processor 100 may receive information from the detector 300, process the information from the detector 300, and process an AR graphic for display on the display unit 200. The processor 100 may select an AR graphic from an AR graphic library, and the selected graphic is three-dimensionally matched to the characteristics perceived by the vehicle user. However, this approach may require a large number of such graphics to match a large number of possible three-dimensional matches.

[0099] Alternatively, the processor 100 may adjust the AR graphics displayed on the display unit 200 to ensure that the AR graphics remain three-dimensionally matched to the characteristics perceived by the user of the vehicle. For example, this may be done using a mathematical or graphical transformation program. In other words, the AR graphics are scaled according to the detected characteristics. This may require a smaller number of characteristics to be stored compared to the library approach.

[0100] Of course, those skilled in the art will understand how to display AR graphics on the display unit 200, and the foregoing provides an illustrative example of how to display three-dimensional matching AR graphics on the display unit 200.

[0101] The processor 100 is configured to process the AR graphics such that changes in the three-dimensional nature of a feature (e.g., isolated or relative to a vehicle) are represented by a three-dimensionally matched AR graphic. Figures 3 to 5 You can understand this point.

[0102] Reference Figure 3 When the detected vehicle 1000 is at a first position, the first AR graphic 3010 is three-dimensionally matched to the detected vehicle 1000 at the first position.

[0103] Reference Figure 4 , when the detected vehicle 1000 is at a second location, the second AR graphic 4010 is three-dimensionally matched to the detected vehicle 1000 at the second location. In this case, the second location is also a second, different proximity. As shown, at the second location, the detected vehicle 1000 is closer to the user / vehicle than at the first location. Therefore, the vehicle in the FoV 4200 has a larger size. The second AR graphic 4010 represents this change in the three-dimensional nature of the feature by enlarging and significantly increasing the size of the second graphic portion 4012. That is, the AR graphic is scaled according to the detected feature.

[0104] Reference Figure 5 , when the detected vehicle 1000 is at a third position, the third AR graphic 5010 is three-dimensionally matched to the detected vehicle 1000 at the third position. As shown, at the third position, the detected vehicle 1000 is closer to the user / vehicle than at the first and second positions. Therefore, the vehicle in the FoV 5200 has a larger size. The third AR graphic 5010 represents this change in the three-dimensional nature of the feature by becoming larger again, significantly increasing the size of the second graphic portion 5010b. That is, the AR graphic is scaled according to the detected feature.

[0105] In this way, the detected three-dimensional characteristic changes of the vehicle 1000 are represented by three-dimensional matched AR graphics. As described above, this type of functionality means that the AR graphics displayed to the user can better represent the real-life three-dimensional nature of the characteristic. In addition, the three-dimensional matched AR graphics provide a higher level of information to the user because the matched AR graphics can indicate the location or orientation of the characteristic, not just the location of the characteristic. The three-dimensional matched AR graphics indicate the location or orientation of the characteristic with high accuracy. Ultimately, the user experience of the system 10 is thereby improved. The safety of the user is also improved.

[0106] Now return to reference Figures 3 to 5As will be apparent from the description of processing the AR graphics provided above so that the change in the three-dimensional nature of the feature is represented by the three-dimensionally matched AR graphics, the processor 100 is configured to process the AR graphics for display at the display unit 200, the AR graphics being three-dimensionally matched to the detected proximity of the vehicle 1000. Figures 3 to 5 As shown, when the proximity of the detected vehicle 1000 changes, the AR graphic will match accordingly. The proximity can be determined or inferred. For example, the proximity can be measured by a sensor and any matching can be performed based on the determination. Alternatively, the proximity can be inferred by the change in the detected size of the detected vehicle (or other object) in the FoV. That is, if a vehicle (or other object) is detected or tracked, and its apparent size changes, it can be inferred (usually) that the detected vehicle (or other object) has changed its proximity.

[0107] In this way, the AR graphic represents the proximity of the detected vehicle 1000, thereby improving user safety and experience.

[0108] In addition, the processor 1000 is configured to process an AR graphic for display at the display unit 200, the AR graphic being scaled to the detected vehicle 1000 in the FoV. Figures 3 to 5 As shown, the AR graphic is scaled to the apparent size of the detected vehicle 1000 in the FoV. For example, the size (eg, width) of the AR graphic may be proportional or matched to the same size of the feature in the FoV.

[0109] In this way, the AR graphics accurately represent the real-life location of the detected vehicle 1000. Since the AR graphics effectively expand the FoV and do not distract the user, the user's safety and experience are improved.

[0110] In addition, the processor 1000 is configured to process an AR graphic for display at the display unit 200, the AR graphic being directed to the detected vehicle 1000 in the FoV. Figures 3 to 5 As shown, the AR graphic is pointed to indicate that the detected vehicle 1000 is away from the vehicle and is traveling directly in front of the vehicle. Although not shown in the figure, if the detected vehicle 1000 is at a certain angle to the vehicle (for example, the detected vehicle 1000 is changing lanes in front of the vehicle), the AR graphic is also pointed accordingly.

[0111] In this way, the AR graphics accurately represent the real-life direction of the detected vehicle 1000. Since the AR graphics effectively expand the FoV and do not distract the user, the user's safety and experience are improved.

[0112] The processor 100 is configured to update the scale or orientation of the AR graphics in real time based on the detected proximity or orientation of the vehicle 1000. Figures 3 to 5 As shown. In this way, the AR graphics can correctly indicate the change of proximity or orientation. The AR graphics correctly represent the real life proximity or orientation of the resulting vehicle 1000. Since the AR graphics effectively expand the FoV and do not distract the user's attention, the user's safety and experience are improved.

[0113] Now return to reference Figures 3 to 5 , the processor 100 is configured to process a first type of AR graphic 3010, 4010, 5010 for display at the display unit 200 if the feature is in FoV. The first type of AR graphic may be a "staple" shaped graphic, such as Figures 3 to 5 3010, 4010 and 5010 shown have a first graphic portion 3012, 4012, 5012 with a fixed apparent depth and a second graphic portion 3014, 4014, 5014 with increasing apparent depth.

[0114] By displaying the first type of AR graphic if the detected feature is in the FoV, user interaction with the system 10 is improved as the user understands that the first type of AR graphic is displayed to provide information about the feature in the FoV.

[0115] Reference Figure 7 , the processor 100 is configured to process a second type of AR graphic 7010 for display at the display unit 200 if the feature is outside the FoV 7200. The second type of AR graphic may be an "arrow" shaped graphic, such as Figure 7 7010 shown. The arrow-shaped graphic indicates that there are features outside of FoV 7200. Here, "outside the user's field of view" may mean not completely within FoV 7200, for example Figure 7 By scaling or pointing the AR graphic 7010 relative to the position or orientation of the detected vehicle 1000 , the arrow-shaped AR graphic 7010 is three-dimensionally matched to the detected vehicle 1000 .

[0116] If the detected feature is outside (or not completely inside) the FoV 7200, the second type of AR graphic is displayed, and the user's interaction with the system 10 is improved because the user is aware that the second type of AR graphic is displayed to provide information about the feature outside (or not completely inside) the FoV 7200. In other words, the first type of AR graphic may not be displayed, but the second type of AR graphic is still displayable to provide useful information to the user. Because the AR graphic effectively augments the FoV 7200 and alerts the user to functions outside the FoV 7200, the user's safety and experience are improved.

[0117] In this way, although the detected vehicle 1000 may be too close to the vehicle (and thus not completely within the FoV 7200) to display the first type of AR graphic at the display unit 200, the second type of AR graphic 7010 can still indicate the presence of the detected vehicle 1000 to the user. This can avoid the need to display an incorrect or out-of-scale first type AR graphic to the user. For example, displaying the first type AR graphic in this scene may take up too much space in the FoV 7200, or may overlap with other features displayed on the display unit 200, such as a speed indicator.

[0118] As if Figures 3 to 7 As is understood, the AR graphics 3010:7010 are used to enhance the visibility and / or user awareness of the feature in the FoV. As shown, by enhancing visibility, the detected feature can stand out from the background (i.e., the road feature), thereby improving the safety of the user and the vehicle. The enhanced user awareness of the feature leads to improved user and vehicle safety, and ultimately improves the user's interaction with the system 10.

[0119] In a second example, the processor 100 is configured to process an AR graphic for display at the display unit 200 , the AR graphic representing the collision time with the feature.

[0120] Here, "time to collision" refers to the predicted or calculated time before the vehicle makes contact with the detected feature of the detector 3000. It will be appreciated that this can be applied to three-dimensional features, such as the detected vehicle 1000. In one example, the time to collision can be the time to reach a specific point or boundary, such as a road lane sign or road intersection, or even a waypoint.

[0121] Advantageously, the AR graphic represents the time to collision with the feature, and the system 10 provides advance warning of a possible collision to the vehicle user. This allows the user to decide whether preventive or corrective action is necessary and should be taken by the user, thus allowing the user to make this decision before engaging the automatic braking system. Furthermore, by providing such an indication as an AR graphic (which may be displayed on a HUD system, such as one incorporated into or adjacent to the windshield), the user does not need to take his attention away from the road. This improves the user experience and, therefore, increases the safety of the user and the vehicle.

[0122] The processor 100 is configured to calculate the time to collision based on the characteristics and relative kinematics of the vehicle. That is, the time to collision may be calculated based on relative position, orientation, velocity, acceleration, motion and / or other kinematic factors. In this way, the time to collision may be calculated with high accuracy, and thus accurate information about the environment may be provided to the user of the vehicle. In addition, the time to collision may be calculated based on road conditions and / or environmental conditions. For example, soft road surfaces and wet weather may increase stopping distances, and therefore may need to be incorporated into the time to collision calculation.

[0123] The processor 100 is configured to process an AR graphic for display at the display unit 200, the AR graphic being proportional to the time to collision with the feature. That is, the AR graphic itself represents the time to collision with the feature. In this way, the user can determine the time to collision with the feature simply by referring to the AR graphic. Advantageously, this enables the user to decide whether preventive or corrective action is necessary and should be taken to avoid a collision, allowing the user to make this decision before activating the automatic braking system. The user does not need to take his attention away from the road while such AR graphics are provided on the HUD system. Thereby improving the user experience and, therefore, improving the safety of the user and the vehicle.

[0124] Reference Figures 3 to 7 , an AR graph representing the time of collision with the characteristic - detected vehicle 1000 - is shown. In the following description, the first, second, third and fourth types of AR graphs are described. It will be understood that the "type" of AR graph can be the same as the "type" of AR graph described above.

[0125] refer to Figure 3 , the collision time with the detected vehicle 1000 is determined to be greater than the first threshold collision time ti. Figure 3 It can be seen that the detected vehicle 1000 is located within the FoV 3200 . The processor 100 is configured to process a first type of AR graphic 3010 for display at the display unit 200 .

[0126] In this way, the user understands that the first AR graphic 3010 indicates that the collision time with the detected vehicle 1000 is greater than the first threshold collision time ti. Therefore, the user can understand the collision time with the detected vehicle 1000, so that the user can have enough time to decide whether to take preventive or corrective measures, for example, whether the detected vehicle 1000 should brake or otherwise change position. More simply, the user understands the relative position, direction, speed, movement and / or acceleration of the vehicle and the detected vehicle 1000 to maintain safety.

[0127] The first AR graphic 3010 is displayed as a single "staple" AR graphic. In other words, the first AR graphic 3010 is a three-dimensional AR graphic. In this way, the AR graphic 3010 can represent the three-dimensional nature of the detected vehicle 1000, which is advantageous as described above. In other words, different examples can be used in an advantageous or synergistic combination.

[0128] The first type of AR graphic 3010 represents a low risk factor. In this way, when seeing the first type of AR graphic, the user can understand that the detected vehicle 1000 poses a low level of risk to the vehicle. In a preferred example, the first type of AR graphic 3010 can be provided in a color indicating a low risk factor, such as green. In this way, the user can quickly and easily interpret and understand the first type of AR graphic 3010 representing a low risk factor.

[0129] Reference Figure 4 , the collision time with the detected vehicle 1000 is determined to be less than the first threshold collision time ti, but greater than a second threshold collision time t2. Figure 4 It can be seen that the detected vehicle 1000 is located within the FoV 4200 . The processor 100 is configured to process a second AR graphic 4010 for display at the display unit 200 .

[0130] In this way, the user learns that the second AR graphic 4010 represents a collision time with the detected vehicle 1000, which is less than the first threshold collision time ti, but greater than a second threshold collision time t2. Therefore, the user can understand the collision time with the detected vehicle 1000, allowing the user to have appropriate time to decide whether to take preventive or corrective measures, for example, whether the detected vehicle 1000 should brake or change its position in other ways.

[0131] More simply, the user understands that the vehicle maintains a reasonably safe relative position, orientation, speed, motion and / or acceleration with respect to the detected vehicle 1000. The user thereby understands that more attention needs to be paid to the road ahead, or alternatively or additionally, the user may wish to take corrective action to reduce the risk level to a lower level. Of course, in doing so, the AR graphics will switch from the second (e.g., 4010) to the first (e.g., 3010).

[0132] The second AR graphic 4010 is displayed as a double "staple" AR graphic. That is, the second AR graphic 4010 is a three-dimensional AR graphic. In this way, the AR graphic 4010 can represent the three-dimensional characteristics of the detected vehicle 1000, which is advantageous as described above.

[0133] The second type of AR graphic 4010 represents a medium risk factor. In this way, when seeing the second type of AR graphic 4010, the user can understand that the detected vehicle 1000 poses a medium risk to the vehicle. In a preferred example, the second type of AR graphic 4010 can be provided in a color indicating a medium risk factor, such as yellow or amber. In this way, the user can quickly and easily interpret and understand the second type of AR graphic 4010 representing a medium risk factor. In another preferred example, the second type of AR graphic 4010 can be a single "staple" shaped AR graphic displayed in yellow or amber, thereby representing a medium risk factor.

[0134] Reference Figure 5 , the collision time with the detected vehicle 1000 is determined to be less than the first threshold collision time ti and also less than the second threshold collision time t2. Figure 5 It can be seen that the detected vehicle 1000 is located within the FoV 5200 . The processor 100 is configured to process a third type of AR graphic 5010 for display at the display unit 200 .

[0135] In this way, the user learns that the third AR graphic 5010 indicates that the collision time with the detected vehicle 1000 is less than the first threshold collision time ti and less than the second threshold collision time t2. The user can therefore understand that the collision time with the detected vehicle 1000 allows the user to have a short time to decide whether preventive or corrective action is required, for example, whether the detected vehicle 1000 should brake or otherwise change position. More simply, the user understands that the vehicle and the detected vehicle 1000 are maintaining a potentially dangerous relative position, direction, speed, movement and / or acceleration. The user thereby understands that a high level of attention needs to be paid to the road ahead, or alternatively or additionally, the user may wish to take corrective action to reduce the risk level to a lower level. Of course, in doing so, the AR graphic will switch from the third type (e.g., 5010) to the second type (e.g., 3010) or the first type (e.g., 2010).

[0136] The third AR graphic 5010 is displayed as a double "cross staple" shape AR graphic. That is, the third AR graphic 5010 is a three-dimensional AR graphic. In this way, the AR graphic 5010 can represent the three-dimensional characteristics of the detected vehicle 1000, which is advantageous as described above.

[0137] The third AR graphic 5010 represents a high risk factor. Thus, when the user sees the third AR graphic 5010, he or she can understand that the detected vehicle 1000 poses a high risk to the vehicle. In a preferred example, the third AR graphic 5010 can be provided in a color (e.g., red) indicating a high risk factor. In this way, the user can quickly and easily interpret and understand the third AR graphic 5010 representing a high risk factor.

[0138] Alternatively, the third AR graphic can be Figure 6 As shown and described at 6010. In this example, the third AR graphic 6010 is a triangular AR graphic 6010. Similarly, the third AR graphic 6010 represents a high risk factor. In this way, when seeing the third AR graphic 6010, the user can understand that the detected vehicle 1000 poses a high risk to the vehicle. In a preferred example, the third AR graphic 6010 can be provided in a color (e.g., red) indicating a high risk factor. In this way, the user can quickly and easily interpret and understand the third AR graphic 6010 representing a high risk factor. In this example, the triangular third AR graphic 6010 can be a two-dimensional AR graphic.

[0139] In addition, although the detected vehicle 1000 may be too close to the vehicle to display a staple-shaped AR graphic at the display unit 200, the triangular AR graphic 6010 may still indicate the presence of the detected vehicle 1000 to the user. This can avoid the need to display an incorrect or inappropriately scaled AR graphic to the user. For example, in this scenario, displaying a staple-shaped AR graphic in the FoV 6200 may take up too much space in the FoV, or may overlap other features displayed on the display unit 200, such as a speed indicator.

[0140] Reference Figure 7 , the collision time with the detected vehicle 1000 is determined to be less than the first threshold collision time ti and also less than the second threshold collision time t2. Figure 7 It can be seen that the detected vehicle 1000 is outside the FoV 7200 . The processor 100 is configured to process a fourth type of AR graphics 7010 for display at the display unit 200 .

[0141] In this way, the user is informed that the fourth AR graphic 7010 represents a detected vehicle outside the FoV 7200. Therefore, the user is alerted to potential risks outside the FoV 7200. In a preferred example, the fourth AR graphic 7010 is an arrow-shaped graphic pointing to the detected feature.

[0142] When the collision time with the detected vehicle 1000 is greater than the first threshold collision time ti, the fourth AR graphic 7010 may have a first shape or color, such as green. When the collision time with the detected vehicle 1000 is less than the first threshold collision time t1 but greater than the second threshold collision time t2, the fourth AR graphic 7010 may have a second shape or color, such as yellow or amber. When the collision time with the detected vehicle 1000 is less than the first threshold collision time t1 and less than the second threshold collision time t2, the fourth AR graphic may have a third shape or color, such as red.

[0143] In this way, the user can easily understand the risk level posed by the detected feature. The user's attention is thus drawn to the feature, even if it is outside the FoV 7200. This improves the user experience and vehicle safety.

[0144] It will be appreciated that the first and second examples described above, respectively relating to three-dimensional matching and time to collision, may be implemented by the same system 10, 50. The various features may be combined or interchanged as necessary or desired.

[0145] Reference Figure 8, a vehicle 8000 is shown. The vehicle 8000 includes the system 10 or the HUD system 50.

[0146] Reference Fig. 9 , a method for displaying information to a user of a vehicle using augmented reality (AR) is shown. Step S9010 includes interacting with a display unit for displaying information in a field of view associated with a system using AR and a detector for detecting a feature in an environment external to the vehicle. Step S9020 includes processing an AR graphic for display at the display unit, the AR graphic being three-dimensionally matched to the feature detected by the detector. Step S9030 includes causing the display unit to display the AR graphic.

[0147] Although step S9030 may be implemented in actual implementation, it may be considered optional for some embodiments. For example, the core differentiating feature is the processing step. In terms of the key features that distinguish the present invention, the actual display step is optional.

[0148] Reference Fig.10 , a method for displaying information to a user of a vehicle using augmented reality (AR) is shown. Step S10010 includes interacting with a display unit for displaying information in a field of view associated with a system using AR and a detector for detecting a characteristic in an environment external to the vehicle. Step S10020 includes processing an AR graphic for display at the display unit, the AR graphic representing a time to collision for the characteristic. Step S10030 includes causing the display unit to display the AR graphic.

[0149] Although step S10030 may be implemented in actual implementation, it may be considered optional for some embodiments. For example, the core differentiating feature is the processing step. In terms of the key features that distinguish the present invention, the actual display step is optional.

[0150] While preferred embodiments have been shown and described, it will be apparent to those skilled in the art that various changes and modifications may be made without departing from the scope of the invention as defined in the appended claims and as hereinbefore described.

[0151] At least some of the exemplary embodiments described herein may be implemented in part or in whole using dedicated special purpose hardware. Terms such as "components", "modules" or "units" as used herein may include, but are not limited to, hardware devices such as circuits in the form of discrete or integrated components, field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs) that perform some tasks or provide related functions. In some embodiments, the described components may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processors. In some embodiments, these functional elements may include, for example, components such as software components, object-oriented software components, class components and task components, programs, functions, properties, processes, subroutines, code snippets, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. Although the exemplary embodiments have been described with reference to the components, modules and units discussed in this disclosure, these functional elements may be combined into fewer components or separated into additional components. This disclosure has described various combinations of optional features, and it should be understood that the described features can be combined in any appropriate combination. In particular, the features of any exemplary embodiment can be combined with the features of any other embodiment as appropriate, unless such combinations are mutually exclusive. Throughout this specification, the terms "comprise" or "comprising" mean including the indicated elements, but not excluding the presence of other elements.

[0152] Attention is paid to all papers and documents related to this application that were filed concurrently or prior to this specification and are open to public inspection with this specification, and the contents of all these papers and documents are incorporated herein by reference.

[0153] All features disclosed in this specification (including any attached patent claims, abstracts and drawings), and / or all steps of any method or procedure so disclosed may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive.

[0154] Unless otherwise expressly stated, each feature disclosed in this specification (including any attached patent claims, abstracts and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose. Therefore, unless otherwise expressly stated, each feature disclosed is only one example of a series of general equivalent or similar features.

[0155] The present invention is not limited to the details of the foregoing embodiments. The present invention extends to any novel single or any novel combination of the features disclosed in this specification (including any attached claims, abstracts and drawings), or any novel single or any novel combination of the steps of any method or procedure disclosed in this specification (including any attached claims, abstracts and drawings).

Claims

1. A system for displaying information to a vehicle user using augmented reality (AR), the system comprising a processor configured to: interact with a display unit to display information using AR in a field of view associated with the system; and a detector for detecting a characteristic in the external environment of the vehicle; processing an AR graphic for display at the display unit, the AR graphic being three-dimensionally matched to the characteristic detected by the detector; And the display unit displays the AR graphic.

2. The system according to claim 1, characterized in that The processor is configured to: cause the display unit to display AR graphics to augment the feature.

3. The system according to claim 1 or 2, characterized in that The processor is configured to: process the AR graphics so that changes in the three-dimensional nature of the feature are represented by the three-dimensionally matched AR graphics.

4. A system according to any one of the preceding claims, characterised in that The processor is configured to process an AR graphic for display at the display unit, the AR graphic being three-dimensionally matched to the proximity of the feature.

5. System according to any one of the preceding claims, characterized in that The processor is configured to process AR graphics for display at the display unit, the AR graphics being scaled to features in the field of view and / or the AR graphics being pointed at features in the field of view.

6. System according to any one of the preceding claims, characterized in that The processor is configured to process an AR graphic for display at the display unit, the AR graphic having a first graphic portion provided at a fixed apparent depth and a second graphic portion provided at increasing apparent depths, selectively wherein the second graphic portion indicates a pointing direction of the feature.

7. A system according to any one of the preceding claims, characterised in that The processor is configured to process the AR graphics by updating the scale or orientation of the AR graphics in real time based on the proximity or orientation of the features for display on the display unit.

8. System according to any one of the preceding claims, characterized in that The processor is configured to process a first type of AR graphic for display at the display unit if the feature is within the field of view.

9. System according to any one of the preceding claims, characterized in that The processor is configured to process a second type of AR graphic for display at the display unit if the feature is outside the field of view.

10. System according to any one of the preceding claims, characterized in that The processor is configured to process AR graphics for display at the display unit, wherein the AR graphics are used to enhance visibility and / or user awareness of features in the field of view.

11. A system according to any one of the preceding claims, characterised in that The processor is configured to process an AR graphic for display at the display unit, the AR graphic representing a time of collision with the feature.

12. A head-up display (HUD) system comprising the system of any preceding claim.

13. A vehicle comprising a system or a HUD system as claimed in any one of the preceding claims.

14. A method for displaying information to a vehicle user using augmented reality (AR), the method comprising: interacting with a display unit for displaying information in a field of view using AR and a detector for detecting characteristics in an environment external to the vehicle; processing an AR graphic for display at the display unit, the AR graphic being three-dimensionally matched to the characteristics detected by the detector; and The display unit is caused to display the AR graphic.

15. A computer comprising a processor and a memory configured to perform the method according to claim 14, a computer program comprising instructions which, when executed by the computer comprising the processor and the memory, cause the computer to perform the method according to claim 14, or a non-transitory computer-readable storage medium comprising instructions which, when executed by the computer comprising the processor and the memory, cause the computer to perform the method according to claim 14.

Images