System and method for displaying information to driver using augmented reality

Abstract

A system for displaying information to a user of a vehicle using augmented reality (AR), the system comprising a processor, the processor is configured to interact with a display unit for displaying information using AR in a field of view associated with the system, a vehicle operational data source for providing vehicle operational data to the processor, and a multimedia data source for providing multimedia data to the processor; processing an AR image for display on the display unit, wherein the AR image is based on the vehicle operation data and the multimedia data; and causing the display unit to display the AR image.

Description

Technical Field

[0001] The present disclosure relates to a system for displaying information to a user of a transportation vehicle using augmented reality (AR), a head-up display (HUD) system, a method of displaying information using AR, and a computer, a computer program, and a non-transitory computer-readable storage medium. Background Art

[0002] Known heads-up display (HUD) systems used in vehicles such as automobiles and aircraft provide a vehicle user with computer-generated virtual images that enhance the user's visual sensory perception of real-world features or objects visible during vehicle operation. The HUD system is configured to generate virtual images and project the virtual images as images or text through or on a windshield or other display screen so that the user can view information while looking up without diverting attention from real-world features visible during vehicle operation.

[0003] It is desirable to display such computer-generated virtual images in a manner that is consistent with and well-matched to the user's visual sensory perception of real-world features or objects visible during operation of the vehicle. If the images displayed to the user do not match the user's visual sensory perception well, the user may become disoriented or have difficulty understanding or comprehending the vehicle dynamics.

[0004] This is particularly important in the operation of an autonomous vehicle, where the user has minimal involvement in the operation of the vehicle. The user can understand the operation of the vehicle through instruments, which indicate to the user the dynamics of the vehicle. However, if such instruments are not well matched to the vehicle dynamics, it may be difficult for the user to understand or comprehend the autonomous vehicle dynamics. This may lead to a decrease in confidence in the autonomous vehicle and ultimately a worse user experience.

[0005] Another issue is how to display content for user interaction during vehicle operation. Display of content (potentially multimedia content) can enable the user to understand vehicle dynamics and thus serve as an instrument. However, if such content is inconsistent and not well matched to the user's visual sensory perception of real-world features or objects that can be seen during vehicle operation, the user may be disoriented or may have difficulty understanding or comprehending vehicle dynamics. Again, this may result in reduced engagement with the content displayed to the user. Summary of the invention

[0006] It is an object of the present invention to provide, inter alia, an improved system and / or method thereof and / or to address one or more of the problems discussed above or elsewhere, or to at least provide an alternative system and / or method.

[0007] The following overview is about some aspects. These aspects are only aspects of the present invention, wherein the systems and methods are as defined in the appended claims. The reader will understand that features of aspects that do not fall within the scope of the present invention may still be included in aspects of the present invention that do fall within the scope of the present invention. Therefore, any aspects lacking these features are reserved to provide the reader with useful background information.

[0008] A first aspect provides a system for displaying information to a user of a vehicle using augmented reality (AR), the system comprising a processor configured to: interact with a display unit for displaying information using AR in a field of view associated with the system, a vehicle operation data source for providing vehicle operation data to the processor, and a multimedia data source for providing multimedia data to the processor; process an AR image for display on the display unit, the AR image being based on the vehicle operation data and the multimedia data; and cause the display unit to display the AR image.

[0009] In one example, the processor is configured to: process the AR image for display on the display unit, the AR image having a first component, wherein the first component includes applying spatial and temporal changes based on one of the multimedia data.

[0010] In one example, the applied spatial and temporal variations are based on one or more of the following: volume information in the multimedia data; timing information in the multimedia data; frequency information in the multimedia data.

[0011] In one example, the applied spatial and temporal variation is provided in a first dimension. In one example, the first dimension is a vertical height.

[0012] In one example, the processor is configured to: process the AR image for display on the display unit, the AR image having a second component.

[0013] In one example, the second component of the AR image includes one or more image portions.

[0014] In one example, an attribute of the one or more image portions is determined based on the vehicle operation data and the multimedia data. In one example, the attribute is a dimension determined based on the vehicle operation data; and / or the attribute is a display duration of the one or more image portions determined based on the multimedia data. In one example, the display duration is determined based on one or more of the following: lyric timing data in the multimedia data; grammatical and / or pronunciation features of a lyric in the multimedia data; current track timing in the multimedia data.

[0015] In one example, the second component is a text feature.

[0016] In one example, the second component is positioned based on: the distance at which the user should begin interacting with the second component, or the time at which the user should begin interacting with the second component; the distance at which the user should have interacted with the second component, or the time at which the user should have interacted with the second component; and / or a starting position of an animation for removing the second component or an image portion thereof and / or a stop position of an animation for removing the second component or an image portion thereof and / or a starting time of an animation for removing the second component or an image portion thereof and / or a stop time of an animation for removing the second component or an image portion thereof.

[0017] In one example, the processor is configured to process the AR image for display at a visually fixed position relative to a feature in the field of view when the vehicle operating data indicates that the vehicle is above a critical vehicle operating parameter.

[0018] In one example, the processor is configured to process for displaying the AR image at an apparent time-varying position relative to a feature in the field of view when the vehicle operating data indicates that the vehicle is below a critical vehicle operating parameter.

[0019] Another aspect provides a system for displaying information to a user of a vehicle using augmented reality (AR), the system comprising a processor configured to: interact with a display unit for displaying information using AR in a field of view associated with the system and a multimedia data source for providing multimedia data to the processor; process an AR image for display on the display unit, the AR image being based on the multimedia data; and cause the display unit to display the AR image.

[0020] Another aspect provides a system for displaying information to a user of a vehicle using augmented reality (AR), the system comprising a processor configured to: interact with a display unit for displaying information using AR in a field of view associated with the system, a first data source for providing first data to the processor, and a second data source for providing multimedia data to the processor; process an AR image for display on the display unit, the AR image being based on the first data and the multimedia data; and cause the display unit to display the AR image.

[0021] In one example, the first data are vehicle operation data, user data, a user profile, a vehicle profile, environmental data, field of view characteristic data, or other data.

[0022] Another aspect provides a head-up display (HUD) system comprising a system according to any system aspect.

[0023] Another aspect provides an audio / visual (AV) system comprising a system or HUD system according to any of the system or HUD system aspects, the AV system configured to provide an AV output associated with the multimedia data provided by the multimedia data source.

[0024] Another aspect provides a vehicle comprising a system, a HUD system, or an AV system according to any of the system, HUD system, or AV system aspects.

[0025] In one example, the vehicle is an autonomous vehicle.

[0026] Another aspect provides a method for displaying information to a user of a vehicle using augmented reality (AR), the method comprising: interacting with a display unit for displaying information using AR in a field of view, a vehicle operation data source for providing vehicle operation data to the processor, and a multimedia data source for providing multimedia data to the processor; processing an AR image for display on the display unit, the AR image being based on the vehicle operation data and the multimedia data; and causing the display unit to display the AR image.

[0027] Another aspect provides a computer comprising: a processor and a memory configured to implement a method according to the aspect; a computer program comprising instructions that, when executed by a computer comprising a processor and a memory, cause the computer to implement a method according to a method aspect; or a non-transitory computer-readable storage medium comprising instructions that, when executed by a computer comprising a processor and a memory, cause the computer to implement a method according to a method aspect.

[0028] Of course, it should be understood that the features described in relation to one aspect of the present invention may be included in other aspects of the present invention. For example, the method of any aspect of the present invention may include any feature described in relation to the device of any aspect of the present invention, and vice versa.

[0029] Other preferred and advantageous features of the invention will become apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] For a better understanding of the invention, and in order to show how embodiments of the invention may be practiced, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0031] Figure 1 A schematic diagram showing a system for displaying information to a user of a transportation vehicle using augmented reality (AR);

[0032] Figure 2 Display the field of view of the user and the display unit;

[0033] Figure 3 A schematic diagram showing a head-up display (HUD) system;

[0034] Figure 4 Displaying a first view, wherein the AR image is displayed;

[0035] Figure 5 displaying a second view in which the AR image is displayed;

[0036] Figure 6 displaying a third view in which the AR image is displayed;

[0037] Figure 7 displaying a fourth view in which the AR image is displayed;

[0038] Figure 8 displaying a fifth view in which the AR image is displayed;

[0039] Fig. 9 displaying a sixth view, in which the AR image is displayed;

[0040] Fig.10 Displaying a seventh view, in which the AR image is displayed;

[0041] Fig.11 Displaying an eighth view, in which the AR image is displayed;

[0042] Fig.12 Displaying a ninth view, in which the AR image is displayed;

[0043] Fig.13 Display including system or HUD system of the vehicle; and

[0044] Fig.14 Shows general methodological principles. DETAILED DESCRIPTION

[0045] According to the present invention, there is provided a system, a heads-up display system, a method, a computer program and a non-transitory computer readable storage medium as set out in the accompanying claims. Other features of the present invention will become apparent from the dependent claims and the subsequent description.

[0046] The following description describes some embodiments. These embodiments are merely embodiments of the present invention, wherein the systems and methods are as defined in the subsequent claims. The reader will understand that features of embodiments that do not fall within the scope of the present invention may still be included in embodiments of the present invention that do fall within the scope of the present invention. Therefore, the description of embodiments lacking these features is reserved to provide the reader with useful background information.

[0047] refer to Figure 1 , showing a system 10 for displaying information to a user of a transportation vehicle using augmented reality (AR). The system 10 includes a processor 100. The processor 100 is configured to interact with a display unit 200. The processor 100 is configured to interact with a transportation vehicle operation data source 300. The processor 100 is configured to interact with a multimedia data source 400. The processor 100 is configured to interact with the display unit 200, the transportation vehicle operation data source 300, and the multimedia data source 400 may mean that the processor 100 is configured to receive information from the display unit 200, the transportation vehicle operation data source 300, and the multimedia data source 400 and transmit information to the display unit 200, the transportation vehicle operation data source 300, and the multimedia data source 400. Specifically, the transportation vehicle operation data source 300 and the multimedia data source 400 are each configured to provide data to the processor 200. The processor 100 is configured to receive data from each of the transportation vehicle operation data source 300 and the multimedia data source 400, respectively.

[0048] As will be described in further detail below, the system 10 enables the displayed AR images to be well matched to the user's visual sensory perception of real-world features or objects that can be seen during the operation of the vehicle because the AR images are based on the vehicle operation data and multimedia data. In the case of implementation in an autonomous vehicle, the AR images that match the external environment of the vehicle enable the AR images to be used as a form of instrumentation for the user. Whether consciously or subconsciously, the user can use the AR images to understand that the vehicle is, for example, maintaining a constant speed, accelerating, braking, and / or changing direction. This means that the user can receive and process technical information without interacting with a dedicated display or interface (e.g., a speedometer or accelerometer). This means that the user has synergy. The system also has synergy, in which fewer dedicated technical displays or interfaces may be required.

[0049] Furthermore, where the AR image includes content such as multimedia content, the content is well matched to the user's perception of real-world features, thereby improving readability, usability, and user engagement. This is facilitated by processing the AR image based on vehicle operating data to ensure that, for example, changes in the vehicle's speed or position can be included in the display of the AR image. Ensuring user interaction with the autonomous vehicle is important from a safety and redundancy perspective.

[0050] Furthermore, by basing the AR images on the vehicle operation data and the multimedia data, the multimedia content displayed to the user can be positioned, resized, and provided at an appropriate time period based on the vehicle operation. This ensures that the user can easily read or otherwise interact with the multimedia content.

[0051] Here, the "user of the transport" is typically the driver or operator of the transport. However, the user may be one or more other users of the transport, for example, a passenger. In one example, the transport may be operated in a fully autonomous mode (including a mode without any direct control from the user of the transport (e.g., the driver)). In this case, the driver may indeed be a passenger in the transport. In fully autonomous mode, despite not controlling the transport himself, the system 10 (and / or the HUD system 50 described below) can enhance the field of view by displaying AR images on the display unit 200 to encourage the user to keep his attention on the road ahead. This can be called "gamification" of the display. Even more, if the transport is not driven or controlled, games can be played while using the transport. Games may involve the use of AR images.

[0052] For the avoidance of doubt, the system 10 can be modified in accordance with a vehicle comprising a display unit 200, a vehicle operating data source 300 and a multimedia data source 400. In this case, the system 10 itself need not comprise the display unit 200, the vehicle operating data source 300 and the multimedia data source 400. However, in an alternative configuration not involving modification, the system 10 may include the display unit 200, the vehicle operating data source 300 and the multimedia data source 400. This may be a physical modification or a software update or upgrade.

[0053] The display unit 200 is used to display information using AR in a field of view (FoV) associated with the system 10. The display unit 200 may include a holographic display unit or may be in the form of a holographic display unit. A holographic display is a display that uses light diffraction to produce a virtual three-dimensional image. The holographic display unit may include a combiner. The combiner may be a windshield (i.e., all or part of a windshield). That is, the present system 10 is an optical see-through system. Optical see-through systems allow users to "directly" view the real world. These AR systems add virtual content in the form of AR images by adding additional light to the light from the real world.

[0054] refer to Figure 2 , showing a plan view of an exemplary FoV of a user and a FoV associated with the system 100. The FoV of a user (referred to herein as the "user FoV") is indicated at 20, and the FoV associated with the system 100 is indicated at 30.

[0055] In this example, the FoV associated with the system 10 is the FoV of the display unit 200. As shown, the user's FoV is greater than the FoV of the display unit 200. Typically, the user's FoV is in the range of 120 to 200 degrees. The FoV of the display unit has a vertical FoV of approximately 4 degrees and a horizontal FoV in the range of 15 to 20 degrees. As can be understood from this figure, the display unit 200 uses AR to display information in the user's FoV.

[0056] In the description herein, "the FoV of the display unit" means the FoV of the augmentable area. This is the portion of the human field of view in which the display unit 200 can display virtual content to enhance the real-world scene. That is, the FoV of the display unit 200 is the area that is viewable by the user and in which the display unit 200 can enhance the user's real-world scene. In this sense, the FoV of the display unit can be referred to as the "field of view of the user using the system 10" or the "field of view of the user using / viewing the display unit 200". In a HUD system (e.g., the HUD system 50), this can be referred to as the "field of view of the user using the HUD system 50".

[0057] In the following description, the FoV of the display unit 200 will be simply referred to as “FoV”.

[0058] It can be understood that the display unit 200 is located in an area that can be viewed by the user. In a specific example, this means that the display unit 200 is located at or in front of the windshield (that is, facing the user), and the user views the road ahead through the windshield when using the transportation vehicle. In other examples, the display unit 200 can be set on the side or rear of the transportation vehicle, so the FoV can face the side or rear of the transportation vehicle.

[0059] In the figure, the user's FoV and the FoV of the display unit 200 are shown as rectangular FoVs. It can be understood that, in reality, such FoVs are not rectangular.

[0060] As described above, the vehicle operation data source 300 is used to provide data to the processor 100. Similarly, the multimedia data source 400 is used to provide data to the processor 100. The processor 100 is configured to display information to the user at the display unit 200. The information provided to the user is based on the data provided by the vehicle operation data source 300 and the multimedia data source 400. The information provided to the user may be based on, dependent on, or representative of the data provided by the vehicle operation data source 300 and the multimedia data source 400.

[0061] Each of the vehicle operation data source 300 and the multimedia data source 400 can be a system, a component, or can be a data storage. For example, the vehicle operation data source 300 can be a speedometer and / or an accelerometer. The vehicle operation data can be provided directly to the processor 100 by the vehicle system or component. That is, the vehicle operation data source 300 can be a system or component of a vehicle configured to provide vehicle operation data (e.g., speed, acceleration, and / or direction of travel data). Alternatively, the vehicle operation data source 300 can be a global positioning system (GPS), a mapping system / service, or other navigation system. Alternatively, the data source 300 can be a component such as a detector or a memory. The detector can be a camera, a radar, and / or a lidar. Alternatively, the data source 300 can be a data storage such as a data table. The data source 300 can include one or more systems, components, or data storage.

[0062] The vehicle operation data provided to the processor 100 may be speed data, acceleration data, direction of travel data, GPS data, navigation data, route data and / or feature data. The navigation data may be in accordance with the Navigation Data Standard (NDS).

[0063] GPS data may include a set of geographic coordinates received from a mapping service. Navigation data and route data may include data points received from a navigation or set-course system. Navigation data and route data may include a set of data points associated with an intended path of a vehicle.

[0064] The vehicle operating data may be information relating to the dynamics of the vehicle, such as the current and / or future speed of the vehicle and / or the current and / or future acceleration of the vehicle; direction of travel data; position in a lane or relative to road signs or features; road surface features; vehicle position; visible distance boundaries; moving route position; and / or position corrections.

[0065] The feature data may be related to features detected in the environment outside the vehicle. The feature may include another vehicle, for example, a road vehicle or an off-road vehicle. The feature may also include road signs or general road facilities. The feature may be a person or an animal. The feature may include the layout of a road or the layout of road lanes. The feature data may be related to all of these features. The features to which the feature data relates may vary depending on the use or mode or type of system, vehicle or scenario.

[0066] Here, the “environment outside the vehicle” may include features in front of the vehicle, on the side of the vehicle, and behind the vehicle. In other words, the feature data may be related to features outside the FoV.

[0067] In one embodiment, the vehicle operation data provided by the vehicle operation data source 300 is related to the movement of the vehicle, the position of the vehicle relative to the lane, and / or environmental characteristics.

[0068] In one embodiment, the multimedia data provided by the multimedia data source 400 is related to lyrics timing data, grammatical features in lyrics or text, and / or track timing. As described above, the multimedia data source 400 can be a system, a component, or a data storage. The multimedia data source 400 can be a multimedia system, a component such as a memory or a data provider in a multimedia system, or a storage of multimedia data.

[0069] The processor 100 is configured to process an AR image for display on the display unit 200. An AR image is an image displayed using augmented reality (AR). The processor 100 is configured to process an AR image in a number of different ways as will be described in further detail below.

[0070] The processor 100 is configured to cause the display unit 200 to display an AR image.

[0071] Although the processor 100 may cause the display unit 200 to display an AR image in actual implementation, it may be considered optional for some embodiments. For example, the core distinguishing feature is the actual processing of the AR image based on the transportation tool operation data and the multimedia data. In terms of the key features that distinguish the present invention, causing the display unit 200 to display an AR image may be optional.

[0072] Of course, those skilled in the art may understand how an AR image may be displayed on the display unit 200 , and the above provides an illustrative example of how an AR image may be displayed on the display unit 200 .

[0073] refer to Figure 3, a head-up display (HUD) system 50 is shown. The HUD system 50 includes a picture generation unit 500, a combiner unit 600, and a corrector unit 700.

[0074] Combiners and correctors may be used, for example, in holographic systems or other systems that present a see-through surface to a user but in which image features are provided by the system. A combiner may be, for example, an at least partially (e.g., semi-) transparent surface used in a system to overlay an image presented by a projector (which may be part of or separate from the image generating unit 500) onto the user's physical world. The combiner is at least partially transparent and allows the user to see through, while reflecting or otherwise directing light (e.g., AR functionality) to the user. Corrector units may be used to correct aberrations, filters, and / or improve light utilization efficiency, or generally correct for some optical property. In some examples, one or more optical devices or lenses may be used that include filters.

[0075] The system 10 described herein may be incorporated into a HUD system 50. The system 10 and / or the HUD system 50 may be incorporated into or form part of an advanced driver assistance system (ADAS).

[0076] Although the present disclosure is related to the system 10 and HUD system 50 for installation in a vehicle, the present disclosure can also be applied in the context of a head-mounted display (HMD). For example, the system 10 can be incorporated into a pair of smart glasses or other wearable devices.

[0077] Furthermore, the system 10 or the HUD system 50 can be incorporated into an audio / video (AV) system. That is, the AV system can include the system 10 or the HUD system 50 according to the embodiments described herein. The AV system can be connected to or include a multimedia data source according to the embodiments described herein. The AV system is configured to provide an AV output related to the multimedia data provided by the multimedia data source. In an advantageous embodiment, the AV output includes a soundtrack, for example, a music track. The AR image output to the user may include text features related to the words or lyrics of the soundtrack. The soundtrack can be provided to the user by the AV system at the same time. For this purpose, the AV system includes a speaker and an optional amplification device. In some embodiments, the audio system can be connected to the system 10 or the HUD system 50, the system 10 or the HUD system provides a video output, and the audio system provides an audio output. In this way, the AV system can be a karaoke system combined with AR or constitute a part thereof.

[0078] Figures 4 to 12Each shows a front view through the vehicle windshield, wherein the AR image is displayed in the FoV. The operation and functionality of the system 10 will be described with reference to these figures. In each figure, an AR image is displayed, wherein the AR image is based on vehicle operation data from the vehicle operation data source 300 and multimedia data from the multimedia data source 400.

[0079] In one embodiment, the processor 100 is configured to generate an AR image. That is, the processor 100 itself generates the AR image. In this way, additional components for generating AR images may not be required.

[0080] In another embodiment, the processor 100 is configured to receive an AR image. That is, the processor 100 itself does not need to generate the AR image. This can reduce the computational load on the processor 100 while allowing dedicated components to generate the AR image.

[0081] In general, applicable to all figures, a system 10 is described in which multimedia data is output to a user of the system 10. In the example shown, the multimedia data is a soundtrack (e.g., a music track). When the soundtrack is played, words or lyrics are displayed to the user in the form of an AR image containing text information. The words or lyrics are placed above the road surface and provided in an image portion of appropriate length, and are displayed for a duration in which the user can read the words or lyrics while the soundtrack is playing. An AR image component similar to an equalizer is also displayed to the user to illustrate the properties of the soundtrack (e.g., beat, track timing, or frequency). Such a system engages the user of the vehicle and also acts as a form of instrumentation to allow the user to determine the dynamics of the vehicle. For example, when the vehicle is stationary or below a critical vehicle operating parameter, the AR image can have a time-varying position relative to a feature in the field of view. Alternatively, when the vehicle is moving or above a critical vehicle operating parameter, the AR image can have a fixed position relative to a feature in the field of view. In this way, interaction with the user of the vehicle is facilitated, and the user can use the AR image as a form of instrumentation to understand the dynamics of the vehicle. Further details of the operation of the system 10 and the functions of the processor 100 are described below.

[0082] This means that the user can interact with the AR image displayed on the display unit 200. Interacting with the AR image can include, for example, reading the AR image or text features contained in the AR image. The user can read or sing the lyrics after interacting with the AR image. That is, in one embodiment, the system 10 can be a karaoke system that incorporates AR or constitutes a part thereof.

[0083] In another exemplary embodiment, the user may interact with the AR image displayed on the display unit 200 so that the AR image changes its appearance. For example, the user may interact with the AR image by selecting the AR image. This may be accomplished by using a remote device, a vehicle component, or by monitoring the movement of a body part (e.g., a hand or arm) of the user. Such functionality may be incorporated into an AR game, thereby facilitating the user to keep their attention on the road ahead, regardless of the fact that they themselves are not controlling the vehicle. This may allow the user to interact with the environment by controlling the AR game environment. As described above, this may be referred to as "gamification" of the display. Furthermore, if the vehicle is not driven or controlled, the game may be played while using the vehicle.

[0084] refer to Figures 4 to 7 , showing a first view, a second view, a third view and a fourth view, wherein an AR image 4110 is shown. The user FoV is indicated by 4000. The FoV (ie, the FoV of the display unit 200) is indicated by 4100.

[0085] The processor 100 is configured to process the AR image 4110 for display on the display unit 200 based on the vehicle operation data and multimedia data from the data sources 300 and 400. In this way, because the AR image 4110 is based on the vehicle operation data and multimedia data, the AR image 4110 is well matched to the visual sensory perception of the real world features or objects that the user can see during the operation of the vehicle. When implemented in an autonomous vehicle, the AR image 4110 matches the external environment of the vehicle so that the AR image 4110 can be used as a form of instrumentation for the user. Whether consciously or subconsciously, the user can use the AR image 4110 to understand that the vehicle is, for example, maintaining a constant speed, accelerating, braking, and / or changing direction. This means that the user can receive and process technical information without interacting with a dedicated display or interface (e.g., a speedometer or accelerometer). This means that the user has a synergy. The system also has a synergy, in which fewer dedicated technical displays or interfaces may be required.

[0086] Furthermore, where the AR image 4110 includes content such as multimedia content, the content is well matched to the user's perception of real-world features, thereby improving readability, usability, and user engagement. This is facilitated by processing the AR image based on the vehicle operating data to ensure that, for example, changes in the vehicle's speed or position can be included in the display of the AR image 4110. Ensuring user interaction with the autonomous vehicle is important from a safety and redundancy perspective.

[0087] Furthermore, by basing the AR images on the vehicle operation data and the multimedia data, the multimedia content displayed to the user can be positioned, resized, and provided at an appropriate time period based on the vehicle operation. This ensures that the user can easily read or otherwise interact with the multimedia content.

[0088] The processor 100 is configured to process an AR image 4110 for display on the display unit 200, the AR image 4110 having a first component 4112. In addition, the processor 100 is configured to process an AR image for display on the display unit, the AR image 4110 having a second component 4114. Figures 4 to 7 Each displays an AR image including the first component and the second component. Figures 8 to 12 Each shows an AR image containing only the second component, but it can be easily understood that the AR image of the drawings may further contain the first component as described herein.

[0089] The first component 4112 includes applying spatial and temporal changes. Applying spatial and temporal changes is based on multimedia data from the multimedia data source 400.

[0090] In one embodiment, the processor 100 is configured to generate an AR image including applying spatial and temporal changes. That is, the processor 100 itself generates the AR image. In this way, no additional components for generating the AR image may be required.

[0091] In another embodiment, the processor 100 is configured to receive an AR image including application spatial and temporal changes. That is, the processor 100 itself does not need to generate the AR image. This can reduce the computational load on the processor 100 while allowing dedicated components to generate the AR image.

[0092] The spatial and temporal variations may be referred to as "lightning bolt appearance", "lightning bolt variation", or "lightning bolt manner" or may take the form of "lightning", "lightning variation", or "lightning manner". The first component 4112 may be located in the FoV 4100 based on, for example, the expected path of travel of the vehicle. An example of such a fit is a "route spline". The route spline may be based on waypoints, lane markings, or boundaries.

[0093] The applied spatial and temporal variation of the first component 4112 is based on one or more of: volume information (e.g., sound level or intensity) in the multimedia data; timing information in the multimedia data; frequency information in the multimedia data. In some cases, these may be generally referred to as "multimedia dynamics" or "sound dynamics".

[0094] In a particular embodiment, the multimedia data source 400 provides information about a music track to the processor 100. The processor 100 processes the AR image based on the information in the multimedia data. Specifically, the multimedia data may include volume information. The application space and time variation of the first component 4112 is provided in a first dimension. As seen by the user, the first dimension can be a horizontal dimension or amplitude variation, or can be a vertical dimension or amplitude variation. Preferably, the first dimension is a vertical amplitude variation.

[0095] In one example, an increase in the volume of the music track (as indicated by the multimedia data) results in an increase in the vertical amplitude of the applied spatial and temporal variation of the first component 4112. Thus, a decrease in the volume of the music track (as indicated by the multimedia data) results in a decrease in the vertical amplitude of the applied spatial and temporal variation of the first component 4112.

[0096] In another example, the multimedia data may include timing information. Furthermore, as described above, the application spatial and temporal variations of the first component 4112 are provided in the first dimension.

[0097] In one example, the applied spatial and temporal variation of the first component 4112 can be based on timing information, such as the beats per minute (BPM) of a music track. An increase in the BPM of the music track (as indicated by the multimedia data) results in an increase in the frequency of the maximum vertical amplitude of the applied spatial and temporal variation of the first component 4112. In one example, the frequency of the maximum vertical amplitude of the first component 4112 corresponds to the BPM of the music track.

[0098] In one example, the applied spatial and temporal variations of the first component 4112 can be based on frequency information. An increase in the frequency or pitch of the music track (as indicated by the multimedia data) results in an increase in the maximum vertical amplitude of the applied spatial and temporal variations of the first component 4112.

[0099] Generally, the first component 4112 can be considered to act as an "equalizer" that provides a visible indication of the multimedia material, or in one specific example, a music track.

[0100] When the vehicle is moving forward in motion, the user perceives that they are moving toward the first component 4112 of the AR image 4110. At the same time, the first component 4112 has spatial and temporal changes as described above. In this way, the user can understand that the vehicle is moving, and therefore, the first component 4112 can be used as a form of instrument. Accordingly, when the vehicle is stationary, the first component 4112 of the AR image 4110 can be provided at a fixed position relative to the features outside the vehicle. The first component 4112 can only change in vertical height, for example. Therefore, the user can understand that the vehicle is stationary by referring to the first component 4112.

[0101] The processor 100 is configured to process an AR image for display on a display unit, wherein the AR image 4110 has a second component 4114. The second component 4114 includes one or more image parts 4114a, 4114b, 4114c. Figure 4 and 5 In FIG. 4 , it can be seen that the AR image 4110 includes two image parts 4114a and 4114b. Figure 6 and 7 It can be seen that the AR image 4110 includes two image parts 4114b and 4114c.

[0102] Providing image portions such as image portions 4114a, 4114b, 4114c allows multimedia data to be divided into portions of length suitable for user interaction. Furthermore, the image portions can be separated to provide the user with an instrument by illustrating the operation of the vehicle, which enables the user to understand the vehicle dynamics by referring to the image portions. The image portions can be spaced apart in the direction in which the vehicle is traveling. That is, the image portions can be spaced apart in a direction extending away from the vehicle.

[0103] As shown, the second component 4114 is a text feature. For example, the second component 4114 can be an image showing words or lyrics of a music track, a movie, an audio book, or other multimedia.

[0104] The processor 100 is configured to process the AR image 4110 (including the first component 4112 and the second component 4114) to position them in the field of view according to various parameters, including:

[0105] Current vehicle motion from vehicle operation data, e.g. speed;

[0106] Timing information in multimedia data;

[0107] ·Grammar and pronunciation features in multimedia materials;

[0108] Current timing in multimedia materials, for example, audio track timing;

[0109] Lane detection data from transportation vehicle operation data;

[0110] Road surface detection data in transportation operation data;

[0111] The location of the transport in the transport operation data;

[0112] · visible distance boundaries in the vehicle operation data; and

[0113] · Movement route position and position correction in the transport operation data.

[0114] Specifically, the attributes of the image portions 4114a, 4114b, 4114c are determined based on the vehicle operation data and the multimedia data. In this way, the user can consciously or subconsciously understand the vehicle operation from the attributes of the image portions 4114a, 4114b, 4114c. In one embodiment, the attribute is a dimension determined based on the vehicle operation data. The dimension can be the length, width or height of the image portions 4114a, 4114b, 4114c. For example, when the vehicle speed is high, the width of the image portions 4114a, 4114b, 4114c can be greater than the width when the vehicle speed is relatively low. In addition, in the embodiment, the attribute can also be the duration (i.e., the length of time) used to display the image portions 4114a, 4114b, 4114c. The duration is based on the multimedia data. The duration is determined based on one or more of the following: lyrics timing data; grammatical and pronunciation features of the lyrics; current track timing.

[0115] The second component 4114 can be positioned according to some factors. In one embodiment, the second element 4114 is positioned according to the distance at which the user should start interacting with the second component 4114 or the time at which the user should start interacting with the second component 4114. That is, the second component 4114 can be displayed at an appropriate visual distance for the user to interact with the second component 4114 (e.g., read), and can be displayed at an appropriate time for the user to interact with the second component 4114. This distance and / or time can be based on multimedia data. In this way, the system 10 ensures that the user can interact with the second component 4114 and understand the second component 4114. The user can interact with the second component 4114 by reading the second component 4114, or in the system 10 for the karaoke system incorporated with AR, sing in response to reading the second component 4114 to interact with the second component 4114.

[0116] In another embodiment, the second component 4114 is positioned based on the distance at which the user should have interacted with the second component 4114 or the time at which the user should have interacted with the second component 4114. That is, the second component 4114 can be located at a visual position where it will be removed from the display. It is expected that the user has interacted with the second component 4114 (e.g., read), so it is appropriate to remove the second component 4114 from the display to avoid unnecessary AR image parts distracting the user. Likewise, in this way, the system 10 ensures that the user can interact with and understand the second component 4114.

[0117] In another embodiment, the second component 4114 is positioned according to the start position of the animation for removing the second component 4114 or its image portion 4114a, 4114b, 4114c and / or the stop position of the animation for removing the second component 4114 or its image portion 4114a, 4114b, 4114c and / or the start time of the animation for removing the second component 4114 or its image portion 4114a, 4114b, 4114c and / or the stop time of the animation for removing the second component 4114 or its image portion 4114a, 4114b, 4114c. In this way, the second component 4114 or its image portion 4114a, 4114b, 4114c can be removed from the display through a disappearing animation. This may include a "fade-out" animation. Removing AR images in this way can increase user engagement and system clarity.

[0118] The processor 100 is configured to change an attribute or characteristic of the second component 4114 or a portion thereof of the AR image 4110 at a distance or time at which the user wants to interact with the AR image 4110. That is, for example, when the user wants to read a text feature of the second component 4114, the second element 4114 may be highlighted in a different color or enlarged in size.

[0119] The processor 100 is configured to process an AR image for display at a fixed position relative to a feature in the field of view when the vehicle operating data indicates that the vehicle is above a critical vehicle operating parameter. Figure 4 and 5 as well as Figure 6 and 7 In these figures, the vehicle is maintaining a substantially constant speed above the critical speed. Specifically, the vehicle is traveling on a highway / highway. The second component 4114 is displayed at a visually fixed position relative to the feature in the user's field of view. Displaying at a visually fixed position relative to the feature in the user's field of view means that the second component 4114 appears to track or follow the feature regardless of the movement of the vehicle.

[0120] By processing the AR image in the manner described when the vehicle is above a critical vehicle operating parameter, a user can determine that the vehicle is above a critical vehicle operating parameter simply by referring to the AR image. For example, a user can determine that the vehicle is above a critical speed by confirming that the AR image is displayed at a fixed position relative to features in the field of view regardless of the vehicle's motion. This is particularly advantageous in an autonomous vehicle setting where the user does not control the vehicle's operation.

[0121] from Figure 4 It can be seen that image portion 4114a is displayed at a position relative to the environment and at Figure 5 , image portion 4114a remains at that (fixed) position relative to the environment, while the vehicle has moved towards the apparent position of image portion 4114a.

[0122] Likewise, from Figure 6 It can be seen that image portion 4114b is displayed at a position relative to the environment and at Figure 7 , image portion 4114b remains at that (fixed) position relative to the environment, while the vehicle has moved towards the apparent position of image portion 4114b.

[0123] refer to Figures 8 to 12 , showing the fifth, sixth, seventh, eighth and ninth views, in which the AR image 8110 is displayed.

[0124] The user FoV is represented by 8000. The FoV (ie, the FoV of the display unit 200) is represented by 8100.

[0125] The processor 100 is configured to process an AR image 8110 for display on the display unit 200 based on the transportation tool operation data and multimedia data from the data sources 300 , 400 .

[0126] about Figures 4 to 7 The description provided for processing AR image 4110 is also relevant and applicable to describing Figures 8 to 12 Specifically, the processor is configured to process the AR image 8110 for display on the display unit 200 based on the transportation tool operation data and multimedia data from the data sources 300 and 400. As described above, although Figures 8 to 12 Each shows an AR image 8110 including only the second component, but it can be easily understood that the AR image of the figure may further include the first component as described herein.

[0127] The processor 100 is configured to process an AR image for display on a display unit, the AR image 8110 having a second component 8114. The second component 8114 includes one or more image parts 8114a, 8114b, 8114c.

[0128] like Figures 8 to 12 As shown, the processor 100 is configured to process an AR image 8110 for display at an apparent time-varying position relative to a feature in the field of view when the vehicle operating data indicates that the vehicle is below a critical vehicle operating parameter.

[0129] exist Figure 8 , the vehicle is slowing down, but is traveling at a speed above the critical speed. Specifically, the vehicle is traveling in a residential area and is approaching a stop sign. Image portions 8114a, 8114b, 8114c are displayed at visually fixed locations relative to features in the user's field of view in a manner similar to that described above.

[0130] exist Fig. 9 , the vehicle has reached the stop sign and has slowed down to below the critical speed. In this case, the vehicle is stationary at the stop sign. Image portions 8114a, 8114b, 8114c are displayed at apparent time-varying positions relative to features in the user's field of view. That is, image portions 8114a, 8114b, 8114c are displayed at a first visual position and then move toward the user to a second visual position based on multimedia data (e.g., timing information when the user wants to interact with a particular image portion 8114a, 8114b, 8114c). Therefore, the AR image is displayed at an apparent time-varying position relative to the features in the field of view.

[0131] By processing the AR image in the manner described when the vehicle is below a critical vehicle operating parameter, a user can determine that the vehicle is below a critical vehicle operating parameter simply by referring to the AR image. For example, a user can determine that the vehicle is below a critical speed by confirming that the AR image is displayed at an apparent time-varying position relative to features in the field of view. This is particularly advantageous in an autonomous vehicle setting where the user does not control the vehicle operation.

[0132] In this way, when the processor 100 determines that the operating parameter of the transport vehicle (e.g., speed) is below a critical value, the image portion of the second component 8114 may still be displayed to the user in a visible manner, thereby allowing the user to interact with the AR image. Interacting with the AR image may, for example, include reading the AR image, in particular reading text features of the image portion. The user may read the AR image and sing in response to reading the AR image, so that a karaoke function may be provided in this way. The user of the system 10 may interpret the time-varying position of the AR image as indicating that the operating parameter of the transport vehicle is below a critical level. That is, for example, by referring only to the AR image 8110, the user may understand that the transport vehicle is slowing down to a stop or is stationary.

[0133] exist Fig.10 The vehicle remains at the stop sign. The image portion moves toward the user until Figures 9 to 11 That is, the image parts are each moved to their final positions before being removed from the field of view via the fading animation.

[0134] exist Fig.11 In the example, the vehicle remains at the stop sign. Furthermore, the image portion moves toward the user to the final position. By comparing Fig.10 and 11 , it can be seen how image portion 8114f is larger than image portion 8114e (in Fig.10 The visual distance of the vehicle is moved to a visual distance closer to the vehicle.

[0135] exist Fig.12 , the vehicle accelerates from a stop sign. As determined by the processor based on the vehicle operating data, image portions 8114h, 8114i of second component 8114 are again displayed at visually fixed locations relative to features in the user's field of view.

[0136] In summary, a system 10 for displaying information to a user of a vehicle using augmented reality (AR) is described. The system 10 includes a processor 100 configured to: interact with a display unit 200 for displaying information using AR in a field of view associated with the system 10, a vehicle operation data source 300 for providing vehicle operation data to the processor, and a multimedia data source 400 for providing multimedia data to the processor; process an AR image for display on the display unit 200, the AR image being based on the vehicle operation data and the multimedia data; and cause the display unit 200 to display the AR image. In this way, content that the user wants to interact with can be displayed to the user based on the vehicle operation data and the multimedia data. In this way, user engagement can be increased. The user can interact with the system 10 by reading and chanting text features of the AR image, and thus provide a karaoke system combined with AR. Furthermore, the user can consciously or subconsciously use the AR image as a form of technical instrument to understand the dynamics of the vehicle, such as speed, acceleration, or position change. This is particularly advantageous in autonomous vehicle implementations where the user does not control the vehicle dynamics, where providing such AR images to the user can provide visible feedback or indication of the vehicle dynamics.

[0137] Although the description of the invention provided herein describes an advantageous embodiment of a system, wherein the system includes a transport operation data source 300 for providing transport operation data to the processor 100, and processing an AR image, the AR image being based on the transport operation data, in an exemplary embodiment of the system 10 associated with the described embodiment, the transport operation data source 300 may not be present, and thus the AR image may not be based on the transport operation data. Alternatively, in one example, the processor 100 is configured to interact with a multimedia data source 400 (i.e., without the transport operation data source 300), and process an AR image for display on a display unit, the AR image being based on the multimedia data. In this way, an AR image output that is not linked to the transport operation data can be provided, but an AR multimedia output system is provided. In another example, the processor 100 is configured to interact with a data source for providing any operation data or parameters to the processor 100, and the AR image can be based on the operation data or parameters. It can be perceived that the data source can be a transport operation data source, which is advantageous for the reasons provided above. Alternatively, the data source may be any other data source that, for example, provides information related to a user, a user profile, a vehicle profile, the environment, features in the field of view, or other information.

[0138] refer to Fig.13 , showing a vehicle 13000. The vehicle 13000 includes, for example, the system 10 or the HUD system 50 as described above.

[0139] refer to Fig.14 , showing a method for displaying information to a user of a transportation vehicle using augmented reality (AR). Step S14100 includes interacting with a display unit for displaying information using AR in a field of view, a transportation vehicle operation data source for providing transportation vehicle operation data to a processor, and a multimedia data source for providing multimedia data to the processor. Step S14200 includes processing an AR image for display on the display unit, the AR image being based on the transportation vehicle operation data and the multimedia data. Step S14300 includes causing the display unit to display the AR image.

[0140] Although step S14300 may be implemented in actual implementation, it may be considered optional for some embodiments. For example, the core distinguishing feature is the processing step. In terms of the key features that distinguish the present invention, the actual display step may be optional.

[0141] limited

[0142] While preferred embodiments have been shown and described, it will be apparent to those skilled in the art that various changes and modifications may be made without departing from the scope of the invention as defined in the appended claims and described above.

[0143] All features disclosed in this specification (including any accompanying claims, abstract and drawings) and / or all steps of any method or process disclosed as such may be combined in any combination, except combinations in which at least some of these features and / or steps are mutually exclusive.

[0144] Unless otherwise expressly stated, each feature disclosed in this specification (including any attached claims, abstracts and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose. Therefore, unless otherwise expressly stated, each feature disclosed is only an example of a series of general equivalent or similar features.

[0145] The present invention is not limited to the details of the foregoing embodiments. The present invention extends to any novel feature or any novel combination of features disclosed in this specification (including any attached claims, abstract and drawings), or to any novel method or any novel combination of steps of any method or process so disclosed.

Claims

1. A system for displaying information to a user of a transportation vehicle using augmented reality (AR), the system comprising a processor configured to: interacting with a display unit for displaying information using AR in a field of view associated with the system, a vehicle operation data source for providing vehicle operation data to the processor, and a multimedia data source for providing multimedia data to the processor; processing an AR image for display on the display unit, the AR image being based on the transportation tool operation data and the multimedia data; and The display unit is caused to display the AR image.

2. The system of claim 1, wherein the processor is configured to: The AR image is processed for display on the display unit, the AR image having a first component, wherein the first component includes an applied spatial and temporal change based on the multimedia data.

3. The system of claim 2, wherein the applied spatial and temporal variations are based on one or more of the following: Volume information in the multimedia data; Timing information in the multimedia data; The frequency information in the multimedia data.

4. A system as claimed in claim 2 or 3, wherein the applied spatial and temporal variations are optionally provided in a first dimension, wherein the first dimension is a vertical height.

5. A system as claimed in any preceding claim, wherein the processor is configured to: The AR image is processed for display on the display unit, and the AR image has a second component. The system of claim 5 , wherein the second component of the AR image comprises one or more image portions.

7. The system of claim 6, wherein an attribute of the one or more image portions is selectively determined based on the vehicle operation data and the multimedia data, wherein: The attribute is a dimension determined based on the transport operation data; and / or The attribute is a display duration of the one or more image portions optionally determined based on the multimedia data, wherein the display duration is determined based on one or more of the following: Lyrics timing information in the multimedia data; grammatical and / or pronunciation features of a lyric in the multimedia data; The current audio track timing within the multimedia data.

8. A system as claimed in any one of claims 5 to 7, wherein the second component is a text feature.

9. The system of any one of claims 5 to 8, wherein the second component is positioned according to: a distance at which the user should start interacting with the second component, or a time at which the user should start interacting with the second component; the distance that the user should have interacted with the second component, or the time that the user should have interacted with the second component; and / or A starting position of an animation for removing the second component or an image portion thereof and / or a stopping position of an animation for removing the second component or an image portion thereof and / or a starting time of an animation for removing the second component or an image portion thereof and / or a stopping time of an animation for removing the second component or an image portion thereof.

10. A system as claimed in any preceding claim, wherein the processor is configured to: When the vehicle operating data indicates that the vehicle is above a threshold vehicle operating parameter, processing is performed to display the AR image at a visually fixed position relative to a feature in the field of view.

11. A system as claimed in any preceding claim, wherein the processor is configured to: When the vehicle operating data indicates that the vehicle is below a critical vehicle operating parameter, processing is performed to display the AR image at an apparent time-varying position relative to a feature in the field of view.

12. A heads-up display (HUD) system comprising a system as claimed in any preceding claim.

13. An audio / visual (AV) system comprising the system as claimed in any one of the preceding claims or a HUD system, the AV system being configured to provide an AV output associated with the multimedia data provided by the multimedia data source.

14. A vehicle, optionally comprising a system as claimed in any one of the preceding claims, a HUD system or an AV system, wherein the vehicle is an autonomous vehicle.

15. A method for displaying information to a user of a transportation vehicle using augmented reality (AR), the method comprising: interacting with a display unit for displaying information in a field of view using AR, a vehicle operation data source for providing vehicle operation data to the processor, and a multimedia data source for providing multimedia data to the processor; processing an AR image for display on the display unit, the AR image being based on the transportation tool operation data and the multimedia data; and The display unit is caused to display the AR image.

Images