Video display device and method

Abstract

Provided is a technology capable of realizing more suitable display such as reducing misalignment, in relation to a technique for realizing an AR function by using a video display device such as smart glasses in a vehicle. This video display device (AR glasses) is usable in a vehicle. The video display device acquires, from an in-vehicle system of the vehicle, object detection information of an object outside the vehicle obtained using a sensor of the in-vehicle system, and displays a video so as to be superimposed on the object in alignment with a position of the object on the basis of the object detection information. The video display device, at the time of display, corrects a deviation of the position of the superimposed display of the video by using a difference between the origin position coordinates of the vehicle and the origin position coordinates of the video display device (step 7A4).

Description

Image display device and method

[0001] This disclosure relates to image display technology.

[0002] In-vehicle systems of vehicles such as automobiles include various sensors such as cameras and LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), and using these sensors, objects such as people and bicycles around the vehicle are detected, and based on this detection, driving control and assistance are performed.

[0003] Examples of driving assistance include functions such as AR (Augmented Reality) that display images such as alerts and navigation so as to be superimposed on a real image of an object or background around the vehicle on a display provided in the in-vehicle system. Examples of in-vehicle displays include a console / instrument panel, a head-up display (HUD) / virtual image display device, a projector, an airborne floating image display device, and the like.

[0004] As a prior art example, International Publication No. 2019 / 097918 (Patent Document 1) can be cited. Patent Document 1 describes that "Regarding the technology of a HUD device having an AR function, it provides a technology capable of reducing the deviation between an object and a virtual image (AR image) and realizing a suitable AR display. The HUD device 1 extracts a predetermined object 4 based on a captured image of an external camera 2, and includes object information including the object position of the object 4 in space, viewpoint information including the viewpoint position and movement amount of the driver in space based on a captured image of an internal camera 3, and virtual image information including the position of a virtual image region 7 in space. Information is acquired, an image for superimposed display on the object 4 is generated, and the display position of the image (AR image 6) in the virtual image region 7 is corrected including at least the horizontal direction (X).".

[0005] International Publication No. 2019 / 097918

[0006] In the future, it is conceivable that users (drivers, passengers, etc.) will use video display devices (in other words, wearable devices) such as smart glasses inside vehicles. For example, a driver may wear smart glasses or a head-mounted display (HMD) on their head and use the AR function of the smart glasses while driving. For instance, the display surface of the smart glasses (transparent or virtual reality type) may superimpose AR images onto the real-world surroundings.

[0007] In this case, when detecting objects around the vehicle and displaying alerts, etc., as AR images, it is preferable to generate and display AR images using high-precision detection information from various in-vehicle sensors. Therefore, when realizing AR functionality using an image display device such as smart glasses in a vehicle, a system can be conceivable in which the in-vehicle system and the smart glasses cooperate to display AR images on the smart glasses. For example, sensor detection information / object location information, or AR image data based on these, is transmitted from the controller or sensors of the in-vehicle system to the smart glasses, and the smart glasses display AR images based on this data.

[0008] However, in such AR image display systems, there may be a difference between the coordinate system of the vehicle, in-vehicle systems, sensors, etc. (e.g., the vehicle's origin coordinates) and the coordinate system of the image display device such as smart glasses inside the vehicle (e.g., the smart glasses' origin coordinates). If there is such a difference, even if an attempt is made to superimpose the AR image according to the position coordinates of an object detected based on the in-vehicle sensors, a discrepancy may occur between the object and the AR image, making it impossible to achieve a satisfactory display.

[0009] The purpose of this disclosure is to provide a technology that can achieve a more suitable display, such as reducing misalignment, for the technology that realizes AR functionality using video display devices such as smart glasses in a vehicle as described above.

[0010] A typical embodiment of this disclosure has the following configuration. One embodiment is a video display device usable in a vehicle, which acquires object detection information of an object outside the vehicle using a sensor of the vehicle's in-vehicle system, corrects the object detection information based on the vehicle's position coordinates so that it becomes object detection information based on the position coordinates of the video display device, and uses the corrected object detection information to display a video superimposed on the object.

[0011] According to a representative embodiment of this disclosure, a more suitable display can be achieved, such as by reducing misalignment, in relation to the technology for realizing AR functionality using an image display device such as smart glasses in a vehicle as described above. Other issues, configurations, and effects will be shown in the embodiments for carrying out the invention.

[0012] This document shows the system configuration of this embodiment (Embodiment 1). Examples of vehicle information and AR glasses information in this embodiment are shown. An example of the in-vehicle system configuration in this embodiment is shown. An example of the AR glasses configuration in this embodiment is shown. An example of the appearance of the AR glasses in this embodiment is shown. An example of a wired connection between the AR glasses and a smartphone in this embodiment is shown. Regarding the challenges of this embodiment, the detection of objects by the vehicle is shown. Regarding the challenges of this embodiment, the misalignment of the superimposed display of AR images on objects is shown. Regarding the challenges of this embodiment, the appearance of the misalignment from the driver's perspective is shown. The flow from object detection by the vehicle to the superimposed display of AR images on objects is shown in this embodiment. The appearance from the driver's perspective in this embodiment is shown. The basic processing flow of the system in this embodiment is shown. An example of detailed processing in the system in this embodiment is shown. An example of the vehicle origin position in this embodiment is shown. An example of the AR glasses position / user position in this embodiment is shown. An example of multiple sensors in this embodiment is shown. An example of how the AR glasses position / user position is determined in this embodiment is shown. This embodiment shows a specific example of AR display position correction. This embodiment shows an example of the view from the driver (front view). This embodiment shows an example of the view from the driver (blind spot). This embodiment shows an example of the view from the driver in the case of the comparative example HUD. This embodiment shows correction for multiple objects. This embodiment shows the view when object 1 is corrected and object 2 is not corrected. This embodiment shows the view when object 1 is not corrected and object 2 is corrected. This embodiment shows the view when object 1 is corrected and object 2 is corrected. This embodiment shows an example of setting priority. This embodiment shows an example of the view in a control example according to the priority of multiple objects. This embodiment shows an example of the view in a control example using gaze (when gaze is forward). This embodiment shows an example of the view in a control example using gaze (when gaze is directed towards an object). This embodiment shows an example of multiple sensors. This embodiment shows integrated information from multiple sensors, etc. This embodiment shows an example of processing to determine the type of received information. This example shows the communication status between the in-vehicle system and the AR glasses in this embodiment (under normal conditions).This embodiment shows an example of the communication status between the in-vehicle system and the AR glasses (in the case of an abnormality). This embodiment shows an example of control according to the communication status between the in-vehicle system and the AR glasses (in the case of an abnormality). This embodiment shows an example of control according to the communication status between the in-vehicle system and the AR glasses (when it returns to normal). This embodiment shows the case of three-dimensional coordinates. This embodiment shows an example of how the view changes depending on whether or not correction is made in the Z direction. This embodiment shows an example of how the driver and passengers see the view. This embodiment shows an example of how the driver and passengers see the view. Regarding the problems of this embodiment, this shows object detection by the vehicle, the difference between the vehicle origin position and the origin position of the video display device, etc. This embodiment shows an example of when the vehicle has multiple sensors. This embodiment shows an example of when there is two-stage correction. This embodiment shows two types of AR functions. This embodiment shows a modified example of this embodiment in which the in-vehicle system generates AR video data. This embodiment shows an example of differences in vehicle origin position and user position for each vehicle. This embodiment shows an example of GUI display for mode setting. This example shows a hypothetical vehicle origin position. This example shows AR display using a hypothetical vehicle origin position. This example shows a case where the sensor detection direction is different from the vehicle's coordinate system.

[0013] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, the same parts are generally denoted by the same reference numerals, and repeated descriptions are omitted. In the drawings, the representation of components may not show their actual location, size, shape, extent, etc., in order to facilitate understanding of the invention.

[0014] In explanations, when describing program-based processing, the focus may be on the program, functions, or processing units. However, the main hardware component is the processor, or a controller, device, computer, or system composed of such a processor. The computer, through its processor, executes processing according to the program read into memory, using resources such as memory and communication interfaces as appropriate. This realizes the predetermined functions and processing units. The processor includes transistors and other circuits, and is composed of semiconductor devices such as CPUs / MPUs and GPUs, and is considered a circuit or processing circuit. Processing is not limited to software program processing; it can also be implemented using dedicated circuits. Applicable dedicated circuits include FPGAs, ASICs, CPLDs, etc.

[0015] The program may be pre-installed as data on the target computer, or it may be distributed as data to the target computer from the program source. The program source may be a program distribution server on a communication network, or a non-transient computer-readable storage medium, such as a memory card or disk. The program may consist of multiple modules. The computer system may consist of multiple devices. The computer system may consist of a client-server system, a cloud computing system, an IoT system, etc. Various types of data and information are composed of structures such as tables and lists, but are not limited to these. Representations such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0016] [Problems, etc.] In this embodiment, a video display device (wearable terminal) such as smart glasses with AR functionality is worn and used by a user inside a vehicle. This video display device works in conjunction with the vehicle's in-vehicle system. Cameras and sensors such as LiDAR in the vehicle and in-vehicle system detect objects such as pedestrians and bicycles (e.g., obstacles related to driving). In other words, these sensors measure parameters such as the position and distance of the objects. The in-vehicle system provides the video display device with the sensor detection information, or the information resulting from processing based on the detection information (e.g., the position coordinates of the objects). Based on such object detection information, the video display device creates an AR image, such as an alert, that is, an image that is aligned with and superimposed on an object in space, and superimposes the AR image onto the actual scene of the objects on the display surface.

[0017] However, there may be differences between the coordinate system including the origin and axial direction of the vehicle, in-vehicle system, sensors, etc. (sometimes referred to as the first coordinate system) and the coordinate system including the origin and axial direction of the video display device (sometimes referred to as the second coordinate system). If such differences exist, a discrepancy may occur in the position where the video display device superimposes the AR image (AR display position). The position coordinates of objects (e.g., obstacles in front of the vehicle) identified by the vehicle's in-vehicle system based on sensor detection information are based on the vehicle's origin position in the first coordinate system (first origin position). On the other hand, the origin position of the video display device's second coordinate system (second origin position) may differ from the first origin position. Such differences occur depending on the position of the user's head wearing the video display device inside the vehicle. Therefore, when the video display device superimposes the AR image using the object position coordinates received from the in-vehicle system as is, a discrepancy may occur in the position of the superimposed AR image relative to the real-world object. If a discrepancy occurs, the quality of AR and driver assistance will deteriorate.

[0018] Therefore, this embodiment has the following solution. In this embodiment, correction is performed to eliminate or reduce the discrepancy between the position of an object in each coordinate system of the vehicle and the video display device, and the position of the superimposed AR image corresponding to it. In this embodiment, the target video display device is a device whose position in the vehicle may change, such as a wearable terminal such as smart glasses / HMD. However, the correction of this disclosure may also be used to eliminate or reduce the discrepancy between the position of an object in each coordinate system of the vehicle and a device fixedly installed in the vehicle (e.g., console, car navigation system, HUD, etc.), and the position of the superimposed AR image corresponding to it.

[0019] The user's video display device inside the vehicle is connected to the vehicle's and in-vehicle system's communication network. The video display device acquires sensor detection information or object position information from the vehicle and in-vehicle system (e.g., the controller). This object position information is described with reference to the first origin position of the first coordinate system.

[0020] In the system of this embodiment, for example, the vehicle's in-vehicle system acquires information such as the first origin position in the vehicle's first coordinate system and transmits it to the video display device. For example, the video display device acquires information such as the origin position of the video display device within the vehicle. For example, the video display device determines the difference between the first origin position in the vehicle's first coordinate system and the second origin position in the video display device's second coordinate system from this information.

[0021] For example, the video display device corrects (in other words, transforms, etc.) the object position coordinates (coordinates relative to the first origin position) obtained from the vehicle to object position coordinates described relative to the second origin position of its own second coordinate system, based on the difference information mentioned above. The object position coordinates are position coordinates in at least two dimensions. Based on the corrected object position coordinates, the video display device obtains the position for superimposing the AR image on the display surface. Then, the video display device superimposes the AR image onto the real-world scene including the object on the display surface.

[0022] The location of the video display device within the vehicle is generally at the driver's seat if the driver is seated there and has the device installed, but it is not limited to this. For example, if a passenger seated in the front passenger seat has the device installed, it will generally be at the front passenger seat. Furthermore, the number of video display devices within the vehicle is not limited to one; there may be multiple video display devices. The same functions can be applied to each video display device.

[0023] Figure 22 is an explanatory diagram relating to the above-mentioned problems and solutions. Inside vehicle 2, a user U1, such as a driver, is wearing an image display device 1, such as smart glasses 1. The first coordinate system of the vehicle's in-vehicle systems and sensors is shown as S1, and the first origin position (vehicle origin position) is shown as O1. The second coordinate system of the image display device, such as the smart glasses 1, inside vehicle 2 is shown as S2, and the second origin position (user position) is shown as O2. The first coordinate system is shown as a dashed line (X, Y), and the second coordinate system is shown as a solid line (X, Y). In this example, the direction of each coordinate system (corresponding axis) is the same, with the front of vehicle 2 being +Y, the rear being -Y, the right side of the vehicle being +X, and the left side being -X. Although not shown, the vertical direction (height direction) of vehicle 2 is the Z axis. In this example, we consider a two-dimensional coordinate system. The first origin position O1 is, for example, the center position on the left and right sides of vehicle 2, and the front end position on the front and rear sides.

[0024] Let the coordinates of the first origin position O1 (vehicle origin position) of vehicle 2 be (Xv, Yv). The second origin position O2, which is the current position of the video display device 1 inside vehicle 2, is, for example, the position of the driver's seat on the right side. Let the coordinates of the second origin position O2 be (Xu, Yu).

[0025] Examples of objects to which AR images are superimposed by the image display device 1 include object J1 being a bicycle and object J2 being a person (pedestrian). The position of the object is the object position coordinate (vehicle acquired coordinate) detected and acquired by the vehicle 2's sensor, and the coordinates are denoted as (Xo, Yo).

[0026] Let Δ be the difference between the first origin position O1 (Xv, Yv) and the second origin position O2 (Xu, Yu). Let Xd be the difference in the X direction and Yd be the difference in the Y direction. The video display device 1 determines this difference Δ(Xd, Yd) based on information such as the vehicle origin position obtained from the in-vehicle system.

[0027] The video display device 1 receives object detection information from the in-vehicle system, including the object's position coordinates (Xo, Yo). The video display device 1 corrects the object's position coordinates (Xo, Yo) as seen from the first origin position O1 (Xv, Yv) to the object's position coordinates (Xo', Yo') as seen from the second origin position O2 (Xu, Yu) of its own second coordinate system. The information necessary to eliminate the discrepancy during this correction is the difference Δ. After the correction, the object's position coordinates as seen from the video display device 1 become (Xo', Yo') = (Xo + Xd, Yo + Yd).

[0028] The video display device 1 can obtain the superimposed display position of the AR image on the display surface by performing a predetermined transformation from the corrected object position coordinates (Xo', Yo') in the coordinate system (X, Y).

[0029] The correction of the object's position coordinates (corresponding AR display position) is more effective the larger the difference Δ between the first origin position O1 and the second origin position O2. Furthermore, the correction is more effective the closer the distance between the vehicle 2 and the video display device 1 to the object. As an application, the control content regarding the correction of the superimposed display position of the AR image may be changed according to the size and distance of the object as seen from the user and the corresponding video display device 1. For example, for larger objects, the misalignment is less noticeable, so the correction may be omitted or its priority may be lowered. For smaller objects, the misalignment is more noticeable, so it is preferable to perform the correction without omitting it or to prioritize the correction.

[0030] It should be noted that, for example, a sensor may be installed at the first origin position O1, or it may be installed at other locations on the vehicle 2. Multiple sensors may also be installed at different locations. Typically, in most cases, the detection information from each of the multiple sensors on the vehicle 2 is assumed to be aligned based on a unified origin position (for example, the first origin position O1). Therefore, in most cases, the same correction should be applied to the detection information from each sensor.

[0031] If the positions of each sensor in vehicle 2 are different, and the reference origin position differs for each sensor detection information, then correction should be performed for each sensor detection information to match the different origin position (reference position). Figure 23 shows an example where there are two sensors at two different positions, left and right, relative to the vehicle origin position (O1) at the front end of vehicle 2. This system, for example, an in-vehicle system, acquires information on the origin position / reference position for each sensor. The information to be acquired may also be the difference 2301 between the vehicle origin position and the sensor position. For example, with respect to the sensor acquisition value (e.g., object position coordinates) of the left sensor sx1, the reference position is the vehicle origin position (O1). The video display device 1 should similarly perform correction according to the difference between the origin position of each sensor and its own origin position.

[0032] It should be noted that even if vehicle 2 is equipped with multiple sensors (for example, four), it is conceivable that only one representative sensor detection information (for example, object position coordinates) may be provided from the in-vehicle system to the video display device 1. For example, the in-vehicle system calculates the position coordinates of an object from the values ​​acquired by the four sensors and determines a single object position coordinate. In this case, the video display device 1 can perform a correction on the provided sensor detection information (object position coordinates) using the difference from the vehicle origin position. If multiple sensor detection information is provided, the video display device 1 can perform the same correction on each of the multiple sensor detection information using the difference from the vehicle origin position, or it can perform the correction on each sensor using the origin position / reference position information for each sensor.

[0033] As an application, a two-stage correction may also be used. Figure 24 shows a two-stage correction. The first stage is a rough correction targeting all users and arbitrary positions of the video display device 1 within the vehicle 2. The second stage is a correction to achieve higher accuracy, such as for the driver. In the example in Figure 24, the first stage correction assumes that the position of the video display device 1 is at the center position (C0) inside the vehicle. The difference Δ1 for the first stage correction is the difference between the vehicle origin position (O1) and the center position (C0) inside the vehicle. The second stage correction is a correction using the actual driver's position, for example, the user position (C2) in the driver's seat. The difference Δ2 for the second stage correction is the difference between the vehicle origin position (O1) and the user position (C2).

[0034] [System Components] The system of this embodiment is a system in which an in-vehicle system and a video display device work together to display AR images, and has the following components in general.

[0035] (1) The user wears and uses an image display device such as smart glasses with AR functionality inside the vehicle. While riding, the image display device superimposes AR objects (AR images) onto objects present around the vehicle. At that time, the image display device displays the AR image on the display surface based on object position information such as sensor detection information acquired from the vehicle's in-vehicle system.

[0036] The creation of the original AR video data may be performed by the video display device, or it may be performed by the in-vehicle system and the video data may be transmitted to the video display device. In this embodiment, the video display device creates the original AR video data and performs corrections regarding the superimposed display position of the AR video (corresponding object position coordinates).

[0037] (2) Before displaying the object location information acquired from the vehicle (position coordinates described based on the first origin position of the first coordinate system), the video display device performs a correction process to the object location information as seen from the video display device that views the object (position coordinates described based on the second origin position of the second coordinate system). In this correction, the difference Δ information is used, as described above (Figure 22).

[0038] (3) The video display device displays an AR image on the display surface based on the corrected object position information. In other words, the video display device displays the AR image superimposed on the corrected object position coordinates.

[0039] (4) When this system detects multiple objects simultaneously, it determines whether or not to display and correct AR, and the priority of these corrections, based on predetermined conditions / criteria. For example, the system may prioritize correcting the object closest in distance from the position of the video display device / vehicle, or the object closest to the centerline of the vehicle. Alternatively, it may prioritize correcting objects located at the left or right edges of the driver's field of view or objects approaching in the direction of the vehicle's movement (for example, a person or bicycle about to cross the road).

[0040] (5) There may be multiple video display devices in the vehicle. Each video display device acquires object location information, etc., from the vehicle's in-vehicle system and corrects the AR image according to the position of each video display device. The objects to be displayed in AR, the conditions for AR display and correction, etc., may differ for each user and video display device.

[0041] For example, when the AR function is used by users in the driver's seat and the passenger seat, the objects displayed in the AR and the control content can be changed for each user. For instance, for the driver, obstacles to driving are used as objects, and information to support safe driving (such as obstacle alerts or navigation to the destination) is provided as AR video. The superimposed position is corrected with high precision to match the object. For passengers, instead of driving assistance, information such as guides is provided as AR video, targeting, for example, stores and facilities on a map. In this case, the position correction can be of lower precision or priority than for driving assistance.

[0042] (6) When the video display device cannot obtain object position information or the like from the vehicle, for example, in the case of communication failure or the like, the AR video may be displayed based on the information detected by the sensor of the video display device itself. In this case, the coordinates of the object on which the AR video is superimposed are obtained in the second coordinate system and visually recognized based on the same second coordinate system. Therefore, correction such as that in (2) is unnecessary.

[0043] (7) In the case of (6), when the AR video has been displayed based on the information from the vehicle until just before, the AR video created by the video display device itself is also superimposed on the same object as before.

[0044] (8) In the case of (6), when the video display device enters a situation where it can obtain information from the vehicle again, it returns to the state of performing AR display control with correction based on the information from the vehicle.

[0045] (9) This system may perform correction regarding AR display for a limited number of objects. As an application, the presence or absence of correction may be controlled according to the magnitude of the vehicle speed. Also, the presence or absence of correction may be controlled according to the driving route. The presence or absence of correction may be controlled according to the distance to the object. The presence or absence of correction may be controlled according to the size of the object. The presence or absence of correction may be controlled according to the direction in which the driver is facing (such as the line-of-sight direction). For example, correction may not be performed for an object in a direction in which the driver is not facing, and correction may be performed when the driver faces the direction of that object.

[0046] Furthermore, this system can display AR images for objects that are in a blind spot (an area with obstructions) from the perspective of the driver or other user; in other words, it can visualize objects in blind spots using AR display. The vehicle detects objects in blind spots using sensors. The vehicle transmits object location information, etc., to the video display device. In the case of video display devices such as smart glasses, when the user turns their head towards the blind spot, the system can correct the display based on the object's location information, etc., and display AR images for that object. In the case of fixedly installed video display devices such as HUDs (where the display area is fixed), such AR display covering blind spots cannot be performed unless there is a mechanism to move the display area.

[0047] [Two Types of AR Functions] In this system, since the in-vehicle system and smart glasses work together, two types of AR functions (in other words, functions that superimpose images onto objects around the vehicle) are possible. The first AR function is a function in which the smart glasses display AR images based on sensor detection information from the in-vehicle system (sometimes referred to as a vehicle-based AR function). The second AR function is an AR function that is implemented by the smart glasses with AR capabilities (in other words, AR glasses) themselves, and is a function that displays AR images based on sensor detection information from the smart glasses themselves (sometimes referred to as a glasses-based AR function).

[0048] Figure 25 is an explanatory diagram regarding two types of AR functions. State A is a state where the first AR function is used as the first mode, and state B is a state where the second AR function is used as the second mode. In the first mode, object detection information 4A such as in-vehicle sensor detection information is transmitted from the in-vehicle system of vehicle 2 to smart glass 1. Based on the object detection information 4A and the like acquired from vehicle 2, smart glass 1 superimposes and displays an AR video on target object J1. At that time, smart glass 1 corrects the object position coordinates based on the difference (Δ) between two coordinate systems. In the second mode, smart glass 1 superimposes and displays an AR video on target object J1 based on the detection information by its own sensors. Smart glass 1 performs correction of the AR display position in the first mode and does not perform correction of the AR display position in the second mode.

[0049] In the system of this embodiment, for example, during normal times, by using the first AR function (vehicle-based AR function) of the first mode, more accurate AR video display can be realized. Also, it can be switched from the first mode to the use of the second mode under predetermined conditions such as during communication failure. Further, in this embodiment, the second AR function of the second mode can also be used at any timing such as user operation and instruction input. As will be described later, which AR function to use can be set by the system and the user.

[0050] Not limited to these, a modification example as shown in Figure 26 is also possible. In this modification example, the in-vehicle system of vehicle 2 generates AR video data 2600 (video frame, etc.) based on object position coordinates and the like based on the detection information of the in-vehicle sensors. The in-vehicle system transmits the object detection information 4A including the object position coordinates and the AR video data 2600 to smart glass 1. Smart glass 1 corrects the AR display position using the difference (Δ) for the received AR video data 2600 and displays the AR video. In any case, the correction holds similarly.

[0051] [Vehicle Origin Position] The vehicle origin position may differ depending on the manufacturer and model of vehicle 2. An example is shown in Figure 27. The positional relationship (difference) between the two coordinate systems is determined according to the combination of vehicle 2 and the smart glasses 1 riding in it. In vehicle 2A in state A, the vehicle origin position is, for example, O1A, located at the front of the vehicle body, and the positional relationship is the same as in Figure 22. The position coordinates of the object J1 are corrected using the difference ΔA to the position coordinates of the smart glasses 1 as viewed from the origin position O2A. The position of the smart glasses 1 within vehicle 2 may also change. For example, if the user wearing the smart glasses 1 moves from the driver's seat to the rear left seat, the origin position of the smart glasses 1 becomes O2Ab. In this case, the object position coordinates are corrected using the difference ΔAb to the position coordinates of the smart glasses 1 as viewed from the origin position O2Ab. Furthermore, the same correction can be made by specifying the position of the smart glasses 1 in both vehicles with the driver's seat on the right side (right-hand drive) and vehicles with the driver's seat on the left side (left-hand drive).

[0052] In vehicle 2B in state B, the vehicle origin is, for example, O1B, which is at the center of the vehicle. Based on the detection information when, for example, an object J1 is detected by a sensor at the front of the vehicle, the object's position coordinates are expressed as the position coordinates of the object J1 as seen from the vehicle origin O1B. Also in state B, if the smart glasses 1 are in the driver's seat as in state A, they are at the origin O2B. The object's position coordinates are corrected using the difference ΔB to the position coordinates of the object J1 as seen from the origin O2B. Also, for example, if the smart glasses 1 are moved to the rear left seat, the origin becomes O2Bb. In this case, the object's position coordinates are corrected using the difference ΔBb to the position coordinates of the object J1 as seen from the origin O2Bb.

[0053] Furthermore, the vehicle origin position (reference position for sensors, etc.) is basically constant for each vehicle. However, in special cases (for example, when the hardware or software of the in-vehicle system is updated), the vehicle origin position may change. Even in this case, if the vehicle origin position and the difference (Δ) in the coordinate system after the change are known, the AR display position correction can be applied in the same way. Also, multiple sensors of multiple types in a vehicle may each have different origin positions (reference positions). In this case, correction should be made according to the origin position of each sensor.

[0054] <Embodiment 1> The system of Embodiment 1 will be described using Figures 1 and subsequent figures. In this system, when a user such as a driver wears and uses AR glasses (smart glasses equipped with AR functionality) inside a vehicle, the in-vehicle system and the AR glasses work together to display AR content. This system has a function to correct the AR display position (corresponding object position coordinates, etc.) based on the positional relationship (difference) between the position of an object outside the vehicle, the reference position of the in-vehicle system's sensors, etc. (vehicle origin position), and the position of the AR glasses.

[0055] [Vehicle / In-vehicle system / AR display system] Figure 1 shows a vehicle 2 (e.g., an automobile) equipped with an in-vehicle system 100, and shows a state in which a user U1 (e.g., a driver) is wearing smart glasses 1 (AR glasses) inside the vehicle 2. In other words, the system in Figure 1 is an AR display system in which the in-vehicle system 100 and the AR glasses 1 work together. For example, user U1, who is the driver, wears the AR glasses 1 and uses the AR function (especially the driving assistance function; the vehicle-based AR function, which is the first AR function in Figure 25) while driving. However, other users, such as a passenger in the front seat, may also wear the AR glasses 1 and use the AR function.

[0056] Figure 1 schematically shows a user U1 seated in a seat 6 of a vehicle 1, wearing AR glasses 1 on their head, and operating the steering wheel 5 with their hands. User U1 can see the real-world scenery in front of them (Y direction) through the windshield 3 of the vehicle 2. User U1 can see the AR image superimposed on the real-world objects in front of them on the display surface of the AR glasses 1. The vehicle 2 is equipped with various sensors 11 such as cameras and LiDAR, and in the example in Figure 1, only the sensors 11 mounted at the front of the vehicle body are shown.

[0057] The in-vehicle system 100 includes a controller 10, etc., which controls the operation of the vehicle 2. The controller 10 is a control device / control unit, also known as an ECU (Electronic Control Unit), and controls the entire in-vehicle system 100 and its various parts. The in-vehicle system 100 may also be an autonomous driving system that controls the autonomous driving of the vehicle 2. The controller 10 of the in-vehicle system 100 is connected to the AR glasses 1 by communication as appropriate. In other words, the in-vehicle system 100 is a vehicle surrounding object detection system, etc., which has the function of detecting objects (in other words, target objects) around the vehicle 2. The controller 10 detects objects around the vehicle 2 using in-vehicle sensors 11, etc.

[0058] AR glasses 1 are smart glasses equipped with AR functionality; in other words, they are wearable devices, head-mounted display devices, image display devices, etc. AR glasses 1 have at least a display function that displays an image on a display surface corresponding to the wearer's field of view, and in particular an AR function that superimposes AR images onto objects. AR glasses 1 are not limited in type or detailed configuration. The display function of AR glasses 1 may be transparent (optical see-through or video see-through) or non-transparent (Virtual Reality) as long as it supports AR functionality. AR glasses 1 are, but are not limited to, a device owned by user U1. AR glasses 1 may be connected to other devices or components not shown, such as remote controls (operators), sensors, chargers, or user U1's mobile devices such as smartphones. AR glasses 1 may also cooperate with these devices.

[0059] AR glasses 1 may have an audio output function and / or an audio input / voice recognition function. The audio output function outputs sound through a speaker or earphone jack. The audio input / voice recognition function recognizes the voice of user U1 input through a microphone and converts it into instructions or other commands. AR glasses 1 performs actions corresponding to the recognized instructions or other commands.

[0060] The AR glasses 1 and the controller 10 may be connected by any technical means. The AR glasses 1 and the controller 10 may be connected directly or via a network. As an example, the AR glasses 1 and the controller 10 may be connected via a CAN (Controller Area Network).

[0061] The controller 10 generates or acquires vehicle information 4 (in other words, in-vehicle system information) as shown in Figure 2. The controller 10 can output the vehicle information 4 to the user U1 via display or sound using the equipment of the in-vehicle system 100, and can also transmit it to the AR glasses 1 via communication. On the other hand, the AR glasses 1 generates or acquires AR glasses information 7 as shown in Figure 2. The AR glasses 1 can transmit the AR glasses information 7 to the controller 10 via communication. In addition, the controller 10 and the AR glasses 1 may exchange arbitrary control information, etc., via communication in order to operate the system in a coordinated manner.

[0062] The controller 10 and the AR glasses 1 may each be connected to external devices, such as the operator's server 8, via an external communication network 9 (e.g., a mobile network, the internet, etc.). The controller 10 and the AR glasses 1 may receive and acquire data from the server 8, or they may send and register data to the server 8. The server 8 may be located inside the vehicle 2. Other devices, such as a car navigation system, may be connected to the controller 10 inside the vehicle 2.

[0063] The images, videos, and audio played on AR glasses 1 can be any images, videos, or audio from any function / application / content. An example of an application is an AR application that superimposes images and videos for driving assistance onto an optical image of the real-world scenery around the vehicle (vehicle surroundings video). For example, a driving assistance AR application could detect objects such as roads, signs, pedestrians, bicycles, and other vehicles, and superimpose AR images for alerts, navigation, etc., onto those objects. Various functions such as AR functions also include a graphical user interface (GUI).

[0064] The video and other data to be played back by the AR glasses 1 may be stored in the memory resources within the AR glasses 1 beforehand, or may be acquired from outside the AR glasses 1, for example, from an in-vehicle system 100 or a server 8. For example, the AR glasses 1 store the source video data (for example, an image representing an alert) for the AR video to be displayed in the AR function in the memory resources within the AR glasses 1 beforehand. The AR glasses 1 then performs AR display position correction, etc., on the source data to generate the AR video data to be displayed.

[0065] Applications and other components installed in the AR glasses 1 perform predetermined processes according to their programs. For example, an AR application controls operation input, screen display, and audio output. Note that the application programs and data may be stored on an external device such as a server 8, and processing / services may be implemented in the form of a client-server system. For example, AR programs and data may be distributed from an external server 8 to the AR glasses 1. Alternatively, as shown in the modified example (Figure 26), the in-vehicle system 100 may primarily generate AR video data, and the AR application may implement the AR functionality.

[0066] In addition to manual operation via a remote control or buttons, the AR glasses 1 may also accept operation based on the user U1's gaze, blinks, hand gestures, etc. The AR glasses 1 detect the user U1's gaze, blinks, and gestures using sensors and perform actions corresponding to the detected gaze, blinks, and gestures. For example, the AR glasses 1 may move a cursor based on gaze movement and select / confirm the cursor location in response to blinks, hand gestures, etc.

[0067] [Vehicle Information and AR Glasses Information] Figure 2 shows an example of vehicle information 4 provided by the in-vehicle system 100 of vehicle 2 and AR glasses information 7 provided by AR glasses 1.

[0068] The vehicle information 4 (in other words, in-vehicle system information) includes object detection information 4A (in other words, object position information, vehicle surrounding object detection related information) based on sensor detection information. Object detection information 4A is one of the source data for generating AR images. Note that sensor detection information (sensor acquired values) may be directly provided as object detection information 4A. Object detection information 4A is, for example, information representing the position of an object (object around the vehicle) detected based on sensor 11, expressed in the coordinate system of vehicle 2 and sensor 11, and includes the object position coordinates as seen from the vehicle origin position coordinates. The controller 10 determines the position, distance, direction, speed, etc. of the object based on the detection information of sensor 11, and as a result obtains object detection information 4A representing them. Object detection information 4A includes information such as the ID and type of the object.

[0069] Furthermore, the vehicle information 4 provided from the controller 10 to the AR glasses 1 includes vehicle information acquisition origin position information 4B (in other words, vehicle coordinate system information). The vehicle information acquisition origin position information 4B includes the vehicle origin position coordinates, etc. The controller 10 transmits the vehicle information acquisition origin position information 4B to the AR glasses 1 at timings such as the first time at startup or each time communication occurs.

[0070] Based on the vehicle information acquisition origin position information 4B obtained from the controller 10, the AR glasses 1 determine the difference (Δ in Figure 22) between the vehicle origin position and the origin position of the AR glasses 1. In other words, the AR glasses 1 determine the difference between the two coordinate systems. The AR glasses 1 uses this difference (Δ) to correct the object position coordinates of the object detection information 4A received from the controller 10.

[0071] The controller 10 acquires and generates vehicle information 4 based on various sensors 11 installed in the vehicle 2. The various sensors 11 detect various events occurring in the vehicle 2 and detect various parameter values ​​related to the vehicle's driving status. For example, cameras and LiDAR detect parameter values ​​of objects around the vehicle. The controller 10 uses the vehicle information 4 to control the driving of the vehicle 2.

[0072] The controller 10 can transmit vehicle information 4 to the AR glasses 1 based on its own judgment or based on a request from the AR glasses 1. The vehicle information 4 to be transmitted may be numerical data or strings of characters, or it may be video data or the like that has been processed to be suitable for display on the AR glasses 1. The AR glasses 1 may also process the vehicle information 4 and generate video data or the like.

[0073] Vehicle information 4 includes, for example, the following information: Vehicle information 4 includes, for example, vehicle speed information (also referred to as vehicle speed), gear information, steering angle information, lamp illumination information, ambient light information, distance information, seat occupancy information, engine ON / OFF information, camera image information, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, vehicle-to-infrastructure communication information, driver input information, etc.

[0074] Vehicle information 4 may include information related to autonomous driving. Vehicle information 4 may also include system notifications and other user interface information.

[0075] Seat occupancy information is information about the occupant's seating status, such as which occupant is seated in which seat 6, obtained based on the seat occupancy sensor. Camera image information is data of images captured by the in-vehicle camera or information resulting from the analysis of those images. Driver input information is arbitrary information entered by the driver through the operation input device provided in the vehicle 2 (in-vehicle system 100). Examples of operation input devices include buttons (vehicle operation switches) installed on the steering wheel 5 and touch operations on the console panel.

[0076] The AR glasses information 7 includes device information (device information, ID, attributes, status, etc.), user information (person information, ID, attributes, status, etc.), wearing information, seating information, etc. Wearing information includes information indicating whether the user is wearing the AR glasses 1 or not. Seating information is information indicating the seat, etc., in which the user using the AR glasses 1 is seated. The AR glasses information 7 may also include user input information, operation information, sensor detection information, etc., for the AR glasses 1.

[0077] [In-vehicle system] Figure 3 shows an example of the configuration of the in-vehicle system 100 of vehicle 2. Examples of sensors 11 (Figure 1) used to detect objects around the vehicle include an external camera 532, a distance measuring sensor 507 (which may be a LiDAR, etc.), an infrared sensor 508, and the like.

[0078] The in-vehicle system 100 has components as shown in the figure that are connected to the controller 10. The in-vehicle system 100 includes a vehicle speed sensor 501, a shift position sensor 502, a steering wheel angle sensor 503, a headlight sensor 504, an illuminance sensor 505, a chromaticity sensor 506, a distance measuring sensor 507, an infrared sensor 508, an engine start sensor 509, a seating sensor 510, a vehicle operation switch 511, an acceleration sensor 512, a gyro sensor 513, a temperature sensor 514, a wireless transceiver 515 for vehicle-to-infrastructure communication, a wireless transceiver 516 for vehicle-to-vehicle communication, a wired wireless communication unit 517 for mobile terminal-to-vehicle communication, a GPS receiver 518, a VICS (Vehicle Information and Communication System, registered trademark) receiver 519, a communication unit 520, a video generation unit 521, a video analysis unit 522, a camera (in-vehicle camera) 531, a camera (out-of-vehicle camera) 532, an audio input device 541 (including a microphone, etc.), an audio output device 542 (including a speaker, etc.), and the like. Sensors and devices, including those listed above, can be added, deleted, replaced, and so on.

[0079] The vehicle speed sensor 501 detects the speed of the vehicle 2 and generates speed information as the detection result. The shift position sensor 502 detects the current gear and generates gear information as the detection result. The steering angle sensor 503 detects the current steering angle of the steering wheel 5 and generates steering angle information as the detection result. The headlight sensor 504 detects whether the headlights are ON / OFF, etc., and generates lamp illumination information as the detection result. The illuminance sensor 505 and the chromaticity sensor 506 detect ambient light and generate ambient light information as the detection result.

[0080] The distance measuring sensor 507 detects the distance between the vehicle 2 and an external object and generates distance information as the detection result. The infrared sensor 508 detects the presence and distance of an object in close proximity to the vehicle 2 and generates infrared information as the detection result. The engine start sensor 509 detects whether the engine is ON or OFF and generates ON / OFF information as the detection result.

[0081] The seating sensor 510 is installed in each seat 6 (Figure 1) in the vehicle 2 and detects whether or not an occupant is seated in the seat 6, and whether or not the occupant is wearing a seat belt. Note that the seating status and whether or not a seat belt is worn may be detected individually. Alternatively, even if there is no seating sensor 510, the seating status of each seat 6 may be determined and detected using sensors such as a camera 531 in the vehicle 2.

[0082] The vehicle operation switch 511 is, for example, a button on the steering wheel 5, and accepts operation from the driver and generates input information from the driver. The acceleration sensor 512 and gyro sensor 513 detect the acceleration and angular velocity of the vehicle 2 and generate acceleration gyro information representing the attitude and behavior of the vehicle 2 as a result of the detection. The temperature sensor 514 detects the temperature inside and outside the vehicle 2 and generates temperature information as a result of the detection.

[0083] The vehicle-to-infrastructure wireless transceiver 515 generates vehicle-to-infrastructure communication information through vehicle-to-infrastructure communication between the vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle wireless transceiver 516 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 2 and other surrounding vehicles. The wired wireless communication unit 517 for mobile terminal-to-vehicle communication generates mobile terminal-to-mobile terminal communication information through communication with mobile terminals. The GPS receiver 518 generates GPS / GNSS information by receiving GPS / GNSS signals from GPS / GNSS satellites. The GPS / GNSS information includes information such as the current time, latitude, and longitude. The VICS receiver 519 generates VICS information obtained by receiving VICS signals. The GPS receiver 518 and the VICS receiver 519 may be provided as part of a navigation system.

[0084] The communication unit 520 is a device or circuit that implements a communication interface with an external communication network and a communication interface (e.g., CAN) with the AR glasses 1. The video generation unit 521 is the part that generates video information when displaying video on the console panel or the AR glasses 1. The video analysis unit 522 is the part that performs video analysis based on the video captured by cameras 531 and 532 and generates analysis result information.

[0085] The in-vehicle camera 531 generates in-vehicle camera video information by photographing the inside of the vehicle 2. The exterior camera 532 generates exterior camera video information by photographing the outside of the vehicle 2. In a specific example, the in-vehicle camera 531 photographs the position, posture, eye position, and movement of the occupants, forming a Driver Monitoring System (DMS). The exterior camera 532 photographs the surrounding environment, such as the front, rear, and sides of the vehicle 2. By analyzing the exterior camera video information, it is possible to determine the presence or absence of other vehicles and people around the vehicle 2, buildings and terrain, road surface conditions such as rain, snow, ice, and unevenness, and road signs. The exterior camera 532 also includes drive recorders that record driving conditions in video.

[0086] The audio input device 541 includes a microphone or the like and inputs audio. The audio output device 542 includes a speaker or the like and outputs audio.

[0087] [AR Glasses] Figure 4 shows an example of the configuration of AR glasses 1. AR glasses 1 includes a processor 101, a storage device 110, an input interface 120, a video input / output device 130, an audio input / output device 140, a sensor group 150, a communication interface 160, an expansion interface 171, a timer 172, an actuator 173, and the like. These components are interconnected via a bus 102.

[0088] The processor 101 controls the entire AR glasses 1 and its various parts. The processor 101 reads programs stored in the non-volatile memory 112 into the volatile memory 111 and executes processing according to the programs. This enables the realization of various functional units. The programs include the basic operation program 112a and the AR program 112b. The AR program 112b realizes the AR function in the AR glasses 1.

[0089] The input interface 120 includes a button switch 121, etc. The video input / output device 130 includes a display 131, an image signal processing unit 132, an out-camera 133, etc. The display 131 is a device that displays images on a display surface corresponding to the field of view. The image signal processing unit 132 performs image signal processing necessary for display based on image and video data.

[0090] The rear camera 133 is a camera that photographs the area in front of the AR glasses 1, etc. The rear camera 133 may consist of multiple cameras that photograph in various directions. Although not shown in the diagram, there may also be an in-camera for detecting the gaze of the driver or other user. Based on the image captured by the in-camera, the gaze detection sensor 156 can detect the state of the user's gaze.

[0091] The audio input / output device 140 includes a speaker 141, an audio signal processing unit 142, a microphone 143, etc. The speaker 141 outputs sound. The audio signal processing unit 142 performs the audio signal processing necessary for audio input and output. The microphone 143 receives sound.

[0092] The sensor group 150 includes a positioning sensor 151, a geomagnetic sensor 152, a distance sensor 153, an acceleration sensor 154, a gyroscope sensor 155, a gaze detection sensor 156, and the like. The positioning sensor 151 determines the position of the AR glasses 1 based on GPS signals, etc. The geomagnetic sensor 152 detects direction. The distance sensor 153 (which may also be a LiDAR sensor, etc.) measures the distance to an object as seen from the AR glasses 1. The acceleration sensor 154 detects acceleration, etc. The gyroscope sensor 155 detects angular velocity, etc. The gaze detection sensor 156 detects the wearer's gaze state (direction of gaze, etc.).

[0093] The communication interface 160 includes a LAN communication interface 161, a short-range wireless communication interface 162, a telephone network communication interface 163, etc. The LAN communication interface 161 corresponds to an interface such as Wi-Fi®. The short-range wireless communication interface 162 corresponds to an interface such as Bluetooth®. The communication interface 160 includes a communication interface with the controller 10, such as the implementation of a CAN interface.

[0094] The actuator 173 transmits physical movements such as vibrations to the user U1 who is wearing it. The actuator 173 may be electrically operated, or it may utilize a piezoelectric element, magnetism, hydraulics, etc., as long as it generates a physical force.

[0095] In this embodiment, when outputting alerts or notifications using the AR function, not only video display on the AR glasses 1's display 131 may be used, but also audio output from the speaker 141 or vibration from the actuator 173 may be used. By outputting audio and vibration to the user simultaneously with or before the video display, it is possible to make it easier for the user to recognize that an alert or other display is being shown.

[0096] Although not shown in Figure 4, the AR glasses may also be equipped with a battery. Furthermore, some of the components shown in the figure may be located in a separate device connected to the AR glasses by wire or wirelessly. For example, the AR glasses may be connected by wire or wirelessly to a separate housing containing a heat-generating component such as a battery. Alternatively, the AR glasses may be connected by wire or wirelessly to a mobile device such as a smartphone, and some or all of the processing of the processor 101 may be performed by the mobile device.

[0097] [AR Glasses: Appearance] Figure 5A shows an example of the configuration of the AR glasses 1. The glass-shaped housing 190 has displays 131L and 131R on the front side that correspond to both eyes, and an out-camera 133F in the center and out-cameras 133L and 133R on the left and right sides. An eye-tracking sensor 156 is located on the inside of the front. On the left and right sides of the housing 190 are left and right speakers 141L and 141R, left and right microphones 143L and 143R, and left and right sensor groups 150L and 150R. Figure 5B shows an example of the configuration when a smartphone is connected to the AR glasses 1 via a wired connection, and the entire processing of the present invention may be realized by sharing the work between the AR glasses 1 and the smartphone. Furthermore, while Figure 5A shows an example where the left display 131L and the right display 131R themselves are transparent displays, the system is not limited to this. As shown in Figure 5B, the display may be built into the frame, and the image from the display may be reflected to allow the user to perceive the image as a virtual image.

[0098] [Problem: AR display position misalignment] Figure 6A and others illustrate the problem. Figure 6A is an explanatory diagram relating to the misalignment of object positions acquired by the AR glasses 1 from the vehicle's in-vehicle system 100, and is an X-Y plan view of space from above. In the drawing, the left-right direction is the X-axis and the front-back direction is the Y-axis. The three-dimensional spatial coordinate system for explanation is shown as (X, Y, Z). Assume that the vehicle 2 is traveling in the forward direction (+Y). For example, assume that user U1, the driver, is seated in the driver's seat on the front right side of the vehicle and is wearing the AR glasses 1. Position 601 is the vehicle origin position (first origin position) in the vehicle 2's coordinate system, or in other words, the vehicle information acquisition origin position coordinate, which is (Xv, Yv). Position 602 is the position of the AR glasses 1 or user U1 wearing the AR glasses 1 (second origin position), which is (Xu, Yu). Object J1 (for example, "Object 1") is an example of an object (for example, a bicycle) detected by the sensor 11 of vehicle 2. Position 603 is the position coordinate detected by the sensor 11 of vehicle 2, and is a position coordinate expressed based on the vehicle origin position (601) of the vehicle coordinate system, and is denoted as (Xo, Yo).

[0099] In Figure 6A, the transfer of object detection information 4A from the in-vehicle system 100 to the AR glasses 1 is explained as follows: In step 6B1, the in-vehicle system 100 of the vehicle 2 detects the coordinates (Xo, Yo) of the position 603 of the object J1 based on the detection information from the sensor 11. In step 6B2, the in-vehicle system 100 transmits object detection information 4A, including the detected position coordinates (Xo, Yo), to the AR glasses 1. In step 6B3, the AR glasses 1 receive and acquire the object detection information 4A, including the position coordinates (Xo, Yo), from the in-vehicle system 100.

[0100] Figure 6B is an explanatory diagram relating to the AR display position shift, following Figure 6A. In step 6B4, the AR glasses 1 superimpose an AR image (virtual image 604A) according to the position coordinates (Xo, Yo) of the target object J1 specified by the object detection information 4A acquired from the vehicle 2. The virtual image 604A is, for example, a ring-shaped alert image that conveys that object 1 is approaching and draws attention. In this case, the AR image (virtual image 604A) will be superimposed at the position 604 shown in the figure. The shape of the virtual image 604A is not limited to the ring shape shown in the figure; for example, J1 (object 1) may be enclosed in a square frame. Position 604 is the coordinates at (Xo, Yo) when viewed from the origin position 602 of the AR glasses 1. The position 604 of this AR image (virtual image 604A) does not match the position 603 of the actual object J1, resulting in a discrepancy between the real image of the object J1 and the AR image (virtual image 604A) as seen by the user U1 of the AR glasses 1 (step 6B5).

[0101] The difference (in other words, the amount of difference) between the position 603 of the object J1 and the display position 604 of the AR image (virtual image 604A) is shown by (Xd, Yd). This amount of difference (Xd, Yd) corresponds to the difference (Δ) between the origin position 601 of the vehicle 2 and the origin position 602 of the AR glasses 1, as in Figure 22.

[0102] Furthermore, it would be even better if the object detection information 4A also provided information such as the type of object. For example, if information that the object type is a bicycle is provided, it would be possible to select and display an AR image corresponding to the object type of a bicycle. An example of changing the appearance of the AR image is to change the shape and color of the alert represented in the AR image according to the object type. It can also be used to determine the priority of AR display based on the object type.

[0103] Figure 6C shows the misalignment of the AR display in Figure 6B as seen from near the viewpoint of the driver wearing the AR glasses 1. The windshield 3 has a region (display region) 3A in which the AR image (virtual image) from the AR glasses 1 can be displayed according to the position and orientation of the driver, user U1's head. The virtual image 604A is displayed within region 3A. The virtual image 604A is not superimposed on the real image of the object J1, and a misalignment occurs between the real image of the object J1 and the virtual image 604A. This is because the amount of misalignment (Xd, Yd) in Figure 6B has not been taken into consideration. In other words, the difference (Δ) between the origin position 601 of the vehicle's coordinate system and the origin position 602 of the AR glasses 1's coordinate system has not been taken into consideration.

[0104] [Solution: Correction of AR display position] Figure 7A and others show the solution and overview in this embodiment. For example, let's assume a situation similar to that described in Figure 6A. Figure 7A shows the flow from object position detection to AR display position correction in this embodiment.

[0105] In step 7A1, the vehicle's onboard system 100 detects the position coordinates (Xo, Yo) of the object J1 based on the detection information from the sensor 11. These position coordinates (Xo, Yo) are relative to the vehicle's origin position 601.

[0106] In step 7A2, the controller 10 of the in-vehicle system 100 transmits object detection information 4A, which includes the position coordinates (Xo, Yo) of the target object J1, and vehicle information acquisition origin position information 4B, which includes the vehicle origin position coordinates (Xv, Yv), to the AR glasses 1.

[0107] In step 7A3, the AR glasses 1 receive and acquire object detection information 4A, which includes the position coordinates (Xo, Yo) of the target object J1, and vehicle information acquisition origin position information 4B, which includes the vehicle origin position coordinates (Xv, Yv), from the vehicle's in-vehicle system 100.

[0108] In step 7A4, the AR glasses 1 determine the difference (Δ) between the vehicle origin position coordinates (Xv, Yv) and its own origin position coordinates (Xu, Yu) from the information acquired from the in-vehicle system 100. Based on the difference (Δ), the AR glasses 1 calculate the position coordinates of the object J1 relative to the origin position 602 of the AR glasses 1, using the position coordinates of the object J1 relative to the vehicle origin position 601 acquired from the in-vehicle system 100. In other words, the AR glasses 1 converts the position coordinates of the object J1 relative to the vehicle origin position 601 to the position coordinates of the object J1 relative to the origin position 602 of the AR glasses 1, based on the difference (Δ). The process in step 7A4 corresponds to the correction of the AR display position. The correction of the position coordinates (corresponding AR display position) here is a correction of the position coordinates in the X-Y plane.

[0109] The difference (Δ) between the vehicle's origin coordinates (Xv, Yv) and the origin coordinates (Xu, Yu) of AR Glass 1 corresponds to the amount of displacement (Xd, Yd). This amount of displacement (corresponding difference Δ) can be expressed as (Xd, Yd) = (Xu - Xv, Yu - Yv).

[0110] If the corrected position coordinates of object J1 (corresponding AR display position) are (X, Y) = (Xo', Yo'), then it can be calculated as (X, Y) = (Xo + Xd, Yo + Yd).

[0111] The AR glasses 1 calculate the position of the superimposed AR image (AR object) on the display surface based on the corrected position coordinates of the object J1 (e.g., position in the X-Y plane). The display position of the AR image in space (e.g., the X-Y plane) and the superimposed display position of the AR image on the display surface have a predetermined relationship and can be converted between each other. Through this conversion, the superimposed display position of the AR image on the display surface of the AR glasses 1 is determined as the superimposed display position after the displacement correction.

[0112] Next, in step 7A5, the AR glasses 1 display the corrected AR image at the superimposed display position on the display surface. 701 shows the superimposed display (on the X-Y plane) of the corrected AR image onto the object J1 as seen from the AR glasses position / user position (origin position 602). 702 is a schematic diagram of the alert image on the X-Y plane as the AR image (virtual image) onto the object J1. In the superimposed display 701 of the corrected AR image (virtual image 702), it is aligned to match the position coordinates 603 (Xo, Yo) of the object J1 as seen from the AR glasses position, with no misalignment in either the X or Y direction.

[0113] Regarding the vehicle information acquisition origin position information 4B, it is possible to transmit it each time object detection information 4A is transmitted or received. However, since information such as the vehicle origin position coordinates is usually a constant value for each vehicle 2, it is not necessary to transmit it each time object detection information 4A is transmitted or received. Therefore, in this system, for example, after the engine of vehicle 2 is started (after its trip is started), the vehicle information acquisition origin position information 4B is notified during the first communication with AR glasses 1. AR glasses 1 keeps the vehicle information acquisition origin position information 4B in memory and refers to and uses it each time object detection information 4A is transmitted or received. This reduces the load compared to transmitting the vehicle information acquisition origin position information 4B each time.

[0114] Figure 7B shows how the superimposed AR image (virtual image 702) of the object J1 appears from the viewpoint of user U1, the driver wearing the AR glasses 1, corresponding to the corrected AR display position (position in the X-Y plane) in Figure 7A. Due to the correction, the virtual image 702 of the alert is superimposed to match the position of the object J1 in the real scene, and there is no misalignment as in Figure 6C. In other words, the AR function of this embodiment makes it possible to more accurately align the superimposed display of the AR image on the object. Therefore, driving assistance using AR images for user U1 becomes more effective.

[0115] In the above explanation, it is assumed that the origin position of the coordinate system of vehicle 2 and the origin position of the coordinate system of AR glasses 1 are described as positions within a common world coordinate system, but this is not limited to this. They may also be described using a common origin position or coordinate system that connects the two coordinate systems, or they may be described as the relative positional relationship between the coordinate system of vehicle 2 and the coordinate system of AR glasses 1 (representation of the origin position of the AR glasses as seen from the coordinate system of vehicle 2, or representation of the origin position of the vehicle as seen from the coordinate system of AR glasses 1).

[0116] [Processing Flow] Figure 8 shows the basic processing flow of this system. The controller 10 and AR glasses 1 in Figure 1 process according to this flow. The AR display position correction in the AR function, which is characteristic of this embodiment, has already been shown in the flow in Figure 7A (steps 7A1 to 7A5), but the flow in Figure 8 encompasses the flow in Figure 7A.

[0117] In step S1, the AR glasses 1 start in driving mode. In this embodiment, there are several operating modes for the AR glasses 1, including an in-vehicle mode for use inside the vehicle 2 and a normal mode for use outside the vehicle 2. The in-vehicle mode may further include a driving mode and a non-driving mode. The driving mode is the mode used when the driver uses the AR glasses 1, and the non-driving mode is the mode used when a non-driver, such as a passenger, uses them. The driving mode may include a manual driving mode and an automatic driving mode. Other modes may include an AR mode that uses the AR function and a non-AR mode that does not use the AR function. In this example, user U1 is the driver and the use of the AR function while driving will be described. Therefore, in step S1, the AR glasses 1 are set to driving mode.

[0118] In step S2, the AR glasses 1 acquire display-related information regarding the AR function from the controller 10 of the in-vehicle system 100. The display-related information includes the vehicle information acquisition origin position information 4B and object detection information 4A mentioned above. The manner in which data and information are exchanged between the controller 10 and the AR glasses 1 is not limited. In one example, using the CAN interface, the AR glasses 1 send a request to the controller 10 when it starts up, and the controller 10 first sends a response including the vehicle information acquisition origin position information 4B. Thereafter, while the communication connection between the two is maintained, the controller 10 continuously (if object detection occurs) sends object detection information 4A to the AR glasses 1 at each point in time (each time) in the time series. Note that the information may be transmitted unidirectionally from the controller 10 to the AR glasses 1, or it may be transmitted bidirectionally. The AR glasses 1 may send a request to the controller 10 and the controller 10 may send a response to the AR glasses 1, or the controller 10 may send a request to the AR glasses 1 and the AR glasses 1 may send a response to the controller 10.

[0119] In step S3, the AR glasses 1 generate AR video data to be superimposed on the target object based on object detection information 4A, which includes the object's position coordinates. The source image / video data (for example, an image representing an alert) may be stored in the memory of the AR glasses 1 in advance. For example, the AR glasses 1 processes this source data appropriately, such as resizing it according to the distance to the target object, to generate AR video data, or in other words, data for displaying an AR video on the display surface. In another modified example, as described above, the in-vehicle system 100 may generate the AR video data and transmit it to the AR glasses 1. In step S3, the AR display position has not yet been corrected.

[0120] In step S4, the AR glasses 1 correct the AR display position based on the object position coordinates obtained from the controller 10. The overview of the correction is shown in Figure 7A, etc. In the flow chart of Figure 8, the display position of the AR video data (e.g., alert image) obtained in step S3 is corrected in step S4 (similar to step 7A4 in Figure 7A). That is, the corrected AR display position (e.g., position in the X-Y plane) is obtained. Using the corrected AR display position, the AR glasses 1 generate data for the display image on the display surface (data to realize a display like that shown in Figure 7B, in other words, video frames) through conversion, etc.

[0121] In this correction, the AR glasses 1 correct at least the superimposed display position of the AR image. However, the size, distortion, color / brightness, etc. of the AR image may also be corrected and adjusted before the AR image is displayed. For example, the alert image's position within the display surface (area 3A in the corresponding Figure 7B) is corrected, and its size is adjusted by scaling to match its depth position. Figure 6B will explain the size adjustment in more detail. When viewing J1 (object 1) from the origin position 602 of the AR glasses 1, J1 (object 1) appears smaller when it is at position 603, which is further away, than when it is at position 604. Therefore, in order for the AR image to be suitably superimposed when J1 (object 1) is viewed from the origin position 602 of the AR glasses 1, the display position is corrected to a position that takes into account the amount of deviation (Xd, Yd) from the coordinates (Xo, Yo), and the size is also adjusted to match the depth position (in this example, since the depth position changes to a farther distance due to the correction, the size of 604A is reduced).

[0122] In step S5, the AR glasses 1 drive the display 131 (Figure 4) based on the data obtained in step S4 to display an AR image on the display surface of the display 131. In step S6, the AR glasses 1 check and determine whether the operation mode is ending, and if so, the operation mode ends in step S7. After that, this flow ends. If the operation mode continues, the process returns to step S2 and repeats in the same manner at each point in time. For example, the display position of the AR image superimposed on the object will also change in accordance with the movement of the object.

[0123] [Detailed Processing Flow] Figure 9 shows a more detailed example of the processing flow, corresponding to the details of steps S2 to S4 in Figure 8. In step S21, the AR glasses 1 acquire non-AR display information from the controller 10. Non-AR display information can be acquired as a type of vehicle information 4 and is information for display in non-AR, such as the vehicle's speed and the distance to the destination. While AR displays an image superimposed according to the position of the object, non-AR displays an image that is not superimposed according to the position of the object. For example, an image representing the vehicle's speed can be displayed at a predetermined position on the display surface. In step S22, the AR glasses 1 acquire AR display information for AR display from the controller 10. AR display information is information such as the alert image mentioned above, which is a warning or caution regarding an object that may be an obstacle.

[0124] In step S21, multiple non-AR display information may be acquired. Similarly, in step S22, multiple AR display information may be acquired. Multiple display information may be exchanged in a single communication between the controller 10 and the AR glasses 1.

[0125] In step S31, the AR glasses 1 determine whether to display and the priority of one or more display information items (including AR information and non-AR information) acquired in step S2. This priority (in other words, display priority) is the priority given to displaying the target display information on the AR glasses 1. For example, for multiple AR images of multiple objects, this determines which one to display first and in what order. In step S31, the AR glasses 1 determine which information (AR images of objects) to display based on priority from the information received and acquired from the controller 10. The AR glasses 1 determine whether to display and the display order for multiple pieces of information based on display priority. In this determination, it is possible to decide to display all information or not to display any information. This display priority may be defined in advance among the display information, or it may be determined each time based on predetermined conditions and algorithms. For example, conditions may be set in advance such that the AR image of an alert has a higher display priority than the non-AR image of vehicle speed.

[0126] In step S32, the AR glasses 1 generate non-AR display video data based on the non-AR display information (such as vehicle speed) that was decided to be displayed in step S31. Also, in step S33, the AR glasses 1 generate AR display video data based on the AR display information (such as alerts) that was decided to be displayed in step S31.

[0127] In step S41, the AR glasses 1 determine whether display correction (including AR display position correction) is necessary for the displayed image data. If it is necessary (Y), the process proceeds to step S42; if it is not necessary (N), step 42 is omitted and the process proceeds to step S5. One method for determining whether AR display position correction is necessary is to determine if correction is necessary when there is a difference (Δ) between the vehicle origin position coordinates (vehicle information acquisition origin position information 4B) obtained from the controller 10 and the position coordinates of the AR glasses 1. Another method is to determine if correction is necessary when the difference (Δ) is greater than or equal to a threshold.

[0128] In step S42, the AR glasses 1 perform display correction (including AR display position correction). The AR display position correction process is as shown in Figure 7A above.

[0129] [Differences in Vehicle Origin Position] Figure 10A is an explanatory diagram regarding differences in the vehicle origin position. The vehicle information acquisition origin position coordinates (vehicle origin position) may differ for each vehicle 2, as described above (Figure 27). Regardless of where the vehicle origin position is, corrections and other measures are possible. The process of correcting the AR display position (corresponding object position coordinates) based on the difference (Δ) between the vehicle origin position and the AR glasses position remains the same. The amount of correction will vary depending on the case.

[0130] Figure 10A illustrates each case. The position of the AR glasses 1, in other words, the user position (the position of the person wearing the AR glasses 1, for example, the position of the driver seated in the driver's seat), is given by coordinates (Xu, Yu). Case 1 is the case where the vehicle origin is shown as A, with coordinates (Xa, Ya), and is similar to Figure 22 mentioned above. The vehicle origin A is located at the center of the front end of the vehicle 2. The correction method (difference) is expressed as (Xu - Xa, Yu - Ya). That is, the correction should reflect the difference (Δ) obtained by subtracting the vehicle origin from the AR glasses position / user position, in other words, the relative positional relationship between the two coordinate systems, in the AR display position (object position coordinates).

[0131] Case 2 is when the vehicle origin is at position B shown in the figure, with coordinates (Xb, Yb). This vehicle origin position B is located in the center of the dashboard at the front of the vehicle. The correction method (difference) is expressed as (Xu - Xb, Yu - Yb). Case 3 is when the vehicle origin is at position C shown in the figure, with coordinates (Xc, Yc). This vehicle origin position C is located in the center of vehicle 2 (vehicle center position C0, described later). The correction method (difference) is expressed as (Xu - Xc, Yu - Yc).

[0132] Case 4 differs significantly from Cases 1-3 in that the AR display is based on the AR glasses position / user position, rather than the vehicle's origin position. In this case, AR display correction is unnecessary.

[0133] [Differences in User Position] Figure 10B is an explanatory diagram illustrating the differences in the position of user U1 wearing AR glasses 1 inside vehicle 2. The vehicle origin position (Xv, Yv) is assumed to be the same as, for example, vehicle origin position A in Figure 10A. User position A is the case of a driver seated in the driver's seat, as described above, and is shown by coordinates (Xua, Yua). User position B is the case of a passenger seated in the passenger seat, and is shown by coordinates (Xub, Yub). User position C is the case of a passenger seated on the right side of the rear, and is shown by coordinates (Xuc, Yuc). User position D is the case of a passenger seated on the left side of the rear, and is shown by coordinates (Xud, Yud). Similarly, for each user position, the AR display position can be corrected using the difference (Δ) from the vehicle origin position. The magnitude of the difference (Δ) from the vehicle origin position differs depending on each user position.

[0134] [Control of correction according to the difference] The larger the difference (difference Δ) between the AR glasses position / user position and the vehicle origin position, the more effective the correction is. As an example of applied control, if the difference (difference Δ) is large, for example above a threshold, a judgment may be made to perform the correction, and if the difference (difference Δ) is small, for example below a threshold, no correction may be performed. In the flow of Figure 8, the AR glasses 1 may decide to perform the correction in step S4 if the difference Δ is greater than or equal to a threshold.

[0135] [Control based on multiple sensors and sensor positions] In addition, multiple sensors mounted on the vehicle 2 may provide multiple pieces of detection information (e.g., object position coordinates) for each object. In this case, normally the controller 10 integrates the object position information from each sensor to calculate / determine a single object position coordinate and transmits that single object position coordinate to the AR glasses 1. However, there may also be cases where multiple pieces of detection information (e.g., object position coordinates) obtained by multiple sensors mounted on the vehicle are transmitted to the AR glasses 1 as they are.

[0136] In the former case, where the controller 10 integrates detection information (e.g., object position coordinates) into one, it is likely that a predetermined method is used to determine which detection information (object position coordinates) is selected. For example, the closest detection information (object position coordinates) among multiple detection information (object position coordinates) acquired by multiple sensors may be selected, or the midpoint of multiple distances may be selected, or the average value of multiple object position coordinates may be calculated. The controller 10 transmits the object position coordinates (sensor integration information) selected / calculated according to the predetermined method to the AR glasses 1.

[0137] As in the former case, when the controller 10 transmits integrated object position coordinates (object detection information 4A), the AR glasses 1 can correct the AR display position using the difference (Δ) between the received and acquired object position coordinates and a single vehicle origin position.

[0138] Furthermore, if the positions of each sensor on the vehicle 2 are different (as described later), it is possible to define and determine one representative sensor position, such as the midpoint or the center of gravity of each sensor position, and consider that representative sensor position to be equivalent to the vehicle origin position. In this case, the controller 10 may use the representative sensor position as the vehicle origin position and transmit the vehicle information acquisition origin position information 4B to the AR glasses 1. Alternatively, the controller 10 may consider the position of each sensor as the vehicle origin position and transmit the sensor position and detection information as a set to the AR glasses 1. Or, information on the positional relationship between the vehicle origin position and the sensor position (for example, the difference, for example, the difference 2301 in Figure 23) may be transmitted to the AR glasses 1. In these cases, the AR glasses 1 may perform correction using the difference (Δ) between the sensor position and the AR glasses position.

[0139] In the latter case, where the detection information (object position coordinates) from each sensor is transmitted directly to the AR glasses 1, the AR glasses 1 corrects the AR display position using the difference (Δ) between the received and acquired object position coordinates and the vehicle origin position. For example, the AR glasses 1 may integrate these multiple object position coordinates into one. Various methods (such as intermediate values) used by the controller 10 can be applied to the integration. The AR glasses 1 may also perform the correction using one object position coordinate selected from the multiple object position coordinates using a predetermined method.

[0140] Figure 10C is an explanatory diagram illustrating a case where a vehicle 2 is equipped with multiple sensors 11 (e.g., cameras and LiDAR), and multiple sensors 11 detect the same object, resulting in multiple detection data (e.g., object position coordinates). In this example, the coordinates of the vehicle origin (Xv, Yv) are assumed to be at the center of the vehicle 2 (vehicle center position: C0), similar to the vehicle origin position C in Figure 10A. The coordinates of the user position / AR glasses position (Xu, Yu) are assumed to be at the driver's seat, similar to the above. In this example, the sensors 11 include sensors sx1 to sx6. Sensors sx1 and sx2 are located on the left and right of the front of the vehicle, sensors sx3 and sx4 are located on the left and right of the sides of the vehicle, and sensors sx5 and sx6 are located on the left and right of the rear of the vehicle. Each sensor is assumed to be a type of sensor capable of measuring the position coordinates of an object. The measured object position coordinates are assumed to be values ​​measured relative to the position of the sensor. The table below shows the sensor position and acquired value (object position coordinates) for each sensor. For example, this is the case when object J1 is detected simultaneously by sensors sx1, sx2, and sx3. The sensor positions are described in the vehicle coordinate system and may also be described as the difference from the vehicle origin.

[0141] The following is an example of how the controller 10 integrates multiple sensor detection information into a single object position coordinate. The controller 10 obtains the acquired values ​​(object position coordinates) from sensors sx1, sx2, and sx3 for, for example, the object J1. When the controller 10 selects one of the three sensor detection information, it may, for example, select and use the acquired value (object position coordinate) from sensor sx1, which is the closest. Alternatively, the controller 10 may calculate the average value of the acquired values ​​from the three sensors. The controller 10 transmits the integrated sensor detection information, for example the acquired value (object position coordinate) from sensor sx1, to the AR glasses 1 as object detection information 4A. At the same time, the controller 10 also transmits the sensor position information of sensor sx1 as a set to the AR glasses 1. The AR glasses 1 perform correction using the acquired value (object position coordinate) and sensor position of sensor sx1 obtained from the controller 10. Specifically, the sensor position can be considered as the vehicle origin position. The AR glasses 1 correct the object position coordinates (Xo1, Yo1), which are acquired values ​​from sensor s1, by reflecting the difference (Δs1) between the user position (Xu, Yu) and the sensor position (Xs1, Ys1).

[0142] The following is an example of how the controller 10 transmits detection information from multiple sensors. The controller 10 obtains detection information (e.g., object position coordinates) from sensors sx1, sx2, and sx3 for, for example, an object J1. The controller 10 transmits the three sensor detection information to the AR glasses 1 as object detection information 4A. At the same time, the controller 10 also transmits the sensor position information of each sensor as a set to the AR glasses 1. When the AR glasses 1 selects one of the three acquired sensor detection information, for example, it may select the acquired value (object position coordinates) from sensor s1, which is the closest. Alternatively, the AR glasses 1 may calculate the average value of the acquired values ​​from the three sensors. The AR glasses 1 performs correction using, for example, the acquired value (object position coordinates) from sensor s1 and the sensor position. Specifically, the sensor position can be considered as the vehicle origin position. The AR glasses 1 correct the object position coordinates (Xo1, Yo1), which are acquired values ​​from sensor sx1, by reflecting the difference (Δs1) between the user position (Xu, Yu) and the sensor position (Xs1, Ys1).

[0143] Another possible method for multiple sensors is as follows: The controller 10 may convert the acquired values ​​(object position coordinates) from each sensor to values ​​relative to the vehicle origin position. For example, the object position coordinates (Xo1, Yo1) detected by sensor sx1 are converted to values ​​(Xo1', Yo1') relative to the vehicle origin position (Xv, Yv). That is, this conversion reflects the difference (Δsx1) between the vehicle origin position (Xv, Yv) and the sensor position (Xs1, Ys1). The controller 10 transmits the converted object position coordinates to the AR glasses 1. In this case, when correcting, the AR glasses 1 only need to reflect the difference (Δ) between the user position and the vehicle origin position.

[0144] [AR Glasses Position] Figure 11 shows a specific example of how to determine the position of the AR glasses 1 (AR glasses position / user position) inside the vehicle 2.

[0145] Figure 11(A) shows the first example. The AR glasses 1 use their own sensors (for example, the rear camera 133 and sensor group 150 in Figure 4) to recognize the space inside the vehicle 2 and determine the seat (seat position) where the user U1 wearing the AR glasses 1 is seated inside the vehicle 2. The AR glasses 1 grasp the determined seat position as the AR glasses position / user position. Position coordinates may be defined in advance for each seat. The AR glasses 1 may also notify the controller 10 of the in-vehicle system 100 of the seat occupancy information representing the determined seat as the AR glasses position / user position (AR glasses information 7 in Figure 2). This allows the controller 10 to grasp the AR glasses position / user position.

[0146] Figure 11(B) shows a second example. The in-vehicle system 100 obtains information from the in-vehicle camera 531 and the seating sensor 510 shown in Figure 3. Based on this information, the controller 10 can determine the position of the AR glasses / user. The AR glasses 1 obtain vehicle information 4, such as in-vehicle camera information and seating information, from the controller 10, and based on the obtained information, determine the seat (seat position) in which the user U1 wearing the AR glasses 1 is seated. The AR glasses 1 recognize the determined seat position as its own AR glasses position / user position.

[0147] In the above example, the seat position was used as the AR glasses position / user position. In the X-Y plane as shown in Figure 22 above, the seat position and the position of the AR glasses 1 / user U1, particularly the position of the head, were considered to be roughly the same. However, it is not limited to this, and a more detailed three-dimensional position including the Z direction may be determined. The AR glasses 1 or the in-vehicle system 100 uses sensors to recognize the head of the AR glasses 1 or user U1 and determines the position coordinates (Xu, Yu, Zu) of the AR glasses 1 / user U1, including the Z direction.

[0148] [Specific Examples of Correction] Figure 12 shows a specific example of correcting the AR display position. Figure 12 shows the difference between the object position coordinates received by the AR glasses 1 from the controller 10 and the AR display position corresponding to the corrected object position coordinates. The table in Figure 12(A) has the following column items: vehicle center coordinates, vehicle origin coordinates (vehicle information acquisition origin coordinates), AR glasses coordinates, object type and object coordinates (before correction), video object (AR video), and AR display coordinates (after correction). The coordinates are in the case of two-dimensional coordinates (X, Y). In this table, three examples are shown in three rows. Note that this table shows an example of data when the vehicle information acquisition origin is described relative to the vehicle center coordinates (C0), as in Case 1 described later. Figure 12(B) is a (X, Y) coordinate diagram as a supplementary diagram corresponding to the example in (A).

[0149] Example 1: With respect to the vehicle center coordinates (0,0), the vehicle origin coordinates (vehicle information acquisition origin coordinates) are (0,20). The AR glasses coordinates (5,0) are (5,0) relative to the vehicle center coordinates (0,0), and correspond to the position of the driver's seat in a right-hand drive vehicle. The object type is a person, and the object coordinates are (-20,50). The object coordinates (-20,50) are the coordinates detected and measured by an on-board sensor with respect to the vehicle origin coordinates (0,20), and are the position before correction. The image object (AR image) generated for this object is an object detection warning (alert image) related to a person. The corrected AR display coordinates (position in the X-Y plane) are (-25,70). The difference (Δ) between the vehicle origin coordinates and the AR glasses coordinates for correction is (5-0,0-20) = (5,-20). The corrected display coordinates change from (-20-5, 50-(-20)) to (-25, 70).

[0150] Example 2: The vehicle origin coordinates (vehicle information acquisition origin coordinates) are the same (0,0) as the vehicle center coordinates (0,0). The AR glasses coordinates (5,0) are the same as in Example 1. The object type is a car, and the object coordinates are (-3,30). The object coordinates (-3,35) are the coordinates detected and measured by the on-board sensor with the vehicle origin coordinates (0,0) as the reference, and are the position before correction. The image object (AR image) generated for this object is an object detection warning (alert image) related to the car. The corrected AR display coordinates (position in the X-Y plane) are (-8,25). The difference (Δ) between the vehicle origin coordinates and the AR glasses coordinates for correction is (5-0,0-0) = (5,0). The corrected display coordinates change from (-3-5,35-0) to (-8,35).

[0151] Example 3: Example 3 is an example where an AR image is displayed using a flag indicating "This is the destination" at the location of a destination set by navigation / map, etc. The object here is the location of the destination in space. Also, this is an example of display by a passenger wearing AR glasses 1, not the driver. The vehicle center coordinates are (0,0) and the vehicle origin coordinates (vehicle information acquisition origin coordinates) are (5,15). The AR glasses coordinates (-5,0) correspond to the position of the passenger seat. The object type is the destination, and the object coordinates are (25,100). The object coordinates (25,100) are coordinates detected and measured relative to the vehicle origin coordinates (5,15), and are the position before correction. The image object (AR image) generated for this object is an image of a flag representing the destination. The corrected AR display coordinates (position in the X-Y plane) are (35,115). The difference (Δ) between the vehicle origin coordinates and the AR glass coordinates for correction is (-5 - 5, 0 - 15) = (-10, -15). The corrected display coordinates change from (25 - (-10), 100 - (-15)) to (35, 115).

[0152] In Example 3, instead of the detection information (object position coordinates) from the sensor 11 of the in-vehicle system 100, the destination position coordinates information from the navigation / map function of the in-vehicle system 100 is used, and the same correction can be applied. In other words, this correction is a conversion from position information based on the vehicle 2 to position information based on the AR glasses 1.

[0153] The information regarding the location of the destination is transmitted from the controller 10 to the AR glasses 1, as in the previous case. The controller 10 can determine the current location of the vehicle 2 using the GPS receiver 518 shown in Figure 3, and can determine the location of the destination as seen from the vehicle 2 based on the difference between the location of the vehicle 2 and the location of the set destination. Alternatively, the in-vehicle system 100 may recognize facilities or other structures of the destination from the camera's captured image and calculate the coordinates of the destination's location.

[0154] [Display of Blind Spots] Figures 13A and 13B show features unique to smart glasses, or in other words, functions that can be realized by smart glasses, which differ from devices fixed to the vehicle 2, such as HUDs. These figures are schematic diagrams of the view from inside the vehicle 2 (near the center) looking forward through the windshield 3, etc. When using AR glasses 1 inside the vehicle 2, the display scope (area 3A) changes according to the orientation of the head and face of the user U1 wearing the AR glasses 1. The vehicle 2's sensor 11 can also detect blind spots (objects in blind spots) of the vehicle 2 as seen by the driver, etc., and the controller 10 can notify the AR glasses 1 of object detection information 4A. If this notification is received, the AR glasses 1 can treat objects in blind spots as targets for AR display.

[0155] Figure 13A schematically shows a state in which user U1, who is the driver, is seated in the driver's seat and wearing AR glasses 1, and user U1's head 1301 is facing forward 1302 (+Y direction). Corresponding to the forward 1302 is the driver's field of view 1303. Corresponding to the field of view 1303 is a region 3A (1304) corresponding to the display surface of the AR glasses 1. If there is an object 1305 in the forward region 3A (1304), as described above, AR images 1306 such as alerts can be superimposed and displayed along with correction.

[0156] Figure 13B schematically shows the case where user U1 has their head 1301 facing diagonally to the front left 1307. Corresponding to the diagonally to the front left 1307 is the driver's field of view 1308. Corresponding to the field of view 1308 is the area 3A (1309) corresponding to the display surface of the AR glasses 1. The pillar, etc., diagonally to the front left from the driver's perspective is a blind spot. In the diagonally to the front left area 3A (1309), if there is an object 1310 (for example, a pedestrian or other object that requires attention) in the blind spot, and if the object 1310 can be detected by the vehicle 2's sensor 11, the AR glasses 1 can receive a notification and, along with correction, superimpose an AR image 1311 such as an alert.

[0157] Figure 13C shows a comparative example of a HUD. A video display device fixed to the vehicle 2, such as a HUD, has a fixed display area 1313 relative to the vehicle 2, regardless of the driver's head orientation. As shown below, the HUD 1312 is fixedly installed on the dashboard or other location on the vehicle 2. The projection direction of the video light from the HUD 1312 is basically fixed. A virtual image display area 1313 is set on the windshield 3 corresponding to the projection area 1314 of the HUD 1312. This display area 1313 is fixed and basically cannot be moved. If there is an object 1310 in the blind spot diagonally to the front left, it will not fit within the display area 1313, so it is not possible to superimpose a virtual image onto the object 1310. Thus, it is not possible to superimpose AR images onto objects 1310 that do not fit within the display area 1313. However, as an alternative, information such as the location and type of the detected object 1310, as well as non-AR images to draw attention, may be displayed within the display area 1313. Similarly, in the AR glasses of the present invention, information display using non-AR images can be used to draw attention to the driver. For example, when the driver is not facing the object 1310 (facing direction 1302), attention may be drawn with information display using non-AR images, and when the driver faces the object 1310 (facing direction 1307), attention may be drawn with superimposed AR images.

[0158] [Multiple Objects and Correction] Figure 14A shows the case where there are multiple objects to be corrected (AR display objects). Even when there are multiple objects at the same time, it is possible to control the correction of the AR display position. In this example, at a certain point in time, "object 1" (e.g., a bicycle) and "object 2" (e.g., a person) are detected as objects. Object detection information 4A for each object detected by the in-vehicle system 100 is notified to the AR glasses 1. The vehicle origin position 1401 and the AR glasses position 1402 are the same as in the example in Figure 7A above.

[0159] Position p1 is the position of object 1 before correction, and the virtual image 1411 is a schematic diagram (showing the position on the X-Y plane) of the AR image (for example, an alert image for bicycle object detection warning) superimposed on position p1 before correction. Position p2 is the position of object 2 before correction, and the virtual image 1421 is a schematic diagram of the AR image (for example, an alert image for person object detection warning) superimposed on position p2 before correction. When correcting the virtual image 1411 of object 1 or the virtual image 1421 of object 2, the AR display position is corrected using a difference (Δ). Position p1' is the corrected position, and the virtual image 1412 is the AR image superimposed on the corrected position p1'. Position p2' is the corrected position, and the virtual image 1422 is the AR image superimposed on the corrected position p2'.

[0160] The virtual image in Figure 14A is a schematic representation of the virtual image at a position on the X-Y plane for illustrative purposes; the actual virtual image on the display surface of the AR glasses 1 is as shown in Figure 14B, etc.

[0161] AR glasses 1 determine whether to perform corrections on the two objects that have received notifications in step S41 of Figure 9. For example, AR glasses 1 makes this determination based on display priority, etc. Figures 14B, 14C, and 14D show examples of this control.

[0162] Figure 14B shows the case where the virtual image 1411 of object 1 is corrected, but the virtual image 1421 of object 2 is not corrected. The virtual image 1411 for object 1 has its display position corrected in the display area and is superimposed as the corrected virtual image 1412 to match the position of object 1. The virtual image 1421 for object 2 is not corrected and is not superimposed on the position of object 2, so there is a misalignment.

[0163] Figure 14C shows the case where the virtual image 1421 of object 2 is corrected, but the virtual image 1411 of object 1 is not corrected. The virtual image 1421 for object 2 has its display position corrected in the display area and is superimposed as the corrected virtual image 1422 to match the position of object 2. The virtual image 1411 for object 1 is not corrected and is not superimposed on the position of object 1, so there is a misalignment.

[0164] Figure 14D shows the case where both the virtual image 1411 of object 1 and the virtual image 1421 of object 2 are corrected.

[0165] Examples of how to decide whether or not to apply correction to multiple targets are as follows. As an example of control, correction may be applied to all targets, but from the perspective of processing load, the number of targets to be corrected may be limited to a predetermined maximum number. A priority order for correction may also be determined for multiple targets.

[0166] In the example shown in Figure 14A, the object position coordinates detected relative to the vehicle origin position 1401 (i.e., the position information before correction) show that object 1 is farther away from object 2 in the depth direction (Y direction), resulting in a greater distance. This is also true when considering the distance from the AR glasses position 1402 (i.e., the position information after correction). From the perspective of user U1 at the AR glasses position 1402, the shift in the virtual image for closer objects is more noticeable, while the shift in the virtual image for farther objects is less noticeable. Furthermore, considering safety, it is desirable to pay more attention to closer objects. Therefore, one approach is to prioritize the closer object for multiple objects, correct the AR display position, and superimpose the AR image with high accuracy. The distance determination in this case can be based on absolute distance or relative distance.

[0167] For example, using a distance threshold 1410 in the depth direction, if the distance to object 1 is greater than or equal to the threshold, and the distance to object 2 is less than the threshold, it may be determined that object 2 is corrected and object 1 is not. In this case, the appearance will be as shown in Figure 14C.

[0168] [Priority] Figure 15 shows a specific example of priority. Table (A) in Figure 15 shows an example of the definition of priority. Here, priority is the priority regarding whether or not to correct the AR display position. This correction priority is set in advance. In this example, there are three priority values: high, medium, and low, but it is not limited to these. The higher the priority, the greater the need for correction of the AR display position. For example, if the priority is high, correction is basically required, if it is medium, correction may be performed on a case-by-case basis, and if it is low, correction may basically not be required. Each column item in the table, such as "Distance to object," indicates a viewpoint or condition for determining priority, and at least one of them may be applied. Multiple viewpoints and conditions may be combined and applied. In addition, priority may be applied not only to whether or not to correct, but also to the order of correction processing, and corrections may be performed first for items with higher priority, and the display may reflect the correction results.

[0169] First, as explained in the example in Figure 14A, the priority may be changed according to the distance to the object as seen from vehicle 2. In the example where "distance to the object" is used, a closer distance results in a higher priority, and a farther distance results in a lower priority.

[0170] Furthermore, the priority may be changed depending on the type of object detected. In the example where "object type" is used, for example, a person may be given a high priority, a bicycle a medium priority, and a car a low priority. In the example where the object type is a person or a bicycle, if the age group of the person or bicycle rider can be determined, the priority may be changed according to the age group. Moreover, the priority may be changed according to the orientation of the object. If the detected object is in an orientation that makes it difficult to see the vehicle (for example, the vehicle is behind the object), the priority may be set high, and if it is in an orientation that makes it easy to see (for example, the vehicle is in front of the object), the priority may be set low.

[0171] Furthermore, the priority may be changed depending on the position of the object as seen from vehicle 2 (in other words, its positional relationship in the left-right direction). In the example where "position of the object" is used, for example, if the object is directly in front of the vehicle's origin position / AR glasses position, the priority may be high; if the object is to the left or right of the front, the priority may be low; and if the object is in the vehicle's blind spot, the priority may be medium.

[0172] Furthermore, the priority may be changed depending on the movement of the object. In the example where "movement of the object" is used, for example, if the object is moving (in other words, in motion), the priority may be set high, and if it is not moving (in other words, stationary), the priority may be set low. Moreover, in the example where the object is moving, the priority may be changed depending on the direction of the object's movement.

[0173] Furthermore, the priority may be changed according to the object's speed. In the example where "object's speed" is used, for example, if the object's speed is fast, the priority may be high; if it is normal, the priority may be medium; and if it is slow, the priority may be low. Speeds such as fast / slow may be distinguished by thresholds, for example. Also, the object's speed may be defined as the speed relative to the speed of vehicle 2 (i.e., the speed of AR glasses 1).

[0174] This system may have a default priority setting as shown in the table in Figure 15. Furthermore, users may be able to change the priority settings in their user settings. Conditions such as "distance to the object" may also be able to be turned on or off. Additionally, when multiple conditions apply, the system will consider which condition the AR glasses 1 should prioritize.

[0175] The table in (B) of Figure 15 shows an example of setting priorities (referred to as condition priorities) for multiple conditions (in other words, the state of the object) in (A). For example, there are condition priorities from 1 to 5, with 1 being the highest and 5 being the lowest. For example, condition priority = 1 is the "distance to the object" condition, and condition priority = 2 is the "position of the object" condition. In step S41 of Figure 9, the AR glasses 1 determine whether or not correction is necessary, and whether or not, while considering these condition priorities.

[0176] The table in (C) of Figure 15 shows examples of other setting items. Setting item 1 allows you to set the upper limit of the number of AR objects that can be displayed simultaneously. In step S31 of Figure 9, the AR glasses 1 refer to this upper limit to determine which AR objects to display. Setting item 2 allows you to set the display priority of AR objects according to the type of object detected. In step S31 of Figure 9, the AR glasses 1 refer to this display priority to determine which AR objects to display. For example, the display priority, in descending order, is people > bicycles > cars. Setting item 3 allows you to set the upper limit of the number of corrections when correcting the AR display position for multiple AR objects simultaneously. In step S41 of Figure 9, the AR glasses 1 refer to this upper limit to determine which AR objects to correct.

[0177] Display priority and correction priority are independent concepts. For detected objects, display priority concerns whether or not to display the AR object itself, while correction priority concerns whether or not to correct the AR display position of the displayed AR object. One control example is to determine the correction priority (whether or not to correct) in association with the display priority (whether or not to display). Objects with a high display priority could be automatically corrected.

[0178] Figure 16 shows examples of whether AR objects are displayed and corrected. Correction is performed on objects with a high correction priority. In the example in Figure 16, objects j1, j2, and j3 are detected in front of vehicle 2. For example, object j1 has the shortest distance in the depth direction, and is set to have a high display priority and a high correction priority. The virtual image 1601 for object j1 is displayed and corrected, and is superimposed to match object j1. Object j2 has a relatively large distance in the depth direction, and is set to have a medium display priority and a medium correction priority. The virtual image 1602 for object j2 is displayed but not corrected, and there is a misalignment with object j2. Although there is a misalignment in the virtual image 1602, it is not very noticeable and still functions as an alert. Object j3 has a large distance in the depth direction, and is set to have a low display priority and a low correction priority. The virtual image for object j3 is not displayed.

[0179] [Correction Timing and Gaze] Figures 17A and 17B show the timing for correcting the AR display position and an example of control using gaze. The timing for correction includes when the AR glasses 1 receive information from the vehicle 2, when the gaze of the user U1, such as the driver, looks at the AR object, etc. The latter determination of gaze may be achieved by detecting the gaze (direction of gaze, etc.) using the in-camera or gaze detection sensor 156 of the AR glasses 1, as described above.

[0180] In the example shown in Figure 17A, an obstacle object J17 (e.g., a person) is detected in front of the vehicle 2. User U1's head 1701 is facing forward 1702, and the display area 3A of the AR glasses 1 is also in front. The AR glasses 1 detect user U1's line of sight. The line of sight range 1703 is the range corresponding to the detected line of sight (direction of gaze) (e.g., a circular area centered on the point of fixation at the end of the line of sight). At this point, user U1 does not see the object J17, and the object J17 (object position coordinates) is outside the line of sight range 1703. At this point, the AR glasses 1 display an AR object (virtual image 1704) for the object J17, but do not correct the AR display position because it is outside the line of sight range 1703.

[0181] The example in Figure 17B shows the case where user U1's head 1701 is turned slightly to the left and their gaze approaches the object J17, or where the head 1701 does not move but the gaze is turned slightly to the left and approaches the object J17. The AR glasses 1 detect the orientation of the AR glasses 1 or the gaze of user U1. The detected gaze range 1705 has moved to the left of the gaze range 1703 in Figure 17A. At this point, user U1 is seeing the object J17, and at least a part of the object J17 is within the gaze range 1703. At this point, the AR glasses 1 display an AR object (virtual image 1706) for the object J17 and correct the AR display position because it is within the gaze range 1705.

[0182] As in the example above, the determination of whether or not to apply AR correction is not limited to determining whether or not the line of sight is inside or outside the line of sight, but may also be made based on the distance between the line of sight and the object. For example, correction may be applied when the distance between the point of fixation and the object is within a threshold.

[0183] As shown in Figure 17A, etc., AR display may be controlled using gaze information. Correction may be omitted when user U1's gaze is not directed at an object, in other words, when the object is outside the line of sight. Correction may be performed if it is determined that the object is within user U1's peripheral vision even if it is outside the line of sight. The size of the control line of sight may be adjusted to suit user U1.

[0184] [Sensor Information] Figures 18A to 18C show control examples of what information the AR glasses 1 receive from the controller 10 and how the received information is processed. In particular, they show which sensor information is used when the vehicle 2 is equipped with multiple sensors 11. When the AR glasses 1 have detection information from multiple sensors 11, they refer to the sensor that has obtained a valid value (corresponding sensor acquired value) and ignore the sensor that is invalid or has no value (corresponding sensor acquired value). The AR glasses 1 correct the AR display position based on the position (reference point) of the referenced sensor 11. When the AR glasses 1 refer to multiple sensors 11, they correct based on the position of each sensor 11. When the values ​​of multiple sensors 11 have been integrated into one by the in-vehicle system 100, the AR glasses 1 correct based on the integrated position and value.

[0185] In the X-Y plan view of vehicle 2 in Figure 18A, the vehicle has four sensors sx1, sx2, sx3, and sx4 as sensors 11 relative to the vehicle's origin position 1801. Sensors sx1 and sx2 are located on the left and right sides of the front of the vehicle, sensor sx3 is located on the front of the left side of the vehicle, and sensor sx4 is located on the front of the right side of the vehicle. The detection axis of each sensor is, for example, in the diagonal forward direction, but is not limited to this.

[0186] The table in Figure 18B shows detection information and integrated information for multiple sensors 11 (sx1 to sx4). #1 is sensor integrated information, and the reference point is the coordinates (Xv, Yv) of the vehicle origin position 1801. Sensor integrated information is information that integrates the detection values ​​and positions of the four sensors sx1 to sx4 into one. When this sensor integrated information is transmitted to the AR glasses 1, the detection information of the four sensors sx1 to sx4 shown in #2 to #5 is not transmitted. #2 is the sensor sx1 acquired value, and the reference point is the installation position of this sensor sx1 (Xs1, Ys1). Sensor sx1 is a sensor (e.g., a camera) installed on the left front end of the vehicle. The sensor sx1 acquired value in #2 is treated as a set with the other sensor acquired values ​​in #3 to #5. That is, the detection information of the four sensors sx1 to sx4 is treated as a set. If integrated information is not used, the set of detection information of the four sensors is transmitted to the AR glasses 1. Similarly, #3, #4, and #5 are the values ​​obtained from each sensor.

[0187] When the information from multiple sensors 11 (sx1 to sx4) is handled individually without being integrated, the controller 10 transmits detection information, including the acquired values ​​from each sensor 11, to the AR glasses 1 as object detection information 4A. For example, the detection information from four sensors 11 (sx1 to sx4) is treated as a set. For example, sensor sx1 has a reference point (Xs1, Ys1) corresponding to its position (sensor position) on the vehicle 2. When sensor sx1 detects the position coordinates of an object, the object position coordinates, which are the acquired values ​​from sensor sx1, are the position coordinates as seen from the reference point (Xs1, Ys1). Along with the object detection information 4A based on the acquired values ​​from sensor sx1, the controller 10 provides information on the reference point (Xs1, Ys1) of sensor sx1, which is the sensor position, as vehicle information acquisition origin position information 4B. When the AR glasses 1 receive the sensor sx1 acquired value and sensor position information, it should correct the AR display position using the difference (Δ) between the AR glasses position (1802) and the reference point (Xs1, Ys1) of sensor s1, which is the sensor position. In other words, the AR glasses 1 should use the reference point (Xs1, Ys1) of sensor s1 as the vehicle origin position as described above.

[0188] When integrating information from multiple sensors 11 (sx1 to sx4) into a single piece of data, the controller 10 creates a single piece of data (sensor integration information) by combining the detection information, including the acquired values ​​from each sensor 11. The controller 10 transmits the sensor integration information to the AR glasses 1 as object detection information 4A. In this vehicle 2 example, the reference point (Xv, Yv) used when integrating the detection information from the four sensors 11 (sx1 to sx4) in the in-vehicle system 100 is the vehicle origin position 1801. For example, when sensor sx1 detects the position coordinates of an object, the object position coordinates, which are the acquired values ​​from sensor sx1, are converted to a value based on the vehicle origin position 1801. The method of integration is not limited to any particular method, and may include methods such as selecting from multiple sensors as described above (Figure 10C) or calculating the average value.

[0189] The controller 10 transmits sensor integration information, which is the object position coordinates that have been unified into a single object, to the AR glasses 1 as object detection information 4A, and also provides the AR glasses 1 with vehicle information acquisition origin position information 4B, which includes the vehicle origin position 1801 (Xv, Yv). For example, the AR glasses 1 at the user position 1802 receive the object position coordinates based on the sensor integration information with respect to the vehicle origin position 1801. The AR glasses 1 can then correct these object position coordinates using the difference (Δ) between the user position 1802 and the vehicle origin position 1801.

[0190] Figure 18C shows the flow of a control example. The AR glasses 1 determine the type of information from the object detection information 4A received and acquired from the controller 10. Figure 18C corresponds to a detailed processing example of the display-related information acquisition process in step S2 in Figure 8. In step 18C1, the AR glasses 1 determine the type of information received from the controller 10. In this example, the types of information are sensor integrated information (#1) and a set of multiple (four) sensor acquisition values ​​(#2 to #5), as shown in the table in Figure 18B. If the type of information is sensor integrated information (#1), proceed to steps S2 and S3 described above, determine the display image (AR object) using the object position coordinates indicated by the integrated information, and perform the correction process as described above.

[0191] If the type of information is a set of multiple (four) sensor acquisition values ​​(#2 to #5), the process proceeds to step 18C2. In step 18C2, the AR glasses 1 determine which of the multiple (four) sensor acquisition values ​​will be used for control (corresponding sensor acquisition value). There are various methods for determination, similar to the integration method on the in-vehicle system 100 side, and are not limited to any particular method. For example, as described above (Figure 10C), the method of selecting the sensor with the closest object position coordinates (sensor acquisition value) may be used. After determination, the AR glasses 1 proceeds to steps S2 and S3, where the sensor acquisition value of the sensor used for control is used to determine the display image (AR object) and corrects the AR display position.

[0192] The example in Figure 18A shows a case where multiple (four) sensors 11 each detect and measure the position of an object, but the same principle applies when multiple sensors 11 perform a single detection and measurement. For example, this principle applies when measuring the position of an object using a stereo camera consisting of a pair of sensors sx1 and sx2. Also, the example in Figure 18A shows a case where the four sensors 11 are of the same type, but the same principle applies even if the sensors are of different types.

[0193] As shown in the example in Figure 18A, even when the sensor positions (reference points) of multiple sensors 11 vary, it is possible to correct the AR display position by referring to the sensor position (reference point) information for each sensor. Furthermore, as will be described later, it is possible to handle cases where the detection axis / detection direction differs for each sensor (in other words, for each sensor acquired value), or in other words, when the axis directions of the vehicle coordinate system / sensor coordinate system differ.

[0194] [Communication Status] In this system (Figure 1), for example, initially, an AR image was superimposed on an object based on the object's position information from vehicle 2. If the video display device 1 cannot obtain the object's position information from the in-vehicle system 100 due to a communication malfunction such as a CAN within vehicle 2, or if the information cannot be received at a predetermined time interval, i.e., if there is a delay, the video display device 1 itself may detect and measure the object and display the AR image based on the detected information. In this case, correction of the AR display position is unnecessary. The display of the AR image can continue for the same object that was initially displaying the AR image. Once the CAN malfunction is resolved, the display returns to one with corrections applied to the information from vehicle 2. Here, if a communication malfunction occurs when AR images are superimposed on multiple objects, the video display device 1 itself may determine which objects to display the AR image on according to priority.

[0195] Figures 19A and 19B show control examples depending on the communication status between the AR glasses 1 and the controller 10 of the in-vehicle system 100. Figure 19A shows the case where the communication status is normal / good. When CAN and other communication is normal / good, the AR glasses 1 can acquire vehicle-related information (object detection information 4A and vehicle information acquisition origin position information 4B) from the controller 10 of the vehicle 2. In this case, the AR glasses 1 can use the acquired information to perform a normal / good AR display with correction. For example, it can accurately superimpose an alert virtual image 19A1 for the target object J19.

[0196] Figure 19B shows a situation where communication is abnormal or malfunctioning. Examples include a situation where transmission throughput is reduced due to a malfunction or increased load on a network such as CAN, or a situation where transmission and reception are completely impossible due to an abnormality. Because communication is abnormal or malfunctioning, the controller 10 of vehicle 2 may fail to transmit display-related information (object detection information 4A and vehicle information acquisition origin position information 4B). In other words, the AR glasses 1 may fail to receive and acquire vehicle-related information from the controller 10. In this case, the AR glasses 1 cannot acquire the necessary information and therefore cannot perform AR display or update the AR display. Also, if there is a delay in communication between the controller 10 and the AR glasses 1, the timing of acquiring vehicle-related information in the AR glasses 1 will be delayed, causing a delay in the AR display and preventing a suitable AR display. For example, at a certain point in time, it may not be possible to update the superimposed display state of the virtual alert image 19B1 for the object J19, resulting in a discrepancy between the moving object J19 and the unupdated virtual alert image 19B1.

[0197] It is necessary to consider how to respond during malfunctions / delays as shown in Figure 19B, and how to respond when the communication status returns to normal (when vehicle-related information can be acquired again).

[0198] Figures 19C and 19D show examples of control responses according to the communication status in this embodiment.

[0199] Figure 19C shows an abnormal / malfunctioning state in the communication between the controller 10 and the AR glasses 1 (and this state is ongoing). The controller 10 or the AR glasses 1 determine / recognize this abnormal / malfunctioning state. The controller 10 may determine the abnormal / malfunctioning state and notify the AR glasses 1 of this abnormal / malfunctioning state, or the AR glasses 1 itself may determine the abnormal / malfunctioning state. When the AR glasses 1 determine / recognize an abnormal / malfunctioning state, it switches from a mode that displays AR based on vehicle-related information from the controller 10 (first AR function, first mode in Figure 25) to a mode that displays AR based on information from the AR glasses 1 itself (second AR function, second mode in Figure 25). The controller 10 may decide to switch from the first mode to the second mode and notify the AR glasses 1 of this, or the AR glasses 1 itself may decide the mode. The AR glasses 1 use their built-in sensors to detect and recognize objects around the vehicle 2, generate display-related information, and display an AR image (for example, a virtual image 19C1 of an alert) without correction.

[0200] Figure 19D shows the communication status when it returns from an abnormal / malfunctioning state to a normal / good state. The controller 10 or AR glasses 1 determine / recognize this situation. The controller 10 and AR glasses 1 switch from the second mode to the first mode. In the first mode, the controller 10 resumes object recognition and generation / transmission of display-related information. In the first mode, the AR glasses 1 perform AR display with correction based on the display-related information received and acquired from the controller 10. Possible timings for recovery include, for example, immediately after the communication status returns from an abnormal / malfunctioning state to a normal / good state, or after confirming that the normal / good state has continued for a predetermined period of time, when the object on which the AR image is superimposed changes.

[0201] The first mode and first AR function of the in-vehicle system 100 detects objects around the vehicle 2 using the vehicle's sensors 11 and displays AR information based on that detection. In contrast, the second mode and second AR function of the AR glasses 1 detect objects around the vehicle 2 using the AR glasses 1's own sensors and displays AR information based on that detection. In the first mode, the vehicle's high-performance sensors 11 can be used, enabling highly accurate object detection. In the second mode, the AR glasses 1's sensors need to detect objects around the vehicle 2 via the vehicle body, etc., so performance is more limited than in the first mode, but communication with the controller 10 can be eliminated.

[0202] [Mode Settings] System settings and user settings are also possible for the two AR functions / modes shown in Figure 19C above. Figure 28 shows an example of a screen display for user settings related to AR functions / modes in the graphical user interface of the AR glasses 1 or the in-vehicle system 100. For example, normally, the first mode (in other words, the vehicle-based AR function) is set to be used. If user U1 wants to use the second mode (in other words, the glasses-based AR function) temporarily, it is possible to set or instruct the system to use the second mode. In addition, there is a mode switching function as described above, so if this function is set to be used, the mode will be automatically switched in the event of a communication failure, etc.

[0203] [3D Correction] Figure 20A shows a case where corrections are made to the AR display position not only in the X and Y directions (horizontal direction) of space, but also in the Z direction (vertical direction). In this control example, the origin position of vehicle 2 and the origin position of AR glasses 1 are determined in 3D, including the position in the Z direction, which is the height direction, as well as the position coordinates (X, Y, Z). When detecting and measuring an object, vehicle 2 (in-vehicle system 100) or AR glasses 1 detect and measure the position coordinates (X, Y, Z) in 3D, including the position in the Z direction. Correction of the 3D position coordinates is also performed when correcting the AR display position.

[0204] The in-vehicle system 100 or AR glasses 1 of this system corrects the difference (ΔZ) in height (in the Z direction) between the reference height (Zv) of the vehicle 2 and the reference height (Zu) of the driver's viewpoint in the AR glasses 1. In Figure 20A, each reference height (Zv, Zu) is the height from the road surface. Here, the vehicle reference height (Zv) corresponds to the Z coordinate of the vehicle origin position 2001 (e.g., the front of the vehicle), which is the reference point for acquiring vehicle information. The reference height of the driver's viewpoint corresponds to the Z coordinate: Zu of the user position / AR glasses position 2002. As shown in Figure 20A, for example, the reference height (Zv) of the vehicle 2 is about the height of the hood. In comparison, the reference height (Zu) of the viewpoint / head / AR glasses 1 of the driver, user U1, is at a higher position. The difference ΔZ of the reference heights is ΔZ = Zu - Zv.

[0205] Figure 20B, corresponding to Figure 20A, shows an example of how the superimposed AR image appears. The upper figure (A) shows the case where correction is applied in the X and Y directions, but not in the Z direction (ΔZ) for the difference in reference height. The virtual image 20B1 is shifted downwards relative to the object J20. The lower figure (B) shows the case where correction is also applied for the Z direction (ΔZ) for the difference in reference height. The virtual image 20B2 is not superimposed on the object J20. It is possible to omit the correction in the Z direction and only correct in the X and Y directions, but correcting in the Z direction as well allows for a more suitable AR display. Furthermore, by acquiring the three-dimensional movement (pitch, roll, yaw) of the vehicle itself from vehicle information, and acquiring the three-dimensional movement (or the head of the user wearing it) of the AR glasses 1 from the sensor group 150 equipped in the AR glasses 1, and applying additional corrections in conjunction with the positional misalignment correction described above, it is possible to display the AR image superimposed on the target object with higher accuracy. These additional corrections may be performed, for example, only when the three-dimensional movement is greater than a threshold.

[0206] [Passenger] Figures 21A and 21B show cases where a passenger in the front passenger seat or someone other than the driver in the driver's seat wears and uses the AR glasses 1. Figures 21A and 21B also show examples of AR and correction for objects different from those used for AR display of obstacles in the road such as people, bicycles, and cars as described above. These objects are, for example, stores, facilities, destinations on a map, and the examples show speech bubble images representing information about the stores, etc., as AR images.

[0207] The correction of the AR display position may be controlled by conditions such as the following: Correction should be enabled when precise information about the object is desired or when the vehicle is getting closer. Correction should not be enabled when general information about the object is desired (when the AR display position deviation is not as important as when driving) or when the vehicle is far away.

[0208] Figure 21A shows how the AR image appears from the viewpoint (user position 2101) of a driver (user U1) wearing AR glasses 1 (1a) in the upper diagram (A). A virtual image is displayed in area 3a of AR glasses 1a. The lower diagram (B) shows how the AR image appears from the viewpoint (user position 2102) of a passenger (user U2) wearing AR glasses 1 (1b). A virtual image is displayed in area 3b of AR glasses 1b. The AR glasses 1b worn by the passenger are different from the AR glasses 1a worn by the driver. Multiple AR glasses 1 in the vehicle 2 may have similar functions or some different functions. For example, the driver uses the AR function in driving mode. This AR function includes a function to display alerts for obstacles, etc., for driving assistance. A passenger in the front passenger seat uses the AR function, for example, in non-driving mode. This AR function includes, for example, the ability to display information about stores and other locations on a map.

[0209] In the example shown in Figure 21A, for an object J21 (e.g., a bicycle) as seen through the driver's AR glasses 1a, the AR display position of the alert's virtual image 21A1 is corrected in order to achieve high positional accuracy for driving assistance. On the other hand, for an object J21 (e.g., a store) as seen through the passenger's AR glasses 1b, general information is sufficient and high positional accuracy is not required, so the AR display position of the virtual image 21B1 representing the store's information is not corrected.

[0210] Figure 21B(C) shows an example of what the passenger sees through the AR glasses 1b after some time has passed since the state in Figure 21A(B). The vehicle 2 is now closer to the object J22 (store, etc.). When the distance to the object J22 is sufficiently small (for example, below a threshold), the AR glasses 1b changes the virtual image 21B1 of the object J22, refines the information, and corrects the AR display position. The virtual image 21B1 changes to virtual image 21C1 through refinement. The virtual image 21C1 displays detailed information about the store (for example, "Restaurant ABC, XX building, 3F"). For example, if this store (restaurant) is on the 3rd floor of the building, the virtual image 21B1 is superimposed to match the position on the 3rd floor of that building. At that time, correction is made to use the position of the AR glasses 1b as a reference. This makes the positional accuracy of the virtual image 21B1 more precise.

[0211] Figure 21B(D) shows another example, an example of how it looks from the driver's AR glasses 1a, and an example of displaying a flag according to the destination based on a map and GPS location information. The destination may simply be coordinates of latitude, longitude, and altitude, and there may not be a specific object corresponding to the destination. In this example, the destination is an intersection, and the virtual image of the flag 21D1 is displayed according to the position coordinates of that intersection. For the virtual image of the flag 21D1, if the distance from the vehicle 2 to the destination is large, no correction is made to the AR display position, and if the distance is small (for example, below a threshold), correction is made to the AR display position.

[0212] The above examples are not exhaustive; for instance, a passenger in the front passenger seat may use AR features, including driver assistance functions, together with the driver in the driver's seat (e.g., a novice driver).

[0213] [Means for determining the vehicle's origin position] We will provide additional information on the means for determining the vehicle's origin position (in other words, the reference position) for each vehicle (for example, each vehicle type).

[0214] Case 1: When vehicle 2 (in-vehicle system 100) can acquire the following information ("information regarding the vehicle's origin position"), and AR glasses 1 can acquire this information from vehicle 2. In Case 1, when user U1 gets into vehicle 2 and puts on AR glasses 1, it is sufficient that AR glasses 1 can receive and acquire this information from controller 10 during the initial communication. The following information is set and stored in advance in a predetermined storage location on a server (server 8 in Figure 1) for each vehicle 2. Based on information such as its vehicle ID, the in-vehicle system 100 of vehicle 2 can specify and request the relevant information ("information regarding the vehicle's origin position") from the server via communication and acquire the information.

[0215] "Information regarding the vehicle's origin position" - (1) Definition information of the coordinates (X, Y) or (X, Y, Z) related to the vehicle's origin position (in other words, coordinate definition information): For example, definition information such as +X to the right of vehicle 2, -X to the left, +Y in front, and -Y behind for each axis of the 2D coordinate system (X, Y).

[0216] (2) Coordinates of the center of vehicle 2 ("vehicle center coordinates"): For example, the center C0 in Figure 24 is defined as the "vehicle center coordinates" and is set as the origin coordinates (0,0) in a two-dimensional coordinate system of (X,Y).

[0217] ・(3) Vehicle Information Acquisition Origin Coordinates: The "vehicle origin position" mentioned above. These are the coordinates that serve as the reference when detecting, measuring, and acquiring vehicle information (position of objects, etc.) using on-board sensors. These coordinates may vary depending on the vehicle type and manufacturer. The vehicle origin position (vehicle information acquisition origin coordinates) is described as coordinates with the vehicle center coordinates as the reference (origin). For example, position A or position B in Figure 10A is the vehicle origin position (vehicle information acquisition origin coordinates), and is described as (0,20) relative to the vehicle center coordinates (C0) = (0,0). It is also possible that the vehicle center coordinates = vehicle information acquisition origin coordinates.

[0218] The in-vehicle system 100 and AR glasses 1 can acquire the above-mentioned "information regarding the vehicle origin position" (coordinate definition information, vehicle center coordinates, vehicle information acquisition origin coordinates). From this information, the in-vehicle system 100 and AR glasses 1 can determine that the vehicle origin position (for example, C1) is in front of the vehicle center coordinates (C0), etc.

[0219] Furthermore, vehicle 2 (in-vehicle system 100) may acquire the following information.

[0220] (4) Position of AR glasses wearer / each occupant within vehicle 2: The position of the AR glasses / user position as described above. For example, the coordinates of each seat, such as the driver's seat. For example, the coordinates of the driver's seat are described as (10,10) relative to the vehicle center coordinate (0,0). From this information, the in-vehicle system 100 and the AR glasses 1 can determine, for example, that the position of the driver wearing the AR glasses 1 (for example, center C2 in Figure 24) is to the right and rear of the vehicle origin position (for example, C1). Also, as described above (Figure 11 (A)), the in-vehicle system 100 can estimate the presence and position of the AR glasses 1 and each occupant from the information of the in-vehicle camera and seat sensors.

[0221] AR glasses 1 can determine the difference (Δ) from the vehicle origin position and the positional relationship based on the "information regarding the vehicle origin position" as described above and the information regarding the position of the AR glasses themselves. Based on this, AR glasses 1 can correct the AR display position for the object position coordinates acquired from the in-vehicle system 100.

[0222] Case 1 above shows a case where the vehicle center coordinate (C0) is used as the origin of the (X,Y) coordinate system (in other words, the origin for describing both the vehicle origin position and the AR glass origin position), but it is not limited to this.

[0223] Case 2: When sufficient information is not provided from vehicle 2. In other words, when AR glasses 1 cannot obtain sufficient information from vehicle 2. In Case 2, AR glasses 1 estimates the necessary information from the limited information available. AR glasses 1 estimates the vehicle origin position (vehicle information acquisition origin coordinates) even if it is unknown or unclear. In this case, for example, AR glasses 1 may use a provisional vehicle origin position to perform AR display and adjust the AR display position based on feedback from user U1.

[0224] In cases where only limited information can be obtained from vehicle 2 (in-vehicle system 100), such as the following:

[0225] (1) The position of the person wearing the AR glasses / each occupant inside the vehicle 2. For example, the position of each seat. Similar to the information in (4) of Case 1. The in-vehicle system 100 estimates the presence and position of occupants such as the driver wearing the AR glasses 1 using, for example, an in-vehicle camera or seat sensors. For example, the position coordinates of the driver's seat where the driver wearing the AR glasses 1 is seated are described as (10,10) relative to the vehicle center coordinates (0,0). Alternatively, as described above (Figure 11 (B)), the AR glasses 1 themselves may estimate the presence and position of the AR glasses 1 and each occupant inside the vehicle 2 using sensors.

[0226] The AR glasses 1 can correct the AR display position using estimation based on the object position coordinates acquired from the in-vehicle system 100, as follows. For example, let's assume that the AR glasses 1 acquires (0, 100) as the object position coordinates from the in-vehicle system 100.

[0227] AR Glasses 1 first assumes that the reference position for the object's position coordinates (vehicle origin position / vehicle information acquisition origin coordinates) is the vehicle's center position (center C0 in Figure 24). AR Glasses 1 then corrects the display position of the AR image to be superimposed on the object using the difference (Δ) between the vehicle's center position (C0) and its own AR Glasses position, and attempts to display the AR image.

[0228] AR glasses 1, using a user interface, ask and confirm with user U1 whether the AR image is superimposed on the object. User U1's response may be a simple choice between two options, such as "the position is correct" or "the position is incorrect."

[0229] If user U1's input is (a) the position is correct, the AR glasses 1 will identify the vehicle origin position as the assumed vehicle center position (C0) and will use this vehicle origin position in subsequent AR display and correction.

[0230] If user U1's input is (b) the position is off, the AR glasses 1 know that the vehicle origin is not the assumed vehicle center position (C0). The AR glasses 1 select a position other than the vehicle center position (C0) (for example, the front of the vehicle) from the candidates, and set it as the next provisional vehicle origin position. Then, the AR glasses 1 repeats the process of displaying the AR at the provisional position, confirming with the user, and providing feedback, as described above. If the vehicle origin position can be determined as a result, the origin position information is stored in a storage medium that can be accessed at any time, such as the storage device 110 of the AR glasses or external storage. Subsequently, when displaying an AR image using the AR glasses 1 in the vehicle, the stored information is read out, making it possible to display a suitable AR image without having to estimate the vehicle origin position as described above. For the candidate vehicle origin position, for example, multiple position coordinates may be defined in advance and made referable and selectable, as shown in Figure 10A. Alternatively, another method for determining the temporary vehicle origin position is to shift the current temporary position by one step in a predetermined direction and distance, and set that position as the next temporary position.

[0231] Figures 29A and 29B show a specific example of the method of using a provisional vehicle origin position in Case 2 described above. Figure 29A shows the vehicle center coordinates C0 (0,0) of vehicle 2, candidate vehicle origin positions (e.g., CA, CB), AR glasses position C2, object position coordinates, etc. The candidate vehicle origin position CA is at the front of the vehicle, for example, coordinates (0,20). The candidate vehicle origin position CB is near the dashboard, for example, coordinates (0,10). The AR glasses position C2 is for example, coordinates (5,0). The object position coordinates of the object J (e.g., a stationary cone) detected by the in-vehicle system 100 are for example (10,40). The correct vehicle origin position (sensor reference position) is assumed to be vehicle origin position CB. In other words, (10,40) is the relative position from vehicle origin position CB.

[0232] Figure 29B shows an example of the corresponding AR display. Initially, as shown in (A), the AR glasses 1 use the vehicle center coordinate C0 (0,0) as a temporary vehicle origin position and display an AR image (virtual image A1) along with a correction for the difference (Δ0). Virtual image A1 is shifted downwards relative to the object J. User U1 inputs "shifted" in response to confirmation on the interface. Next, as shown in (B), the AR glasses 1 use position CA (0,20) as a temporary vehicle origin position and display an AR image (virtual image A2) along with a correction for the difference (ΔA). Virtual image A2 is shifted upwards relative to the object J. User U1 inputs "shifted" in response to confirmation on the interface. Next, as shown in (C), the AR glasses 1 use position CB (0,10) as a temporary vehicle origin position and display an AR image (virtual image A3) along with a correction for the difference (ΔB). The virtual image A3 matches the object J. User U1 inputs "matches" in response to the interface confirmation.

[0233] The above example uses a stationary object for clarity, but similar adjustments are possible even when displaying AR at different times for moving objects such as people. However, prompting user U1 for input during driving or other situations requiring concentration may interfere with safe driving, so it may be better to limit this to situations such as when the vehicle is stopped.

[0234] [Sensor Orientation] In the example shown in Figure 7A above, the vehicle coordinate system with respect to the vehicle's origin position, for example (X, Y), is shown as being fixed with the X-axis representing the left-right direction of the vehicle 2 and the Y-axis representing the front-rear direction of the vehicle 2. However, the correction can be applied similarly even if the orientation of the coordinate axes relative to the vehicle 2 varies in various ways.

[0235] Figure 30 shows an example where the axis direction of the coordinate system is different from that described above. The examples of the center (C0), vehicle origin position (3001), and AR glasses position (3002) in vehicle 2 are the same as described above. In this example, the on-board sensor 11 is located to the right of the vehicle origin position (3001). The direction of the detection axis of this sensor 11 is oblique to the (X, Y) axis direction of the vehicle, as shown in the figure. In other words, when considering a sensor coordinate system based on the sensor 11, it has a coordinate system of (Xsx, Ysx) as shown in the figure. The sensor values ​​acquired by this sensor 11 are assumed to be coordinate values ​​in this sensor coordinate system (Xsx, Ysx).

[0236] The controller 10 of the in-vehicle system 100 transmits the sensor values ​​acquired by the sensor 11 and related information regarding the coordinate system to the AR glasses 1. Alternatively, the sensor 11 may also transmit information to the AR glasses 1. Based on the acquired sensor values ​​and other information, the AR glasses 1 perform processing to convert the (Xsx, Ysx) axes of the sensor coordinate system to the (X, Y) axes related to the vehicle origin position. The following is an example of this processing.

[0237] (1) Sensor 11 detects the object position coordinates of the target object J30. The object position coordinates obtained by the sensor are, for example, (-10, -10) with respect to (Xsx, Ysx). Controller 10 transmits the sensor values ​​obtained by sensor 11 (object detection information 4A) and related information regarding the coordinate system (vehicle information acquisition origin position information 4B) to AR glasses 1. AR glasses 1 receive and acquire the information from controller 10.

[0238] (2) The AR glasses 1 perform a coordinate transformation, taking into account the difference in axis inclination (direction) between the coordinate system (X, Y) of the vehicle 2 and the coordinate system (Xsx, Ysx) of the sensor 11. At this time, the AR glasses 1 grasp the difference in axis inclination (direction), specifically how much the (Xsx, Ysx) axis is inclined with respect to the (X, Y) axis, based on information from the controller 10. Based on this difference between the two coordinate systems, the AR glasses 1 convert the object position coordinates in the sensor coordinate system (Xsx, Ysx) to the object position coordinates in the vehicle coordinate system (X, Y). The object position coordinates in the vehicle coordinate system (X, Y) may be values ​​with the vehicle origin position 3001 as (0,0) or values ​​with the sensor position as (0,0). In this example, the latter is used.

[0239] (3) The AR glasses 1 correct the AR display position based on the difference (Δ) between the vehicle origin position (the position of sensor 11 in this example) and the AR glasses position, as described above, for the object position coordinates in (X, Y) after conversion. The AR glasses 1 obtain the display position on the display surface from the corrected AR display position (position in the X-Y plane).

[0240] (4) The AR glasses 1 display an AR image (e.g., an alert image) superimposed on the object J30 at the corrected display position.

[0241] The above example shows a case where the AR glasses 1 perform a transformation between two coordinate systems, but the process is not limited to this; the in-vehicle system 100 (controller 10) may also perform the transformation between the two coordinate systems.

[0242] Furthermore, the same conversion and correction can be applied to, for example, the sensor 11b located on the right side of the vehicle 2 facing to the right, and the sensor 11c located on the rear side of the vehicle 2 facing backward.

[0243] [Effects, etc.] According to the above embodiment, regarding the technology for realizing AR functionality using an image display device such as smart glasses in a vehicle, it is possible to achieve a more suitable display, such as reducing the deviation in the superimposed position of the AR image. The above embodiment describes the case in which smart glasses 1 are used as the image display device 1 in the vehicle 2, but it is not limited to smart glasses 1 and can be similarly applied to various in-vehicle image display devices, such as consoles / instrument panels, head-up displays (HUDs) / virtual image display devices, projectors, and aerial levitation image display devices. Even in the case of these image display devices, if there is a difference between the origin position / reference position of the vehicle / sensor and the position of the image display device, it is possible to correct the AR display position to reduce the deviation caused by that difference. In particular, in the case of wearable devices such as smart glasses 1, as shown in Figure 13B, the position and orientation of the wearable device in the vehicle 2 change with the movement of the user U1 and head. At that time, correction of the display position of the AR image can be achieved in accordance with that movement, which is highly effective.

[0244] Although embodiments of this disclosure have been described in detail above, the invention is not limited to the embodiments described above and can be modified in various ways without departing from the gist of the invention. Each embodiment can be modified by adding, deleting, or replacing components, except for essential components. Unless otherwise specified, each component may be singular or plural. Combinations of each embodiment and its variations are also possible.

[0245] 1...Smart glasses (AR glasses, wearable device, video display device), 2...Vehicle, 4A...Object detection information, 4B...Vehicle information acquisition origin position information, 601...Vehicle origin position (Vehicle information acquisition origin coordinates), 602...User position (AR glasses position), U1...User, J1...Object, Δ...Difference.

Claims

1. An image display device comprising: a display; a communication interface for communicating with an in-vehicle system of a vehicle; and a processor, wherein the processor acquires object detection information of an object outside the vehicle from the in-vehicle system via the communication interface; corrects the object detection information, which is based on the position coordinates of the vehicle, to information based on the position coordinates of the image display device; and uses the corrected information to display an image on the display so as to be superimposed on the object outside the vehicle.

2. A video display device usable inside a vehicle, which acquires object detection information of an object outside the vehicle using sensors of the vehicle's onboard system from the onboard system, corrects the object detection information based on the vehicle's position coordinates so that it becomes object detection information based on the position coordinates of the video display device, and displays an image superimposed on the object using the corrected object detection information.

3. The video display device according to claim 2, wherein the video display device acquires vehicle information from the in-vehicle system that allows for the identification of the vehicle's position coordinates.

4. The video display device according to claim 2, wherein the video display device is mounted on the head of a user inside the vehicle and displays the video on a display surface corresponding to the user's field of view.

5. The video display device according to claim 2, wherein the position coordinates of the vehicle and the position coordinates of the video display device are described with the center position of the vehicle as the origin.

6. The video display device according to claim 2, comprising: a first mode for displaying the video based on object detection information from the vehicle; and a second mode for displaying the video based on object detection information of an object outside the vehicle using a sensor of the video display device, wherein the first mode and the second mode can be switched.

7. The video display device according to claim 2, wherein the object detection information of the vehicle includes object position coordinates based on the position of the sensor in the vehicle or the position coordinates of the vehicle, and the correction is performed using the position of the sensor or the position coordinates of the vehicle associated with the object position coordinates.

8. The video display device according to claim 2, wherein the video display device uses a sensor of the video display device to determine the position coordinates of the video display device in the space inside the vehicle.

9. An image display device according to claim 2, wherein the image display device acquires information from the in-vehicle system that can identify the position coordinates of the image display device in the space inside the vehicle using sensors of the in-vehicle system, and determines the position coordinates of the image display device based on the acquired information.

10. An image display device according to claim 2, wherein for each object to be displayed, the presence or absence of the correction or its priority is determined.

11. The video display device according to claim 10, wherein the video display device determines whether or not to perform the correction or the priority of the correction according to the distance between the target object and the vehicle or the video display device.

12. An image display device according to claim 10, wherein the image display device determines whether or not to perform the correction or the priority of the correction according to the type of object being targeted.

13. An image display device according to claim 10, wherein the presence or absence of correction or priority of correction is determined according to the left-right position of the target object with respect to the position of the image display device.

14. An image display device according to claim 10, wherein the image display device determines whether or not the correction is performed or its priority according to whether or not the target object is moving or its speed of movement.

15. An image display device according to claim 2, wherein the image display device determines whether or not the correction is performed according to the state of the user's gaze.

16. A video display device according to claim 6, wherein the first mode is used when the communication status between the in-vehicle system and the video display device is good, and the second mode is used when the communication status is poor.

17. The video display device according to claim 2, wherein, when displaying, a temporary coordinate is set as the position coordinate of the vehicle, the correction is performed, and a first video is displayed as the video; the user is asked to confirm whether the position of the superimposed display of the first video is correct or misaligned; and if the result of the confirmation is that it is misaligned, the correction and display using the temporary coordinate are repeated in the same manner.

18. A method for displaying an image in an image display device usable in a vehicle, comprising: a step of the image display device acquiring object detection information of an object outside the vehicle using a sensor of the vehicle's in-vehicle system from the in-vehicle system; a step of the image display device displaying an image based on the object detection information so as to be aligned with and superimposed on the object; and a step of the image display device correcting the positional misalignment of the superimposed image during the display using the difference between the position coordinates of the vehicle and the position coordinates of the image display device.

Images