Image display device

Abstract

To more appropriately determine a display position of a target image.SOLUTION: An image display device has: a display that is mounted on the head of a user who is a driver of a vehicle and displays a target image in the visual field of the user in a superimposed manner; and a camera for SLAM that is fixed to the display and acquires an image for SLAM obtained by photographing the surroundings of the display. The image display device comprises: extraction means that extracts a characteristic part from the acquired image for SLAM; display position determination means that determines a display position of the target image by using the extracted characteristic part; and exclusion means that excludes the extracted characteristic part when the characteristic part belongs to a steering wheel.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] This specification discloses an image display device that displays a target image in a superimposed manner in the field of view of a user who is a vehicle occupant. [Background technology]

[0002] Conventionally, a technology is known in which a predetermined image is superimposed on the user's field of vision, thereby allowing the user to recognize that a virtual object represented by the image actually exists. For example, Patent Document 1 discloses a technology in which a driver wears smart glasses, which are glasses-type displays, and an image of a preceding vehicle leading the vehicle in which the driver is riding is displayed on the smart glasses. In Patent Document 1, the preceding vehicle represented by the image moves so as to lead the vehicle in which the driver is riding to the destination. Therefore, the driver can move to the destination by performing a driving operation to follow the preceding vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-129406 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order to make the user perceive a virtual object as if it actually exists, the display position of the image (hereinafter referred to as the "target image") displayed on the display must be determined based on the position in real space of the virtual object represented by the target image and the position of the display in real space.

[0005] Patent Document 1 shows that in order to identify the position of the display in real space, the shape of the front window is detected based on an image obtained by a camera, and these are used as markers. Then, the camera captures an image of a scene including the markers, and the position of the display in real space is identified based on the obtained image.

[0006] There is a demand for markers to be easy to recognize in images captured by a camera, but there are some images that are easy to recognize but are not suitable as markers. [Means for solving the problem]

[0007] The image display device disclosed in this specification includes a display device that is worn on the head of a user who is a vehicle occupant and displays a target image superimposed on the user's field of vision, a slam camera that is fixed to the display and obtains a slam image showing the surroundings of the display, an extraction means that performs image recognition processing on the obtained slam image and extracts a characteristic shape as a characteristic part, a determination means that uses stored information about the shape of the steering wheel to determine whether the extracted characteristic part belongs to the steering wheel, an exclusion means that excludes the extracted characteristic part when the extracted characteristic part belongs to the steering wheel, a registration means that registers the characteristic part that does not belong to the steering wheel as an AR marker, and a registration The above AR The display device further comprises a display position determining means for determining a display position of the target image by using a marker.

[0008] An object on the rear side of the steering wheel that can be seen through the steering wheel is set as a target for extracting the characteristic portion. Effect of the Invention

[0009] According to the technology disclosed in this specification, the display position of the target image can be determined more appropriately. [Brief description of the drawings]

[0010] [Figure 1]FIG. 1 is a block diagram showing a configuration of an image display device. [Diagram 2] FIG. 1 is a diagram showing a state in which a user is wearing a wearable device. [Diagram 3] FIG. 2 is a diagram showing a schematic view of a driver who is a user. [Figure 4] FIG. 13 is an image diagram for explaining a space-fixed display and a device-fixed display. [Diagram 5] FIG. 2 is a diagram illustrating a user's field of view when a target image is displayed. [Figure 6] 13 is a flowchart showing a process of capturing the shape of an interior part as an AR marker in the device controller. [Figure 7] FIG. 13 is a diagram for explaining area specification when excluding a steering wheel. [Figure 8] 10 is a flowchart showing a process for displaying a target image while driving. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The configuration of the image display device will be described below with reference to the drawings. In the description, specific aspects are shown to facilitate understanding, but these are merely examples and can be modified as appropriate. FIG. 1 is a block diagram showing the configuration of an image display device 10. In this embodiment, the image display device 10 is configured by a wearable device 12.

[0012] The wearable device 12 is a device that a vehicle occupant (e.g., a driver) wears on his / her head, and is, for example, a glasses- or goggle-type device. The wearable device 12 has a display 14, a slam camera 16, a pupil position sensor 18, and a device controller 20 to function as the image display device 10.

[0013] The wearable device 12 will be described with reference to Fig. 2. Fig. 2 is a diagram showing a state in which a user 100, who is a vehicle occupant, wears the wearable device 12. The wearable device 12 is a device formed in the shape of glasses, and is called smart glasses or AR glasses. The wearable device 12 includes temples 26, which are linear frames for hanging on the ears, and rims 24, which are frames formed in a shape that surrounds the eyes and can be hung on the nose.

[0014] The display 14 displays an image in the field of view of the user 100 wearing the wearable device 12. In this example, the display 14 is an organic EL display or liquid crystal display having a display area 22 arranged inside the rim 24, and displays an image in a part or all of the display area 22. The display area 22 has high transparency. Therefore, when no image is displayed in the display area 22, the user 100 (i.e., the passenger) can see the scenery ahead through the display area 22. Also, when an image is displayed only in a part of the display area 22, the user 100 can see the scenery in the field of view ahead and the displayed image at the same time. In this case, the image may be opaque or translucent. In the following description, the image displayed on the display 14 is called a "target image" to distinguish it from other images. Also, the object virtually represented by the target image is called a "target object."

[0015] The slum camera 16 is a camera fixed to the display 14 and captures an image of the periphery of the display 14. The slum camera 16 is fixed facing forward, for example, near the front end of the temple 26, and captures an image of a range similar to the visual field of the user 100. Hereinafter, an image captured by the slum camera 16 will be referred to as a "slum image." The device controller 20, which will be described later, identifies the position and orientation of the display 14 in real space based on an AR marker captured in the slum image, which will be described later.

[0016] The pupil position sensor 18 is a sensor that detects the positions of the pupils of the right and left eyes of the user 100, and is fixed, for example, near the center of the rim 24. This pupil position sensor 18 can be formed, for example, by using a camera or the like.

[0017] The device controller 20 controls the operation of the wearable device 12. The device controller 20 acquires the image and position information acquired by the slam camera 16 and the pupil position sensor 18, processes this information, and causes the display 14 to display a target image.

[0018] The device controller 20 is physically a computer having a processor 20a, a memory 20b, and a communication I / F 20c. This "computer" also includes a microcontroller in which a computer system is incorporated into a single integrated circuit. The processor 20a refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0019] The memory 20b holds digital data required for computer processing. The memory 20b includes at least one of a main memory connected to the processor 20a via a memory bus and a secondary storage device accessed by the processor 20a via an input / output channel. The memory 20b can be composed of a semiconductor memory (e.g., RAM, ROM, solid-state drive, etc.).

[0020] The communication I / F 20c is wirelessly connected to other electronic devices, specifically, the in-vehicle system 28, and can be connected to various sites via the Internet or the like. In particular, it is capable of communicating with an information center 30 that provides vehicle information. Also, for example, the communication I / F 20c may transmit and receive data to and from the in-vehicle system 28 via short-range wireless communication such as CAN (Controller Area Network) communication, Bluetooth (registered trademark), Wi-Fi (registered trademark), and infrared communication.

[0021] The functions of the device controller 20 may be realized by an external system such as a computer of the in-vehicle system 28, a computer of the information center 30, or another portable computer (such as a smartphone). In this case, the device controller 20 sends information from the slam camera 16 and the pupil position sensor 18 to an external system such as the in-vehicle system 28, receives image data as a result of processing, and displays it on the display 14. Also, some of these processes can be executed in the external system.

[0022] The in-vehicle system 28 is a system mounted on the vehicle and controls various in-vehicle devices. As shown in Fig. 3, the in-vehicle system 28 includes, as interior parts, a meter display 40a provided on the instrument panel, a multi-display 40b provided on the center console, and an electronic inner mirror 40c provided inside the upper part of the windshield. In this embodiment, these interior parts are used as the AR marker 60.

[0023] 3 is a diagram showing a schematic view of the field of view of the driver, who is the user 100. The meter display 40a is a display that displays information about the vehicle state, such as the vehicle speed and fuel efficiency. As shown in FIG. 3, the meter display 40a is disposed on the rear side of the steering wheel 56 as seen from the driver, and the driver can view the display area of ​​the meter display 40a through the steering wheel 56.

[0024] The multi-display 40b is a display that displays information related to in-vehicle electronic devices (such as a navigation device or an audio device). As shown in Fig. 3, the multi-display 40b is disposed in the center of the instrument panel in the vehicle width direction, that is, in a location generally called the center console.

[0025] The electronic inner mirror 40c is a display that displays an image of the rear of the vehicle captured by a backup camera (not shown). The electronic inner mirror 40c is used instead of a rearview mirror (room mirror) that displays the rear of the vehicle by optical reflection. The electronic inner mirror 40c may be switchable between a digital mode that displays an image and a mirror mode that displays the rear by optical reflection. As shown in FIG. 3, the electronic inner mirror 40c is disposed in the same position as a general rearview mirror, that is, near the upper end of the windshield glass. Note that a general room mirror may be used instead of the electronic inner mirror.

[0026] As described above, the device controller 20 generates data of a target image to be displayed on the display 14. Here, the display form of the target image on the display 14 can be either "device-fixed display" or "space-fixed display", but in this embodiment, "space-fixed display" is used. This space-fixed display is a display form in which a target image representing a specific object is displayed as if it exists in real space.

[0027] For example, consider a case where user 100 is looking at a real space in which a table 80 actually exists through display area 22 of display device 14, as shown in Fig. 4. In this case, when a target image 50 representing a "ball" is displayed in display area 22 as shown in state S1 in Fig. 4, naturally, the real space including table 80 and target image 50 of the "ball" are simultaneously reflected in user 100's visual field.

[0028] In the case of device-fixed display, the display position of the target object 72 (a "sphere" in the example of FIG. 4) shown in the target image 50 is determined regardless of the real space. Therefore, in the device-fixed display, as shown in state S2 in FIG. 4, even if the viewpoint of the user 100 moves, the display position, size, and shape of the target image 50 in the display area 22 do not change.

[0029] On the other hand, in a spatially fixed display, the location of the target object 72 (a "ball" in the example of FIG. 4) shown in the target image 50 in real space is identified, and the target image 50 is displayed as if it really exists in the identified location. For example, in a spatially fixed display, consider a case in which the "ball" target object 72 is assumed to be located on a table 80 in real space. In this case, as shown in state S3, the display position, size, and shape of the "ball" in the display area 22 are changed so that the "ball" appears to be located on the table 80 even if the viewpoint of the user 100 moves.

[0030] In this way, by displaying the target image 50 in a spatially fixed display, the user 100 has the illusion that the target object 72 shown in the target image 50 actually exists. In other words, by displaying the target image 50 in a spatially fixed display, it is possible to add, remove, emphasize, or attenuate information in the real environment, thereby expanding the real world as seen by humans. Such technology is generally called "augmented reality" or "AR."

[0031] Next, an example of the display of the target image 50 in this example will be described. FIG. 5 is a diagram that shows a schematic view of the user 100 (the driver in this example) when the target images 50a and 50b are displayed. In the example of FIG. 5, the target image 50a showing the traveling direction of the vehicle and the target image 50b showing a message to the driver are displayed in a fixed manner in space. These target images 50a and 50b are displayed on the display 14 in the same position and size as when the target object shown by the image actually exists. For example, the target image 50a is displayed in the display area 22 in the same position and size as when the arrow-shaped object represented by the target image 50a actually exists on the road surface in front of the vehicle that actually exists. In addition, the target image 50b is displayed in the display area 22 in the same position and size as when the text object represented by the target image 50b actually exists, to the upper right of the steering wheel 56 that actually exists. Therefore, if the line of sight of the user 100 moves, the display position and size of these target images 50a and 50b in the display area 22 also change.

[0032] In this way, in the case of spatially fixed display, the target image 50 can be displayed taking into consideration the arrangement of real objects, so that it is possible to reliably prevent the target image 50 from interfering with driving operations. Also, in the case of spatially fixed display, the target image 50 can be displayed at a position that is correlated with a real object (e.g., a pedestrian, etc.), so that the attention of the user 100 can be effectively directed to the object.

[0033] Incidentally, in order to perform spatially fixed display, it is necessary to accurately detect the position of the pupil relative to the display 14, and the position and orientation of the display 14 in real space. The device controller 20 determines the position of the target image 50 within the display area 22, etc., based on the position and orientation of the target object in real space, the position and orientation of the display 14 in real space, and the position of the pupil relative to the display 14. Of these, the position of the pupil relative to the display 14 is detected by the pupil position sensor 18, as described above.

[0034] The position and orientation of the display 14 in the real space are calculated by the device controller 20 performing Visual Simultaneous Localization and Mapping based on the SLAM image obtained by the SLAM camera 16. Visual SLAM is a technology for simultaneously estimating the three-dimensional information of the environment and the position and orientation of the camera from the video captured by the camera. To perform this visual SLAM, the characteristic shapes of a plurality of interior parts in the vehicle are recognized as AR markers 60 (see FIG. 3). The device controller 20 extracts a plurality of AR markers 60 from the SLAM image captured by the SLAM camera 16, and can calculate the position and orientation of the display 14 in the real space from the positional relationship in these SLAM images and the like. Also, based on the coordinates, size, distortion, etc. of one AR marker 60 within the SLAM image, the position and orientation of the display 14 in the real space can be calculated.

[0035] To perform this visual SLAM, there are AR markers 60 (see FIG. 3) in the vehicle. In the present embodiment, the AR marker 60 is specified by image recognition from the shape of the interior part. The shape, size, and installation position of the AR marker 60 are registered in advance by initial setting. Therefore, the device controller 20 extracts the image of the AR marker 60 from the SLAM image captured by the SLAM camera 16. Then, based on the coordinates, size, distortion, etc. of the image of this AR marker 60 within the SLAM image, the position and orientation of the display 14 in the real space can be calculated. Also, by performing image recognition on a plurality of AR markers 60 and referring to the installation positions of the plurality of AR markers 60, the calculation of the position and orientation of the display 14 can be easily performed. Note that as the AR marker 60, instead of an interior part, a marker having a predetermined shape and size and installed at a predetermined position in the vehicle can also be used. Furthermore, these different types of AR markers can be used in combination.

[0036] <Registration of AR Marker> 6 is a flowchart showing a process for acquiring the shape of an interior part of a vehicle as the AR marker 60 in the device controller 20.

[0037] First, the wearable device 12 is brought into the vehicle, and for example, a user puts the wearable device 12 on his / her head in the driver's seat, turns on the power, and sets the AR marker registration mode. To set the AR marker registration mode, the wearable device 12 may have an operation button or the like, or may be operable from a terminal such as a smartphone. By setting the AR marker registration mode, an AR marker recognition program is executed in the device controller 20.

[0038] In the AR marker registration mode, first, an image is captured by the slam camera 16 (S11). This image is the same as the image seen by the driver in the driver's seat. At this time, it is advisable to have the user stand still in a position looking forward in the driver's seat, and then move their head to obtain images and videos from different positions.

[0039] The area in which feature parts are detected is limited from the captured image (S12). For example, when a candidate to be used as the AR marker 60 is known in advance, the recognition area is limited to a recognition area 70 including the position where the candidate exists (see FIG. 3). In this example, the area is limited to the upper central area where the electronic inner mirror 40c exists, and the instrument panel area including the meter display 40a and the multi-display 40b. Also, when an AR marker 60 has been recognized and registered in advance, it is advisable to use that data to limit the target area. Note that such a limitation may not be imposed, and the entire captured image may be subject to feature recognition.

[0040] Next, image recognition processing is performed on the target image to detect characteristic shapes, i.e., characteristic parts (S13). This can be done by detecting contours and extracting shapes using edge detection or the like. If the target is a display frame or the like, recognition may be limited to shapes similar to that. Furthermore, characteristic parts may be recognized using information on previously registered AR markers.

[0041] As described above, when wearable device 12 is moving and acquiring images, it is advisable to exclude feature parts outside the vehicle by detecting the movement of the detected feature parts or by comparing two images of wearable device 12 in different positions to calculate the distance to the feature parts.

[0042] Then, one characteristic part is extracted (S14), and it is determined whether or not it belongs to the steering wheel 56 (S15). The steering wheel 56 is basically a circular ring shape, and has characteristics different from others, so it can be easily determined. Note that information about the shape of the steering wheel 56 may be stored and used.

[0043] Then, the characteristic parts of the steering wheel 56 that do not belong to the vehicle are registered as AR markers (S16).

[0044] On the other hand, if it is determined in S15 that the target feature part belongs to the steering wheel 56, S15 is skipped and it is determined whether extraction of the feature parts has been completed (S17). However, if the partial extraction of the feature parts has not been completed and there are still features remaining, the process returns to S14 and the next feature part is processed, and this is repeated until there are no more features remaining.

[0045] 7 is a diagram for explaining area specification when excluding the steering wheel 56. In the figure, the area hatched with diagonal lines is the excluded area to which the steering wheel 56 belongs and is excluded. In this way, the upper half of the steering wheel 56 where the rear is visible is included in the target area. As a result, the meter display 40a located on the rear side of the steering wheel 56 becomes the extraction target, and can be registered as the AR marker 60.

[0046] In this way, the device controller 20 functions as an exclusion means for excluding the steering wheel from the characteristic parts, an extraction means for extracting the characteristic parts, and a display position determination means for determining the display position of the target image 50.

[0047] <Display of target image> FIG. 8 is a flowchart showing a process for displaying a target image while driving.

[0048] First, an image is captured by the slam camera 16 (S21). Using information on the registered AR marker 60, the AR marker 60 is detected from the image (S22). Then, using position information on the recognized AR marker 60, the display position of the target image 50 is determined (S23), and the target image 50 is displayed at the determined position (S24).

[0049] In this manner, according to the present embodiment, the steering wheel 56 is excluded from the interior parts to be registered as the AR marker 60.

[0050] Because the steering wheel 56 moves or is hidden by the hands, excluding it reduces the processing burden when recognizing the AR marker 60 and also reduces the possibility that the AR marker 60 will not be recognized. [Explanation of symbols]

[0051] 10 image display device, 12 wearable device, 14 display, 16 slam camera, 18 pupil position sensor, 20 device controller, 20a processor, 20b memory, 20c communication I / F, 22 display area, 24 rim, 26 temple, 28 in-vehicle system, 30 information center, 40a meter display, 40b multi-display, 40c electronic inner mirror, 50 target image, 56 steering wheel, 60 AR marker, 70 recognition area, 72 target object, 80 table, 100 user.

Claims

1. A display device that is mounted on the head of a vehicle occupant and displays a target image in a superimposed manner in the user's field of vision; a slum camera fixed to the display and configured to capture a slum image of the periphery of the display; An extraction means for performing image recognition processing on the obtained slam image and extracting a characteristic shape as a characteristic portion; a determination means for determining whether the extracted feature portion belongs to a steering wheel; an exclusion means for excluding the extracted feature portion when the extracted feature portion belongs to a steering wheel; a registration means for registering a characteristic portion not belonging to the steering wheel as an AR marker; a display position determination means for determining a display position of the target image by using the registered AR marker; An image display device comprising:

2. The image display device according to claim 1 , An object on the rear side of the steering wheel that can be seen through the steering wheel is set as a target for extracting the characteristic portion. Image display device.

Images