Image display device and image display system

Abstract

An image display device of the present disclosure comprises a camera, a range finding sensor, an illuminance sensor, a light / dark determination device, an image generation device, an image storage device, an image search device, a visual recognition image determination device, and a display device. The camera images a real object in front of a user and obtains a camera image. The image generation device generates, on the basis of data from the range finding sensor, a distance image that corresponds to the camera image and that represents the distance to each of multiple positions of the real object. The light / dark determination device determines, on the basis of data from the illuminance sensor, whether the location where the user is present is a light location or a dark location. The image storage device stores, as a set, a camera image and a distance image obtained at a light location. At a dark location, the image search device searches and identifies, from among distance images of light locations already stored, a distance image corresponding to an obtained distance image of a dark location. The visual recognition image determination device determines a visual recognition image on the basis of the camera image included in the same set as the identified distance image. The display device displays the visual recognition image so as to be visually recognized by the user.

Description

[Technical field]

[0001] The present invention relates to an image display device and an image display system that support visibility in dark places. [Background technology]

[0002] Augmented reality technology, which adds AR (Argument Reality) objects created with CG (Computer Graphics) to real space or background images, is used in content such as games and maintenance work. Users can experience augmented reality by viewing content images synthesized on the display screen using a head mounted display (hereinafter also referred to as HMD (Head Mounted Display)), a portable information terminal, etc.

[0003] Patent Document 1 discloses an HMD equipped with a camera that captures an image of the background and a distance measuring device that measures the distance to a real object in the background. It also discloses a technology that executes a process to extract a real object from an image obtained by the camera and assigns an AR object in association with the real object.

[0004] An example of a distance measurement sensor is disclosed in Patent Document 2. Patent Document 2 discloses a distance measurement sensor (LiDAR: Light Detection and Ranging) in the field of autonomous driving that scans laser light and measures the arrival time or phase difference of the reflected light to obtain data indicating the distance to an object in the front direction. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2020 / 115784 [Patent Document 2] International Publication No. 2019 / 082926 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 does not take into consideration use in dark places. For example, in dark places where it is difficult to capture images with a camera, such as places where the lights are turned off due to a power outage, construction or maintenance sites at night where lighting cannot be used due to consideration for the surrounding environment, and inside buildings where the lights are not turned on, the user has poor visibility ahead, making it difficult to use the technology.

[0007] For these reasons, it is desirable to improve the user's forward visibility in dark places. [Means for solving the problem]

[0008] Among the inventions disclosed in this application, a brief summary of representative inventions is as follows.

[0009] An image display device according to a representative embodiment of the present invention is an image display device that displays a visual image for a user to visually recognize, comprising: a camera that captures an image in front of the user to obtain a camera image; a distance measurement sensor that obtains data representing the distance from the user to each position in a real object included in the field of view of the camera; an illuminance sensor that obtains data representing the brightness of the location where the user is located; a generation device that generates a distance image corresponding to the field of view and in which each pixel represents the distance to each of the positions based on the data obtained by the distance measurement sensor; a determination device that determines whether the location of the user is in a bright or dark place based on the data obtained by the illuminance sensor; and a camera image obtained by the camera when the determination device determines that the location is in a bright place. The image display device includes a storage device that stores an image of the object and a distance image obtained by the generation device as an image set including a bright camera image and a bright distance image; a recognition device that recognizes the distance image obtained by the generation device as a dark distance image when the determination device determines that the object is in a dark place; a search device that identifies a bright distance image corresponding to the recognized dark distance image by comparing the bright distance image stored in the storage device with the dark distance image recognized by the recognition device; a determination device that determines a visual image to be viewed by the user based on a bright camera image included in the same image set as the bright distance image identified by the search device; and a display device that displays the visual image. Effect of the Invention

[0010] The effects obtained by the representative inventions among those disclosed in this application will be briefly described as follows.

[0011] According to a representative embodiment of the present invention, it is possible to improve a user's forward visibility in a dark place. [Brief description of the drawings]

[0012] [Figure 1] 1 is an external view of an HMD according to a first embodiment. [Diagram 2]FIG. 2 is a block diagram showing a hardware configuration of the HMD according to the first embodiment. [Diagram 3] FIG. 2 is a functional block diagram showing the functional configuration of the HMD according to the first embodiment. [Figure 4] FIG. 4 is a process flow diagram of a dark place guide program by the HMD according to the first embodiment. [Diagram 5] FIG. 4 is a process flow diagram of image set storage and management by the HMD according to the first embodiment. [Figure 6] 3A to 3C are diagrams showing examples of images obtained by the HMD according to the first embodiment. [Figure 7A] FIG. 11 is a diagram illustrating a first example of the configuration of an HMD system according to a second embodiment. [Figure 7B] FIG. 11 is a diagram illustrating a second configuration example of the HMD system according to the second embodiment. [Figure 8] FIG. 2 is a block diagram showing a hardware configuration of an image storage service server. [Figure 9] FIG. 2 is a diagram illustrating an example of a data structure of an image set. [Figure 10] FIG. 11 is a functional block diagram showing the functional configuration of an HMD according to a third embodiment. [Figure 11A] FIG. 11 is a flow diagram of a visual recognition image determination process according to the third embodiment. [Figure 11B] FIG. 11 is a flow diagram of a visual recognition image determination process according to the third embodiment. [Figure 12] 13 is a diagram for explaining a first example of a process for determining a visual image by an HMD according to the third embodiment. FIG. [Figure 13A] 13 is a diagram for explaining a second example of a process for determining a visual image by an HMD according to the third embodiment. FIG. [Figure 13B] 13 is a diagram for explaining a third example of a process for determining a visual image by an HMD according to the third embodiment. FIG. [Figure 14] FIG. 13 is a flow diagram of a process for comparing distance images and determining whether they match each other according to the fourth embodiment. [Figure 15A] FIG. 13 is a diagram for explaining a translation process in the third embodiment. [Figure 15B]FIG. 13 is a diagram for explaining a scaling process in the third embodiment. [Figure 15C] FIG. 13 is a diagram for explaining a combination process using a plurality of distance images in the third embodiment. [Figure 16] FIG. 13 is an external view of an HMD according to a fifth embodiment. [Figure 17] FIG. 13 is a functional block diagram of an HMD according to a fifth embodiment. [Figure 18] FIG. 13 is a flow diagram of a visual image generation and editing process according to the fifth embodiment. [Figure 19] 13A to 13C are diagrams showing examples of a camera image and a visually recognized image obtained according to the fifth embodiment. [Figure 20A] 13 is a diagram showing a first example of the appearance of an HMD according to a sixth embodiment. FIG. [Figure 20B] 13 is a diagram showing a first example of the appearance of an HMD according to a sixth embodiment. FIG. [Figure 21A] 13 is a diagram showing a second example of the appearance of the HMD according to the sixth embodiment. FIG. [Figure 21B] 13 is a diagram showing a second example of the appearance of the HMD according to the sixth embodiment. FIG. [Figure 21C] FIG. 2 is a diagram showing a gesture operation area by a user's finger. [Figure 22] FIG. 2 is a diagram showing an example of a user interface screen in an HMD. [Diagram 23] FIG. 13 is an external view of an HMD according to an eighth embodiment. [Figure 24A] 1A to 1C are diagrams illustrating an example of the angle of view of a camera and a measurement range of a distance measuring sensor of a portable information terminal. [Figure 24B] FIG. 13 is a diagram showing an example of a state in which a dark distance image is displayed on a portable information terminal. [Figure 24C] FIG. 11 is a diagram showing an example of a state in which a bright place camera image is displayed on a portable information terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Next, an embodiment of the present invention will be described. By providing the technology shown in each of the following embodiments, it becomes possible to ensure safety in dark places based on highly accurate distance measurement. This distance measurement technology contributes to "9. Build resilient infrastructure, promote industry, innovation and infrastructure" and "11. Make cities and towns sustainable" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0014] It should be noted that each embodiment described below is an example for realizing the present invention, and does not limit the technical scope of the present invention.

[0015] In addition, in the following embodiments, components having the same functions are denoted by the same reference numerals, and repeated explanations thereof will be omitted unless particularly necessary.

[0016] In addition, in the following description of each embodiment, expressions such as "image matching" or "the same image" are included, but "matching" or "identical" here does not mean 100% perfect matching or perfect identity. "Matching" or "identical" here means allowing for deviations in the linear direction, rotational direction, and enlargement / reduction direction of the images, differences in color, differences in shading (light and dark), and the like, within a range that does not cause practical problems.

[0017] In the following description of each embodiment, a "bright distance image corresponding to a dark distance image" or "another bright distance image corresponding to a bright distance image" refers to an image in which the images are substantially identical except for some clearly different areas. Here, "substantially identical" means that deviations in the linear direction, rotational direction, and zoom direction of the images, differences in color, and differences in shading (light and dark), etc., are allowed within a range that does not cause practical problems. For example, when the degree of identity or similarity of the images is equal to or exceeds a certain threshold value, the images are determined to be identical.

[0018] (Embodiment 1) An HMD according to a first embodiment of the present invention will be described.

[0019] Overview of First Embodiment In a bright place, the HMD of the first embodiment obtains a bright camera image taken by a camera and a bright distance image corresponding to the camera image and based on the output of a distance measuring sensor, and saves and accumulates these images as a set. Meanwhile, in a dark place, a dark distance image based on the output of the distance measuring sensor is obtained, and a bright distance image corresponding to (substantially identical to) the dark distance image is searched for and specified. Then, based on the bright camera image included in the same set as the bright distance image, a visual image to be viewed by the user is determined and displayed. This allows the user to view a bright image of the same location obtained in the past in a dark place, improving the user's visibility.

[0020] <HMD hardware configuration> Fig. 1 is an external view of an HMD according to embodiment 1. As shown in Fig. 1, the HMD 1 according to this embodiment includes a camera 11, a distance measurement sensor 12, a right-eye projector 13, a left-eye projector 14, an image screen (image display surface) 15, a nose pad 16, a controller 17, a microphone 18, a speaker 19, and frame housings 20a to 20c. A user wears the HMD 1 on his / her face using the frame housings 20a, 20b and the nose pad 16.

[0021] Right-eye projector 13, left-eye projector 14, and image screen 15 constitute a display device of HMD 1. Right-eye projector 13 and left-eye projector 14 are also referred to as projectors 13, 14 hereinafter.

[0022] In this embodiment, the image screen 15 is a semi-transmissive screen that transmits light from in front of the user. However, as described below, the image screen 15 may be a non-transmissive display that does not transmit light.

[0023] With a semi-transparent screen, the user can view the situation in front of them through the semi-transparent screen, whereas with a non-transparent display, the user can view the situation in front of them by displaying a camera image captured in front of them on a display inside a non-transparent HMD.

[0024] The camera 11 is attached so as to capture an image of the real space in front of the user. The camera 11 is a so-called digital camera, and captures an image of its own visual field to obtain image data corresponding to the visual field. In this embodiment, the image represented by this image data is called a camera image. The camera 11 repeats capturing images at a predetermined timing under the control of the controller 17. This capturing may be performed manually or automatically. When capturing images automatically, for example, the image is captured at a constant frame rate, and the frame rate may be appropriately adjusted according to the processing speed and memory capacity of the controller 17. The frame rate may be, for example, about 1 to 30 fps (frames per second).

[0025] The distance measurement sensor 12 measures the distance from the sensor itself to the real object corresponding to each position in the field of view of the camera 11, and obtains distance data representing these distances. When the user wears the HMD 1, the distance measurement sensor 12 is essentially equivalent to obtaining distance data representing the distance from the user to the real object corresponding to each of the above positions. The distance measurement sensor 12 is capable of measuring distances even in dark places, and is configured to emit infrared light and receive reflected light from a real object, such as the LiDAR described above.

[0026] Projectors 13 and 14 project CG images, camera images, etc. onto image screen 15 and superimpose them on the background that is viewed through image screen 15. When projecting a CG image, projectors 13 and 14 project a left-eye image and a right-eye image that are generated taking parallax into consideration onto image screen 15. This makes it possible to display the CG image three-dimensionally as if it were at a specified distance in real space.

[0027] The controller 17 takes in image data obtained by the camera 11 and distance data obtained by the distance measuring sensor 12, and supplies these data to an internal memory or processor. The controller 17 also has built-in sensors such as a GPS (Global Positioning System) sensor, an illuminance sensor, an acceleration sensor, a gyro sensor, and a direction (magnetic) sensor. The controller 17 also creates images to be projected by the projectors 13 and 14, sounds to be output by the speaker 19, and the like. The controller 17, the camera 11, the distance measuring sensor 12, the microphone 18, and the speaker 19 are disposed in the frame housings 20a to 20c. Note that the locations of these components shown in FIG. 1 are merely an example, and may not be exactly the same.

[0028] Fig. 2 is a block diagram showing a hardware configuration of the HMD according to the first embodiment. In Fig. 2, the same components as those shown in Fig. 1 are given the same numbers. As shown in Fig. 2, the controller 17 of the HMD 1 according to this embodiment includes an internal bus 30, a GPS sensor 31, an illuminance sensor 32, an acceleration sensor 33, a gyro sensor 34, a direction sensor 35, a processor 36, a memory 37, an image memory 38, a non-volatile storage device 39, and a communication device 40. The blocks 31 to 40 are connected to each other via the internal bus 30 and operate in cooperation with each other.

[0029] The processor 36 is configured, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit).

[0030] The memory 37 and the image memory 38 are configured, for example, by a RAM (Random Access Memory) which is a semiconductor storage device.

[0031] The nonvolatile storage device 39 is composed of a single or multiple nonvolatile memory media. For example, a programmable ROM (Read Only Memory) can be considered as a nonvolatile memory medium. The programmable ROM is composed of, for example, an EEPROM (Electronically Erasable and Programmable ROM) or a FROM (Flash ROM). The nonvolatile storage device 39 stores a basic operation program 41 and a dark place guide program 42 as processing programs. In addition, a saved data area 43 is assigned to the nonvolatile storage device 39. The saved data area 43 stores data and image data required to execute the processing program.

[0032] The communication device 40 includes a mobile communication device such as 4G (Generation) or 5G, a wireless LAN communication device, etc. The communication device 40 selects an appropriate process from among communication processes as necessary, and connects the HMD 1 to a network.

[0033] Image data to be sent to the projectors 13 and 14 is stored in an image memory 38 and then read out.

[0034] The processor 36 controls and manages basic operations and realizes the dark place guide function by expanding into the memory 37 a basic operation program 41 and a dark place guide program 42 stored in the non-volatile storage device 39 and executing them.

[0035] <HMD functional block configuration> Fig. 3 is a functional block diagram showing the functional configuration of the HMD according to embodiment 1. Each functional block shown in Fig. 3 is realized by the processor 36 expanding a dark place guide program in the memory 37, executing it, and cooperating with various sensors and various devices included in the HMD 1.

[0036] As shown in FIG. 3, the HMD1 of this embodiment includes, as functional blocks, a position information acquisition device 51, a time information acquisition device 52, an orientation information acquisition device 53, an attitude information acquisition device 54, a camera image acquisition device 55, a distance image generation device (generation device) 56, a bright place / dark place determination device (determination device) 57, a bright place image storage management device (storage device) 58, a bright place image storage device 59, a dark place distance image recognition device (recognition device) 60, a comparison image narrowing down device 61, a bright place image search device 62, a visual image determination device 63, and a visual image display device (display device) 64.

[0037] The position information acquisition device 51 acquires coordinate data obtained by the GPS sensor 31, and generates position information representing the location of the user based on the coordinate data. The time information acquisition device 52 acquires clock data including date and time from the electronic clock of the processor 36, and generates time information representing the date and time when the image was acquired. The orientation information acquisition device 53 acquires orientation data obtained by the orientation sensor 35, and generates orientation information representing the horizontal direction of the user's face, i.e., the orientation based on the orientation data. The posture information acquisition device 54 acquires data obtained by the acceleration sensor 33 and the gyro sensor 34, and generates posture information representing the vertical direction of the user's face based on the orientation data.

[0038] The camera image acquisition device 55 acquires image data obtained by the camera 11, and performs image processing such as noise removal, interpolation, size adjustment, etc. on the data as necessary to generate a camera image. The camera image acquisition device 55 continuously and repeatedly acquires image data at time intervals, and generates camera images in time series.

[0039] The distance image generating device 56 acquires distance data obtained by the distance measuring sensor 12, and generates a distance image based on the data. The distance image is an image having an area corresponding to the field of view of the camera image. The distance image is an image in which the distance from the distance measuring sensor 12 to a real object corresponding to each position in the field of view is reflected in the pixel value (color or shade) of the pixel corresponding to each position. When the bright / dark place determining device 57 described later determines that the location where the user is located is a bright place, the distance image generating device 56 generates a distance image corresponding to each generated camera image for each of the camera images. When the location where the user is located is determined to be a dark place, the distance image generating device 56 generates a distance image regardless of the presence or absence of a camera image.

[0040] The bright and dark place determination device 57 acquires illuminance data obtained by the illuminance sensor 32, and determines whether the location of the user is a bright place or a dark place based on the data. The bright and dark place determination device 57 performs the determination of a bright place or a dark place, for example, by judging the threshold value of the illuminance represented by the illuminance data.

[0041] The determination of whether the location is bright or dark may be made based on the brightness of the acquired camera image, or may be made using both the illuminance data and the brightness of the camera image.

[0042] The bright and dark place determination may be performed by supplementing the position information and time information. For example, when the illuminance data indicates that the illuminance is exactly halfway between a bright place and a dark place, the bright and dark place determination device 57 refers to the position information and time information. Then, based on this information, the location and date and time of the user are detected, and whether or not it is after the evening time preset for each location is determined. If it is before the evening time, it is determined to be a bright place, and if it is after the evening time, it is determined to be a dark place.

[0043] The bright place image storage management device 58 stores the camera image and distance image obtained when the bright place dark place determination device 57 determines that the location of the user is a bright place as an image set including the bright place camera image and the bright place distance image. When storing the image set, the bright place image storage management device 58 stores the position information, time information, orientation information, and posture information when the image set was obtained as metadata in association with the image set. The image set and metadata are stored in the bright place image storage device 59 described later. Here, it has been described that the image set is stored when it is determined to be a bright place, but a flag for distinguishing between a bright place and a dark place may be added to store either the bright place image set or the dark place image set. In this case, if nothing is captured in the camera image in the dark place, only the time information may be stored and the camera image may not be stored.

[0044] The bright light image storage device 59 stores the image sets and metadata stored by the bright light image storage management device 58. That is, the bright light image storage device 59 sequentially stores and accumulates image sets for each location where the user was and for each orientation (direction) and posture of the user.

[0045] The dark place distance image recognition device 60 recognizes, as a dark place distance image, a distance image obtained when the bright place / dark place determination device 57 determines that the place where the user is located is a dark place.

[0046] When a dark distance image is recognized, the comparison image narrowing device 61 narrows down the bright distance images stored in the bright image storage device 59 to bright distance images to be compared with the dark distance image. The narrowing down of the bright distance images is performed using associated metadata. That is, based on the position information, time information, orientation information, and posture information when the recognized dark distance image was obtained, the comparison image narrows down the bright distance images that are similar to the location where the user is, the date on which the user was present, the orientation of the user, and the posture of the user within a predetermined range. In this embodiment, the comparison image narrows down the images based on the position information, time information, orientation information, and posture information, but the comparison image narrows down the images based on only the position information, only the position information and orientation information, or only the position information, orientation information, and posture information. Alternatively, the comparison image narrowing down by the comparison image narrowing device 61 may be omitted.

[0047] Bright place image search device 62 sequentially reads out bright place distance images to be compared with the recognized dark place distance images, and compares the dark place distance images with the read out bright place distance images. Then, based on the result of this comparison, it searches for and identifies a bright place distance image corresponding to the dark place distance image, i.e., a bright place distance image that matches within an allowable range. The comparison between the dark place distance image and the bright place distance image is performed by performing translation, rotation (tilt) movement, enlargement / reduction, pixel value adjustment, etc., on at least one of the distance images.

[0048] To determine whether there is a match within an acceptable range, for example, a method can be used in which the degree of match between the recognized dark distance image and the read out bright distance image is calculated, and if the degree of match is above a threshold, it is determined that there is a match, and if the degree of match is below the threshold, it is determined that there is no match.

[0049] As a specific example, for each position or region in the distance image, the pixel value levels between corresponding pixels or pixel groups, for example the difference between the average pixel values, are obtained, the magnitude of these differences is comprehensively evaluated to calculate the degree of match, and the degree of match is determined by a threshold judgment. The degree of match may be, for example, a value obtained by multiplying the variance or deviation value of the pixel value level difference at each position or region by a negative coefficient, or the reciprocal of the variance or deviation value.

[0050] In addition, when comparing distance images, if it is recognized that there is a difference only in a part of the entire image area, it is deemed that there is a match within an allowable range. In other words, the match determination is performed excluding the difference area described later.

[0051] In addition, the determination of whether or not there is a match within an acceptable range may be performed using, for example, artificial intelligence, which may be trained on matches of images representing objects that may exist in real space.

[0052] The visual image determination device 63 determines a visual image to be viewed by the user based on the result of the comparison between the distance images by the bright image search device 62. When a bright distance image that substantially matches the dark distance image is specified, the visual image determination device 63 determines a visual image based on a bright camera image included in the same image set as the bright distance image. When several matching distance images are specified, the user may be allowed to select the most recent one or the brightest one. On the other hand, when a bright distance image that substantially matches the dark distance image is not specified, the visual image is determined based on the dark distance image. The visual image may be the bright camera image itself or the dark distance image itself, or in some cases, it may be obtained by performing image processing such as processing or synthesis on the bright camera image or the dark distance image.

[0053] The visual recognition image display device 64 displays the determined visual recognition image by projecting it onto the image screen 15 using the projectors 13 and 14 so that the user can visually recognize it. Here, when the user is moving, it may be difficult to display the bright camera image of the same position as a moving image without delay while detecting the position. In such a case, it may be possible to display a still image according to the user's moving position.

[0054] <Dark Place Guide Program Processing Flow> Next, the dark place guide program process will be described.

[0055] FIG. 4 is a process flow diagram of a dark place guide program by the HMD according to the first embodiment.

[0056] In step S1, a process of acquiring a camera image is performed. Specifically, the camera image acquisition device 55 acquires image data obtained by the camera 11. The camera image may be acquired in synchronization with the timing of the camera image capture, or the image may be captured at any timing while continuously capturing images. When capturing images continuously, the image capture is performed at a frame rate of, for example, about 10 to 60 fps (frames per second).

[0057] In step S2, a process of acquiring distance data is performed. Specifically, the distance image generating device 56 acquires distance data representing the distance from the distance measuring sensor to a real object included in the viewing area of ​​the camera 11.

[0058] In step S3, a process of generating a distance image is performed. Specifically, the distance image generating device 56 generates a distance image corresponding to the viewing area of ​​the camera 11 based on the acquired distance data.

[0059] In step S4, a process is performed to determine whether the place is bright or dark. Specifically, the bright / dark place determination device 57 acquires illuminance data from the illuminance sensor 32, and determines whether the place where the user is located is bright or dark based on the illuminance data. As described above, this determination may be made based on the brightness of the acquired camera image, or may be made by supplementing the use of position information, time information, and the like. If it is determined that the place is bright (S4, Yes), the process proceeds to step S5. On the other hand, if it is determined that the place is dark (S4, No), the process proceeds to step S7.

[0060] In step S5, a process of acquiring metadata is performed. Specifically, the bright place image storage management device 58 acquires time information, position information, orientation information, and posture information as metadata to be associated with the image set.

[0061] In step S6, the image set is stored, updated, and managed. Specifically, the bright-light image storage management device 58 treats the obtained camera image and distance image as a bright-light camera image and a bright-light distance image. Then, the image set including these images is associated with the acquired metadata and stored in the bright-light image storage device 59. Then, the process proceeds to step S17.

[0062] When saving an image set, if a bright distance image corresponding to the bright distance image to be saved has already been saved, the image set may be overwritten. Alternatively, for images that were acquired recently, only the recording date and time may be added, and the images may not be overwritten.

[0063] Also, image sets that were acquired a certain time ago (first time) or more ago, so-called old image sets, may be erased (deleted) from the bright light image storage device 59. The certain time may be, for example, about one month to one year.

[0064] The image set storage and management process in step S6 will be described in detail later.

[0065] In step S7, a process of recognizing a dark distance image is performed. Specifically, when the dark distance image recognition device 60 determines that the location of the user is a dark place, it recognizes the obtained distance image as a dark distance image.

[0066] In step S8, a process of narrowing down bright distance images to be compared is performed. Specifically, the comparison image narrowing device 61 narrows down the bright distance images to be compared with the recognized dark distance image from among the past bright distance images stored in the bright distance image storage device 59. The narrowing down of the bright distance images is performed using associated metadata. That is, based on the position information when the recognized dark distance image was obtained, bright distance images in which the location of the user matches or is similar within a predetermined range are searched for and narrowed down. In addition to the position information, orientation information or posture information may also be used to search for and narrow down bright distance images in which the orientation in which the user is facing, the posture of the user (tilt of the face in the vertical direction), or both of these match or are similar within a predetermined range.

[0067] In step S9, a process of reading out a bright range image is performed. Specifically, bright range image search device 62 reads out one bright range image from the narrowed down bright range images.

[0068] In step S10, a process of comparing the dark distance image and the bright distance image is performed. Specifically, the bright image search device 62 compares the recognized dark distance image with the read bright distance image, and calculates an evaluation value reflecting the degree of match between the distance images. Here, if the bright distance image is not saved and only the camera image is available, the features of the dark distance image and the features of the bright camera image may be extracted and compared.

[0069] In step S11, a process is performed to determine whether the compared distance images correspond to each other, i.e., whether they substantially match. Specifically, the bright place image search device 62 determines whether the recognized dark place distance image and the read bright place distance image substantially match by threshold judgment of the calculated evaluation value. If it is determined that they match (S11, Yes), the process proceeds to step S12. On the other hand, if it is determined that they do not match (S11, No), the process proceeds to step S13.

[0070] In step S12, a process is performed in which the bright camera image is used as the basis of the visual image. Specifically, the visual image determination device 63 sets the bright camera image included in the same image set as the bright distance image determined to match in step S11 as the basis of the visual image to be visually recognized by the user. Then, the process proceeds to step S15.

[0071] In step S13, a process is performed to determine whether all comparisons of the narrowed-down bright range images have been completed. Specifically, bright range image search device 62 performs comparisons on all narrowed-down bright range images and determines whether there are any bright range images remaining to be compared next. If it is determined that all comparisons have been completed (S13, Yes), the process proceeds to step S14. On the other hand, if it is determined that all comparisons have not been completed (S13, No), the process returns to step S9, and the next bright range image to be compared is read out.

[0072] In step S14, a process is performed in which the dark distance image is used as a basis for a visual image. Specifically, the visual image determination device 63 sets the recognized dark distance image as a basis for a visual image to be visually recognized by the user.

[0073] In step S15, a process of generating and editing a visual image is performed. Specifically, the visual image determination device 63 generates and determines a visual image based on a bright camera image or a dark distance image set as a base. At this time, as necessary, editing is performed such as pasting a CG image or an AR object image onto the image set as a base, or emphasizing a part of the image set as a base.

[0074] In step S16, a process of displaying the visual image is performed. Specifically, the visual image display device 64 projects and displays the visual image on the image screen 15 so that the determined visual image is visually recognized by the user.

[0075] In step S17, a process is performed to determine whether or not to continue the dark place guide program. For example, when moving from a dark place to a bright place, when moving to an environment where the execution of the dark place guide program is deemed unnecessary, when a command to stop or end the dark place guide program is input by the user, or when an internal processing error or a communication error occurs, it is determined not to continue (S17, No) and the dark place guide program is terminated. On the other hand, when there is no particular reason to end it, it is determined to continue (S17, Yes), and the process returns to step S1 and the process by the dark place guide program is continued.

[0076] <Image set storage and management processing> Here, the flow of the image set storage and management process in step S6 above will be described in detail.

[0077] FIG. 5 is a process flow diagram of image set storage and management by the HMD according to the first embodiment.

[0078] In step S61, a process of narrowing down the past image sets to be compared is performed. Specifically, the bright light image storage management device 58 narrows down the past bright light distance images stored in the bright light image storage device 59 to bright light distance images that match within a predetermined range the currently acquired image set's imaging location, user's direction (orientation), and posture, based on the currently acquired metadata, as comparison targets. Note that this narrowing down may be performed based only on the imaging location, without considering the orientation and posture at the time of imaging.

[0079] In step S62, a process of reading one past bright field distance image is performed. Specifically, bright field image storage management device 58 reads one of the past bright field distance images narrowed down in step S61. At this time, the locations recognized by the location information included in the metadata may be read in order from the location and direction closest to the current location. This may allow necessary processes such as new saving of an image set, overwriting saving, adding recording date and time, and erasing old image sets, which will be described later, to be completed at a relatively early stage. As a result, even if the image set saving and management process is interrupted for some reason, the risk of adverse effects occurring can be reduced.

[0080] In step S63, a process of comparing the current bright field distance image with the previous bright field distance image is performed. Specifically, bright field image storage management device 58 compares the acquired current bright field distance image with the previous bright field distance image read in step S62.

[0081] In step S64, a process is performed to determine whether the bright distance images to be compared are identical. Specifically, bright image storage management device 58 determines whether the acquired current bright distance image and the previous bright distance image read in step S62 match within an allowable range, i.e., whether they are substantially identical. If the distance images are completely different or deviate by a predetermined level or more, they are determined to be not identical. If it is determined that they are the same (S64, Yes), the process proceeds to step S65. On the other hand, if it is determined that they are not the same (S64, No), the process returns to step S62, where the next bright distance image to be compared is read and the process continues.

[0082] In step S65, a process of recognizing the recording date and time of the past bright distance image is performed. Specifically, the bright distance image storage management device 58 reads and recognizes the recording date and time included in the metadata of the past bright distance image read in step S62.

[0083] In step S66, a process is performed to determine whether the read past bright distance image is old or not. Specifically, bright image storage management device 58 determines whether the recording date and time recognized in step S65 is a certain period of time or more before the current time, i.e., whether it is old or not. If it is determined that it is old (S66, Yes), the process proceeds to step S67. On the other hand, if it is determined that it is not old (S66, No), the process proceeds to step S68.

[0084] In step S67, a process of deleting the past image set is performed. Specifically, the bright place image storage management device 58 deletes the image set including the past bright place distance image read in step S62. Then, the process proceeds to step S70.

[0085] In step S68, a process is performed to determine whether the recognized recording date and time is approximately the same as the current date and time. Specifically, the bright place image storage management device 58 determines whether the time difference between the recording date and time recognized in step S65 and the current date and time is within a relatively short time set in advance, that is, whether the dates and times are substantially approximately the same. The preset time can be, for example, about one hour to one day, but is not limited to this. If it is determined that the dates and times are approximately the same (S68, Yes), the process proceeds to step S69. On the other hand, if it is determined that the dates and times are not the same (S68, No), the process proceeds to step S70.

[0086] In step S69, a process of overwriting or adding the recording date and time is performed. Specifically, the bright image storage management device 58 overwrites and stores a new image set of the acquired bright distance image and bright camera image to the image set including the past bright distance image read in step S62. Alternatively, for the image set including the past bright distance image read in step S62, only the recording date and time, which is metadata, is updated to the current date and time. In this case, since there is no substantial difference in the image itself, there is no problem in updating only the recording date and time, which can simplify the process.

[0087] However, if the past image set including the read past bright field distance image has an advantage or is easier to use, it is preferable to update the recording date and time without overwriting. For example, if the bright field distance image or bright field camera image included in the past image set has a relatively higher resolution than the current image, or if the aspect ratio of the image is well balanced, updating the recording date and time is selected. After the process of step S69 is completed, the process proceeds to step S70.

[0088] In step S70, a process is performed to determine whether there are any past bright field distance images to be compared. Specifically, bright field image storage management device 58 determines whether or not all comparisons have been completed for the past bright field distance images narrowed down to be compared in step S61, and whether or not there are any past bright field distance images remaining to be compared. If it is determined that there are no past bright field distance images remaining (S70, Yes), the process proceeds to step S71. On the other hand, if it is determined that there are past bright field distance images remaining (S70, No), the process returns to step S62, where the next past bright field distance image is read out and the process continues.

[0089] In step S71, a process is performed to determine whether the image set has been overwritten or saved. Specifically, the bright place image saving management device 58 determines whether the process of step S69, i.e., whether the image set currently acquired has been overwritten or the recording date and time has been added, has been performed. In this determination, if it is determined that the image set has been overwritten or saved (S71, Yes), the image set saving and management process is terminated. On the other hand, if it is determined that the image set has not been overwritten or saved (S71, No), the process proceeds to step S72.

[0090] In step S72, a process of newly saving the image set is performed. Specifically, the bright place image saving management device 58 associates the acquired image set with the currently obtained metadata and newly saves it in the bright place image storage device 59. Then, the image set saving and management process ends.

[0091] According to this image set storage and management process, past image sets that are substantially the same as the acquired image set are searched for and identified. Then, of the identified image sets, old ones are deleted, and those that are not old but whose recording dates and times differ by a certain amount or more are left, and those whose recording dates and times are substantially the same are overwritten and saved, or the recording dates and times are appended, i.e., updated. Furthermore, if the recording dates and times or time periods are different from those of past image sets, or if at least a portion of the images differ by a certain level or more, the acquired image set is newly saved. This makes it possible to hold the necessary image sets for the necessary period of time, and as a result, it is possible to prevent the endless increase in the storage capacity of image sets and to maintain the storage capacity appropriately.

[0092] <Examples of camera images, range images, and visual images> Fig. 6 is a diagram showing an example of an image obtained by the HMD according to the first embodiment. Fig. 6 shows an example of a camera image, a distance image, and a visual image obtained in a bright place and a dark place. In Fig. 6, the top row shows an image recorded in a bright place in the past, and the middle row shows an image recorded in a dark place at the present time. The left side shows a camera image, and the right side shows a distance image. Generally, a distance image is shown in different colors depending on the distance, but here it is shown pseudo in black and white (grayscale).

[0093] In Fig. 6, 80A is a bright place camera image captured clearly, and 81A is a bright place distance image corresponding to this bright place camera image 80A. 82B is a dark place camera image obtained at the current time, which is unclear due to the dark surroundings. On the other hand, 83B is a dark place distance image corresponding to the dark place camera image 82B, which was obtained almost simultaneously with the dark place camera image, but which has been obtained with clarity comparable to that of the bright place distance image 81A.

[0094] The user views the scenery in front of him through the semi-transparent image screen of the HMD, but in a dark place, it is difficult to confirm the situation ahead, as with the dark place camera image 82B. Therefore, as shown in the lower part of FIG. 6, the camera image in a bright place with a matching distance image is determined as the visual image 84B to be visually recognized by the user. Then, the visual image is displayed on the display surface of the HMD. This allows the user to confirm the situation at the site in a dark place.

[0095] Also, for example, the contour of an object may be extracted from the dark distance image 83B and compared with the contour of the object extracted from the bright distance image 81A or the bright camera image 80A, thereby making it possible to confirm that the images are of the field of view from the same position. Even in such a case, the user can check the situation at the site in a dark place by displaying the camera image in a bright place on the HMD.

[0096] According to the first embodiment, in a dark place where it is difficult to see ahead, a camera image in a bright place with the same field of view that was previously saved can be displayed to assist the user in seeing ahead. For example, in a dark work site with poor visibility, by providing a clear camera image obtained in a bright place instead of a camera image taken in a dark place, the user can confirm the presence of objects such as obstacles, and can work more safely.

[0097] In addition, in the first embodiment, when an image set is saved, if a corresponding image, i.e., a substantially matching bright distance image, has already been saved, the following process is performed. That is, if the acquisition date and time of the already saved corresponding image is within a certain period of time from the acquisition date and time of the image to be saved and is relatively new, the corresponding image is not overwritten and only the date and time of the corresponding image is added to the current date and time. Also, if the acquisition date and time of the corresponding image is older than the acquisition date and time of the image to be saved and is more than a certain period of time ago, the corresponding image is deleted. In this way, the process can be simplified to speed up the process, reduce energy consumption, and use the storage area efficiently.

[0098] Furthermore, in the first embodiment, the current location of the user is identified by GPS or the like, and a search is performed from bright distance images narrowed down to that location, thereby shortening the time required for image search.

[0099] In addition, in the first embodiment, a semi-transmissive screen that transmits light from the front of the user is used as the image screen, which is the image display surface. This allows the user to directly view real objects in front of the user while wearing the HMD, providing a natural sensation as if the user were wearing glasses.

[0100] In the first embodiment, in steps S10 and S11, the dark distance image and the bright distance image are compared to determine whether they match. However, it is possible that the bright distance image has not been recorded for some reason. In such a case, the dark distance image and the bright camera image may be compared to determine whether they match. For example, the contour of the object may be extracted from the dark distance image, the contour of the object may be extracted from the bright camera image, and the two contours may be compared to determine whether they match. This makes it possible to provide the user with an image with good visibility even when there is no bright distance image and a bright camera image is saved.

[0101] In addition, in the first embodiment, if the location of the user is limited and there are few bright distance images to compare, or if the processing speed is high, the process of narrowing down the bright distance images to be compared may be omitted. This simplifies the algorithm of the dark place guide program and reduces development costs.

[0102] Also, in the first embodiment, if the user is indoors and GPS location information is unavailable, and the metadata of the image set includes location information and orientation data from a network access point or UWB (Ultra Wide Band), these may be used. That is, a bright distance image having, as metadata, location information and orientation information that is substantially the same as the location information obtained from an access point or the orientation information detected by an orientation sensor at the user's location may be searched for preferentially. This allows the search time for bright distance images to be shortened even when GPS location information is unavailable.

[0103] In addition, in the first embodiment, if the user's initial position can be determined by the GPS or a wireless communication access point, the user's location after moving can be identified using a method such as Pedestrian Dead Reckoning (PDR) and a bright distance image corresponding to the line of sight from that position can be searched for. This allows the dark place guide program to be continuously executed even when the user moves from outside to indoors.

[0104] In the first embodiment, the specified bright camera image is used as the visual image as it is, but the specified bright camera image may be changed in brightness or color, or processed by image processing to obtain an image. For example, the color of the specified bright camera image may be converted to black and white or sepia, or may be arranged in vivid colors used in CG or animation. A virtual reality space corresponding to the user's location, the direction the user is facing, and the user's posture may be displayed, and a CG image artificially created based on the specified bright camera image may be superimposed on the virtual reality space.

[0105] (Embodiment 2) A second embodiment of the present invention will now be described.

[0106] Overview of the Second Embodiment The second embodiment is an HMD system in which an HMD and a server are connected via a network. In this embodiment, the HMD and the server cooperate with each other to provide an image storage service including a dark place guide function to a user.

[0107] <HMD system configuration> The configuration of an HMD system according to the second embodiment will be described.

[0108] Fig. 7A is a diagram showing a first configuration example of an HMD system according to embodiment 2. Fig. 7A shows the configuration of an HMD system 100 in which a single HMD and a server are connected via a network.

[0109] 7A, the HMD system 100 according to this embodiment includes an HMD 1, a user 70, wireless communication signals 71a and 71b using electromagnetic waves or the like, an access point 72, a network 73, and an image storage service server (server) 74. The network 73 is, for example, a wide area communication network, and specifically, the Internet, Ethernet, an industrial communication network, or the like can be considered.

[0110] Configuration of the Image Storage Service Server Fig. 8 is a block diagram showing the hardware configuration of the image storage service server. As shown in Fig. 8, the image storage service server 74 includes an internal bus 740, a network I / F (Interface) 741, a processor 742, a memory 743, and a storage 744. The storage 744 stores a basic operation program 745, and is provided with an image data storage area 746. The storage 744 is formed of, for example, a hard disk or a semiconductor memory.

[0111] An image set including a distance image from the HMD 1 is stored in a storage 744 via a network I / F 741, a processor 742, etc. A basic operation program 745 is executed by the processor 742 and a memory 743, and management processing for multiple users is performed.

[0112] How the image storage service works A user 70 wears an HMD 1 on his / her head and looks ahead. The HMD 1 is connected to a network 73 via communication signals 71a and 71b and an access point 72, and the network 73 is connected to an image storage service server 74.

[0113] The HMD 1 executes the dark place guide program, but the image sets and metadata based on the bright place distance images and bright place camera images are stored in the image data storage area 746 of the image storage service server 74, not in the HMD 1, and are read out as necessary. Alternatively, a group of necessary image sets corresponding to the location is downloaded in advance from the image storage service server 74 to the HMD 1 and used. This is the image storage service.

[0114] That is, this HMD system includes an image storage service server 74 connected to a network 73 , and an HMD 1 connected to the network 73 and communicating with the image storage service server 74 .

[0115] The HMD 1 also includes a camera that captures an image of the area in front of the user 70, a distance measuring sensor that obtains data representing the distance from the user 70 to a real object corresponding to each position in the camera's field of view, an illuminance sensor that obtains data representing the brightness of the location where the user 70 is located, and a display device that displays a visual image for the user 70 to view.

[0116] Then, the image storage service server 74 and the HMD 1 cooperate with each other to execute various processes as described below.

[0117] [Process 1-1] A process for generating a distance image in which each pixel represents the distance to each of the above positions corresponding to the camera's field of view based on data obtained by the distance sensor

[0118] [Process 1-2] A process for determining whether the user 70 is in a bright or dark place based on data obtained by the illuminance sensor.

[0119] [Process 1-3] A storage process for storing the camera image obtained by the camera and the distance image obtained by the generation process as an image set of a bright camera image and a bright distance image when the determination process determines that the subject is in a bright place.

[0120] [Process 1-4] A recognition process for recognizing the distance image obtained by the generation process as a dark place distance image when the determination process determines that the place is dark.

[0121] [Process 1-5] A search process for identifying a bright distance image corresponding to the recognized dark distance image by comparing the bright distance image stored in the storage process with the dark distance image recognized in the recognition process.

[0122] [Process 1-6] A determination process for determining a visual image based on a bright camera image included in the same image set as the bright range image identified by the identification process.

[0123] [Process 1-7] Display process for displaying the visual image determined by the determination process so that the user can visually recognize it.

[0124] By performing such processing, image sets and metadata, which tend to become large in size, can be stored in the image storage service server 74 rather than inside the HMD 1, thereby reducing the storage capacity of the HMD 1.

[0125] Furthermore, it is possible to have the image storage service server 74 perform processing with a high load, rather than the HMD 1, and to send only the results of that processing to the HMD 1, thereby lowering the processing specifications of the HMD 1 and achieving high-speed processing.

[0126] Fig. 7B is a diagram showing a second configuration example of the HMD system according to the second embodiment. Fig. 7B shows the configuration of an HMD system 101 in which a plurality of HMDs and a server are simultaneously connected via a network. The example shown in Fig. 7B is an example in which there are users wearing HMDs in a plurality of locations. There may be a plurality of users in each of the plurality of locations. The plurality of users share the image storage service provided by the image storage service server 74.

[0127] In FIG. 7B, 1a and 1b are HMDs, 70a and 70b are users, 71c, 71d, and 71e are communication signals, 72 is an access point, 73 is a network, and 74 is an image storage service server. User 70a is at Site abc (75a) and uses the image storage service, and user 70b is at Site aaa (75b) and uses the image storage service. Each user stores a bright camera image and a bright distance image obtained in a bright place as an image set according to the position and orientation of each user, together with metadata, in the image storage service server 74. In addition, each user can access the stored image set from various locations to call up the latest required image set.

[0128] That is, this system includes an image storage service server (server) 74 connected to a network 73, and a plurality of HMDs (mobile image display devices) 1a and 1b that are connected to the network 73 and communicate with the image storage service server 74.

[0129] In addition, each of the multiple HMDs 1a, 1b is equipped with a camera that captures an image of the area in front of the users 70a, 70b, a distance measurement sensor that obtains data representing the distance from the camera to a real object corresponding to each position in the camera's field of view, an illuminance sensor that obtains data representing the brightness of the location where the users 70a, 70b are located, a display device that displays a visual image to be viewed by the users 70a, 70b, and a position information acquisition device that acquires position information of the HMDs 1a, 1b.

[0130] In each of the multiple HMDs 1a and 1b, various processes as described below are executed in cooperation with the image storage service server 74 and the HMD.

[0131] [Process 2-1] A first generation process for generating a distance image in which each pixel represents the distance to each of the above positions corresponding to the field of view of the camera based on data obtained by the distance measuring sensor.

[0132] [Process 2-2] The first determination process determines whether the user is in a bright or dark place based on the data obtained by the illuminance sensor.

[0133] [Process 2-3] When the first determination process determines that the location is in a bright place, the camera image obtained by the camera and the distance image obtained by the first generation process are treated as an image set of a bright place camera image and a bright place distance image, and the image set is associated with the location information obtained by the location information acquisition device and stored in the image storage service server 74.

[0134] In addition, the image storage service server 74 cooperates with one of the plurality of HMDs 1a and 1b to execute various processes as described below.

[0135] [Process 2-4] A second generation process for generating a distance image that corresponds to the above-mentioned field of view and indicates the distance to each of the above-mentioned positions based on data obtained by a distance measuring sensor.

[0136] [Process 2-5] The second determination process determines whether the user is in a bright or dark place based on the data obtained by the illuminance sensor.

[0137] [Process 2-6] A recognition process for recognizing the distance image obtained by the second generation process as a dark place distance image when the second determination process determines that the place is dark.

[0138] [Process 2-7] Identification process for identifying bright distance images stored in the image storage service server 74, in which the position represented by the associated position information corresponds to the position represented by the position information acquired by the position information acquisition device of the one HMD.

[0139] [Process 2-8] A search process for identifying bright distance images having a degree of coincidence with the recognized dark distance images that is equal to or exceeds a threshold by comparing the bright distance images identified by the identification process with the dark distance images recognized by the recognition process.

[0140] [Process 2-9] A process of reading out from the image storage service server 74 a bright camera image included in the same image set as the bright distance image identified by the search process, and determining a visual image to be displayed by the one HMD based on the read out bright camera image.

[0141] [Process 2-10] Display process for displaying the determined visual image so that the user can visually recognize it

[0142] By executing such a process, image sets are successively accumulated by a plurality of users, and the accumulated image sets are shared.

[0143] In this embodiment, the HMD is connected to the network wirelessly, but may be connected via a wire.

[0144] Example of image set data structure Here, an example of the data structure of an image set is shown.

[0145] Fig. 9 is a diagram showing an example of the data configuration of an image set. As shown in Fig. 9, an image set including a camera image and a range image and its metadata are associated with each other to form a data set T10.

[0146] The data set T10 includes a service ID DT11 for an image storage service, and location information T12a and T12b (including site information (location information) if it can be specified). The location information is GPS coordinates obtained from a GPS sensor, and the site information is the name of a site, building, etc. specified from the GPS coordinates. The data of the image set is thus classified by location or location information.

[0147] The data set T10 includes the dates and times T13a and T13b when the image set was acquired. The data set T10 may also include orientation data indicating the direction in which the user is facing. The data set T10 further includes camera image data T14a and T14b and distance image data T15a and T15b. The camera image and the distance image are managed as a pair of data.

[0148] In the example shown in FIG. 9, the site (location) is shown as an indoor location of a building, but outdoor location information detected by a GPS or the like may also be used.

[0149] In addition, by registering image data that does not include distance images, such as camera images captured by a smartphone, it is possible to widely collect and store camera images. In this case, the distance images may be compared with stored camera images to estimate the degree of location match. For example, the comparison may be performed using the contour components of the camera images and the distance images.

[0150] According to the second embodiment, an HMD system having the same effect as the HMD of the first embodiment can be realized, and the implementation cost of the HMD can be reduced by storing image data in a server. In addition, images stored by multiple users can be shared and used by multiple users, which creates a synergistic effect and expands the range of places where the system can be used or the range of users.

[0151] (Embodiment 3) A third embodiment of the present invention will now be described.

[0152] Overview of the Third Embodiment The third embodiment is an HMD that detects a difference area between a dark distance image and a corresponding bright distance image, and determines a visual image according to the detection result. That is, when there is a difference area between a current dark distance image and a bright distance image corresponding to the dark distance image, the HMD according to this embodiment determines which distance image is causing the difference, performs processing according to the result, and determines a more suitable visual image.

[0153] It should be noted that the hardware configuration of the HMD according to the third embodiment is similar to that of the HMD according to the first embodiment, and therefore a description thereof will be omitted.

[0154] <HMD functional configuration> FIG. 10 is a functional block diagram showing the functional configuration of the HMD according to the third embodiment.

[0155] As shown in FIG. 10, the HMD 1c of embodiment 3 has a configuration based on the functional configuration of the HMD 1 of embodiment 1, and further includes a difference area detection device (detection device) 65, a difference factor discrimination device (discrimination device) 66, and a bright distance image selection device (selection device) 67.

[0156] The difference area detection device 65 detects difference areas between the current dark distance image and the identified previous bright distance image.

[0157] When a difference area is detected between the current dark distance image and a specified past bright distance image, the difference factor discrimination device 66 compares and analyzes the two distance images to determine whether the cause of the difference lies on the side of the dark distance image or the side of the bright distance image.

[0158] The bright distance image selection device 67 determines whether a bright distance image corresponding to the current dark distance image and acquired further back in time than the previously identified bright distance image is stored, and selects and reads out such a bright distance image as an even earlier bright distance image.

[0159] The visual image determination device 63 edits and determines the visual image depending on the presence or absence of differences between the dark distance image and the bright distance image, the cause of the differences if any, and the presence or absence of further past bright distance images.

[0160] <Flow of visual recognition image determination process> The process flow of the dark place guide program according to the third embodiment is basically the same as that of the first embodiment, but differs only in the process of determining the visual image. Therefore, only the process of determining the visual image will be described here, and the description of the other processes will be omitted.

[0161] 11A and 11B are flow diagrams of a visual recognition image determination process according to embodiment 3. This flow diagram corresponds to step S12 in the process flow according to embodiment 1 shown in FIG.

[0162] In step S121, a process of comparing the bright distance image with the dark distance image is performed. Specifically, different area detection device 65 compares the bright distance image with the dark distance image.

[0163] In step S122, a process is performed to determine whether there is a difference between the two compared distance images based on the comparison result in step S121. Specifically, the difference area detection device 65 determines whether there is a difference between the past bright distance image and the current dark distance image from the comparison result. If it is determined that there is no difference, that is, that the two distance images are substantially identical over the entire image area (S122, No), the process proceeds to step S123. On the other hand, if it is determined that there is a difference (S122, Yes), the process proceeds to step S124.

[0164] In step S123, the bright camera image is determined as the visual image. Specifically, the visual image determination device 63 determines the bright camera image as the visual image. This corresponds to the visual image determination example described with reference to FIG. 6. This ends the visual image determination process.

[0165] In step S124, the characteristics of the difference area are analyzed and the cause of the difference area is recognized. Specifically, the difference cause discrimination device 66 analyzes the characteristics of the difference area in both distance images. Then, based on the analysis result, it is determined whether the cause of the difference area is on the dark distance image side or the bright distance image side.

[0166] For example, when discontinuous distance data appears in a region of the dark distance image, it can be determined that a sudden displacement has occurred in the dark distance image, and conversely, when discontinuous distance data appears in a region of the bright distance image, it can be determined that a sudden displacement has occurred in the bright distance image.

[0167] Furthermore, for example, when the object included in the difference area is a person, a movable object, an artificial object, or the like, it can be considered that the difference area may have arisen due to differences caused by these objects moving.

[0168] Subsequent processing will differ depending on whether the cause of the difference is in the present or the past, and on what object is contained in the difference area.

[0169] In step S125, a process is performed to determine whether or not there is a cause of difference in the dark distance image. Specifically, the difference cause determination device 66 determines whether the cause of difference is in the dark distance image or the bright distance image based on the above analysis results. In this determination, if it is determined that there is a cause of difference in the dark distance image (S125, Yes), the process proceeds to step S126. On the other hand, in this determination, if it is determined that there is a cause of difference in the bright distance image (S125, No), the process proceeds to step S127.

[0170] In step S126, a process is performed to obtain a visual image by inserting a mask image or an AR object into the difference area of ​​the bright camera image. Specifically, the visual image determination device 63 obtains a visual image by performing an emphasis process on the area corresponding to the difference area in the specified bright camera image. For example, a mask image or an AR object is inserted into the area corresponding to the difference area of ​​the bright camera image. If an object included in the difference area can be recognized, an AR object corresponding to the object is inserted. If an object included in the difference area cannot be recognized or if such an AR object is not prepared, a mask image is superimposed on the area corresponding to the difference area of ​​the bright camera image, and an emphasis process is performed to obtain a visual image. This ends the visual image determination process.

[0171] Here, an example of determining the visual recognition image in step S126 will be described with reference to the drawings.

[0172] Fig. 12 is a diagram for explaining a first example of a process for determining a visual image by an HMD according to embodiment 3. Fig. 12 shows an example in which a visual image is determined by superimposing a mask image or an AR object on a bright camera image. This example is a case in which the difference between distance images is caused by the current dark distance image.

[0173] The current dark distance image 83E has a person present as a difference region 87. The corresponding bright distance image 81B has no person present. If the entire bright camera image 80B is used as the visual image, it is not appropriate because the user may collide with a person when moving in real space. Therefore, for example, an emphasized mask 88A is added to the corresponding region 87C of the difference region based on the bright camera image 80B. According to this embodiment, the user can be alerted.

[0174] Alternatively, when it is recognized that the difference region 87 is a person by analyzing the characteristics of the difference region 87, an AR object 88B of a person is added to a corresponding region 87C of the difference region based on the bright camera image 80B. According to this embodiment, it is possible to convey the details of the difference to the user.

[0175] In step S127, a process is performed to determine whether there is a bright distance image that corresponds to the current dark distance image and was acquired further back in time. Specifically, bright distance image selection device 67 determines whether there is a bright distance image that corresponds to the current dark distance image and was acquired further back in time than the previously specified bright distance image. If it is determined in this determination that there is no such bright distance image from a further back in time (S127, No), the process proceeds to step S128. On the other hand, if it is determined in this determination that there is such a bright distance image from a further back in time (S127, Yes), the process proceeds to step S129.

[0176] In step S128, a process is performed to determine a visual image by inserting an image portion of the difference area of ​​the dark distance image into the difference area of ​​the bright camera image. Specifically, the visual image determination device 63 acquires a past bright camera image, and determines an image obtained by inserting or pasting an image portion of the difference area of ​​the dark distance image into the difference area as the visual image.

[0177] Here, an example of determining the visual recognition image in step S128 will be described with reference to the drawings.

[0178] FIG. 13A is a diagram for explaining a second example of a process for determining a visual image by an HMD according to the third embodiment. FIG. 13A shows an example in which a part of a dark distance image is pasted onto a bright camera image to obtain a visual image. In FIG. 13A, 81C is a stored bright distance image, 80C is a stored bright camera image, 83C is a dark distance image, and 84C is a visual image to be visually recognized by a user. There are some differences between the distance images 81C and 83C. The area RA of the dark distance image 83C is a difference area, and a person is present in the area corresponding to the difference area in the bright distance image 81C. In this case, if the bright camera image 80C is applied to the entire area as a visual image, the user is misled into thinking that a person who is not currently present is present.

[0179] Therefore, an image portion of difference area RA in dark distance image 83C is superimposed on a corresponding area in bright camera image 80C to obtain visually recognized image 84C.

[0180] In step S129, a process of reading out bright distance images going back even further in time is performed. Specifically, bright distance image selection device 67 reads out bright distance images going back even further in time that correspond to the current dark distance image.

[0181] In step S130, the dark distance image is compared with the previous bright distance image and analyzed. Specifically, the visual image determination device 63 compares the current dark distance image with the previous bright distance image that was read out. Then, the necessary analysis is performed to determine whether there is a difference between these two distance images.

[0182] In step S131, it is determined whether there is a difference between the dark distance image and the further previous bright distance image. Specifically, based on the comparison and analysis results in step S130, visual image determination device 63 determines whether there is a difference between the current dark distance image and the further previous bright distance image that has been read out. If it is determined that there is a difference (S131, No), the process proceeds to step S128. On the other hand, if it is determined that there is no difference (S131, Yes), the process proceeds to step S132.

[0183] In step S132, a process of reading out a further previous bright-light camera image is performed. Specifically, the visual image determination device 63 reads out from the bright-light image storage device 59 a further previous bright-light camera image included in the same image set as the further previous bright-light distance image.

[0184] In step S133, a process is performed to determine a visual image by inserting an image portion of a difference area of ​​the further past bright camera image into the difference area of ​​the bright camera image. Specifically, the visual image determination device 63 determines an image obtained by inserting or pasting an image portion of a difference area of ​​the further past bright camera image into the difference area of ​​the specified past bright camera image as the visual image.

[0185] Here, an example of determining the visual recognition image in step S133 will be described with reference to the drawings.

[0186] FIG. 13B is a diagram for explaining a third example of the process of determining a visual image by the HMD according to the third embodiment. FIG. 13B shows an example of determining a visual image by pasting a part of a further past bright camera image onto a past bright camera image. In FIG. 13B, the situation is the same as the example shown in FIG. 13A, but the visual image is determined by referring to a further past bright camera image 80D saved at a time further back in time than the past bright camera image 80C. That is, an image portion corresponding to the area RB of the further past bright camera image 80D is fitted into the past bright camera image 80C to obtain a visual image 84D. The further past bright camera image 80D is a camera image captured when no person is present. An image portion of the area RB corresponding to the area RA in this further past bright camera image 80D is superimposed on the corresponding area of ​​the bright camera image 80C to obtain a visual image 84D.

[0187] Here, the entire image of the past bright camera image 80D may be used as the visual image as is, but a clearer image can be obtained by using the clearest of the stored camera images as the entire image and fitting only the partially different areas from other past bright camera images.

[0188] Alternatively, a bright image from a time period desired by the user may be selected and used as the full-range image, for example an image from the evening may be used as the full-range image.

[0189] In this embodiment, a person is used as an example of a difference region, but in reality, the difference region is not limited to a person. For example, the difference region may be a living thing such as an animal, a moving object such as a car, a bicycle, a motorcycle, a dolly, or a handcart, a newly installed facility, or a placed component. As a non-moving object, a building or a road sign may be considered. Furthermore, in order to recognize these, machine learning technology may be used to distinguish the difference object.

[0190] Furthermore, if new obstacles appear in past bright camera images, they may be displayed in an emphasized manner to draw attention to them.

[0191] Furthermore, if it is possible to determine what the object is, a CG image, sign, or text data may be displayed instead.

[0192] By performing this type of processing, a safe route can be ensured even in dark places without bumping into or tripping over obstacles.

[0193] In addition, the difference between the distance images may be caused by a factor other than the new appearance of an object in the current dark distance image. For example, at a disaster site, something that should be there may be missing, such as a collapsed road or a collapsed bridge. Even in such a case, as described in Figures 11A and 11B, it is possible to process the features according to the difference region 87, recognize what the difference region is, and superimpose the corresponding AR object to convey the details of the difference to the user.

[0194] According to the third embodiment, it is possible to determine a visual image in accordance with the occurrence state of the difference region, and high performance can be achieved in generating a visual image for the dark place guide function.

[0195] (Embodiment 4) A fourth embodiment of the present invention will now be described.

[0196] Overview of Fourth Embodiment The fourth embodiment is an HMD that includes processes such as translation, rotation, and scaling of distance images in the process of comparing distance images and determining whether they match. Note that the hardware configuration of the HMD according to the fourth embodiment is similar to that of the HMD according to the first embodiment, and therefore a description thereof will be omitted.

[0197] <Processing flow for comparing distance images and determining whether they match> The processing flow of the dark place guide program according to the fourth embodiment is basically the same as that of the first embodiment, but differs only in the processing of comparing distance images and determining whether they match. Therefore, only the processing flow of comparing distance images and determining whether they match will be described here, and descriptions of other flows will be omitted.

[0198] Fig. 14 is a flow diagram of a process for comparing distance images and determining whether they match according to embodiment 4. This flow diagram corresponds to steps S9 to S11 and S13 in the overall processing flow of Fig. 4, which are enclosed by a dashed line in Fig. 14. Also, Figs. 15A to 15C are diagrams for explaining an example of a process used for comparing distance images and determining whether they match according to embodiment 4.

[0199] In step S21, a process of extracting features of the dark distance image is performed. Specifically, the bright image search device 62 extracts features of the recognized dark distance image.

[0200] In step S22, a process of acquiring a bright light distance image is performed. Specifically, the bright light image search device 62 reads out one of the narrowed down bright light distance images.

[0201] In step S23, a process of extracting features of the bright range image is performed. Specifically, the distance image search device 62 extracts features of the read bright range image.

[0202] In step S24, the extracted features are compared and a process of calculating the distance image movement amount and the scaling factor is performed. Specifically, the distance image search device 62 compares the features of the dark distance image with the features of the bright distance image, and calculates the distance image movement amount and the distance image scaling factor by focusing on the same features.

[0203] In step S25, the dark distance image is moved and scaled. Specifically, distance image search device 62 moves and scales the dark distance image by the calculated movement amount and scaling magnification. Here, in steps S24 and S25, if the distance from the shooting position changes due to scaling, it is easier to compare distance images if the color indicating the perspective of the distance image is changed according to the scaling magnification.

[0204] In step S26, a process of comparing the distance images is performed. Specifically, distance image search device 62 compares the shifted and enlarged / reduced dark distance image with the bright distance image.

[0205] In step S28, a process is performed to determine whether the distance images match. Specifically, distance image search device 62 determines whether the dark distance image and the bright distance image match based on the above comparison result. If it is determined that they match (S28, Yes), the process proceeds to step S29. On the other hand, if it is determined that they do not match (S28, No), the process proceeds to step S30.

[0206] In step S29, a process is performed to determine whether the matching area is sufficient. Specifically, distance image search device 62 determines whether the coverage rate of the area where the recognized dark distance image and the read bright distance image match with respect to the entire area is greater than a specified value. If it is determined that the coverage rate is greater than the specified value (S29, Yes), the process proceeds to step S12. On the other hand, if it is determined that the coverage rate is equal to or less than the specified value (S29, No), the process proceeds to step S30.

[0207] In step S30, a process is performed to determine whether all comparisons of the narrowed-down bright distance images have been completed. Specifically, distance image search device 62 determines whether all comparisons have been completed and there are no bright distance images remaining to be compared next. If it is determined that all comparisons have been completed, i.e., there are no bright distance images remaining to be compared (S30, Yes), the process proceeds to step S14. On the other hand, if it is determined that all comparisons have not been completed, i.e., there are still bright distance images remaining to be compared (S30, No), the process returns to step S22 and the next bright distance image to be compared is read out.

[0208] Fig. 15A is a diagram for explaining a translation process in embodiment 3. Fig. 15B is a diagram for explaining a scaling process in embodiment 3. Fig. 15C is a diagram for explaining a combination process using a plurality of distance images in embodiment 3.

[0209] In Fig. 15A, the thick frame indicated by 81F is the dark distance image, and the thin frame indicated by 83F is the bright distance image. In Fig. 15A, when the dark distance image is shifted by a distance "a", if a match between the two distance images is found in the hatched area, the bright camera image in the hatched area is used to determine the visual image.

[0210] In FIG. 15B, when dark distance image 81F is multiplied by "z", if the two distance images match, the bright camera image is multiplied by "1 / z" and used to determine the visual image.

[0211] In FIGS. 15A and 15B, blank areas may be filled with, for example, a dark distance image or another past camera image.

[0212] 15C shows a case where multiple bright distance images 83J, 83K, and 83L are compared with dark distance image 81F. Although this example is not described in the above flow, such processing may be incorporated. In this example, when dark distance image 81F and bright distance images 83J, 83K, and 83L partially match, bright camera images corresponding to the matching areas are cut out and synthesized to generate a visual image.

[0213] At this time, the bright distance images 83J, 83K, and 83L may be superimposed so that a clearer image is prioritized, that is, the clearer image is positioned higher (closer). Alternatively, the images may be superimposed so that an image with a more recent acquisition date and time is prioritized, that is, the images with a more recent acquisition date and time are positioned higher. In the example shown in FIG. 15C, the order of clarity is bright distance image 83J>bright distance image 83K>bright distance image 83L, and the bright distance images 83J, 83K, and 83L are superimposed from the top in this order, with priority given to the clearer image.

[0214] In this embodiment, the target of the processing such as translation, rotation, and scaling is a dark distance image, but the processing may be applied to a bright distance image to perform a comparison process between the distance images. In addition, when a wide-angle image is captured using a 360-degree camera, an image in front of the user's HMD may be cut out and saved at a predetermined angle of view based on the position and direction of the HMD. Furthermore, when recording multiple captured images, the direction of the captured images at the same position may be aligned in advance, and images with approximately the same position and direction may be recorded in association with each other.

[0215] According to the fourth embodiment, it is possible to easily find an image in a bright place that substantially matches a dark place distance image.

[0216] (Embodiment 5) A fifth embodiment of the present invention will now be described.

[0217] Overview of Fifth Embodiment The fifth embodiment is an HMD that performs processing suitable for a case where the user is in a dark place but a part of the area in front of the user is illuminated by a light source.

[0218] Configuration of the HMD according to the fifth embodiment

[0219] Fig. 16 is an external view of an HMD according to embodiment 5. The same components as those in the HMD 1 shown in Fig. 1 are given the same numbers, and duplicated explanations will be omitted. The HMD 1d shown in Fig. 16 has a configuration in which a headlight 22 is added as an auxiliary light source to the HMD 1 shown in Fig. 1.

[0220] Fig. 17 is a functional block diagram of an HMD 1d according to embodiment 5. The same blocks as those in the functional block diagram shown in Fig. 3 are given the same numbers, and duplicated explanations will be omitted. The functional block diagram shown in Fig. 17 has a configuration in which an irradiation area detection device (detection device) 68 is added to the functional block diagram shown in Fig. 3.

[0221] The illumination area detection device 68 detects a spot illumination area, which is a partial area that is illuminated by a light source and is bright, in a dark place camera image obtained by the camera.

[0222] The visual image determination device 63 pastes an image portion of the spot illumination area in the dark camera image to a corresponding area of ​​the detected spot illumination area in the bright camera image or the dark distance image determined as the basis of the visual image, thereby determining the visual image.

[0223] The irradiation area detection device 68 is realized by the processor 36 executing the dark place guide program 42 using the memory 37 and the like.

[0224] In this embodiment, it is assumed that a part of the area in front of the user is illuminated by the headlight 22 of the HMD 1d. However, even if a flashlight held by the user or a distant illumination light illuminates a part of the area in front of the user, it can be considered as the same environment. Therefore, the fifth embodiment can be similarly applied to these cases.

[0225] Fig. 18 is a flow diagram of a visual image generation / editing process according to embodiment 5. This process flow corresponds to step S15 in the overall process flow according to embodiment 1 shown in Fig. 4, and is a modified example of embodiment 1.

[0226] In step S31, a process is performed to determine whether or not there is a spot illumination area. Specifically, the visual image determination device 63 determines whether or not there is a spot illumination area in the obtained dark place camera image. In this determination, if it is determined that there is a spot illumination area (S31, Yes), the process proceeds to step S32. On the other hand, in this determination, if it is determined that there is no spot illumination area (S31, No), the process proceeds to step S34.

[0227] In step S32, a process of cutting out an image portion of the spot illumination region is performed. Specifically, the visual image determination device 63 cuts out an image portion of the spot illumination region illuminated by a light source such as a headlight from the dark place camera image.

[0228] In step S33, a process of synthesizing the cut-out image portion is performed. Specifically, the visual image determination device 63 determines the visual image by pasting the cut-out image portion to an area corresponding to the spot irradiation area in the image set as the basis of the visual image. At this time, the pasting position and size of the cut-out image portion are adjusted so that the image set as the basis and the cut-out image portion are smoothly connected and look natural. After that, the generation and editing process of the visual image is completed, and the process proceeds to step S16 shown in FIG. 4.

[0229] In step S34, the visual recognition image is determined by the method according to the above-mentioned embodiments 1 to 4. After that, the visual recognition image generation and editing process ends, and the process proceeds to step S16 shown in FIG.

[0230] Fig. 19 is a diagram showing an example of a camera image and a visually recognized image obtained by the embodiment 5. In Fig. 19, 82G shows a dark place camera image, and 84G shows a visually recognized image. In the dark place camera image 82G, the spot illumination area 89 illuminated by the headlight 22 shows the situation ahead.

[0231] The visually recognized image 84G is an image obtained by pasting an image portion of the spot illumination region 89 in the dark place camera image onto a corresponding region of the spot illumination region 89 in the bright place camera image.

[0232] For example, consider a case where there is an unknown difference area in the visual confirmation image 84G, and the difference area is masked. In this case, the user can directly visually confirm what is in the difference area as a camera image by directing the headlights 22 toward the real space area corresponding to the mask display area. Here, in this example, one spot area is pasted and displayed, but an image of a wide area of ​​the illumination area can be obtained by the user changing the spot illumination direction to move the illumination area. An image of such a wide spot illumination area may be pasted.

[0233] According to the fifth embodiment, when there is a difference in front of the user in a dark place that is not captured in the past bright camera image, the difference can be confirmed by the visual image by illuminating the difference area with a light source. This is particularly effective when the display surface of the visual image has light non-transmittance, as described later.

[0234] (Embodiment 6) A sixth embodiment of the present invention will now be described. The sixth embodiment is an HMD that enables operation by the user's gestures.

[0235] 20A and 20B are diagrams showing a first example of the appearance of an HMD according to embodiment 6. HMD 1e shown in Fig. 20A and 20B includes a distance measurement sensor 23 suitable for measuring short distances (hereinafter referred to as a short distance measurement sensor) and a distance measurement sensor 24 suitable for measuring medium or long distances (hereinafter referred to as a medium to long distance measurement sensor). These distance measurement sensors are connected to controller 17, and controller 17 acquires distance data obtained by these distance measurement sensors.

[0236] The mid-to-long distance measuring sensor 24 is for obtaining distance data for the real space in front of the user's line of sight. The short distance measuring sensor 23 is for detecting gestures made by the user's fingers. A table in which distance data corresponding to the type of gesture and the type of operation are associated is stored in the saved data area in the controller 17. The controller 17 refers to the table based on the distance data obtained by the short distance measuring sensor 23, recognizes the type of gesture made by the user, and accepts the corresponding operation. Examples of the operation include turning on and off a dark place guide program, and switching a visual image to be displayed.

[0237] The mid- to long-distance measuring sensor 24 is installed at the end in the width direction of the upper frame of the lens (display) of the HMD 1e, for example, as shown in FIG. 20A. The short-distance measuring sensor 23 is installed at the center of the upper frame. In this case, the distance measuring central axis direction 24c of the mid- to long-distance measuring sensor 24 and the distance measuring central axis direction 23c of the short-distance measuring sensor 23 are approximately the same. The user puts his or her fingers out in front of his or her eyes to perform a gesture operation. The short-distance measuring sensor 23 obtains distance data corresponding to the user's gesture operation.

[0238] Incidentally, when a user looks at a place close to the user in front of the user, the user's line of sight is often lower than when visually recognizing an object in front. Also, when a user makes a gesture with his / her fingers, it is natural for the user to look at the fingers. Therefore, the short-distance measuring sensor 23 may be installed so that its distance measuring central axis direction 23c is inclined downward by a predetermined angle (e.g., 30°) from the distance measuring central axis direction 24c of the medium-to-long-distance measuring sensor 24, for example, as shown in FIG. 20B. This allows the user to perform a gesture operation with his / her fingers while checking it in a natural line of sight, enabling a more natural and stress-free operation.

[0239] The arrangement of the short-range distance measuring sensor 23 and the medium- to long-range distance measuring sensor 24 is not limited to the above example, and various arrangements can be considered depending on the design concept or specifications.

[0240] 21A and 21B are diagrams showing a second example of the appearance of the HMD according to embodiment 6. Also, Fig. 21C is a diagram showing a gesture operation area by a user's finger.

[0241] 21A and 21B show a modification in which the arrangement of the short-distance measuring sensor is changed from that of the first example.

[0242] The HMD 1f according to this modification is an example in which the short distance measuring sensor 23 is disposed on a portion of the frame housing 20 shaped like the temple of glasses. The central axis direction 24c of the medium to long distance measuring sensor 24 is the same as in the previous example, but the central axis direction 23c of the short distance measuring sensor 23 extends toward the side of the user's head. In this case, the user performs gesture operations with his / her fingers on the side of the user's head.

[0243] For example, as shown in Fig. 21C, the gesture operation space area is defined by an xyz coordinate system, and the gesture operation space area is divided into a plurality of partial areas. The short-distance measuring sensor 23 obtains distance measurement values ​​in each of the divided partial areas, and the controller 17 detects the gesture operation based on the distribution state of the distance measurement values ​​or their change over time. Examples of gesture operations include a method in which a finger is moved in the x direction to instruct the display screen to move forward or backward, moved in the y direction to instruct the display screen to move up or down, and moved in the z direction to enlarge or reduce the display screen.

[0244] This allows the user to perform gesture operations with their fingers near the side of their head, and allows operations to be performed without the gestures interfering with the field of view.

[0245] According to the sixth embodiment, the image display device can be operated by the user's gesture operations to turn on and off the dark place guide program, switch the displayed image, etc., and an easier-to-use image display device can be realized. For example, even when the user is working on-site wearing gloves, the image display device can be operated without operating a button-type or touch-type operation unit that requires relatively delicate operation.

[0246] (Embodiment 7) A seventh embodiment of the present invention will now be described. The seventh embodiment is an example of an HMD configured to display a user interface screen.

[0247] The controller 17 of the HMD controls the display device to display a user interface screen on the display surface, and also accepts predetermined operations such as the gesture operations shown in the sixth embodiment.

[0248] Fig. 22 is a diagram showing an example of a user interface screen in the HMD. For example, when the illuminance sensor detects that the user's field of vision has become dark, the HMD controller 17 controls the display device to display a screen display switching selection screen 90. For example, a selection screen as shown in Fig. 22 is displayed, allowing the user to select whether or not to switch the display screen to a bright image mode.

[0249] If the user does not want to switch the screen, that is, selects "No," then, for example, a distance measurement image is displayed. On the other hand, if the user wants to switch the screen, that is, selects "Yes," then a selection screen is displayed for selecting several ways to display the screen. For example, a selection screen such as that shown in the lower part of Fig. 22 is displayed, allowing the user to select the image to be displayed from among "daytime image," "partial image interpolation," and "arranged image."

[0250] This allows the display of bright images captured in bright places, and when a portion of the field of view is dark, the display of only that dark portion can be replaced with an image captured in bright places. By selecting an arranged image, the image can be processed and displayed by image processing. For example, an image can be displayed that has been purposely converted to black and white or sepia, or an image that has been arranged in vivid colors such as those used in CG or animation.

[0251] According to the seventh embodiment, the video can be switched according to the user's intentions and preferences.

[0252] (Embodiment 8) An eighth embodiment of the present invention will now be described. In the first to seventh embodiments, an HMD having an optically transparent image display surface that transmits light from the front of the user is used as an image display device. In the present embodiment, an HMD having an optically non-transparent image display surface that does not transmit light from the front of the user is used as an image display device.

[0253] FIG. 23 is an external view of an HMD according to the eighth embodiment as an image display device according to the first to seventh embodiments. FIG. 23 is a diagram showing an example of a goggle-type HMD having a display surface that is non-transparent to light from the front of the user. This is a usage mode called a video see-through mode in which images captured by a camera are displayed on the left and right displays. The HMD 1z shown in FIG. 23 includes a housing 20d molded to cover the user's field of view, and a fixture 20e for fixing the housing 20d so as to cover the user's field of view. A left-eye display 25 and a right-eye display 26 are provided on the surface of the housing 20d that covers the user's eyes. In addition, the housing 20d is equipped with a right camera 11a, a left camera 11b, a distance measuring sensor 12, a controller 17, a microphone 18, a speaker 19, a battery 21, a headlight 22, and a short-distance measuring sensor 23. The illuminance sensor is built into the controller 17, but may be provided on the upper part of the housing 20d.

[0254] The left-eye display 25 and the right-eye display 26 are configured, for example, by a liquid crystal panel, an organic EL (Electro Luminescence) panel, a plasma display, etc. Of course, the left-eye and right-eye displays 25 and 26 may each be configured by an image projection device using a projector.

[0255] The user wears the HMD1z on their head and views an image displayed on the left-eye display with their left eye and an image displayed on the right-eye display with their right eye, allowing them to view a camera image, distance image, or visual image in three dimensions.

[0256] According to the eighth embodiment, a more robust HMD that fits closely to the face can be used as the image display device, and the user's eyes can be protected from dangerous obstacles, gases, or liquids that may be present at the site. In addition, since it is no longer necessary to limit the display device to a projector type, the degree of freedom in designing the image display device is increased.

[0257] (Embodiment 9) A ninth embodiment of the present invention will now be described. In the first to eighth embodiments, an HMD is used as the image display device. In the ninth embodiment, a portable information terminal is used as the image display device. Examples of the portable information terminal include a smartphone, a tablet terminal device, and a notebook computer.

[0258] There are many models of portable information terminals that have a camera and a distance measuring sensor on the side opposite to the display screen (back side). The distance measuring sensor may be added later. Such portable information terminals can operate in the same way as the HMDs described above.

[0259] Fig. 24A is a diagram showing an example of the angle of view of the camera of the portable information terminal and the measurement range of the distance measuring sensor. Fig. 24B is a diagram showing an example of a state in which a dark distance image is displayed on the portable information terminal. Fig. 24C is a diagram showing an example of a state in which a bright camera image is displayed on the portable information terminal.

[0260] The portable information terminal 1s is configured to define x and y coordinate axes as shown by the dashed lines in Fig. 24A, and to have a camera angle of view 91 and a distance measuring sensor measurement range 92 as shown by the dashed lines. The portable information terminal 1s can also display a dark distance image on the display screen as shown in Fig. 24B, or a bright camera image corresponding to the dark distance image as shown in Fig. 24C.

[0261] According to the ninth embodiment, it is possible to perform the same operation as the HMD according to the first to eighth embodiments by using a versatile and widely used portable information terminal without using a specially manufactured image display device such as an HMD. This makes it possible to reduce the development cost of the image display device or the cost of the device itself. In addition, it is possible to expand the users to those who already own a portable information terminal, and it is possible to efficiently accumulate image sets in any location. In addition, the user can obtain visual images such as bright camera images in any location.

[0262] Although various embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modified examples are included. For example, the same effect can be obtained by replacing the generation of a distance image by a distance sensor with the generation of an infrared camera image by an infrared camera. Furthermore, the above-mentioned embodiments have been described in detail to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all of the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the scope of the present invention. Furthermore, the numerical values, messages, etc. included in the text and figures are merely examples, and the effect of the present invention is not impaired even if different ones are used.

[0263] In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them as an integrated circuit, for example. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor such as an MPU or a CPU interpreting and executing a program that realizes each function. In addition, the range of functions realized by software is not limited, and hardware and software may be used together. Information such as programs, tables, files, etc. that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

[0264] Possible aspects of the present invention will be described below.

[0265] [Appendix 1] An image display device that displays a visual image for a user to visually recognize, A camera that captures an image in front of the user and obtains a camera image; a distance measuring sensor for obtaining data representing a distance from the user to each position on a real object included in a viewing area of ​​the camera; an illuminance sensor for obtaining data representing the brightness of a location where the user is located; a generating device that generates a distance image corresponding to the field of view and in which each pixel represents a distance to each of the positions based on data obtained by the distance measuring sensor; a determination device that determines whether the location of the user is a bright place or a dark place based on data obtained by the illuminance sensor; a storage device that stores the camera image obtained by the camera and the distance image obtained by the generation device as an image set including a bright place camera image and a bright place distance image when the determination device determines that the place is bright; a recognition device that recognizes the distance image obtained by the generation device as a dark place distance image when the determination device determines that the place is a dark place; a search device for identifying a bright distance image corresponding to the recognized dark distance image by comparing the bright distance image stored by the storage device with the dark distance image recognized by the recognition device; a determination device that determines a visual image to be visually recognized by the user based on a bright camera image included in the same image set as the bright range image identified by the search device; A display device that displays the visual image; An image display device comprising:

[0266] [Appendix 2] A program for causing a computer to function as a generating device, a determining device, a storing device, a recognizing device, a searching device, and a deciding device in the image display device described in [Appendix 1].

[0267] [Appendix 3] A computer-readable recording medium having recorded thereon a program for causing a computer to function as a generating device, a determining device, a storing device, a recognizing device, a searching device, and a deciding device in the image display device described in [Appendix 1].

[0268] [Appendix 4] In the image display device described in Appendix 1, the process of comparing a dark distance image and a bright distance image applies at least one of translation, rotation, and zoom to at least one of the dark distance image and the bright distance image.

[0269] [Appendix 5] In the image display device according to claim 1, The image display device uses a smartphone, a tablet terminal device, or a notebook computer as the image display device.

[0270] [Appendix 6] In the image display device according to claim 1, The camera also serves as the illuminance sensor.

[0271] [Appendix 7] In the image display device according to claim 1, The search device performs at least one of translation, rotation, and enlargement / reduction on at least one of the recognized dark distance image and the identified bright distance image to compare them.

[0272] [Appendix 8] In the image display device according to claim 1, The display device is an image display device having a screen onto which the visual image is projected.

[0273] [Appendix 9] In the image display device according to claim 1, The display device is an image display device having a display device that displays the visual image.

[0274] [Appendix 10] In the image display device according to claim 1, The search device is an image display device that identifies a bright place distance image that corresponds to the dark place distance image and has a similarity or match with the dark place distance image that is equal to or greater than a threshold value.

[0275] [Appendix 11] In the image display device according to claim 1, The search device is an image display device that identifies a bright distance image that is recognized by artificial intelligence as matching, approximating or similar to the dark distance image as a bright distance image corresponding to the dark distance image.

[0276] [Appendix 12] In the image display device according to claim 1, A time information acquisition device is provided, the time information acquisition device acquires time information including at least a date; The storage device includes: A process of extracting a bright range image corresponding to a bright range image to be stored by the storage device from among the stored bright range images; a process of storing the image set in association with time information acquired by the time information acquisition device; An image display device that, when saving the image set, performs a process of overwriting the image set including the bright distance image extracted by the extraction process, or a process of erasing the image set including the extracted bright distance image if the time represented by the time information associated with the extracted bright distance image is a first time or more earlier than the time represented by the time information associated with the bright distance image to be saved.

[0277] [Appendix 13] In the image display device according to claim 12, An image display device, wherein the extraction process is a process of extracting a bright distance image corresponding to the bright distance image to be stored, the bright distance image having a similarity or degree of agreement between the bright distance image to be stored being equal to or greater than a threshold value.

[0278] [Appendix 14] In the image display device according to claim 12, An image display device, wherein the extraction process is a process of extracting a bright distance image that corresponds to the bright distance image to be stored and that is recognized by artificial intelligence as matching, approximating, or similar to the bright distance image to be stored. [Explanation of symbols]

[0279] 1, 1a, 1b, 1h...transmissive head mounted display, 1z...non-transmissive head mounted display, 1s...portable information terminal, 11...camera, 12...distance measuring sensor, 13...right eye projector, 14...left eye projector, 15...image screen, 16...nose pad, 17...controller, 18...microphone, 19...speaker, 20, 20a to 20c...frame housing, 21...battery, 22...headlight, 23...short distance measuring sensor, 24...medium to long distance measuring sensor, 30...internal bus, 31...GPS sensor, 32...illuminance sensor, 33...acceleration sensor, 34...gyro sensor, 35...orientation sensor, 36...processor, 37...memory, 38...image memory, 39...non-volatile storage device, 40...communication device, 41...base This operation program, 42... dark place guide program, 43... saved data area, 51... position information acquisition device, 52... time information acquisition device, 53... orientation information acquisition device, 54... attitude information acquisition device, 55... camera image acquisition device, 56... distance image generation device, 57... bright place and dark place judgment device, 58... bright place image storage management device, 59... bright place image storage device, 60... dark place distance image recognition device, 61... comparison image narrowing down device, 62... bright place image search device, 63... visual image determination device, 64... visual image display device, 65... difference area detection device, 66... ​​difference factor discrimination device, 67... bright place distance image selection device, 68... irradiation area detection device, 70... user, 72... access point, 73... network, 74... image storage service server, 100, 101... HMD system.

Claims

1. An image display device that displays a visual image for a user to visually recognize, A camera that captures an image in front of the user and obtains a camera image; a distance measuring sensor for obtaining data representing a distance from the user to each position on a real object included in a viewing area of ​​the camera; an illuminance sensor for obtaining data representing the brightness of a location where the user is located; a generating device that generates a distance image corresponding to the field of view and in which each pixel represents a distance to each of the positions based on data obtained by the distance measuring sensor; a determination device that determines whether the location of the user is a bright place or a dark place based on data obtained by the illuminance sensor; a storage device that stores the camera image obtained by the camera and the distance image obtained by the generation device as an image set including a bright place camera image and a bright place distance image when the determination device determines that the place is bright; a recognition device that recognizes the distance image obtained by the generation device as a dark place distance image when the determination device determines that the place is a dark place; a search device for identifying a bright distance image corresponding to the recognized dark distance image by comparing the bright distance image stored by the storage device with the dark distance image recognized by the recognition device; a determination device that determines a visual image to be visually recognized by the user based on a bright camera image included in the same image set as the bright range image identified by the search device; A display device that displays the visual image; An image display device comprising:

2. 2. The image display device according to claim 1, The present invention includes a location information acquisition device and a selection device, The location information acquisition device acquires location information of the user, When storing the image set, the storage device stores the image set in association with the location information acquired by the location information acquisition device; The narrowing-down device narrows down the bright field distance images to be compared by the search device based on position information associated with the image set.

3. 2. The image display device according to claim 1, A time information acquisition device is provided, the time information acquisition device acquires time information including at least a date; The storage device includes: an extraction process for extracting a bright range image corresponding to the bright range image to be stored from among the stored bright range images; a storage process for storing the image set in association with time information acquired by the time information acquisition device; When saving the image set, the image display device performs an overwrite process to overwrite the image set including the bright distance image extracted by the extraction process, or an erasure process to erase the image set including the extracted bright distance image if the time represented by the time information corresponding to the extracted bright distance image is a first time or more earlier than the time represented by the time information corresponding to the bright distance image to be saved.

4. 2. The image display device according to claim 1, A detection device is provided, The detection device detects a difference area between the recognized dark distance image and the specified bright distance image, The determination device determines the visible image by performing an insertion process to insert an image corresponding to the difference area in the recognized dark distance image into a corresponding area of ​​the difference area in a bright camera image included in the same image set as the identified bright distance image.

5. 5. The image display device according to claim 4, A discrimination device is provided, the discrimination device discriminates whether a cause of occurrence of the difference region is in the recognized dark distance image or the specified bright distance image; The determination device executes the insertion process when the discrimination device determines that the occurrence cause is in the specified bright distance image.

6. 2. The image display device according to claim 1, A detection device and a selection device are provided, The detection device detects a difference area between the recognized dark distance image and the specified bright distance image, the selection device compares the bright distance image stored by the storage device with the recognized dark distance image to select a bright distance image that corresponds to the recognized dark distance image and is different from the specified bright distance image; The determination device determines the visible image by performing an insertion process to insert an image corresponding to the detected difference area in a bright camera image included in the same image set as the bright distance image selected by the selection device into a corresponding area of ​​the detected difference area in a bright camera image included in the same image set as the identified bright distance image.

7. 2. The image display device according to claim 1, A detection device and a discrimination device are provided, The detection device detects a difference area between the recognized dark distance image and the specified bright distance image, the discrimination device discriminates whether a cause of occurrence of the difference area detected by the detection device is in the recognized dark distance image or the specified bright distance image; The determination device, when the discrimination device determines that the cause of the occurrence is in the recognized dark distance image, determines the visible image by performing an enhancement process that enhances the corresponding area of ​​the detected difference area in a bright camera image included in the same image set as the identified bright distance image.

8. 2. The image display device according to claim 1, A detection device and a discrimination device are provided, The detection device detects a difference area between the recognized dark distance image and the specified bright distance image, the discrimination device discriminates whether a cause of occurrence of the difference area detected by the detection device is in the recognized dark distance image or the specified bright distance image; When the discrimination device determines that the cause of the occurrence is in the recognized dark distance image, the determination device determines the visible image by performing an insertion process to insert an artificial image corresponding to the characteristics of the detected difference area in a bright camera image included in the same image set as the identified bright distance image.

9. 2. The image display device according to claim 1, The search device identifies a plurality of photopic distance images; The determination device determines the visual image based on a plurality of bright field distance images identified by the search device.

10. 2. The image display device according to claim 1, A light source and a detector, The light source illuminates a portion of the viewing area, The detection device detects an area illuminated by the light source in a dark place camera image obtained by the camera in a dark place, The determination device determines the visible image by performing an insertion process to insert a corresponding image of the detected illuminated area in a dark camera image included in the same image set as the recognized dark distance image into a corresponding area of ​​the illuminated area detected by the detection device in a bright camera image included in the same image set as the identified bright distance image.

11. 2. The image display device according to claim 1, Includes a head mounted display, The head mounted display is an image display device having the camera, the distance measurement sensor, the illuminance sensor, and the display device.

12. An image display system comprising: a server connected to a network; and a portable image display device connected to the network and communicating with the server, The image display device includes: A camera that captures an image in front of a user and obtains a camera image; a distance measuring sensor for obtaining data representing a distance from the user to each position on a real object included in a viewing area of ​​the camera; an illuminance sensor for obtaining data representing the brightness of a location where the user is located; a display device that displays a visual image to be visually recognized by the user; By cooperation between the server and the image display device, a generation process for generating a distance image corresponding to the field of view and in which each pixel represents a distance to each of the positions, based on data obtained by the distance measuring sensor; A determination process for determining whether the location of the user is a bright place or a dark place based on data obtained by the illuminance sensor; a storage process for storing the camera image obtained by the camera and the distance image obtained by the generation process as an image set of a bright place camera image and a bright place distance image when the determination process determines that the location is a bright place; a recognition process for recognizing the distance image obtained by the generation process as a dark place distance image when the determination process determines that the place is a dark place; a search process for identifying a bright place distance image corresponding to the recognized dark place distance image by comparing the bright place distance image stored by the storage process with the dark place distance image recognized by the recognition process; A determination process for determining the visual image based on a bright camera image included in the same image set as the bright range image identified by the search process; An image display system that performs the above steps.

13. An image display system comprising: a server connected to a network; and a plurality of portable image display devices connected to the network and communicating with the server, Each of the plurality of image display devices includes A camera that captures an image in front of a user and obtains a camera image; a distance measuring sensor for obtaining data representing a distance from the user to each position on a real object included in a viewing area of ​​the camera; an illuminance sensor for obtaining data representing the brightness of a location where the user is located; a display device that displays a visual image to be visually recognized by the user; a location information acquisition device for acquiring location information of the user, In each of the plurality of image display devices, the server and the image display device cooperate with each other to a first generation process for generating a distance image corresponding to the field of view and in which each pixel represents a distance to each of the positions, based on data obtained by the distance measuring sensor; a first determination process for determining whether a location of the user is a bright place or a dark place based on data obtained by the illuminance sensor; a storage process for storing, when it is determined that the first determination process is a bright place, a camera image obtained by the camera and a distance image obtained by the first generation process as an image set including a bright place camera image and a bright place distance image, and storing the image set in the server in association with location information obtained by the location information acquisition device; By cooperation between the server and one of the plurality of image display devices, a second generation process for generating a distance image corresponding to the field of view and representing a distance to each of the positions based on data obtained by the distance measuring sensor; a second determination process for determining whether the location of the user is a bright place or a dark place based on data obtained by the illuminance sensor; a recognition process for recognizing, when the second determination process determines that the location is a dark place, the distance image obtained by the second generation process as a dark place distance image; an identification process for identifying a bright distance image among the bright distance images stored in the server, the bright distance image being one in which a position indicated by associated position information corresponds to a position indicated by position information acquired by the position information acquisition device of the one image display device; a search process for comparing the bright distance image identified by the identification process with the dark distance image recognized by the recognition process to identify a bright distance image corresponding to the recognized dark distance image; The image display system executes a determination process of reading out from the server a bright camera image included in the same image set as the bright distance image identified by the search process, and determining the visual image to be displayed by the one image display device based on the read bright camera image.

14. 13. The image display system according to claim 12, The image display device has a head-mounted display shape, An image display system, wherein the display device has an image display surface that covers part or all of the user's field of vision and is transparent to light from in front of the user.

15. 13. The image display system according to claim 12, The image display device has a head-mounted display shape, An image display system, wherein the display device has an image display surface that covers part or all of the user's field of vision and is non-transparent to light from in front of the user.

16. A portable image display device that communicates with a server connected to a network, The image display device includes: A camera that captures an image in front of a user and obtains a camera image; a distance measuring sensor for obtaining data representing a distance from the user to each position on a real object included in a viewing area of ​​the camera; an illuminance sensor for obtaining data representing the brightness of a location where the user is located; a display device that displays a visual image to be visually recognized by the user; By cooperation between the server and the image display device, a generating device that generates a distance image corresponding to the field of view and in which each pixel represents a distance to each of the positions based on data obtained by the distance measuring sensor; a determination device that determines whether the location of the user is a bright place or a dark place based on data obtained by the illuminance sensor; a communication device that outputs, when the determination device determines that the location is a bright place, a camera image obtained by the camera and a distance image obtained by the generation process as an image set of a bright place camera image and a bright place distance image to be stored on the server; a recognition device that recognizes the distance image obtained by the generation device as a dark place distance image when the determination device determines that the place is a dark place; identifying a bright distance image corresponding to the recognized dark distance image by comparing the bright distance image stored in the server with the dark distance image recognized by the recognition process; receiving, at the communication device, the visual image processed based on a bright camera image included in the same image set as the specified bright range image; An image display device that displays the received visual image.

Images