Image display device and image display system

Abstract

To provide an image display device and an image display system which improve visibility in front of a user at a dark place.SOLUTION: An image display device comprises a camera, a ranging sensor, an illuminance sensor, a brightness and darkness determination device, an image generation device, an image storage device, an image retrieval device, a visually recognized image determination device and a display device. The camera images a real object in front of a user to obtain a camera image. The image generation device generates a distance image representing distance to each position of the real object for the camera image on the basis of data of the ranging sensor. The brightness and darkness determination device determines whether a place at which the user exists is a bright place or a dark place on the basis of data of the illuminance sensor. The image storage device stores the camera image and the distance image obtained at the bright place as a set. The image retrieval device retrieves and specifies the distance image corresponding to the obtained distance image of the dark place from among already stored distance images of the bright place, at the dark place. The visually recognized image determination device determines a visually recognized image on the basis of the camera image included in the same set as the specified distance image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image display device and an image display system that support visibility in dark places. [Background technology]

[0002] Augmented reality technology, which adds AR (Argument Reality) objects created with CG (Computer Graphics) to real space or background images, is used in content such as games and maintenance work. Users can experience augmented reality by viewing content images synthesized on the display screen using a head-mounted display (HMD), a portable information terminal, or the like.

[0003] Patent Document 1 discloses an HMD equipped with a camera that captures an image of the background and a distance measuring device that measures the distance to a real object in the background. It also discloses a technology that executes a process to extract a real object from an image obtained by the camera and assigns an AR object in association with the real object.

[0004] An example of a distance measurement sensor is disclosed in Patent Document 2. Patent Document 2 discloses a distance measurement sensor (LiDAR: Light Detection and Ranging) in the field of autonomous driving, which scans laser light and measures the arrival time or phase difference of the reflected light to obtain data indicating the distance to an object in the front direction. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2020 / 115784 [Patent Document 2] International Publication No. 2019 / 082926 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 does not take into consideration use in dark places. For example, in dark places where it is difficult to capture images with a camera, such as places where the lights are turned off due to a power outage or the like, construction or maintenance sites at night where lighting cannot be used due to consideration for the surrounding environment, or inside buildings where the lights are not on, the user's forward visibility is poor, making it difficult to use the technology.

[0007] Under these circumstances, it is desirable to improve the forward visibility of the user in dark places. [Means for solving the problem]

[0008] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0009] An image display device according to a representative embodiment of the present invention is an image display device that displays a visual image for a user to visually recognize, and includes a camera that captures an image in front of the user and obtains a camera image, a distance measurement sensor that obtains data representing the distance from the user to each position in a real object included in the field of view of the camera, an illuminance sensor that obtains data representing the brightness of the location where the user is located, a generation device that generates a distance image corresponding to the field of view and in which each pixel represents the distance to each position based on the data obtained by the distance measurement sensor, a determination device that determines whether the location where the user is located is a bright place or a dark place based on the data obtained by the illuminance sensor, and a camera image obtained by the camera when the determination device determines that the location is a bright place. The image display device includes a storage device that stores the image and the distance image obtained by the generation device as an image set including a bright place camera image and a bright place distance image; a recognition device that recognizes the distance image obtained by the generation device as a dark place distance image when the determination device determines that the place is dark; a search device that identifies a bright place distance image corresponding to the recognized dark place distance image by comparing the bright place distance image stored in the storage device with the dark place distance image recognized by the recognition device; a determination device that determines a visual image to be viewed by the user based on a bright place camera image included in the same image set as the bright place distance image identified by the search device; and a display device that displays the visual image. [Effects of the Invention]

[0010] The effects obtained by the representative inventions disclosed in this application will be briefly explained as follows.

[0011] According to the exemplary embodiment of the present invention, it is possible to improve the forward visibility of a user in a dark place. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is an external view of an HMD according to a first embodiment. [Figure 2]FIG. 1 is a block diagram showing the hardware configuration of an HMD according to a first embodiment. [Figure 3] FIG. 2 is a functional block diagram showing the functional configuration of the HMD according to the first embodiment. [Figure 4] FIG. 4 is a process flow diagram of a dark place guide program using an HMD according to the first embodiment. [Figure 5] FIG. 10 is a process flow diagram of image set storage and management by the HMD according to the first embodiment. [Figure 6] 3A and 3B are diagrams showing examples of images obtained by the HMD according to the first embodiment. [Figure 7A] FIG. 10 is a diagram illustrating a first example of the configuration of an HMD system according to a second embodiment. [Figure 7B] FIG. 10 is a diagram illustrating a second configuration example of the HMD system according to the second embodiment. [Figure 8] FIG. 2 is a block diagram showing the hardware configuration of an image storage service server. [Figure 9] FIG. 10 is a diagram illustrating an example of the data configuration of an image set. [Figure 10] FIG. 10 is a functional block diagram showing the functional configuration of an HMD according to a third embodiment. [Figure 11A] FIG. 11 is a flowchart of a visual recognition image determination process according to the third embodiment. [Figure 11B] FIG. 11 is a flowchart of a visual recognition image determination process according to the third embodiment. [Figure 12] 10A and 10B are diagrams for explaining a first example of a process for determining a visual image by an HMD according to a third embodiment. [Figure 13A] 10A and 10B are diagrams for explaining a second example of the process of determining a visual image by the HMD according to the third embodiment. [Figure 13B] 13A and 13B are diagrams for explaining a third example of the process of determining a visual image by the HMD according to the third embodiment. [Figure 14] FIG. 11 is a flowchart of a process for comparing distance images and determining whether they match according to the fourth embodiment. [Figure 15A] FIG. 11 is a diagram for explaining a parallel movement process in the third embodiment. [Figure 15B]FIG. 11 is a diagram for explaining scaling processing in the third embodiment. [Figure 15C] 13A and 13B are diagrams for explaining a combination process using a plurality of distance images in the third embodiment. [Figure 16] FIG. 10 is an external view of an HMD according to a fifth embodiment. [Figure 17] FIG. 10 is a functional block diagram of an HMD according to a fifth embodiment. [Figure 18] FIG. 13 is a flowchart of a visual image generation and editing process according to the fifth embodiment. [Figure 19] 13A and 13B are diagrams showing examples of a camera image and a visually recognized image obtained in the fifth embodiment. [Figure 20A] FIG. 13 is a diagram showing a first example of the appearance of an HMD according to a sixth embodiment. [Figure 20B] FIG. 13 is a diagram showing a first example of the appearance of an HMD according to a sixth embodiment. [Figure 21A] FIG. 20 is a diagram showing a second example of the appearance of the HMD according to the sixth embodiment. [Figure 21B] FIG. 20 is a diagram showing a second example of the appearance of the HMD according to the sixth embodiment. [Figure 21C] FIG. 10 is a diagram showing a gesture operation area by a user's finger. [Figure 22] FIG. 2 is a diagram showing an example of a user interface screen in an HMD. [Figure 23] FIG. 13 is an external view of an HMD according to an eighth embodiment. [Figure 24A] 10A and 10B are diagrams illustrating an example of the angle of view of a camera and the measurement range of a distance measuring sensor of a portable information terminal. [Figure 24B] FIG. 10 is a diagram showing an example of a state in which a dark distance image is displayed on a portable information terminal. [Figure 24C] FIG. 10 is a diagram showing an example of a state in which a bright-light camera image is displayed on a portable information terminal. DETAILED DESCRIPTION OF THE INVENTION

[0013] Next, we will explain the embodiments of the present invention. By providing the technology shown in each of the following embodiments, it becomes possible to ensure safety in dark places based on highly accurate distance measurement. This distance measurement technology contributes to "9. Build resilient infrastructure, promote inclusive and sustainable industrialization, promote innovation and foster resilience" and "11. Make cities and towns inclusive and sustainable" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0014] It should be noted that the embodiments described below are merely examples for realizing the present invention and do not limit the technical scope of the present invention.

[0015] In addition, in the following embodiments, components having the same functions are denoted by the same reference numerals, and repeated explanations thereof will be omitted unless particularly necessary.

[0016] Furthermore, in the following description of each embodiment, expressions such as "image matching" or "the same image" are used, but "matching" or "identical" here does not mean 100% perfect matching or perfect identity. "Matching" or "identical" here allows for deviations in the linear direction, rotational direction, and scaling direction of the images, as well as differences in color and shading (light and dark), within a range that does not cause practical problems.

[0017] In the following description of each embodiment, a "bright distance image corresponding to a dark distance image" or "another bright distance image corresponding to a bright distance image" refers to images that are substantially identical except for some clearly different areas. Here, "substantially identical" means that deviations in the linear, rotational, and scaling directions of the images, as well as differences in color and shading (light and dark), are allowed within practically acceptable limits. For example, an image is determined to match if its degree of identity or similarity is equal to or exceeds a certain threshold.

[0018] (Embodiment 1) An HMD according to a first embodiment of the present invention will be described.

[0019] Overview of First Embodiment In a bright place, the HMD of the first embodiment obtains a bright place camera image from a camera and a bright place distance image corresponding to the camera image and based on the output of a distance sensor, and saves and accumulates these images as a set. Meanwhile, in a dark place, it obtains a dark place distance image based on the output of the distance sensor, and searches for and identifies a bright place distance image corresponding to (substantially matching) the dark place distance image. Then, based on the bright place camera image included in the same set as the bright place distance image, it determines and displays a visual image to be viewed by the user. This allows the user to view a bright place image of the same location obtained previously in a dark place, improving the user's visibility.

[0020] <HMD hardware configuration> Fig. 1 is an external view of an HMD according to embodiment 1. As shown in Fig. 1, the HMD 1 according to this embodiment includes a camera 11, a distance measurement sensor 12, a right-eye projector 13, a left-eye projector 14, an image screen (image display surface) 15, nose pads 16, a controller 17, a microphone 18, a speaker 19, and frame housings 20a to 20c. A user wears the HMD 1 on their face using the frame housings 20a and 20b and the nose pads 16.

[0021] Right-eye projector 13, left-eye projector 14, and image screen 15 constitute a display device of HMD 1. Right-eye projector 13 and left-eye projector 14 will also be referred to as projectors 13, 14 hereinafter.

[0022] In this embodiment, the image screen 15 is a semi-transmissive screen that transmits light from in front of the user. However, as will be described later, the image screen 15 may also be a non-transmissive display that does not transmit light.

[0023] With a semi-transparent screen, the user can see the situation ahead through the semi-transparent screen, but with a non-transparent display, the user can check the situation by displaying a camera image of the front view on a display inside the non-transparent HMD.

[0024] The camera 11 is mounted so as to capture an image of the real space in front of the user. The camera 11 is a so-called digital camera that captures an image of its own field of view to obtain image data corresponding to the field of view. In this embodiment, the image represented by this image data is referred to as a camera image. The camera 11 is controlled by the controller 17 and repeatedly captures images at a predetermined timing. This capturing may be performed manually or automatically. When capturing images automatically, for example, the image is captured at a constant frame rate, which may be adjusted appropriately depending on the processing speed and memory capacity of the controller 17. The frame rate may be, for example, about 1 to 30 fps (frames per second).

[0025] The ranging sensor 12 measures the distance from itself to a real object corresponding to each position in the field of view of the camera 11, and obtains distance data representing these distances. Note that when the user wears the HMD 1, the ranging sensor 12 is essentially equivalent to obtaining distance data representing the distance from the user to the real object corresponding to each of the above positions. The ranging sensor 12 is capable of measuring distances even in dark places, and is configured, for example, as with the LiDAR described above, by emitting infrared light and receiving reflected light from a real object.

[0026] Projectors 13 and 14 project CG images, camera images, etc. onto image screen 15 and superimpose them on a background that is visible through image screen 15. When projecting a CG image, projectors 13 and 14 project an image for the left eye and an image for the right eye, which are generated taking parallax into consideration, onto image screen 15. This makes it possible to display the CG image three-dimensionally as if it were located at a predetermined distance in real space.

[0027] Controller 17 takes in image data obtained by camera 11 and distance data obtained by distance measurement sensor 12, and supplies these data to an internal memory or processor. Controller 17 also has built-in sensors such as a GPS (Global Positioning System) sensor, an illuminance sensor, an acceleration sensor, a gyro sensor, and a direction (magnetic) sensor. Controller 17 also generates images to be projected by projectors 13 and 14, sounds to be output by speaker 19, and the like. Controller 17, camera 11, distance measurement sensor 12, microphone 18, and speaker 19 are arranged in frame housings 20a to 20c. Note that the locations of these components shown in FIG. 1 are merely an example and may not be exactly as shown.

[0028] Fig. 2 is a block diagram showing the hardware configuration of the HMD according to the first embodiment. Note that in Fig. 2, the same components as those shown in Fig. 1 are assigned the same numbers. As shown in Fig. 2, the controller 17 of the HMD 1 according to this embodiment internally includes an internal bus 30, a GPS sensor 31, an illuminance sensor 32, an acceleration sensor 33, a gyro sensor 34, a direction sensor 35, a processor 36, a memory 37, an image memory 38, a nonvolatile storage device 39, and a communication device 40. Furthermore, the respective blocks 31 to 40 are connected via the internal bus 30 and operate in cooperation with one another.

[0029] The processor 36 is configured, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit).

[0030] The memory 37 and the image memory 38 are configured by, for example, a RAM (Random Access Memory) which is a semiconductor storage device.

[0031] The nonvolatile storage device 39 is composed of one or more nonvolatile memory media. An example of a nonvolatile memory medium is a programmable ROM (Read Only Memory). The programmable ROM is composed of, for example, an EEPROM (Electronically Erasable and Programmable ROM) or a FROM (Flash ROM). The nonvolatile storage device 39 stores a basic operation program 41 and a dark place guide program 42 as processing programs. A saved data area 43 is also allocated to the nonvolatile storage device 39. The saved data area 43 stores data and image data required to execute the processing programs.

[0032] The communication device 40 includes a mobile communication device such as 4G (Generation) or 5G, a wireless LAN communication device, etc. The communication device 40 selects an appropriate process from among communication processes as necessary, and connects the HMD 1 to a network.

[0033] Image data to be sent to the projectors 13 and 14 is stored in an image memory 38 and then read out.

[0034] The processor 36 controls and manages basic operations and realizes the dark place guide function by expanding the basic operation program 41 and the dark place guide program 42 stored in the non-volatile storage device 39 into the memory 37 and executing them.

[0035] <HMD functional block configuration> Fig. 3 is a functional block diagram showing the functional configuration of the HMD according to embodiment 1. Each functional block shown in Fig. 3 is realized by the processor 36 expanding and executing a dark place guide program in the memory 37, and by cooperating with various sensors and devices included in the HMD 1.

[0036] As shown in FIG. 3, the HMD1 according to this embodiment includes, as functional blocks, a position information acquisition device 51, a time information acquisition device 52, an orientation information acquisition device 53, an attitude information acquisition device 54, a camera image acquisition device 55, a distance image generation device (generation device) 56, a bright place / dark place determination device (determination device) 57, a bright place image storage management device (storage device) 58, a bright place image storage device 59, a dark place distance image recognition device (recognition device) 60, a comparison image narrowing device 61, a bright place image search device 62, a visual image determination device 63, and a visual image display device (display device) 64.

[0037] The position information acquisition device 51 acquires coordinate data obtained by the GPS sensor 31 and generates position information indicating the user's location based on the data. The time information acquisition device 52 acquires clock data including the date and time from the electronic clock of the processor 36 and generates time information indicating the date and time when the image was acquired. The orientation information acquisition device 53 acquires orientation data obtained by the orientation sensor 35 and generates orientation information indicating the horizontal orientation, i.e., orientation, of the user's face based on the data. The posture information acquisition device 54 acquires data obtained by the acceleration sensor 33 and gyro sensor 34 and generates posture information indicating the vertical orientation of the user's face based on the data.

[0038] The camera image acquisition device 55 acquires image data obtained by the camera 11 and performs image processing such as noise removal, interpolation, and size adjustment on the data as necessary to generate a camera image. The camera image acquisition device 55 continuously and repeatedly acquires image data at time intervals to generate camera images in time series.

[0039] The distance image generating device 56 acquires distance data obtained by the distance measuring sensor 12 and generates a distance image based on that data. The distance image is an image having an area corresponding to the field of view of the camera image. The distance image is an image in which the distance from the distance measuring sensor 12 to a real object corresponding to each position in the field of view is reflected in the pixel value (color or shade) of the pixel corresponding to each position. If the bright / dark place determining device 57 (described later) determines that the location where the user is located is a bright place, the distance image generating device 56 generates a distance image corresponding to each generated camera image. If the location where the user is located is determined to be a dark place, the distance image generating device 56 generates a distance image regardless of whether a camera image is present.

[0040] The bright / dark place determination device 57 acquires illuminance data obtained by the illuminance sensor 32 and determines whether the location of the user is in a bright place or a dark place based on the data. The bright / dark place determination device 57 determines whether the location is a bright place or a dark place, for example, by determining a threshold value of the illuminance represented by the illuminance data.

[0041] The determination of whether the location is bright or dark may be made based on the brightness of the acquired camera image, or may be made using both the illuminance data and the brightness of the camera image.

[0042] Furthermore, the bright place / dark place determination may be performed by supplementing location information and time information. For example, if the illuminance data indicates that the illuminance is exactly halfway between a bright place and a dark place, the bright place / dark place determination device 57 refers to the location information and time information. Then, based on this information, it detects the user's location and date and time, and determines whether it is after the evening time preset for each location. If it is before evening, it is determined to be a bright place, and if it is after evening, it is determined to be a dark place.

[0043] The bright place image storage management device 58 stores camera images and distance images obtained when the bright place / dark place determination device 57 determines that the user's location is a bright place as an image set including bright place camera images and bright place distance images. When storing the image set, the bright place image storage management device 58 stores the location information, time information, orientation information, and posture information at the time the image set was obtained as metadata, associated with the image set. The image set and metadata are stored in the bright place image storage device 59, which will be described later. Here, it has been described that the image set is stored when it is determined to be a bright place, but a flag for distinguishing between bright and dark places may be added so that either the bright place image set or the dark place image set is stored. In this case, if nothing is captured in the dark place camera images, only the time information may be stored, and the camera images may not be stored.

[0044] The bright light image storage device 59 stores the image sets and metadata saved by the bright light image saving management device 58. That is, the bright light image storage device 59 sequentially saves and accumulates image sets for each location where the user was and for each orientation (direction) and posture of the user.

[0045] The dark place distance image recognition device 60 recognizes, as a dark place distance image, a distance image obtained when the bright place / dark place determination device 57 determines that the location where the user is located is a dark place.

[0046] When a dark distance image is recognized, the comparison image narrowing device 61 narrows down the bright distance images stored in the bright distance image storage device 59 to bright distance images to be compared with the recognized dark distance image. The narrowing down of bright distance images is performed using associated metadata. That is, based on the position information, time information, orientation information, and posture information at the time the recognized dark distance image was obtained, the comparison image narrowing device 61 searches for and narrows down the bright distance images whose location, date, orientation, and posture are similar within a predetermined range to the user's location, date, and posture. In this embodiment, the comparison image narrowing down is performed based on the position information, time information, orientation information, and posture information. However, the comparison image narrowing down may be performed based on only the position information, only the position information and orientation information, or only the position information, orientation information, and posture information. Alternatively, the comparison image narrowing down by the comparison image narrowing device 61 may be omitted.

[0047] Bright image search device 62 sequentially reads out bright distance images to be compared with the recognized dark distance images, and compares the read out bright distance images with the dark distance images. Based on the results of this comparison, it searches for and identifies bright distance images that correspond to the dark distance images, i.e., bright distance images that match within an acceptable range. The comparison of the dark distance images and the bright distance images is performed by performing translation, rotation (tilt), scaling, pixel value adjustment, etc. on at least one of the distance images.

[0048] To determine whether or not there is a match within an acceptable range, for example, a method can be used in which the degree of match between the recognized dark distance image and the read-out bright distance image is calculated, and if the degree of match is above a threshold, it is determined that there is a match, and if the degree of match is below the threshold, it is determined that there is no match.

[0049] As a specific example, for each position or region in the distance image, the pixel value levels between corresponding pixels or pixel groups, such as the difference in average pixel values, are obtained, and the magnitude of these differences is comprehensively evaluated to calculate the degree of match, which is then subjected to threshold judgment.The degree of match can be, for example, a value obtained by multiplying the variance or deviation value of the pixel value level difference at each position or region by a negative coefficient, or the reciprocal of the variance or deviation value.

[0050] When comparing distance images, if it is recognized that there is a difference only in a part of the entire image area, the images are considered to match within an acceptable range. In other words, the match determination is performed excluding the difference area, which will be described later.

[0051] Furthermore, the determination of whether or not the match is within an acceptable range may be performed using, for example, artificial intelligence, which may be trained on the match of images representing objects that may exist in real space.

[0052] The visible image determination device 63 determines the visible image to be viewed by the user based on the results of the comparison of distance images by the bright image search device 62. When a bright distance image that substantially matches the dark distance image is identified, the visible image determination device 63 determines the visible image based on a bright camera image included in the same image set as the bright distance image. When several matching distance images are identified, the user may be allowed to select the most recent one or the brightest one. On the other hand, when no bright distance image that substantially matches the dark distance image is identified, the visible image is determined based on the dark distance image. The visible image may be the bright camera image itself or the dark distance image itself, or in some cases, it may be obtained by performing image processing such as processing or compositing on the bright camera image or the dark distance image.

[0053] The visual image display device 64 projects the determined visual image onto the image screen 15 using the projectors 13 and 14 so that the user can visually recognize the image. Here, if the user is moving, it may be difficult to detect the position and display the bright camera image of the same position as a moving image without delay. In such a case, it may be possible to display a still image according to the user's moving position.

[0054] <Dark Place Guide Program Processing Flow> The dark place guide program processing will now be described.

[0055] FIG. 4 is a process flow diagram of a dark place guide program by an HMD according to the first embodiment.

[0056] In step S1, a process of acquiring camera images is performed. Specifically, the camera image acquisition device 55 acquires image data obtained by the camera 11. The camera images may be acquired in synchronization with the timing of camera image capture, or images may be acquired by capturing images at any timing while continuously capturing images. When continuously capturing images, the images are captured at a frame rate of, for example, about 10 to 60 fps (frames per second).

[0057] In step S2, a process of acquiring distance data is performed. Specifically, the distance image generating device 56 acquires distance data representing the distance from the distance measuring sensor to a physical object included in the field of view of the camera 11.

[0058] In step S3, a distance image is generated. Specifically, distance image generating device 56 generates a distance image corresponding to the field of view of camera 11 based on the acquired distance data.

[0059] In step S4, a process for determining whether the location is bright or dark is performed. Specifically, the bright / dark location determination device 57 acquires illuminance data from the illuminance sensor 32 and determines whether the location of the user is bright or dark based on the illuminance data. As described above, this determination may be made based on the brightness of the acquired camera image, or may be made with the aid of location information, time information, and the like. If this determination determines that the location is bright (S4, Yes), the process proceeds to step S5. On the other hand, if the location is determined to be dark (S4, No), the process proceeds to step S7.

[0060] In step S5, a process of acquiring metadata is performed. Specifically, the bright place image storage management device 58 acquires time information, position information, orientation information, and posture information as metadata to be associated with the image set.

[0061] In step S6, the image set is saved, updated, and managed. Specifically, the bright-light image saving and managing device 58 treats the acquired camera images and distance images as bright-light camera images and bright-light distance images. The image set including these images is then associated with the acquired metadata and saved in the bright-light image storage device 59. Then, the process proceeds to step S17.

[0062] When saving an image set, if a bright field distance image corresponding to the bright field distance image to be saved has already been saved, the image set may be overwritten. Alternatively, for images acquired at a similar date and time, only the recording date and time may be added, and the image may not be overwritten.

[0063] Furthermore, image sets that were acquired a certain time (first time) or more ago, so-called old image sets, may be erased (deleted) from the bright light image storage device 59. The certain time period may be, for example, about one month to one year.

[0064] Details of the image set storage and management process in step S6 will be described later.

[0065] In step S7, a process of recognizing a dark distance image is performed. Specifically, when dark distance image recognition device 60 determines that the location of the user is a dark place, it recognizes the obtained distance image as a dark distance image.

[0066] In step S8, a process is performed to narrow down the bright range images to be compared. Specifically, comparison image narrowing device 61 narrows down the bright range images to be compared with the recognized dark range image from among past bright range images stored in bright range image storage device 59. The narrowing down of the bright range images is performed using associated metadata. That is, based on the position information when the recognized dark range image was obtained, bright range images whose location of the user matches or is similar within a predetermined range are searched for and narrowed down. Note that, in addition to position information, orientation information or posture information may also be used to search for and narrow down bright range images whose orientation of the user, the user's posture (tilt of the face in the vertical direction), or both match or are similar within a predetermined range.

[0067] In step S9, a process of reading out a bright spot distance image is performed. Specifically, bright spot image search device 62 reads out one bright spot distance image from the narrowed down bright spot distance images.

[0068] In step S10, the dark distance image and the bright distance image are compared. Specifically, bright image search device 62 compares the recognized dark distance image with the read bright distance image and calculates an evaluation value that reflects the degree of match between the distance images. Here, if the bright distance image is not saved and only the camera image is available, the features of the dark distance image and the bright camera image may be extracted and compared.

[0069] In step S11, a process is performed to determine whether the compared distance images correspond to each other, i.e., whether they substantially match. Specifically, bright place image search device 62 determines whether the recognized dark place distance image and the read bright place distance image substantially match by threshold-based determination of the calculated evaluation value. If this determination determines that they match (S11, Yes), the process proceeds to step S12. On the other hand, if it determines that they do not match (S11, No), the process proceeds to step S13.

[0070] In step S12, the process of using the bright camera image as the basis of the visual image is performed. Specifically, the visual image determination device 63 sets the bright camera image included in the same image set as the bright distance image determined to match in step S11 as the basis of the visual image to be viewed by the user. Then, the process proceeds to step S15.

[0071] In step S13, a process is performed to determine whether all comparisons of the narrowed-down bright range images have been completed. Specifically, bright range image search device 62 performs comparisons of all narrowed-down bright range images and determines whether there are any bright range images remaining to be compared next. If it is determined that all comparisons have been completed (S13, Yes), the process proceeds to step S14. On the other hand, if it is determined that all comparisons have not been completed (S13, No), the process returns to step S9, and the next bright range image to be compared is read.

[0072] In step S14, the dark distance image is used as the basis for the visual image. Specifically, the visual image determination device 63 sets the recognized dark distance image as the basis for the visual image to be viewed by the user.

[0073] In step S15, a process of generating and editing a visual image is performed. Specifically, the visual image determination device 63 generates and determines a visual image based on a bright camera image or a dark distance image set as a base. At this time, editing is performed, such as pasting a CG image or an AR object image onto the image set as a base, or emphasizing a part of the image set as a base, as necessary.

[0074] In step S16, a process of displaying the visual image is performed. Specifically, the visual image display device 64 projects and displays the determined visual image on the image screen 15 so that the user can visually recognize the visual image.

[0075] In step S17, a process is performed to determine whether or not to continue the dark place guide program. For example, when moving from a dark place to a bright place, when moving to an environment where execution of the dark place guide program is deemed unnecessary, when a command to stop or end the dark place guide program is input by the user, or when an internal processing error or communication error occurs, it is determined not to continue (S17, No), and the dark place guide program is terminated. On the other hand, if there is no particular reason to end it, it is determined to continue (S17, Yes), and the process returns to step S1, and processing by the dark place guide program continues.

[0076] <Image set storage and management processing> Here, the flow of the image set storage and management process in step S6 above will be described in detail.

[0077] FIG. 5 is a process flow diagram of image set storage and management by the HMD according to the first embodiment.

[0078] In step S61, a process of narrowing down the past image sets to be compared is performed. Specifically, bright light image storage management device 58 narrows down the past bright light distance images stored in bright light image storage device 59 to those bright light distance images to be compared that match, within a predetermined range, the image capture location, user direction (orientation), and posture of the currently acquired image set, based on the currently acquired metadata. Note that this narrowing down may be performed based only on the image capture location, without considering the orientation and posture at the time of image capture.

[0079] In step S62, a process of reading one past bright field distance image is performed. Specifically, bright field image storage and management device 58 reads one of the past bright field distance images narrowed down in step S61. At this time, the images may be read in order, starting with the image closest to the current location and orientation, as recognized by the location information included in the metadata. This may allow necessary processes, such as new image set saving, overwriting, adding recording date and time, and deleting old image sets, to be completed relatively quickly. As a result, even if the image set saving and management process is interrupted for some reason, the risk of adverse effects can be reduced.

[0080] In step S63, the current bright field distance image is compared with the previous bright field distance image. Specifically, bright field image storage management device 58 compares the acquired current bright field distance image with the previous bright field distance image read in step S62.

[0081] In step S64, a process is performed to determine whether the compared bright field distance images are identical. Specifically, bright field image storage management device 58 determines whether the acquired current bright field distance image and the previous bright field distance image read in step S62 match within an acceptable range, i.e., whether they are substantially identical. If the distance images are completely different or deviate by more than a predetermined level, they are determined to be not identical. If this determination determines that they are identical (S64, Yes), the process proceeds to step S65. On the other hand, if this determination determines that they are not identical (S64, No), the process returns to step S62, where the next bright field distance image to be compared is read and processing continues.

[0082] In step S65, a process is performed to recognize the recording date and time of the past bright field distance image. Specifically, bright field image storage management device 58 reads and recognizes the recording date and time included in the metadata of the past bright field distance image read in step S62.

[0083] In step S66, a process is performed to determine whether the read previous bright light distance image is old. Specifically, bright light image storage management device 58 determines whether the recording date and time recognized in step S65 is older than a certain period of time, i.e., whether it is old. If it is determined to be old (S66, Yes), the process proceeds to step S67. On the other hand, if it is determined not to be old (S66, No), the process proceeds to step S68.

[0084] In step S67, a process of deleting the past image set is performed. Specifically, the bright light image storage management device 58 deletes the image set including the past bright light distance image read in step S62. Then, the process proceeds to step S70.

[0085] In step S68, a process is performed to determine whether the recognized recording date and time is approximately the same as the current date and time. Specifically, the bright light image storage management device 58 determines whether the time difference between the recording date and time recognized in step S65 and the current date and time is within a relatively short period of time set in advance, i.e., whether the dates and times are substantially approximately the same. The predetermined period of time can be, for example, about one hour to one day, but is not limited to this. If it is determined that the dates and times are approximately the same (S68, Yes), the process proceeds to step S69. On the other hand, if it is determined that the dates and times are not the same (S68, No), the process proceeds to step S70.

[0086] In step S69, a process of overwriting or adding the recording date and time is performed. Specifically, the bright light image storage management device 58 overwrites and saves a new image set consisting of the acquired bright light distance images and bright light camera images over the image set including the past bright light distance images read in step S62. Alternatively, for the image set including the past bright light distance images read in step S62, only the recording date and time, which is metadata, is updated to the current date and time. In this case, since there is no substantial difference in the images themselves, there is no problem with updating only the recording date and time, which can simplify the process.

[0087] However, if the past image set containing the retrieved past bright field distance image has advantages or is easier to use, it is preferable to update the recording date and time without overwriting. For example, if the bright field distance image or bright field camera image contained in the past image set has a relatively higher resolution than the current image, or if the aspect ratio of the image is well balanced, updating the recording date and time is selected. After the processing of step S69 is completed, proceed to step S70.

[0088] In step S70, a process is performed to determine whether there are any previous bright field distance images to be compared. Specifically, bright field image storage management device 58 determines whether all comparisons have been completed for the previous bright field distance images narrowed down to be compared in step S61, and whether there are any previous bright field distance images remaining to be compared. If it is determined that there are no previous bright field distance images remaining (S70, Yes), the process proceeds to step S71. On the other hand, if it is determined that there are previous bright field distance images remaining (S70, No), the process returns to step S62, where the next previous bright field distance image is read and processing continues.

[0089] In step S71, a process is performed to determine whether the image set has been overwritten or saved. Specifically, the bright light image saving management device 58 determines whether the process of step S69, i.e., whether the currently acquired image set has been overwritten or the recording date and time has been added, has already been performed. If it is determined that the image set has been overwritten or saved (Yes in S71), the image set saving and management process ends. On the other hand, if it is determined that the image set has not been overwritten or saved (No in S71), the process proceeds to step S72.

[0090] In step S72, a process for newly saving the image set is performed. Specifically, the bright photopic image saving management device 58 associates the acquired image set with the currently obtained metadata and newly saves it in the bright photopic image storage device 59. Then, the image set saving and management process ends.

[0091] According to this image set storage and management process, past image sets that are substantially identical to the acquired image set are searched for and identified. Then, of the identified image sets, old ones are deleted, and those that are not old but whose recording dates and times differ by more than a certain amount of time are retained. For those with substantially the same recording dates and times, the old image sets are overwritten or their recording dates and times are appended, i.e., updated. Furthermore, if the recording dates and times or time periods differ from those of past image sets, or if at least some of the images differ by more than a certain level, the acquired image set is newly saved. This allows necessary image sets to be retained for only the required period of time, thereby preventing the inexhaustible growth of image set storage capacity and maintaining appropriate storage capacity.

[0092] <Examples of camera images, range images, and visual images> Fig. 6 is a diagram showing an example of an image obtained by the HMD according to the first embodiment. Fig. 6 shows examples of a camera image, a distance image, and a visual image obtained in a bright place and a dark place. In Fig. 6, the top row shows an image recorded in a bright place in the past, and the middle row shows an image currently taken in a dark place. The left side is a camera image, and the right side is a distance image. Generally, distance images are displayed in different colors depending on the distance, but here they are shown in black and white (grayscale) for a pseudo effect.

[0093] In Figure 6, 80A is a clearly captured bright place camera image, and 81A is a bright place distance image corresponding to bright place camera image 80A. 82B is a dark place camera image currently acquired, which is unclear due to the dark surroundings. On the other hand, 83B is a dark place distance image corresponding to dark place camera image 82B, acquired almost simultaneously with the dark place camera image, but obtained with clarity comparable to that of bright place distance image 81A.

[0094] The user views the scenery in front of them through the semi-transparent image screen of the HMD, but in dark places it is difficult to confirm the situation ahead, just as with the dark place camera image 82B. Therefore, as shown in the lower part of Figure 6, the camera image taken in a bright place with a matching distance image is determined as the visual image 84B to be visually recognized by the user. This visual image is then displayed on the display surface of the HMD. This allows the user to confirm the situation at the site even in dark places.

[0095] In addition, for example, the outline of an object can be extracted from the dark distance image 83B and compared with the outline of the object extracted from the bright distance image 81A or the bright camera image 80A, thereby confirming that the images are of the field of view from the same position. Even in such cases, the user can check the situation at the site in a dark place by displaying the camera image in a bright place on the HMD.

[0096] According to the first embodiment, in a dark place where it is difficult to see ahead, it is possible to support the user in seeing ahead by displaying a previously saved camera image of the same field of view taken in a bright place. For example, in a dark work site where visibility is poor, by providing a clear camera image taken in a bright place instead of a camera image taken in a dark place, the user can confirm the presence of objects such as obstacles and work more safely.

[0097] Furthermore, in the first embodiment, when saving an image set, if a corresponding image, i.e., a substantially matching bright field distance image, has already been saved, the following processing is performed. That is, if the acquisition date and time of the already saved corresponding image is within a certain period of time and is relatively newer than the acquisition date and time of the image to be saved, the corresponding image is not overwritten, and only the date and time of the corresponding image is added to the current date and time. Also, if the acquisition date and time of the corresponding image is older than the acquisition date and time of the image to be saved and is more than a certain period of time earlier, the corresponding image is deleted. This simplifies processing, thereby enabling faster processing, reduced energy consumption, and more efficient use of storage area.

[0098] Furthermore, in the first embodiment, the current location of the user is identified using a GPS or the like, and a search is performed from bright field distance images narrowed down to that location, thereby reducing the time required for image search.

[0099] In addition, in the first embodiment, a semi-transparent screen that transmits light from in front of the user is used as the image screen, which is the image display surface. This allows the user to directly view real objects in front of the user while wearing the HMD, providing a natural feeling as if wearing glasses.

[0100] In the first embodiment, steps S10 and S11 compare the dark distance image and the bright distance image to determine whether they match. However, it is possible that the bright distance image is not recorded for some reason. In such a case, the dark distance image and the bright camera image may be compared to determine whether they match. For example, the contour of the object may be extracted from the dark distance image, the contour of the object may be extracted from the bright camera image, and the two contours may be compared to determine whether they match. This allows the user to be provided with an image with good visibility even when a bright camera image is saved but no bright distance image is available.

[0101] In addition, in embodiment 1, if the user's location is limited and there are few bright distance images to compare, or if the processing speed is high, the process of narrowing down the bright distance images to be compared may be omitted. This simplifies the algorithm of the dark place guide program and reduces development costs.

[0102] Furthermore, in the first embodiment, if the user is indoors and GPS location information is unavailable, and the metadata of the image set includes location information and orientation data from a network access point or UWB (Ultra Wide Band), these may be used. That is, a search may be prioritized for bright range images whose metadata contains location information and orientation information that are substantially the same as the location information obtained from an access point or orientation information detected by an orientation sensor at the user's location. This allows the search time for bright range images to be shortened even when GPS location information is unavailable.

[0103] Furthermore, in the first embodiment, if the user's initial position can be determined using GPS or a wireless communication access point, a method such as Pedestrian Dead Reckoning (PDR) can then be used to identify the user's location after moving, and a bright place distance image can be searched for according to the line of sight from that location. This allows the dark place guide program to continue running even when the user moves from outdoors to indoors.

[0104] In addition, in the first embodiment, the identified bright camera image is used as the visual image as is. However, the identified bright camera image may be used as the visual image by changing the brightness or color or by processing the image. For example, the color of the identified bright camera image may be converted to black and white or sepia, or may be arranged in vivid colors such as those used in CG or animation. A virtual reality space corresponding to the user's location, the direction the user is facing, and the user's posture may be displayed, and an artificial CG image based on the identified bright camera image may be superimposed on the displayed image.

[0105] (Embodiment 2) A second embodiment of the present invention will now be described.

[0106] Overview of Second Embodiment The second embodiment is an HMD system in which an HMD and a server are connected via a network. In this embodiment, the HMD and the server cooperate to provide the user with an image storage service including a dark place guide function.

[0107] <HMD system configuration> The configuration of the HMD system according to the second embodiment will be described.

[0108] Fig. 7A is a diagram showing a first example of the configuration of an HMD system according to embodiment 2. Fig. 7A shows the configuration of an HMD system 100 in which a single HMD and a server are connected via a network.

[0109] 7A, the HMD system 100 according to this embodiment includes an HMD 1, a user 70, wireless communication signals 71a and 71b using electromagnetic waves or the like, an access point 72, a network 73, and an image storage service server (server) 74. The network 73 is, for example, a wide area communication network, and specifically, the Internet, Ethernet, an industrial communication network, or the like can be considered.

[0110] <Configuration of the image storage service server> Fig. 8 is a block diagram showing the hardware configuration of an image storage service server. As shown in Fig. 8, the image storage service server 74 includes an internal bus 740, a network I / F (Interface) 741, a processor 742, a memory 743, and a storage 744. The storage 744 stores a basic operation program 745 and is provided with an image data storage area 746. The storage 744 is configured, for example, by a hard disk or a semiconductor memory.

[0111] An image set including a distance image from the HMD 1 is stored in a storage 744 via a network I / F 741, a processor 742, etc. A basic operation program 745 is executed using the processor 742 and memory 743, and management processing for multiple users is performed.

[0112] <How the image storage service works> A user 70 wears an HMD 1 on his / her head and looks ahead. The HMD 1 is connected to a network 73 via communication signals 71a and 71b and an access point 72, and the network 73 is connected to an image storage service server 74.

[0113] The HMD 1 executes the dark place guide program, but the image sets and metadata based on the bright place distance images and bright place camera images are not stored in the HMD 1 but in the image data storage area 746 of the image storage service server 74, and are read out as needed. Alternatively, the necessary image sets corresponding to the location are downloaded in advance from the image storage service server 74 to the HMD 1 and used. This is the image storage service.

[0114] That is, this HMD system includes an image storage service server 74 connected to a network 73 , and an HMD 1 connected to the network 73 and communicating with the image storage service server 74 .

[0115] The HMD1 also includes a camera that captures an image in front of the user 70 and obtains a camera image, a distance measurement sensor that obtains data representing the distance from the user 70 to a real object corresponding to each position in the camera's field of view, an illuminance sensor that obtains data representing the brightness of the location where the user 70 is located, and a display device that displays a visual image for the user 70 to view.

[0116] Then, the image storage service server 74 and the HMD 1 cooperate to execute various processes as follows.

[0117] [Process 1-1] A generation process that generates a distance image based on data obtained by the distance sensor, where each pixel corresponds to the camera's field of view and represents the distance to each of the above positions.

[0118] [Process 1-2] A process for determining whether the user 70 is in a bright or dark place based on data obtained from the illuminance sensor.

[0119] [Process 1-3] If the determination process determines that the location is in a bright place, the camera image obtained by the camera and the distance image obtained by the generation process are saved as an image set consisting of a bright place camera image and a bright place distance image.

[0120] [Process 1-4] A recognition process for recognizing the distance image obtained by the generation process as a dark distance image when the determination process determines that the location is dark.

[0121] [Process 1-5] A search process for identifying a bright distance image corresponding to the recognized dark distance image by comparing the bright distance image stored in the storage process with the dark distance image recognized in the recognition process.

[0122] [Process 1-6] A determination process for determining a visual image based on a bright camera image included in the same image set as the bright range image identified by the identification process.

[0123] [Process 1-7] Display process for displaying the visual image determined by the determination process so that the user can visually recognize it.

[0124] By performing such processing, image sets and metadata, which tend to become large in size, can be stored in the image storage service server 74 rather than inside the HMD 1, thereby reducing the storage capacity of the HMD 1.

[0125] Furthermore, it is possible to have the image storage service server 74 perform processing that imposes a heavy load, rather than the HMD 1, and send only the results of that processing to the HMD 1, thereby lowering the processing specifications of the HMD 1 and achieving high-speed processing.

[0126] Fig. 7B is a diagram showing a second example of the configuration of the HMD system according to the second embodiment. Fig. 7B shows the configuration of an HMD system 101 in which multiple HMDs and a server are simultaneously connected via a network. The example shown in Fig. 7B is an example in which there are users wearing HMDs at multiple locations. There may be multiple users at each of the multiple locations. The multiple users share the image storage service provided by the image storage service server 74.

[0127] In FIG. 7B, 1a and 1b are HMDs, 70a and 70b are users, 71c, 71d, and 71e are communication signals, 72 is an access point, 73 is a network, and 74 is an image storage service server. User 70a is at Site abc (75a) and uses the image storage service, while user 70b is at Site aaa (75b) and uses the image storage service. Each user stores bright camera images and bright distance images acquired in a bright location as an image set according to their respective positions and orientations, along with metadata, in the image storage service server 74. Furthermore, each user can access the stored image sets from various locations to retrieve the latest required image set.

[0128] That is, this system includes an image storage service server (server) 74 connected to a network 73, and a plurality of HMDs (mobile image display devices) 1a and 1b connected to the network 73 and communicating with the image storage service server 74.

[0129] Each of the multiple HMDs 1a, 1b is equipped with a camera that captures an image of what is in front of the users 70a, 70b to obtain a camera image, a distance measurement sensor that obtains data representing the distance from the camera to a real object corresponding to each position in the camera's field of view, an illuminance sensor that obtains data representing the brightness of the location where the users 70a, 70b are located, a display device that displays a visual image for the users 70a, 70b to view, and a position information acquisition device that acquires position information of the HMDs 1a, 1b.

[0130] Then, in each of the plurality of HMDs 1a and 1b, the image storage service server 74 and the HMDs cooperate to execute various processes as described below.

[0131] [Process 2-1] A first generation process for generating a distance image based on data obtained by a distance sensor, where each pixel corresponds to the camera's field of view and represents the distance to each of the above positions.

[0132] [Process 2-2] First determination process to determine whether the user is in a bright or dark place based on data obtained from the illuminance sensor

[0133] [Process 2-3] If the first determination process determines that the location is bright, the camera image obtained by the camera and the distance image obtained by the first generation process are combined into an image set of a bright camera image and a bright distance image, and the image set is associated with the location information obtained by the location information acquisition device and stored in the image storage service server 74.

[0134] Furthermore, the image storage service server 74 cooperates with one of the plurality of HMDs 1a and 1b to execute various processes as described below.

[0135] [Process 2-4] A second generation process for generating a distance image that corresponds to the field of view and indicates the distance to each of the above positions based on data obtained by the distance sensor.

[0136] [Process 2-5] A second determination process that determines whether the user is in a bright or dark place based on data obtained from the illuminance sensor.

[0137] [Process 2-6] A recognition process for recognizing the distance image obtained by the second generation process as a dark place distance image when the second determination process determines that the place is dark.

[0138] [Process 2-7] Identification process for identifying bright field distance images stored in the image storage service server 74, in which the position indicated by the associated position information corresponds to the position indicated by the position information acquired by the position information acquisition device of the one HMD.

[0139] [Process 2-8] A search process for comparing the bright range image identified by the identification process with the dark range image recognized by the recognition process to identify a bright range image whose degree of match with the recognized dark range image is equal to or greater than a threshold.

[0140] [Process 2-9] A determination process for reading out from the image storage service server 74 a bright camera image included in the same image set as the bright camera distance image identified by the search process, and determining a visual image to be displayed by the one HMD based on the read out bright camera image.

[0141] [Process 2-10] Display process for displaying the determined visual recognition image so that the user can visually recognize it.

[0142] By executing such processing, image sets are successively accumulated by a plurality of users, and the accumulated image sets are shared.

[0143] In this embodiment, the HMD is connected to the network wirelessly, but may be connected via a wire.

[0144] <Example of image set data structure> Here, an example of the data structure of an image set is shown.

[0145] Fig. 9 is a diagram showing an example of the data structure of an image set. As shown in Fig. 9, an image set including camera images and range images and their metadata are associated with each other to form a data set T10.

[0146] The data set T10 includes a service ID T11 for an image storage service, and location information T12a and T12b (including site information (location information) if it can be identified). The location information is GPS coordinates obtained from a GPS sensor, and the site information is the name of the site, building, etc. identified from the GPS coordinates. In this way, the data in the image set is classified by location or location information.

[0147] The data set T10 includes the dates and times T13a and T13b when the image set was acquired. The data set T10 may also include orientation data indicating the direction the user is facing. The data set T10 further includes camera image data T14a and T14b and range image data T15a and T15b. The camera image and range image are managed as a pair of data.

[0148] In the example shown in FIG. 9, the site (place) is shown as an indoor location in a building, but outdoor location information detected by GPS or the like may also be used.

[0149] Furthermore, by registering image data that does not include distance images, such as camera images taken with a smartphone, it is possible to collect and store a wide range of camera images. In this case, the distance images may be compared with stored camera images to estimate the degree of location match. For example, the comparison may be performed using the contour components of the camera image and the distance image.

[0150] According to the second embodiment, an HMD system having the same effect as the HMD of the first embodiment can be realized, and the implementation cost of the HMD can be reduced by storing image data on a server. Furthermore, images stored by multiple users can be shared and used by multiple users, which creates a synergistic effect and expands the range of places where the system can be used or the range of users.

[0151] (Embodiment 3) A third embodiment of the present invention will now be described.

[0152] Overview of the Third Embodiment The third embodiment is an HMD that detects a difference area between a dark distance image and a corresponding bright distance image, and determines a visual image based on the detection result. That is, when there is a difference area between the current dark distance image and the corresponding bright distance image, the HMD according to this embodiment determines which distance image is causing the difference, performs processing based on the result, and determines a more appropriate visual image.

[0153] The hardware configuration of the HMD according to the third embodiment is the same as that of the HMD according to the first embodiment, and therefore a description thereof will be omitted.

[0154] <HMD functional configuration> FIG. 10 is a functional block diagram showing the functional configuration of the HMD according to the third embodiment.

[0155] As shown in Figure 10, the HMD 1c of embodiment 3 has a configuration based on the functional configuration of the HMD 1 of embodiment 1, and further includes a difference area detection device (detection device) 65, a difference factor discrimination device (discrimination device) 66, and a bright field distance image selection device (selection device) 67.

[0156] The difference region detection device 65 detects difference regions between the current dark distance image and the identified previous bright distance image.

[0157] When a difference area is detected between the current dark distance image and a specified past bright distance image, the difference factor discrimination device 66 compares and analyzes the two distance images to determine whether the cause of the difference lies on the dark distance image side or the bright distance image side.

[0158] The bright range image selection device 67 determines whether a bright range image corresponding to the current dark range image and acquired further back in time than the previously identified bright range image is stored, and selects and reads out such a bright range image as the further previous bright range image.

[0159] The visual image determination device 63 edits and determines the visual image depending on whether there is a difference between the dark distance image and the bright distance image, the cause of the difference if there is one, and whether there are any further past bright distance images.

[0160] <Flow of visual recognition image determination process> The processing flow of the dark place guide program according to the third embodiment is basically the same as that of the first embodiment, but differs only in the process of determining the visual image. Therefore, only the process of determining the visual image will be described here, and the description of the other flows will be omitted.

[0161] 11A and 11B are flowcharts of the visual recognition image determination process according to embodiment 3. This flowchart corresponds to step S12 in the processing flow according to embodiment 1 shown in FIG.

[0162] In step S121, the bright place distance image and the dark place distance image are compared. Specifically, different area detection device 65 compares the bright place distance image and the dark place distance image.

[0163] In step S122, a process is performed to determine whether there is a difference between the two compared distance images based on the comparison result in step S121. Specifically, different area detection device 65 determines whether there is a difference between the previous bright distance image and the current dark distance image based on the comparison result. If it is determined that there is no difference, that is, that the two distance images are substantially identical across the entire image area (S122, No), the process proceeds to step S123. On the other hand, if it is determined that there is a difference (S122, Yes), the process proceeds to step S124.

[0164] In step S123, a process is performed in which the brightly lit camera image is determined as the visually recognized image. Specifically, the visually recognized image determination device 63 determines this brightly lit camera image as the visually recognized image. This corresponds to the example of determining the visually recognized image described with reference to FIG. 6. This completes the visually recognized image determination process.

[0165] In step S124, the characteristics of the difference area are analyzed and the cause of its occurrence is recognized. Specifically, the difference cause determination device 66 analyzes the characteristics of the difference area in each of the distance images. Then, based on the analysis results, it is determined whether the cause of the difference area is in the dark distance image or the bright distance image.

[0166] For example, when discontinuous distance data appears in a region of the dark distance image, it can be determined that a sudden displacement has occurred in the dark distance image. Conversely, when discontinuous distance data appears in a region of the bright distance image, it can be determined that a sudden displacement has occurred in the bright distance image.

[0167] Furthermore, for example, if the object included in the difference area is a person, a movable object, an artificial object, or the like, it can be considered that the difference area may have arisen due to differences caused by the movement of such an object.

[0168] Subsequent processing will differ depending on whether the cause of the difference is in the present or the past, and what object is included in the difference area.

[0169] In step S125, a process is performed to determine whether there is a cause of difference in the dark distance image. Specifically, difference cause determination device 66 determines whether the cause of difference is in the dark distance image or the bright distance image based on the analysis results. If it is determined that there is a cause of difference in the dark distance image (S125, Yes), the process proceeds to step S126. On the other hand, if it is determined that there is a cause of difference in the bright distance image (S125, No), the process proceeds to step S127.

[0170] In step S126, a process is performed to obtain a visual image by inserting a mask image or an AR object into the difference region of the bright camera image. Specifically, the visual image determination device 63 obtains a visual image by performing an enhancement process on the region corresponding to the difference region in the identified bright camera image. For example, a mask image or an AR object is inserted into the region corresponding to the difference region of the bright camera image. If an object included in the difference region can be recognized, an AR object corresponding to the object is inserted. If an object included in the difference region cannot be recognized or if such an AR object is not prepared, an enhancement process, such as superimposing a mask image, is performed on the region corresponding to the difference region of the bright camera image to obtain a visual image. This completes the visual image determination process.

[0171] An example of determining the visual recognition image in step S126 will now be described with reference to the drawings.

[0172] Fig. 12 is a diagram illustrating a first example of a process for determining a visual image by an HMD according to embodiment 3. Fig. 12 illustrates an example in which a visual image is determined by superimposing a mask image or an AR object on a brightly lit camera image. This example is a case in which the difference between distance images is due to the current dark distance image.

[0173] The current dark distance image 83E contains a person as a difference region 87. The corresponding bright distance image 81B does not contain a person. Using the entire bright camera image 80B as the visual image would be inappropriate, as it could lead to a collision with a person when the user moves around in real space. Therefore, while using the bright camera image 80B as the basis, an emphasized mask 88A, for example, is added to the corresponding region 87C of the difference region. This embodiment alerts the user.

[0174] Alternatively, if the characteristics of difference region 87 are analyzed and it is recognized that difference region 87 is a person, AR object 88B of the person is added to corresponding region 87C of that difference region based on bright camera image 80B. According to this embodiment, it is possible to convey details of the differences to the user.

[0175] In step S127, a process is performed to determine whether there is a bright distance image that corresponds to the current dark distance image and was acquired further back in time. Specifically, bright distance image selection device 67 determines whether there is a saved bright distance image that corresponds to the current dark distance image and was acquired further back in time than the previously identified bright distance image. If this determination determines that there is no such bright distance image (S127, No), the process proceeds to step S128. On the other hand, if this determination determines that there is such a bright distance image that went further back in time (S127, Yes), the process proceeds to step S129.

[0176] In step S128, a process is performed to determine a visible image by inserting an image portion of the difference region of the dark distance image into the difference region of the bright camera image. Specifically, the visible image determination device 63 acquires a past bright camera image, and determines an image obtained by inserting or pasting an image portion of the difference region of the dark distance image into the difference region of the bright camera image as the visible image.

[0177] An example of determining the visual recognition image in step S128 will now be described with reference to the drawings.

[0178] FIG. 13A is a diagram illustrating a second example of a process for determining a visual image using an HMD according to the third embodiment. FIG. 13A illustrates an example in which a visual image is obtained by pasting a portion of a dark distance image onto a bright camera image. In FIG. 13A, 81C is a saved bright distance image, 80C is a saved bright camera image, 83C is a dark distance image, and 84C is a visual image to be viewed by the user. There are some differences between the distance images 81C and 83C. Region RA in dark distance image 83C is a difference region, and a person is present in the region corresponding to the difference region in bright distance image 81C. In this case, if bright camera image 80C is applied to the entire area as the visual image, the user may be misled into thinking that a person who is not currently present is present.

[0179] Therefore, the image portion of the difference area RA in the dark distance image 83C is superimposed on the corresponding area in the bright camera image 80C to obtain a visually recognized image 84C.

[0180] In step S129, a process of reading out a bright distance image going back further in time is performed. Specifically, bright distance image selection device 67 reads out a bright distance image going back further in time that corresponds to the current dark distance image.

[0181] In step S130, the dark distance image is compared with the previous bright distance image and analyzed. Specifically, the visual image determination device 63 compares the current dark distance image with the previously read bright distance image. Then, it performs the analysis necessary to determine whether there is a difference between these two distance images.

[0182] In step S131, it is determined whether there is a difference between the dark distance image and the previous bright distance image. Specifically, based on the comparison and analysis results in step S130, visual image determination device 63 determines whether there is a difference between the current dark distance image and the previous bright distance image that was read out. If this determination determines that there is a difference (S131, No), the process proceeds to step S128. On the other hand, if this determination determines that there is no difference (S131, Yes), the process proceeds to step S132.

[0183] In step S132, a process of reading out further previous brightly-lit camera images is performed. Specifically, the visible image determination device 63 reads out further previous brightly-lit camera images included in the same image set as the further previous brightly-lit distance image from the brightly-lit image storage device 59.

[0184] In step S133, a process is performed to determine a visual image by inserting an image portion of a difference region of a further previous bright camera image into the difference region of the bright camera image. Specifically, the visual image determination device 63 determines an image obtained by inserting or pasting an image portion of a difference region of the further previous bright camera image into the difference region of the identified past bright camera image as the visual image.

[0185] Here, an example of determining the visual recognition image in step S133 will be described with reference to the drawings.

[0186] FIG. 13B is a diagram illustrating a third example of a process for determining a visual image using an HMD according to the third embodiment. FIG. 13B illustrates an example in which a visual image is determined by pasting a portion of an even earlier bright-light camera image onto a previous bright-light camera image. In FIG. 13B, the situation is the same as the example shown in FIG. 13A, but the visual image is determined by referring to an even earlier bright-light camera image 80D, which was saved at a time even earlier than the previous bright-light camera image 80C. That is, an image portion corresponding to region RB of the even earlier bright-light camera image 80D is inserted into the previous bright-light camera image 80C to obtain visual image 84D. The even earlier bright-light camera image 80D is a camera image captured when no people are present. The image portion of region RB in this even earlier bright-light camera image 80D, which corresponds to region RA, is overlaid on the corresponding region of the bright-light camera image 80C to obtain visual image 84D.

[0187] Here, the entire image of the past bright camera image 80D can be used as the visual image as is, but a clearer image can be obtained by using the clearest of the saved camera images as the entire image and inserting only the partially different areas from other past bright camera images.

[0188] Alternatively, a brightly lit image taken during a time period desired by the user may be selected and used as the full-range image, for example, an image taken in the evening.

[0189] In this embodiment, a person has been described as an example of a difference region, but in reality, this is not limited to a person. For example, possible difference regions include living things such as animals, moving objects such as automobiles, bicycles, motorcycles, carts, and handcarts, newly installed equipment, placed components, etc. Examples of non-moving objects include buildings and road signs. Furthermore, in order to recognize these, machine learning technology may be used to distinguish between different objects.

[0190] Furthermore, if new obstacles appear in the past bright camera images, they may be displayed in an emphasized manner to draw attention to them.

[0191] Furthermore, if it is possible to determine what the object is, a CG image, sign, or text data may be displayed as an alternative.

[0192] By performing this type of processing, a safe path can be ensured even in dark places without bumping into obstacles or tripping.

[0193] Furthermore, differences between distance images may be caused by factors other than the appearance of a new object in the current dark distance image. For example, at a disaster site, something that should be there may be missing, such as a collapsed road or a collapsed bridge. Even in such cases, as described in Figures 11A and 11B, it is possible to process features in accordance with the difference region 87, recognize the difference region, and superimpose the corresponding AR object to convey details of the difference to the user.

[0194] According to the third embodiment, it is possible to determine a visual image that matches the occurrence of a difference region, and high performance can be achieved in generating a visual image for the dark place guide function.

[0195] (Embodiment 4) A fourth embodiment of the present invention will now be described.

[0196] Overview of Fourth Embodiment The fourth embodiment is an HMD that includes processes such as translation, rotation, and scaling of distance images in the process of comparing distance images and determining whether they match. Note that the hardware configuration of the HMD according to the fourth embodiment is the same as that of the HMD according to the first embodiment, and therefore a description thereof will be omitted.

[0197] <Processing flow for comparing distance images and determining whether they match> The processing flow of the dark place guide program according to the fourth embodiment is basically the same as that of the first embodiment, except for the processing of comparing distance images and determining whether they match. Therefore, only the processing flow of comparing distance images and determining whether they match will be described here, and descriptions of the other flows will be omitted.

[0198] Fig. 14 is a flowchart of the process of comparing distance images and determining whether they match according to embodiment 4. This flowchart corresponds to steps S9 to S11 and S13 in the overall processing flow of Fig. 4, which are enclosed by a dashed line in Fig. 14. Figs. 15A to 15C are diagrams for explaining an example of the process used to compare distance images and determine whether they match according to embodiment 4.

[0199] In step S21, a process of extracting features of the dark distance image is performed. Specifically, the bright place image search device 62 extracts features of the recognized dark distance image.

[0200] In step S22, a process of acquiring a bright spot distance image is performed. Specifically, bright spot image search device 62 reads out one of the narrowed down bright spot distance images.

[0201] In step S23, a process of extracting features of the bright range image is performed. Specifically, distance image search device 62 extracts features of the read bright range image.

[0202] In step S24, the extracted features are compared and the distance image movement amount and scaling factor are calculated. Specifically, distance image search device 62 compares the features of the dark distance image with the features of the bright distance image, and calculates the distance image movement amount and the distance image scaling factor by focusing on the same features.

[0203] In step S25, the dark distance image is moved and scaled. Specifically, distance image search device 62 moves and scales the dark distance image by the calculated movement amount and scaling factor. Here, in steps S24 and S25, if the distance from the shooting position changes due to scaling, changing the color indicating perspective in the distance image in accordance with the scaling factor makes it easier to compare distance images.

[0204] In step S26, the distance images are compared with each other. Specifically, distance image search device 62 compares the shifted and scaled dark distance image with the bright distance image.

[0205] In step S28, a process is performed to determine whether the distance images match. Specifically, distance image search device 62 determines whether the dark distance image and the bright distance image match based on the comparison result. If it is determined that they match (S28, Yes), the process proceeds to step S29. On the other hand, if it is determined that they do not match (S28, No), the process proceeds to step S30.

[0206] In step S29, a process is performed to determine whether the matching area is sufficient. Specifically, distance image search device 62 determines whether the coverage rate of the matching area between the recognized dark distance image and the read bright distance image with respect to the entire area is greater than a specified value. If it is determined that the coverage rate is greater than the specified value (S29, Yes), the process proceeds to step S12. On the other hand, if it is determined that the coverage rate is equal to or less than the specified value (S29, No), the process proceeds to step S30.

[0207] In step S30, a process is performed to determine whether all comparisons of the narrowed-down bright range images have been completed. Specifically, distance image search device 62 determines whether all comparisons have been completed and whether there are any bright range images remaining to be compared next. If it is determined that all comparisons have been completed, i.e., that there are no more bright range images remaining to be compared (S30, Yes), the process proceeds to step S14. On the other hand, if it is determined that all comparisons have not been completed, i.e., that there are still more bright range images remaining to be compared (S30, No), the process returns to step S22, and the next bright range image to be compared is read.

[0208] Fig. 15A is a diagram for explaining translation processing in embodiment 3. Fig. 15B is a diagram for explaining scaling processing in embodiment 3. Fig. 15C is a diagram for explaining combination processing using multiple distance images in embodiment 3.

[0209] In Figure 15A, the thick frame indicated by 81F is the dark distance image, and the thin frame indicated by 83F is the bright distance image. In Figure 15A, when the dark distance image is shifted by a distance "a," if a match between the two distance images is found in the hatched area, the bright camera image in the hatched area is used to determine the visual image.

[0210] In FIG. 15B, if the two distance images match when dark distance image 81F is multiplied by "z", the bright camera image is multiplied by "1 / z" and used to determine the visual image.

[0211] In FIGS. 15A and 15B, blank areas may be filled with, for example, a dark distance image or another past camera image.

[0212] 15C shows a case where multiple bright distance images 83J, 83K, and 83L are compared with dark distance image 81F. Although this example is not described in the above flow, such processing may be incorporated. In this example, when dark distance image 81F and bright distance images 83J, 83K, and 83L partially match, bright camera images corresponding to the matching areas are cut out and combined to generate a visual image.

[0213] In this case, the overlapping of bright distance images 83J, 83K, and 83L may be performed so that priority is given to the clearer image, i.e., the clearer image is positioned higher (closer). Alternatively, the overlapping may be performed so that priority is given to the image with the most recent acquisition date and time, i.e., the image with the most recent acquisition date and time is positioned higher. In the example shown in FIG. 15C, the order of clarity is bright distance image 83J > bright distance image 83K > bright distance image 83L, and the images are arranged so that priority is given to the clearer image, with bright distance image 83J, bright distance image 83K, and bright distance image 83L overlapping in this order from top to bottom.

[0214] In this embodiment, the target of processing such as translation, rotation, and scaling is a dark distance image, but these processes may also be applied to a bright distance image to compare the distance images. Furthermore, when capturing wide-angle images such as with a 360-degree camera, an image in front of the user's HMD may be cropped and saved at a predetermined angle of view based on the position and orientation of the HMD. Furthermore, when recording multiple captured images, the orientation of the captured images from the same position may be aligned in advance, and images with approximately the same position and orientation may be recorded in association with each other.

[0215] According to the fourth embodiment, it is possible to easily find an image in a bright place that substantially matches a dark place distance image.

[0216] (Embodiment 5) A fifth embodiment of the present invention will now be described.

[0217] Overview of Fifth Embodiment The fifth embodiment is an HMD that performs processing suitable for a case where the user is in a dark place but a part of the area in front of the user is illuminated by a light source.

[0218] Configuration of HMD according to the fifth embodiment

[0219] Fig. 16 is an external view of an HMD according to embodiment 5. The same components as those in the HMD 1 shown in Fig. 1 are assigned the same numbers, and duplicated explanations will be omitted. The HMD 1d shown in Fig. 16 has a configuration in which a headlight 22 is added as an auxiliary light source to the HMD 1 shown in Fig. 1.

[0220] Fig. 17 is a functional block diagram of an HMD 1d according to embodiment 5. The same blocks as those in the functional block diagram shown in Fig. 3 are assigned the same numbers, and duplicated explanations will be omitted. The functional block diagram shown in Fig. 17 has a configuration in which an irradiation area detection device (detection device) 68 is added to the functional block diagram shown in Fig. 3.

[0221] The illumination area detection device 68 detects a spot illumination area, which is a partial area illuminated by a light source and brightened in a dark place camera image obtained by a camera.

[0222] The visual image determination device 63 pastes the image portion of the spot illumination area in the dark place camera image to the corresponding area of ​​the detected spot illumination area in the bright place camera image or the dark place distance image determined as the basis of the visual image, thereby determining the visual image.

[0223] The irradiation area detection device 68 is realized by the processor 36 executing the dark place guide program 42 using the memory 37 and the like.

[0224] In this embodiment, it is assumed that a part of the area in front of the user is illuminated by the headlight 22 of the HMD 1d. However, even if a flashlight held by the user or a distant illumination light illuminates a part of the area in front of the user, this can be considered to be the same environment. Therefore, the fifth embodiment can be similarly applied to these cases.

[0225] 18 is a flowchart of a visual image generation and editing process according to embodiment 5. This process flow corresponds to step S15 in the overall process flow according to embodiment 1 shown in FIG. 4, and is a modified example of embodiment 1.

[0226] In step S31, a process is performed to determine whether or not a spot illumination area exists. Specifically, the visual image determination device 63 determines whether or not a spot illumination area exists in the obtained dark place camera image. If it is determined that a spot illumination area exists (S31, Yes), the process proceeds to step S32. On the other hand, if it is determined that a spot illumination area does not exist (S31, No), the process proceeds to step S34.

[0227] In step S32, a process of cutting out an image portion of the spot illumination area is performed. Specifically, the visible image determination device 63 cuts out an image portion of the spot illumination area illuminated by a light source such as a headlight from the dark place camera image.

[0228] In step S33, a process of synthesizing the cut-out image portion is performed. Specifically, the visual image determination device 63 determines the visual image by pasting the cut-out image portion to an area corresponding to the spotlight area in the image set as the base of the visual image. At this time, the pasting position and size of the cut-out image portion are adjusted so that the image set as the base and the cut-out image portion are smoothly connected and look natural. Thereafter, the visual image generation / editing process ends, and the process proceeds to step S16 shown in FIG. 4.

[0229] In step S34, the visual image is determined by the method according to any one of the above-described embodiments 1 to 4. After that, the visual image generation and editing process ends, and the process proceeds to step S16 shown in FIG.

[0230] Fig. 19 is a diagram showing examples of a camera image and a visually recognized image obtained in embodiment 5. In Fig. 19, 82G shows a dark place camera image, and 84G shows a visually recognized image. In dark place camera image 82G, a spot illumination area 89 illuminated by headlight 22 shows the situation ahead.

[0231] The visually recognized image 84G is an image obtained by pasting an image portion of the spot-illuminated region 89 in the dark place camera image onto a region corresponding to the spot-illuminated region 89 in the bright place camera image.

[0232] For example, consider a case where the visual confirmation image 84G contains an unknown difference area and the difference area is masked. In this case, the user can point the headlights 22 at the real space area corresponding to the masked area and directly visually confirm what is in the difference area in the camera image. Here, in this example, one spot area is pasted and displayed, but the user can obtain an image of a wider illumination area by changing the spot illumination direction and moving the illumination area. An image of such a wide spot illumination area may also be pasted.

[0233] According to the fifth embodiment, when a difference that is not captured in the previous bright camera image is present in front of the user in a dark place, the difference area can be illuminated with a light source, allowing the difference to be confirmed in the visual image. This is particularly effective when the display surface of the visual image is non-transparent, as will be described later.

[0234] (Embodiment 6) A sixth embodiment of the present invention will now be described. The sixth embodiment is an HMD that allows operation by the user's gestures.

[0235] 20A and 20B are diagrams showing a first example of the appearance of an HMD according to embodiment 6. HMD 1e shown in Fig. 20A and 20B includes a distance measurement sensor 23 suitable for measuring short distances (hereinafter referred to as short-distance distance measurement sensor) and a distance measurement sensor 24 suitable for measuring medium or long distances (hereinafter referred to as medium-to-long distance distance measurement sensor). These distance measurement sensors are connected to controller 17, which acquires distance data obtained by these distance measurement sensors.

[0236] The medium- to long-distance distance sensor 24 is used to obtain distance data relative to the real space in front of the user's line of sight. The short-distance distance sensor 23 is used to detect gestures made by the user's fingers. A table in which distance data corresponding to types of gestures and types of operations are associated is stored in a saved data area in the controller 17. Based on the distance data obtained by the short-distance distance sensor 23, the controller 17 refers to the table to recognize the type of user gesture and accepts the corresponding operation. Examples of operations include turning on and off a dark place guide program and switching the visual image to be displayed.

[0237] The medium- to long-distance ranging sensor 24 is installed at the widthwise end of the upper frame of the lens (display) of the HMD 1e, for example, as shown in FIG. 20A. The short-distance ranging sensor 23 is installed at the center of the upper frame. In this case, the distance measurement central axis direction 24c of the medium- to long-distance ranging sensor 24 and the distance measurement central axis direction 23c of the short-distance ranging sensor 23 are approximately the same. The user extends their fingers in front of their eyes and performs a gesture operation. The short-distance ranging sensor 23 obtains distance data corresponding to the user's gesture operation.

[0238] Incidentally, when a user looks at a location close to the user in front of the user, the user's line of sight is often downward compared to when viewing an object ahead. Furthermore, when making a gesture with a finger, the user naturally looks at the finger. Therefore, the short-distance ranging sensor 23 may be installed so that its ranging central axis direction 23c is tilted downward by a predetermined angle (e.g., 30°) from the ranging central axis direction 24c of the medium- to long-distance ranging sensor 24, as shown in FIG. 20B . This allows the user to perform a gesture operation with their finger while checking the natural line of sight, enabling a more natural and stress-free operation.

[0239] The arrangement of the short-distance measuring sensor 23 and the medium- to long-distance measuring sensor 24 is not limited to the above example, and various arrangements can be considered depending on the design concept or specifications.

[0240] 21A and 21B are diagrams showing a second example of the appearance of the HMD according to embodiment 6. Also, Fig. 21C is a diagram showing a gesture operation area for a user's fingers.

[0241] 21A and 21B show a modification in which the arrangement of the short-distance measuring sensor is changed from the first example.

[0242] The HMD 1f according to this modification is an example in which the short-distance ranging sensor 23 is arranged in a portion of the frame housing 20 that is shaped like the temples of glasses. The central ranging axis direction 24c of the medium- to long-distance ranging sensor 24 is the same as in the previous example, but the central ranging axis direction 23c of the short-distance ranging sensor 23 extends toward the side of the user's head. In this case, the user performs gesture operations with their fingers on the side of the user's head.

[0243] For example, as shown in Fig. 21C, the gesture operation space area is defined by an xyz coordinate system, and the gesture operation space area is subdivided into a plurality of partial areas. The short-distance sensor 23 obtains distance measurements in each of the subdivided partial areas, and the controller 17 detects gesture operations based on the distribution of the distance measurements or their changes over time. Examples of gesture operations include moving a finger in the x direction to instruct forward or backward movement of the display screen, moving in the y direction to instruct up or down movement, and moving in the z direction to zoom in or out of the display screen.

[0244] This allows the user to perform gesture operations with their fingers near the side of their head, and allows operations to be performed without the gestures interfering with their field of vision.

[0245] According to the sixth embodiment, the image display device can be operated by a user's gesture operations to turn on / off the dark place guide program, switch the displayed image, etc., thereby realizing an image display device that is easier to use. For example, even when a user is working on-site wearing gloves, the user can operate the image display device without operating a button-type or touch-type operation unit that requires relatively delicate operation.

[0246] (Embodiment 7) A seventh embodiment of the present invention will now be described. The seventh embodiment is an example of an HMD configured to display a user interface screen.

[0247] The controller 17 of the HMD controls the display device to display a user interface screen on the display surface, and also accepts predetermined operations such as the gesture operations shown in the sixth embodiment.

[0248] Fig. 22 is a diagram showing an example of a user interface screen in an HMD. For example, when an illuminance sensor detects that the user's field of vision has become dark, the HMD controller 17 controls the display device to display a screen display switching selection screen 90. For example, a selection screen such as that shown in Fig. 22 may be displayed, allowing the user to select whether or not to switch the display screen to a bright video mode.

[0249] If the user does not want to switch the screen, i.e., selects "No," then, for example, a distance measurement image is displayed. On the other hand, if the user wants to switch the screen, i.e., selects "Yes," then a selection screen is displayed for selecting several ways to display the screen. For example, a selection screen like the one shown in the lower part of Fig. 22 is displayed, allowing the user to select the image to be displayed from "daytime image," "partially interpolated image," and "arranged image."

[0250] This allows the display of bright images captured in bright places, or if a portion of the field of view is dark, the display of only that dark portion can be replaced with an image captured in bright places. By selecting an arranged image, the image can be processed and displayed. For example, an image can be displayed that has been deliberately converted to black and white or sepia, or an image that has been arranged in vivid colors like those used in CG or animation.

[0251] According to the seventh embodiment, the video can be switched according to the user's intentions and preferences.

[0252] (Embodiment 8) An eighth embodiment of the present invention will now be described. In the first to seventh embodiments, an HMD having a light-transmitting image display surface that transmits light from the front of the user is used as the image display device. In the present embodiment, an HMD having a light-impermeable image display surface that does not transmit light from the front of the user is used as the image display device.

[0253] FIG. 23 is an external view of an HMD according to an eighth embodiment as an image display device according to any one of the first to seventh embodiments. FIG. 23 illustrates an example of a goggle-type HMD having a display surface that is opaque to light from the front of the user. This HMD is used in a video see-through mode, in which images captured by a camera are displayed on the left and right displays. The HMD 1z shown in FIG. 23 includes a housing 20d molded to cover the user's field of view and a fixture 20e for fixing the housing 20d so as to cover the user's field of view. A left-eye display 25 and a right-eye display 26 are provided on the surface of the housing 20d that covers the user's eyes. The housing 20d also includes a right camera 11a, a left camera 11b, a distance sensor 12, a controller 17, a microphone 18, a speaker 19, a battery 21, a headlight 22, and a short-distance distance sensor 23. The illuminance sensor is built into the controller 17, but may be provided on the top of the housing 20d.

[0254] Left-eye display 25 and right-eye display 26 are configured, for example, by a liquid crystal panel, an organic EL (Electro Luminescence) panel, a plasma display, etc. Of course, these left-eye and right-eye displays 25 and 26 may each be configured by an image projection device using a projector.

[0255] The user wears the HMD1z on their head and views the image displayed on the left-eye display with their left eye and the image displayed on the right-eye display with their right eye, allowing them to view camera images, distance images, or visual images in three dimensions.

[0256] According to the eighth embodiment, a more robust HMD that fits closely to the face can be used as the image display device, and the user's eyes can be protected from dangerous obstacles, gases, or liquids that may be present at the site. Also, since the display device does not need to be limited to a projector type, the degree of freedom in designing the image display device increases.

[0257] (Embodiment 9) A ninth embodiment of the present invention will now be described. In the first to eighth embodiments, an HMD is used as the image display device. In the ninth embodiment, a portable information terminal is used as the image display device. Examples of the portable information terminal include a smartphone, a tablet terminal, and a notebook computer.

[0258] There are many models of portable information terminals that have a camera and a distance sensor on the back side opposite the display screen. The distance sensor can also be added later. Such portable information terminals can be made to operate in the same way as the HMDs described above.

[0259] Fig. 24A is a diagram showing an example of the angle of view of the camera and the measurement range of the distance measuring sensor of the portable information terminal. Fig. 24B is a diagram showing an example of a dark distance image displayed on the portable information terminal. Fig. 24C is a diagram showing an example of a bright camera image displayed on the portable information terminal.

[0260] The portable information terminal 1s is configured to define x and y coordinate axes as shown by the dashed lines in Fig. 24A, and to have a camera angle of view 91 and a distance measuring sensor measurement range 92 as shown by the dashed lines. The portable information terminal 1s can also display a dark distance image on the display screen as shown in Fig. 24B, or a bright camera image corresponding to the dark distance image as shown in Fig. 24C.

[0261] According to the ninth embodiment, it is possible to perform the same operations as the HMDs according to the first to eighth embodiments using a highly versatile and widely used portable information terminal, without using a specially manufactured image display device such as an HMD. This reduces the development costs of the image display device or the cost of the device itself. Furthermore, it is possible to expand the user base to those who already own portable information terminals, and it is possible to efficiently accumulate image sets from any location. Furthermore, users can obtain visual images, such as brightly lit camera images, in any location.

[0262] Although various embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the same effect can be achieved by replacing the generation of a distance image using a distance sensor with the generation of an infrared camera image using an infrared camera. Furthermore, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. included in the text and figures are merely examples, and the effects of the present invention will not be impaired if different ones are used.

[0263] Furthermore, some of the configurations of each embodiment may be added to, deleted from, or replaced with other configurations. Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be implemented in hardware, in part or in whole, by, for example, designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor, such as an MPU or CPU, interpreting and executing a program that implements each function. Furthermore, the scope of functions implemented by software is not limited, and hardware and software may be used together. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device, such as a hard disk or SSD (Solid State Drive), or a recording medium, such as an IC card, SD card, or DVD.

[0264] Possible aspects of the present invention will be described below.

[0265] [Appendix 1] An image display device that displays a visual image for a user to visually recognize, a camera that captures an image in front of the user and obtains a camera image; a distance measurement sensor for obtaining data representing a distance from the user to each position on a real object included in a field of view of the camera; an illuminance sensor that obtains data representing the brightness of a location where the user is located; a generating device that generates a distance image corresponding to the field of view, each pixel of which represents a distance to each of the positions, based on data obtained by the distance measuring sensor; a determination device that determines whether the location of the user is a bright place or a dark place based on data obtained by the illuminance sensor; a storage device that stores the camera image obtained by the camera and the distance image obtained by the generation device as an image set including a bright place camera image and a bright place distance image when the determination device determines that the location is a bright place; a recognition device that recognizes the distance image obtained by the generation device as a dark place distance image when the determination device determines that the place is dark; a search device that identifies a bright place distance image corresponding to the recognized dark place distance image by comparing the bright place distance image stored by the storage device with the dark place distance image recognized by the recognition device; a determination device that determines a visual image to be viewed by the user based on a bright camera image included in the same image set as the bright distance image identified by the search device; a display device that displays the visual image; An image display device comprising:

[0266] [Appendix 2] A program for causing a computer to function as a generating device, a determining device, a storing device, a recognizing device, a searching device, and a determining device in the image display device described in [Appendix 1].

[0267] [Appendix 3] A computer-readable recording medium having recorded thereon a program for causing a computer to function as a generating device, a determining device, a storing device, a recognizing device, a searching device, and a deciding device in the image display device described in [Appendix 1].

[0268] [Appendix 4] In the image display device described in Appendix 1, the process of comparing a dark distance image and a bright distance image applies at least one of parallel translation, rotation, and scaling to at least one of the dark distance image and the bright distance image.

[0269] [Appendix 5] In the image display device according to Supplementary Note 1, The image display device uses a smartphone, a tablet terminal device, or a notebook computer.

[0270] [Appendix 6] In the image display device according to Supplementary Note 1, The camera also serves as the illuminance sensor.

[0271] [Appendix 7] In the image display device according to Supplementary Note 1, The search device performs at least one of translation, rotation, and scaling on at least one of the recognized dark distance image and the identified bright distance image to compare them.

[0272] [Appendix 8] In the image display device according to Supplementary Note 1, The image display device includes a screen onto which the visual image is projected.

[0273] [Appendix 9] In the image display device according to Supplementary Note 1, The display device includes a display device that displays the visual image.

[0274] [Appendix 10] In the image display device according to Supplementary Note 1, The search device is an image display device that identifies a bright place distance image that corresponds to the dark place distance image and has a similarity or match with the dark place distance image that is equal to or greater than a threshold.

[0275] [Appendix 11] In the image display device according to Supplementary Note 1, The search device is an image display device that identifies a bright distance image that is recognized by artificial intelligence as matching, approximating, or similar to the dark distance image as a bright distance image corresponding to the dark distance image.

[0276] [Appendix 12] In the image display device according to Supplementary Note 1, A time information acquisition device is provided, the time information acquisition device acquires time information including at least a date; The storage device is extracting, from the stored bright field distance images, a bright field distance image corresponding to the bright field distance image to be stored by the storage device; a process of storing the image set in association with time information acquired by the time information acquisition device; An image display device that, when saving the image set, performs a process of overwriting the image set including the bright distance image extracted by the extraction process, or a process of erasing the image set including the extracted bright distance image if the time represented by the time information associated with the extracted bright distance image is more than a first time before the time represented by the time information associated with the bright distance image to be saved.

[0277] [Appendix 13] 13. The image display device according to claim 12, An image display device, wherein the extraction process is a process of extracting a bright field distance image corresponding to the bright field distance image to be saved, the bright field distance image having a similarity or degree of coincidence with the bright field distance image to be saved that is greater than or equal to a threshold.

[0278] [Appendix 14] 13. The image display device according to claim 12, An image display device, wherein the extraction process is a process of extracting a bright field distance image that corresponds to the bright field distance image to be saved and that is recognized by artificial intelligence as matching, approximating, or similar to the bright field distance image to be saved. [Explanation of symbols]

[0279] 1, 1a, 1b, 1h...Transmissive head-mounted display, 1z...Non-transmissive head-mounted display, 1s...Portable information terminal, 11...Camera, 12...Distance measurement sensor, 13...Right-eye projector, 14...Left-eye projector, 15...Image screen, 16...Nose pad, 17...Controller, 18...Microphone, 19...Speaker, 20, 20a to 20c...Frame housing, 21...Battery, 22...Headlight, 23...Short-range distance measurement sensor, 24...Medium-to-long-range distance measurement sensor, 30...Internal bus, 31...GPS sensor, 32...Illuminance sensor, 33...Acceleration sensor, 34...Gyro sensor, 35...Orientation sensor, 36...Processor, 37...Memory, 38...Image memory, 39...Non-volatile storage device, 40...Communication device, 41...Base This operating program, 42... dark place guide program, 43... saved data area, 51... position information acquisition device, 52... time information acquisition device, 53... orientation information acquisition device, 54... attitude information acquisition device, 55... camera image acquisition device, 56... distance image generation device, 57... bright place and dark place judgment device, 58... bright place image storage management device, 59... bright place image storage device, 60... dark place distance image recognition device, 61... comparison image narrowing down device, 62... bright place image search device, 63... visual image determination device, 64... visual image display device, 65... difference area detection device, 66... ​​difference factor discrimination device, 67... bright place distance image selection device, 68... irradiation area detection device, 70... user, 72... access point, 73... network, 74... image storage service server, 100, 101... HMD system.

Claims

1. An image display device that displays a visual image for a user to visually recognize, a camera that captures an image in front of the user; an illuminance sensor that detects the brightness of the user's surroundings; A control unit; a display unit, The control unit Controlling to acquire data relating to the brightness of the surroundings of the user detected by the illuminance sensor; Control to determine the brightness of the user's surroundings based on the data; When it is determined that the surroundings of the user are dark in the determination, the image is recognized as a dark place image, and a bright place image corresponding to the dark place image is acquired. Control is performed to determine a visual image to be visually recognized by the user based on the photopic image; Control to detect a difference area between the dark image and the bright image; performing an insertion process of inserting an image corresponding to the difference region in the dark place image into a corresponding region of the difference region in the bright place image, thereby controlling the visual recognition image to be determined; The display unit displays the visual image. Image display device.

2. 2. The image display device according to claim 1, The control unit Controlling to acquire location information of the user; and controlling the image processing unit to narrow down the bright spot images based on position information associated with the images. Image display device.

3. 2. The image display device according to claim 1, The control unit Control is performed to determine whether the cause of the difference region is in the dark image or the bright image; When it is determined that the occurrence cause is in the bright image, control is performed to execute the insertion process. Image display device.

4. An image display device that displays a visual image for a user to view, a camera that captures an image in front of the user; an illuminance sensor that detects the brightness of the user's surroundings; A control unit; a display unit, The control unit Controlling to acquire data relating to the brightness of the surroundings of the user detected by the illuminance sensor; Control to determine the brightness of the user's surroundings based on the data; When it is determined that the surroundings of the user are dark in the determination, the image is recognized as a dark place image, and a bright place image corresponding to the dark place image is acquired. Control is performed to determine a visual image to be visually recognized by the user based on the photopic image; Control to detect a difference area between the dark image and the bright image; Controlling to select a light image corresponding to the dark image and different from the light image; performing an insertion process of inserting an image corresponding to the detected difference region in the selected bright image into a corresponding region of the difference region in the bright image, thereby controlling to determine the visual image; The display unit displays the visual image. Image display device.

5. An image display device that displays a visual image for a user to view, a camera that captures an image in front of the user; an illuminance sensor that detects the brightness of the user's surroundings; A control unit; a display unit, The control unit Controlling to acquire data relating to the brightness of the surroundings of the user detected by the illuminance sensor; Control to determine the brightness of the user's surroundings based on the data; When it is determined that the surroundings of the user are dark in the determination, the image is recognized as a dark place image, and a bright place image corresponding to the dark place image is acquired. Control is performed to determine a visual image to be visually recognized by the user based on the photopic image; Control to detect a difference area between the dark image and the bright image; Control is performed to determine whether the cause of the difference region is in the dark image or the bright image; When it is determined that the cause of the occurrence is in the dark place image, an enhancement process is executed to enhance a corresponding area of ​​the difference area in the bright place image, thereby controlling to determine the visually recognized image; The display unit displays the visual image. Image display device.

6. An image display device that displays a visual image for a user to view, a camera that captures an image in front of the user; an illuminance sensor that detects the brightness of the user's surroundings; A control unit; a display unit, The control unit Controlling to acquire data relating to the brightness of the surroundings of the user detected by the illuminance sensor; Control to determine the brightness of the user's surroundings based on the data; When it is determined that the surroundings of the user are dark in the determination, the image is recognized as a dark place image, and a bright place image corresponding to the dark place image is acquired. Control is performed to determine a visual image to be visually recognized by the user based on the photopic image; Control to detect a difference area between the dark image and the bright image; Control is performed to determine whether the cause of the difference region is in the dark image or the bright image; When it is determined that the cause of the occurrence is in the dark place image, an insertion process is executed to insert an artificial image according to characteristics of the difference region into a corresponding region of the difference region in the bright place image, thereby controlling to determine the visual image; The display unit displays the visual image. Image display device.

7. 2. The image display device according to claim 1, The control unit Control to acquire multiple photopic images, Control is performed to determine the visual image based on the plurality of photopic images. Image display device.

8. An image display device that displays a visual image for a user to visually recognize, a camera that captures an image in front of the user; an illuminance sensor that detects the brightness of the user's surroundings; A control unit; A display unit; a light source; the light source illuminates a portion of the field of view of the camera; The control unit Controlling to acquire data relating to the brightness of the surroundings of the user detected by the illuminance sensor; Control to determine the brightness of the user's surroundings based on the data; When it is determined that the surroundings of the user are dark in the determination, the image is recognized as a dark place image, and a bright place image corresponding to the dark place image is acquired. Control is performed to determine a visual image to be visually recognized by the user based on the photopic image; Controlling the camera to detect an area illuminated by the light source in a dark place image captured by the camera in a dark place; Control is performed to determine the visual image by performing an insertion process of inserting an image corresponding to the illuminated area in the dark place image into a corresponding area of ​​the illuminated area in the bright place image; The display unit displays the visual image. Image display device.

9. 2. The image display device according to claim 1, including a head-mounted display, the head-mounted display includes the camera, the illuminance sensor, and the display unit; Image display device.

Images