Information processing device

Abstract

According to the present invention, a display control unit causes a virtual object to be displayed in a display region of AR glasses. An image acquiring unit acquires a first captured image including a range of real space viewed through the display region by a user wearing the AR glasses. An object detecting unit detects a prescribed object appearing in the first captured image. The AR glasses are configured to reduce light in a prescribed wavelength band among light incident on the display region. The display control unit causes a partial image representing a part of the first captured image corresponding to the prescribed object to be displayed in a position in the display region in which the user views the prescribed object.

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Conventionally, products have been developed that aim to adjust the brightness around a user or reduce the transmittance of light of a specific wavelength such as blue light, etc. For example, Patent Document 1 below discloses a surface protection film in which four layers are laminated in this order: a protective layer made of polyurethane, a transparent substrate film, a blue light blocking layer, and an adhesive layer. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6992063 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, XR glasses that apply XR technologies such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) have become widespread. For example, if a surface protection film such as that described in Patent Document 1 is attached to the XR glasses, the user's entire field of view may become dark, making it difficult to see objects within the field of view.

[0005] The object of the present invention is to control the light entering the XR glasses without obstructing the user's field of vision. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention comprises a display control unit that displays a virtual object in a display area of ​​a see-through head-mounted display, an image acquisition unit that acquires an image from an imaging device that includes a range of real space that a user wearing the see-through head-mounted display sees through the display area, and an object detection unit that detects a predetermined object that appears in the image, wherein the see-through head-mounted display is configured to reduce light of a predetermined wavelength band from the light incident on the display area, and the display control unit displays a partial image representing a portion of the predetermined object in the image at a position in the display area where the user sees the predetermined object. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to control light entering the XR glasses without obstructing the user's field of vision. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing the configuration of a display system 10 according to a first embodiment. [Figure 2] 1 is an explanatory diagram showing the appearance of AR glasses 30. FIG. [Figure 3] FIG. 2 is a block diagram showing the configuration of the AR glasses 30. [Figure 4] FIG. 2 is a block diagram showing the configuration of a terminal device 20. [Figure 5] 10 is a diagram showing an example of a first captured image PA obtained by capturing an image of a real space R. FIG. [Figure 6] 1 is a diagram showing an example of a real space R visually recognized by a user wearing AR glasses 30. FIG. [Figure 7] 10 is a diagram showing an example of a real space R visually recognized by a user when a partial image PD is displayed. FIG. [Figure 8] FIG. 10 is a block diagram showing the configuration of AR glasses 30 according to a second embodiment. [Figure 9] FIG. 10 is a block diagram showing the configuration of a terminal device 20 in a second embodiment. [Figure 10]10 is a diagram showing an example of a real space R visually recognized by a user when a partial image PT is displayed. FIG. [Figure 11] 10 is a diagram showing an example of a real space R visually recognized by a user when a partial image PT is displayed. FIG. [Figure 12] 10 is a diagram showing an example of a real space R visually recognized by a user when an outer edge image PE is displayed. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment The configuration of a display system 10 including an information processing device according to a first embodiment of the present invention will be described below.

[0010] A-1. System Configuration FIG. 1 is a diagram showing the configuration of a display system 10 according to the first embodiment. The display system 10 includes a terminal device 20 and AR glasses 30. The terminal device 20 is an example of an information processing device. The AR glasses 30 are an example of a see-through head-mounted display. The terminal device 20 and the AR glasses 30 are connected to each other so that they can communicate with each other. The terminal device 20 is preferably a mobile terminal device such as a smartphone or a tablet.

[0011] The display system 10 is a system that uses AR technology to present various types of information to a user wearing AR glasses 30 (described below). Here, AR technology refers to a technology that allows a user wearing a device, such as a see-through head-mounted display, to visually recognize information indicated by a virtual object by superimposing the virtual object on a coordinate space of the real world. The AR glasses 30 may display virtual objects and output other types of information, such as audio information. Examples of virtual objects include still images, videos, 3DCG models, and text.

[0012] In this embodiment, the virtual object is described as being a partial image PD or PT, which will be described later. However, the virtual object is not limited to this, and may be displayed based on virtual object data provided from a virtual object server (not shown), for example. In this case, the terminal device 20 is communicably connected to the virtual object server via a wide area communication network (not shown). The terminal device 20 receives virtual object data corresponding to the virtual object from the virtual object server. The terminal device 20 uses the virtual object data to display the virtual object on the AR glasses 30.

[0013] A-2.AR Glasses 30 The AR glasses 30 are eyeglass-type see-through head-mounted displays worn on the user's head. The AR glasses 30 display virtual objects on lenses 310A and 310B under the control of the terminal device 20. Note that, as the see-through head-mounted display, for example, a goggle-shaped see-through head-mounted display having the same functions as the AR glasses 30 may be used.

[0014] 2 is an explanatory diagram showing the appearance of the AR glasses 30. The AR glasses 30 have temples 301 and 302, a bridge 303, frames 304 and 305, rims 306 and 307, lenses 310A and 310B, and an imaging lens LEN that can be seen from the outside.

[0015] The bridge 303 connects the limbs 306 and 307. The bridge 303 is provided with an imaging lens LEN that constitutes the first imaging device 324 shown in FIG.

[0016] The frame 304 is provided with a display panel for the left eye and an optical member for the left eye. The frame 305 is provided with a display panel for the right eye and an optical member for the right eye. The display panel for the left eye and the display panel for the right eye are, for example, liquid crystal panels or organic EL (Electro Luminescence) panels. The display panel for the left eye and the display panel for the right eye display images corresponding to virtual objects, for example, based on control from the terminal device 20 described later. The optical member for the left eye guides light emitted from the display panel for the left eye to the lens 310A. The optical member for the right eye guides light emitted from the display panel for the right eye to the lens 310B. Furthermore, the frames 304 and 305 are provided with a speaker 322 described later.

[0017] Rim 306 holds lens 310A, and rim 307 holds lens 310B.

[0018] Each of the lenses 310A and 310B has a half mirror. The half mirrors of the lenses 310A and 310B transmit light representing the real space R, thereby guiding the light representing the real space R to the user's eyes. The half mirrors of the lenses 310A and 310B also reflect light representing the virtual object, which is guided by the optical member, toward the user's eyes. The light of the real space R that has passed through the half mirror and the light representing the virtual object that has been reflected by the half mirror are superimposed and enter the user's eyes, causing the user to perceive the virtual object as being located in the real space R. In other words, the lenses 310A and 310B function as a transmissive display disposed in front of the user's eyeballs. The lenses 310A and 310B form the display area of ​​the AR glasses 30.

[0019] In this embodiment, blue light cut filters F are attached to the outer surfaces of the lenses 310A and 310B of the AR glasses 30. The outer surfaces of the lenses 310A and 310B are the two surfaces of the planar lenses 310A and 310B opposite the surface facing the eyes of the user wearing the AR glasses 30. The blue light cut filters F are filters that reflect or absorb blue light with short wavelengths of 380 nm to 500 nm, which is visible light (wavelengths of 380 nm to 780 nm). In other words, by including the blue light cut filters F, the AR glasses 30 are configured to reduce light of a predetermined wavelength band among the light incident on the lenses 310A and 310B. The predetermined wavelength band is 380 nm to 500 nm. The blue light cut filters F may also cut ultraviolet light with wavelengths of less than 380 nm, in addition to blue light.

[0020] By using the blue light cut filter F, it is possible to reduce the amount of blue light that enters the eyes of a user wearing the AR glasses 30, thereby reducing eye strain and other burdens on the user. In this embodiment, the blue light cut filter F is attached to the lenses 310A, 310B for the purpose of reducing blue light emitted from the display screens of electronic devices placed around the user wearing the AR glasses 30 and blue light contained in natural light entering through windows, etc.

[0021] Instead of the blue light cut filter F, for example, an ND (Neutral Density) filter that reduces the incident light to the lenses 310A and 310B over the entire band may be used. By using an ND filter, the user can take a rest or concentrate on their work even in bright surroundings. A blue light blocking lens may be used as B. In this case, the lenses 310A and 310B themselves serve as light control members.

[0022] 3 is a block diagram showing the configuration of the AR glasses 30. The AR glasses 30 include the temples 301 and 302, bridge 303, frames 304 and 305, rims 306 and 307, lenses 310A and 310B, and imaging lens LEN described above, as well as a projection device 321, a speaker 322, a communication device 323, a first imaging device 324, a storage device 327, a processing device 328, and a bus 329.

[0023] 3 are stored in, for example, frames 304 and 305. The projection device 321, speaker 322, communication device 323, first image capture device 324, storage device 327, and processing device 328 are connected to one another by a bus 329 for communicating information. The bus 329 may be configured using a single bus, or may be configured using different buses between each element of the device, etc.

[0024] The projection device 321 includes lenses 310A and 310B, a left-eye display panel, an optical member for the left eye, a right-eye display panel, and an optical member for the right eye. As described above, the projection device 321 displays images corresponding to virtual objects on the left-eye display panel and the right-eye display panel based on control from the terminal device 20. Light emitted from the left-eye display panel is guided to the lens 310A by the optical member for the left eye. Light emitted from the right-eye display panel is guided to the lens 310B by the optical member for the right eye. The user views the images corresponding to the virtual objects displayed on the lenses 310A and 310B.

[0025] The speaker 322 is located in each of the frames 304 and 305. The speaker 322 may not be located in each of the frames 304 and 305, but may be located in, for example, one of the frames 304 and 305, at least one of the temples 301 and 302, or at least one of the bridge 303. The speaker 322 is controlled directly by the terminal device 20 or via a processing device 328 of the AR glasses 30. The sound output by the speaker 322 is, for example, sound output in association with a virtual object. The speaker 322 may not be included in the AR glasses 30 and may be separate from the AR glasses 30.

[0026] The communication device 323 includes a communication interface for communicating with other devices. The communication device 323 communicates with the terminal device 20 using wireless communication or wired communication. In this embodiment, the communication device 323 communicates with the communication device 203 (see FIG. 4) of the terminal device 20 using short-range wireless communication such as Bluetooth (registered trademark).

[0027] The first imaging device 324 has, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens LEN (see FIG. 2). For example, the imaging optical system may have various optical elements such as a prism, or may have a zoom lens or a focus lens. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0028] The first imaging device 324 captures an image of a subject and outputs first captured image information indicating the captured image (hereinafter referred to as a "first captured image PA"). In this embodiment, the imaging range of the first imaging device 324 matches or includes the user's field of view. Therefore, the first captured image PA includes the range of real space R that a user wearing the AR glasses 30 views through the lenses 310A and 310B. The correspondence between each pixel of the first captured image PA captured by the first imaging device 324 and each pixel of the see-through display realized by the lenses 310A and 310B is calibrated in advance. In other words, the position of an object captured in the first captured image PA on the lenses 310A and 310B as seen by the user wearing the AR glasses 30 is known. Therefore, the display control unit 214 (described later) can project a virtual object, the display position of which has been determined based on the first captured image PA, onto the lenses 310A and 310B.

[0029] The first captured image PA generated by the first imaging device 324 is transmitted as first captured image information to the terminal device 20 via the communication device 323. The first imaging device 324 repeats imaging at a predetermined imaging interval, and transmits the generated first captured image information to the terminal device 20 each time an image is captured.

[0030] The storage device 327 is a recording medium readable by the processing device 328. The storage device 327 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM). The storage device 327 stores a program PG1. The program PG1 is a program for operating the AR glasses 30.

[0031] The processing unit 328 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0032] The processing device 328 reads the program PG1 from the storage device 327. The processing device 328 executes the program PG1 to function as the operation control unit 330. The operation control unit 330 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0033] The operation control unit 330 controls the operation of the AR glasses 30. For example, the operation control unit 330 provides the projection device 321 with a control signal for image display that the communication device 323 has received from the terminal device 20. The projection device 321 displays an image indicated by the control signal for image display on the lenses 310A and 310B.

[0034] A-3. Terminal device 20 4 is a block diagram showing the configuration of terminal device 20. Terminal device 20 includes a touch panel 201, a communication device 203, a storage device 205, a processing device 206, and a bus 207. Touch panel 201, communication device 203, storage device 205, and processing device 206 are interconnected by bus 207 for communicating information. Bus 207 may be configured using a single bus, or may be configured using different buses for each device.

[0035] The touch panel 201 displays various information to the user and detects touch operations by the user. The touch panel 201 serves as both an input device and an output device. For example, the touch panel 201 is configured by bonding a touch sensor unit capable of detecting touch operations between a cover glass and a display panel such as a liquid crystal display panel or an organic EL display panel. For example, when a user's finger is in contact with the touch panel 201, the touch panel 201 periodically detects the contact position of the user's finger on the touch panel 201 and transmits touch information indicating the detected contact position to the processing device 206.

[0036] The communication device 203 includes a communication interface for communicating with other devices. The communication device 203 communicates with the AR glasses 30 using wireless communication or wired communication. In this embodiment, the communication device 203 communicates with the communication device 323 (see FIG. 3 ) using the same type of short-range wireless communication as the communication device 323 of the AR glasses 30.

[0037] The communication device 203 may also communicate with a virtual object server using wireless or wired communication to receive virtual object data. In this case, the communication device 203 has an interface that can be connected to a wide area network (not shown).

[0038] The storage device 205 is a recording medium readable by the processing device 206. The storage device 205 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 205 stores a program PG2. The program PG2 is a program for operating the terminal device 20.

[0039] Processing unit 206 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of a processor and a CPU is an example of a computer.

[0040] The processing device 206 reads the program PG2 from the storage device 205. By executing the program PG2, the processing device 206 functions as an image acquisition unit 210, an object detection unit 212, and a display control unit 214. At least one of the image acquisition unit 210, the object detection unit 212, and the display control unit 214 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.

[0041] The image acquisition unit 210 acquires a captured image including the range of real space R that is visually recognized through the lenses 310A and 310B by a user wearing the AR glasses 30 from the first imaging device 324. In this embodiment, the image acquisition unit 210 acquires first captured image information corresponding to the first captured image PA captured by the first imaging device 324 of the AR glasses 30 via the communication device 203.

[0042] The object detection unit 212 detects a predetermined object appearing in the first captured image PA. In this embodiment, the "object" is not limited to an inanimate object, but may also be a living object (a person, a plant). In the first embodiment, the predetermined object is a display screen of an electronic device. The object detection unit 212 detects the display screen of the electronic device from the first captured image PA. The display screen of the electronic device may be a device whose main function is to display information to a user, such as a display screen of a television, personal computer, smartphone, tablet, etc. Furthermore, the display screen of the electronic device may be a device whose main function is to receive settings for the electronic device from a user, such as a display screen of a printer, portable music player, cooking appliance, etc.

[0043] The object detection unit 212 detects the display screen of the electronic device using, for example, image analysis. The object detection unit 212 may also detect the brightness or contrast ratio of the display screen shown in the first captured image PA, and if the brightness or contrast ratio is equal to or less than a predetermined value, may determine that the display screen (or the electronic device itself) is not in use and not detect it.

[0044] The display control unit 214 displays virtual objects on the lenses 310A and 310B of the AR glasses 30. For example, the display control unit 214 controls the projection device 321 of the AR glasses 30 using virtual object data acquired from a virtual object server (not shown) to display a virtual object corresponding to the virtual object data.

[0045] In this embodiment, the display control unit 214 displays a partial image PD representing a portion of a predetermined object in the first captured image PA at a position on the lenses 310A, 310B where the user views the predetermined object. In the first embodiment, the partial image PD is a portion of the first captured image PA that represents the display screen of the electronic device. The display control unit 214 displays the partial images PD (e.g., PD1 to PD3) at a position on the lenses 310A, 310B where the user views the display screen.

[0046] The purpose of this display is to improve the visibility of the display screen from the user's perspective. As described above, the blue light cut filter F is attached to the AR glasses 30. Therefore, the visibility of the display screen is reduced compared to when the AR glasses 30 are not worn. On the other hand, in many cases, the display screen is arranged to display information required by the user, and it is desirable for the user to easily grasp the display content of the display screen. For this reason, the display control unit 214 generates a partial image PD representing the display screen of the electronic device from the first captured image PA captured by the first imaging device 324. Then, the display control unit 214 displays the partial image PD superimposed at a position where the user views the display screen. Therefore, the visibility of the display screen is improved compared to when the display screen is viewed through the AR glasses 30. In other words, the display system 10 allows the user to easily grasp the display content of the display screen while reducing blue light.

[0047] FIG. 5 is a diagram showing an example of a first captured image PA captured of real space R. The first captured image PA shown in FIG. 5 is captured so as to include the range of real space R that a user wearing the AR glasses 30 views through the lenses 310A and 310B. The real space R captured in the first captured image PA is an indoor space, and includes a television OB1, a laptop computer OB2, a smartphone OB3, a lighting fixture OB4, and a window OB5. In the first captured image PA, a portion of the display screen of the television OB1 is referred to as a partial image PD1, a portion of the display screen of the laptop computer OB2 is referred to as a partial image PD2, and a portion of the display screen of the smartphone OB3 is referred to as a partial image PD3. Hereinafter, the television OB1, the laptop computer OB2, and the smartphone OB3 may be referred to as electronic devices OB1 to OB3. The colors of the real space R captured in the first captured image PA are substantially the same as those perceived by a user not wearing the AR glasses 30.

[0048] FIG. 6 is a diagram showing an example of a real space R perceived by a user wearing the AR glasses 30. FIG. 6 shows the real space R perceived by the user when the partial image PD is not displayed. The real space R shown in FIG. 6 is perceived by the user through a blue light cut filter F. Therefore, the amount of blue light incident on the user's eyes is reduced. On the other hand, compared to FIG. 5, the real space R appears darker overall.

[0049] 7 is a diagram showing an example of the real space R visually perceived by the user when the partial image PD is displayed. In FIG. 7, the partial image PD1 is displayed superimposed on the display screen of the television OB1, the partial image PD2 is displayed superimposed on the display screen of the laptop computer OB2, and the partial image PD3 is displayed superimposed on the display screen of the smartphone OB3. As described above, the color of the real space R captured in the first captured image PA is substantially the same as the color perceived by the user who is not wearing the AR glasses 30. Therefore, the display screens of the television OB1, the laptop computer OB2, and the smartphone OB3 are visually perceived in the same way as when the user is not wearing the AR glasses 30.

[0050] Each of the partial images PD1 to PD3 is an image of the same size as the display screen of each of the electronic devices OB1 to OB3 visually recognized by the user. That is, the display control unit 214 displays the partial images PD of the same size as the display area visually recognized by the user. This allows the user to visually recognize the partial images PD without feeling uncomfortable, and reduces the user's fatigue associated with using the AR glasses 30.

[0051] The partial image PD may also be displayed in a color tone appropriate for a so-called night mode. In the night mode, the color tone of the image is changed to a warmer tone. This reduces the amount of blue light incident on the user's eyes, preventing subsequent use of the display screen from interfering with sleep. In the night mode, the color temperature of the white point of the display screen is lowered compared to normal. That is, the display control unit 214 may set the color temperature of the white point of the partial image PD displayed on the lenses 310A and 310B to a predetermined temperature or lower. The predetermined temperature, i.e., the color temperature of the white point in the night mode, may be specified by the user. For example, if the color temperature of the white point in normal use is 6500 K (equivalent to daylight white), the color temperature of the white point in the night mode may be 5500 K (equivalent to daylight white) or lower and 2700 K (equivalent to incandescent white) or higher. Displaying the partial image PD in the night mode color tone reduces the amount of blue light incident on the user's eyes, further reducing eye strain and other user stress.

[0052] A-4. Summary of the first embodiment As described above, the display system 10 according to the first embodiment is configured to reduce light in a predetermined wavelength band from among light incident on the display area of ​​the AR glasses 30. Furthermore, the display system 10 displays a partial image PD that shows a predetermined object in real space R at a position in the display area of ​​the AR glasses 30 where the user can view the predetermined object. Therefore, it is possible to control the light that enters the AR glasses 30 without obstructing the user's field of vision.

[0053] The display system 10 according to the first embodiment is configured to reduce light of 380 nm or more and 500 nm or less, which is considered blue light, from among the light incident on the display area of ​​the AR glasses 30. The display system 10 also displays a partial image PD representing the display screen of the electronic device. This reduces blue light emitted from the display screen of the electronic device, and allows the user to easily grasp the content displayed on the display screen. The specified object is not limited to the display screen, and may be an object designated by the user.

[0054] Furthermore, the display system 10 according to the first embodiment displays the partial image PD in night mode. That is, the display system 10 sets the color temperature of the white point of the partial image PD displayed in the display area of ​​the AR glasses 30 to a predetermined temperature or lower. This reduces blue light from the image displayed on the lenses 310A and 310B, further reducing eye strain and other user stress.

[0055] Furthermore, the display system 10 according to the first embodiment displays the partial images PD1 to PD3 with the same size as the display screens of the electronic devices OB1 to OB3 viewed by the user. Therefore, the partial images PD are displayed without creating a sense of discomfort, and fatigue on the user caused by using the AR glasses 30 is reduced.

[0056] B. Second embodiment A second embodiment of the present invention will be described below. In the following description, for the sake of simplicity, the same components as those in the first embodiment will be denoted by the same reference numerals, and a description of their functions may be omitted. In addition, in the following description, for the sake of simplicity, differences between the second embodiment and the first embodiment will be mainly described.

[0057] FIG. 8 is a block diagram showing the configuration of the AR glasses 30 according to the second embodiment. In the second embodiment, the AR glasses 30 include a second imaging device 331 in addition to the configuration of the AR glasses 30 according to the first embodiment shown in FIG. 3. The second imaging device 331 has an imaging lens (not shown) on the surface of the rims 306, 307 that faces the user's eyes when the user wears the AR glasses 30. The second imaging device 331 captures an image including the user's eyes. The image captured by the second imaging device 331 is referred to as the second captured image. The second captured image generated by the second imaging device 331 is transmitted to the terminal device 20 via the communication device 323 as second captured image information. The second imaging device 331 repeatedly captures images at a predetermined imaging interval and transmits the generated second captured image information to the terminal device 20 each time an image is captured.

[0058] 9 is a block diagram showing the configuration of the terminal device 20 in the second embodiment. In the second embodiment, the processing device 206 of the terminal device 20 executes a program PG2 to function as an image acquisition unit 210, an object detection unit 212, a display control unit 214, and a tracking unit 216.

[0059] The image acquisition unit 210 acquires a first captured image PA of an area corresponding to the user's field of view, as in the first embodiment, and also acquires a second captured image capturing an area including the user's eyes. In this embodiment, the image acquisition unit 210 acquires first captured image information corresponding to the first captured image PA captured by the first imaging device 324 of the AR glasses 30, and second captured image information corresponding to the second captured image captured by the second imaging device 331 of the AR glasses 30.

[0060] The tracking unit 216 tracks the movement of the user's eyes. In this embodiment, the tracking unit 216 performs image analysis of the movement of the user's eyes in the second captured image PA to estimate the position in the first captured image PA where the user's gaze is directed. Specifically, the tracking unit 216 uses, for example, the inner corner of the eye as a reference point and estimates the position where the user's gaze is directed based on the position of the iris relative to the inner corner of the eye. For example, if the iris of the right eye is far from the inner corner of the eye, it can be estimated that the user's gaze is directed to the right. Also, if the inner corner of the right eye and the iris are close to each other, it can be estimated that the user's gaze is directed to the left. Note that the tracking unit 216 may track the movement of the user's eyes using, for example, a corneal reflex method.

[0061] The object detection unit 212 detects a first object appearing in the captured image based on the eye movement of the user tracked by the tracking unit 216. The first object is an object that the user is gazing at. That is, in the second embodiment, the predetermined object is the first object that the user is gazing at. The object detection unit 212 detects objects appearing in the first captured image PA, for example, by using image analysis. Then, the object detection unit 212 detects an object that overlaps with the position of the user's line of sight as the first object. Note that the object detection unit 212 may detect a specific object as the first object, for example, when the state in which the user's line of sight is directed at the specific object continues for a predetermined time (for example, 2 seconds) or more.

[0062] In the second embodiment, the display control unit 214 generates a partial image PT (e.g., PT1, PT2) representing the first object in the first captured image PA. The display control unit 214 displays the partial image PT at a position in the display area of ​​the AR glasses 30 where the user views the first object. As described above, the blue light cut filter F is attached to the AR glasses 30. Therefore, the visibility of the first object is reduced compared to a state where the user is not wearing the AR glasses 30. For this reason, the display control unit 214 generates a partial image PT representing the first object in the first captured image PA captured by the first imaging device 324. Then, the display control unit 214 displays the partial image PT at a position where the user views the first object. By displaying the partial image PT, the visibility of the first object is improved compared to when the first object is viewed through the AR glasses 30. In other words, the display system 10 makes it easier for the user to view the first object while reducing blue light.

[0063] 10 and 11 are diagrams showing an example of the real space R visually perceived by the user when the partial image PT is displayed. In FIGS. 10 and 11, the user's point of gaze (the point at which the gaze is directed) is indicated by the symbol TP. FIG. 10 shows a case where the first object is a window OB5. In this case, the partial image PT1 is displayed superimposed on the window OB5. The partial image PT1 is a portion of the first captured image PA shown in FIG. 5 that represents the window OB5. As described above, the color of the real space R captured in the first captured image PA is substantially the same as the color perceived by a user not wearing the AR glasses 30. By displaying the partial image PT1, the window OB5 is visually perceived in the same way as when the user is not wearing the AR glasses 30.

[0064] 11 shows a case where the first object is the display screen of the smartphone OB3. In this case, a partial image PT2 is displayed superimposed on the smartphone OB3. The partial image PT2 is a portion of the first captured image PA shown in FIG. 5 that represents the display screen of the smartphone OB3. By displaying the partial image PT2, the display screen of the smartphone OB3 is viewed in the same way as when the user is not wearing the AR glasses 30.

[0065] The display control unit 214 may display the partial image PT when the first object is a specific object. The specific object is, for example, the display screen of an electronic device. In this case, the display control unit 214 does not display the partial image PT1 when the first object is a window OB5, as shown in FIG. 10, and displays the partial image PT2 when the first object is a smartphone OB3, as shown in FIG. 11. By displaying the partial image PT only when the first object is a specific object, only objects for which the user particularly wants to increase visibility are displayed as partial images PT, thereby reducing the frequency with which the partial images PT are displayed.

[0066] In the second embodiment, the partial image PT may be displayed in the same size as the first object visually recognized by the user. In the second embodiment, the partial image PT may be displayed in the color of the night mode.

[0067] As described above, the display system 10 according to the second embodiment displays a partial image PT in which a first object that a user is gazing at is captured. By displaying the partial image PT in which the first object is captured, blue light generated around the user is reduced and the visibility of the first object for the user is improved.

[0068] C: Modified Example The following are variations of the above-described embodiment. Two or more variations arbitrarily selected from the following variations may be combined as appropriate within the scope of not mutually contradicting each other.

[0069] C1: First modified example In the first and second embodiments, the display control unit 214 displays the partial images PD and PT, each showing the entirety of a predetermined object, at a position on the lenses 310A and 310B where the user views the predetermined object. For example, in the first embodiment, the display control unit 214 displays the partial image PD, each showing the entire display screen of the electronic device, superimposed on a position where the user views the display screen. However, the display control unit 214 may also display, among the partial images PD and PT, an outer edge image PE (e.g., PE1 to PE3) indicating the outer edge of the predetermined object at a position on the lenses 310A and 310B where the user views the outer edge of the predetermined object.

[0070] FIG. 12 is a diagram showing an example of the real space R visually recognized by the user when an outer edge image PE is displayed. In FIG. 12, an outer edge image PE1 is displayed at a position where the user views the outer edge of the display screen of a television OB1. An outer edge image PE2 is displayed at a position where the user views the outer edge of the display screen of a laptop computer OB2. An outer edge image PE3 is displayed at a position where the user views the outer edge of the display screen of a smartphone OB3. By displaying the outer edge image PE, the user can grasp the position of the display screen of the electronic device, making it easier to grasp the content displayed on the display screen compared to when the outer edge image PE is not displayed. Furthermore, the outer edge image PE has a smaller image area than the partial image PD. Therefore, displaying the outer edge image PE places less strain on the user's eyes than when the partial image PD is displayed.

[0071] 12, the outer edge image PE is an image showing a predetermined range inside from the outer edge of the display screen of the electronic device. However, the outer edge image PE is not limited to this, and may be, for example, an image showing a predetermined range outside from the outer edge of the display screen of the electronic device, or an image showing a predetermined range including both the inside and outside of the outer edge of the display screen of the electronic device.

[0072] According to the first modification, the strain on the user's eyes is further reduced, and the visibility of a predetermined object is improved.

[0073] C2: Second modified example In the first and second embodiments, the partial images PD and PT were displayed regardless of the distance between the user and the predetermined object. However, the display of the partial images PD and PT may be switched on or off depending on the distance between the user and the predetermined object. Specifically, when the distance between the user and the predetermined object is less than a predetermined distance, the partial images PD and PT showing the predetermined object are displayed, and when the distance between the user and the predetermined object is equal to or greater than the predetermined distance, the partial images PD and PT showing the predetermined object may not be displayed. Generally, when the distance between the user and the predetermined object is large, the visibility of the predetermined object is low even when the user is not wearing the AR glasses 30. Therefore, even if the partial images PD and PT are displayed, the visibility of the predetermined object is minimally affected. In other words, by displaying the partial images PD and PT only when the distance between the user and the predetermined object is less than the predetermined distance, the effectiveness of displaying the partial images PD and PT can be improved.

[0074] In this case, the object detection unit 212 detects the distance between the user and the predetermined object. For example, if the size of the predetermined object is known, the object detection unit 212 may estimate the distance between the user (more specifically, the AR glasses 30 worn by the user) and the predetermined object based on the size of the predetermined object shown in the first captured image PA. Furthermore, if the AR glasses 30 are equipped with a LiDAR (Light Detection and Ranging) sensor, the object detection unit 212 may detect the distance between the user and the predetermined object using the detection value of the sensor.

[0075] When the distance between the user and the predetermined object is less than a predetermined distance, the display control unit 214 causes the partial images PD and PT to be displayed on the AR glasses 30. When the distance between the user and the predetermined object is equal to or greater than the predetermined distance, the display control unit 214 does not cause the partial images PD and PT to be displayed on the AR glasses 30.

[0076] The predetermined distance may be changed depending on, for example, the size of the object. For example, the display screen of the television OB1 shown in FIG. 5 is larger than the display screen of the smartphone OB3. Therefore, a user may view the display screen of the television OB1 from a position, for example, 3 meters away from the television OB1. On the other hand, even if the display screen of the smartphone OB3 is viewed from a position 3 meters away, it is considered that the content displayed on the display screen is barely discernible. Therefore, the display control unit 214 may increase the predetermined distance as the size of the predetermined object increases.

[0077] According to the second modified example, the partial images PD and PT are displayed only when the distance between the user and the predetermined object is less than a predetermined distance, thereby improving the effectiveness of displaying the partial images PD and PT.

[0078] C3: Third modified example In the first and second embodiments, the partial images PD and PT were displayed at the same size as the predetermined object as seen by the user. For example, in the first embodiment, partial images PD1 to PD3 were displayed at the same size as the display screen in the user's field of view. This is not limiting, and for example, the partial images PD and PT may be larger than the predetermined object as seen by the user. By displaying the partial images PD and PT larger than the predetermined object as seen by the user, the visibility of the partial images PD and PT can be improved, thereby improving user convenience. Furthermore, the ratio between the size of the predetermined object in the user's field of view and the display size of the partial images PD and PT may be set by the user.

[0079] C4: Fourth variant In the first and second embodiments, the blue light cut filters F were attached to the outer surfaces of the lenses 310A and 310B of the AR glasses 30. This is not a limitation, and the blue light cut filters F may be attached to the inner surfaces of the lenses 310A and 310B of the AR glasses 30. However, if the blue light cut filters F are attached to the inner surfaces of the lenses 310A and 310B of the AR glasses 30, the images (virtual objects including the partial images PD and PT) displayed by the projection device 321 become difficult to see. Because it is possible to reduce blue light by displaying the images displayed by the projection device 321 in night mode, for example, it is preferable to attach the blue light cut filters F to the outer surfaces of the lenses 310A and 310B, as described above.

[0080] C5: Fifth variant In the first and second embodiments, the AR glasses 30 and the terminal device 20 are separate entities. However, this is not limiting, and for example, the AR glasses 30 may have the functions of the terminal device 20. In other words, the processing device 328 of the AR glasses 30 may function as the image acquisition unit 210, the object detection unit 212, the display control unit 214, and the tracking unit 216.

[0081] According to the fifth modification, the terminal device 20 is not required, which is advantageous in simplifying the system configuration.

[0082] D:Other (1) Each function illustrated in Figures 3, 4, 8, and 9 is realized by any combination of hardware and software. There are no particular limitations on how each function is realized. Each function may be realized using a single device that is physically or logically coupled, or may be realized using a device that is configured by connecting two or more physically or logically separated devices directly or indirectly (for example, using wires, wirelessly, etc.). Each function may be realized by combining software with the single device or multiple devices.

[0083] (2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

[0084] (3) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the storage device 205 and the storage device 327 may be configured by at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Also, the program may be transmitted from a network via a telecommunications line.

[0085] (4) Each of the first embodiment, the second embodiment, and the first to fifth modifications may be implemented using any of the following standards: LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or a decimal point), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE The present invention may be applied to at least one of systems using 802.20, UWB (Ultra-Wideband), Bluetooth (registered trademark), or other appropriate systems, and next-generation systems that are extended, modified, created, or defined based on these systems. The present invention may also be applied to a combination of multiple systems (e.g., a combination of LTE and / or LTE-A with 5G).

[0086] (5) The order of the exemplary processing procedures, sequences, or flowcharts shown in the first embodiment, second embodiment, and first to fifth modifications may be changed as long as there is no contradiction. For example, the methods described herein present various step elements in an exemplary order and are not limited to the specific order presented.

[0087] (6) In each of the first embodiment, the second embodiment, and the first to fifth modifications, input and output information, etc. may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information, etc. may be overwritten, updated, or added. Output information, etc. may be deleted. Input information, etc. may be transmitted to another device.

[0088] (7) In each of the first embodiment, the second embodiment, and the first to fifth variants, the determination may be made based on a value represented by one bit (0 or 1), a boolean value (true or false), or a comparison of numerical values ​​(e.g., a comparison with a predetermined value).

[0089] (8) The programs exemplified in each of the first embodiment, second embodiment, and first to fifth modifications should be broadly construed to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, or functions, regardless of whether they are called software, firmware, middleware, microcode, hardware description languages, or by other names. Furthermore, software, instructions, or the like, may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, optical fiber cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared, microwave), at least one of these wired technology and wireless technology is included within the definition of a transmission medium.

[0090] (9) The information and the like described in each of the first embodiment, the second embodiment, and the first to fifth modifications may be represented using any of a variety of different technologies. For example, data, information, and the like that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms described in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0091] (10) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the terms "system" and "network" are used interchangeably.

[0092] (11) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the terminal device 20 may be a mobile station, which may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term.

[0093] (12) A mobile station may be referred to as a transmitting device, a receiving device, a communication device, or the like. A mobile station may be a device mounted on a mobile object, or the mobile object itself. A mobile object refers to an object that can move. A mobile object can move at any speed. A mobile object can be stopped. Examples of mobile objects include, but are not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcars, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and objects mounted thereon. A mobile object may also be an object that moves autonomously based on an operation command. A mobile object may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile object (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

[0094] (13) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the term "determining" or "determining" may encompass a wide variety of actions. "Determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching a table, database, or other data structure), ascertaining, and regarding that as a "determination." Also, "determining" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory) as a "judging" or "determining." Furthermore, "decision" can include the act of considering something to be "decided" such as resolving, selecting, choosing, establishing, or comparing. In other words, "decision" can include the act of considering something to be "decided" to be an action. "Decision" can also be interpreted as "assuming," "expecting," or "considering," among others.

[0095] (14) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the term "connected," or any variation thereof, refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more electric wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0096] (15) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0097] (16) As used herein, any reference to elements using designations such as "first" and "second" does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0098] (17) When the words "include," "including," and variations thereof are used in the first embodiment, second embodiment, and first to fifth modifications in this specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this specification or claims is not intended to mean an exclusive logical OR.

[0099] (18) Throughout this application, where articles are added by translation, such as a, an, and the in English, the disclosure may include the plural form of the noun following these articles.

[0100] (19) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended as an illustrative explanation and does not have any limiting meaning on the present invention. Furthermore, multiple embodiments selected from the embodiments exemplified in this specification may be combined. [Explanation of symbols]

[0101] 10...display system, 20...terminal device, 30...AR glasses, 201...touch panel, 203...communication device, 205...storage device, 206...processing device, 207...bus, 210...image acquisition unit, 212...object detection unit, 214...display control unit, 216...tracking unit, 321...projection device, 322...speaker, 323...communication device, 324...first imaging device, 327...storage device, 328...processing device, 329...bus, 330...operation control unit, 331...second imaging device, F...blue light cut filter, LEN...imaging lens, R...real space.

Claims

1. a display control unit that displays a virtual object in a display area of ​​a see-through head-mounted display; an image acquisition unit that acquires, from an imaging device, an imaged image including a range of real space visually recognized through the display area by a user wearing the see-through type head mounted display; an object detection unit that detects a predetermined object appearing in the captured image, the see-through type head mounted display is configured to reduce light of a predetermined wavelength band among light incident on the display area, the object detection unit detects a distance between the user and the predetermined object; The display control unit a partial image representing a part of the predetermined object in the captured image is displayed in a position in the display area where the predetermined object is visually recognized by the user; When the distance between the user and the predetermined object is less than a predetermined distance, the partial image is displayed in the display area; The predetermined distance is changed depending on the size of the predetermined object. Information processing device.

2. the predetermined object is a display screen of an electronic device, the predetermined wavelength band is equal to or greater than 380 nm and equal to or less than 500 nm, the object detection unit detects the display screen from the captured image; the partial image represents the display screen of the captured image, the display control unit displays the partial image at a position in the display area where the user views the display screen.

2. The information processing device according to claim 1.

3. the predetermined object is a first object that the user gazes at, A tracking unit that tracks eye movements of the user, the object detection unit detects the first object from the captured image based on the eye movement of the user tracked by the tracking unit; the partial image represents the first object in the captured image, the display control unit displays the partial image at a position in the display area where the user views the first object.

2. The information processing device according to claim 1.

4. the display control unit sets the color temperature of the white point of the partial image displayed in the display area to a predetermined temperature or lower.

2. The information processing device according to claim 1.

5. the partial image is an image having the same size as the size of the predetermined object visually recognized by the user, the display control unit causes the partial image to be displayed in the display area.

2. The information processing device according to claim 1.

6. the display control unit displays an outer edge image indicating an outer edge of the predetermined object among the partial images at a position in the display area where the user can view the outer edge of the predetermined object.

2. The information processing device according to claim 1.

Images