Display control device and display control method

Abstract

To accomplish both the visibility of a real space and that of a virtual object when a user wears a transmission-type wearable device.SOLUTION: An object determining unit 111 determines whether or not there is a stationary display 21 in the visual field of a user U who is wearing a wearable device 10. A display control unit 112 changes, based on the presence or absence of the stationary display 21 in the visual field of the user U, the display form of a virtual object VO projected on the wearable device 10 between a first display form that displays the virtual object VO based on a user coordinate system with reference to the position of the user U and a second display form that displays the virtual object VO based on a real-space coordinate system with reference to the predetermined position in a real space R.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a display control device and a display control method. [Background technology]

[0002] Conventionally, see-through wearable devices such as MR (Mixed Reality) glasses or AR (Augmented Reality) glasses have become widespread. See-through wearable devices display virtual objects superimposed on a scene in real space that passes through the lenses. Therefore, depending on the situation, visibility of the real space may be reduced, causing inconvenience. To address this issue, for example, Patent Document 1 listed below ensures visibility of an augmented reality image by moving the augmented reality image (virtual object) so that it does not come into contact with an object in real space (e.g., a person) when the object tries to come into contact with the augmented reality image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-47887 Summary of the Invention [Problem to be solved by the invention]

[0004] When see-through wearable devices are used on a daily basis, it is expected that users will, for example, work on a computer while wearing the see-through wearable device, or move around while looking at a smartphone while wearing the see-through wearable device. In the conventional technology described above, virtual objects are displayed so as to avoid objects in real space, but it is desirable to be able to change the display mode of virtual objects to suit various situations.

[0005] An object of the present disclosure is to achieve both visibility of real space and visibility of virtual objects when wearing a see-through wearable device. [Means for solving the problem]

[0006] A display control device according to a preferred embodiment of the present disclosure includes an object determination unit that determines whether a first object is present in the field of view of a user wearing a see-through wearable device, and a display control unit that switches the display mode of a virtual object projected onto the see-through wearable device between a first display mode that displays the virtual object based on a user coordinate system based on the position of the user, and a second display mode that displays the virtual object based on a real space coordinate system based on a predetermined position in real space, based on the presence or absence of the first object in the field of view of the user.

[0007] A display control method according to a preferred embodiment of the present disclosure determines whether a first object is present in the field of view of a user wearing a transparent wearable device, and based on the presence or absence of the first object in the user's field of view, switches the display mode of a virtual object projected onto the transparent wearable device between a first display mode in which the virtual object is displayed based on a user coordinate system based on the position of the user, and a second display mode in which the virtual object is displayed based on a real space coordinate system based on a predetermined position in real space. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to achieve both visibility of real space and visibility of virtual objects when wearing a see-through wearable device. [Brief explanation of the drawings]

[0009] [Figure 1] 1A and 1B are diagrams illustrating a wearable device 10 according to an embodiment in use. [Figure 2] FIG. 1 is a block diagram showing the configuration of a wearable device 10. [Figure 3] 10 is a schematic diagram showing the visual field range SR of a user U when there is no fixed display 21. FIG. [Figure 4] 10 is a schematic diagram showing the visual field range SR of a user U when there is no fixed display 21. FIG. [Figure 5] 10 is a schematic diagram showing the visual field range SR of a user U when there is no fixed display 21. FIG. [Figure 6] 1 is a schematic diagram showing a visual field range SR of a user U when a fixed display 21 is present. [Figure 7] 1 is a schematic diagram showing a visual field range SR of a user U when a fixed display 21 is present. [Figure 8] 1 is a schematic diagram showing a visual field range SR of a user U when a fixed display 21 is present. [Figure 9] 1 is a schematic diagram showing a visual field range SR of a user U when a moving display 31 is present. [Figure 10] 1 is a schematic diagram showing a visual field range SR of a user U when a moving display 31 is present. [Figure 11] 1 is a schematic diagram showing a visual field range SR of a user U when a moving display 31 is present. [Figure 12] 10 is a flowchart showing the operation of the processing device 108. [Figure 13] FIG. 10 is a block diagram showing the configuration of a wearable device 10A according to a first modified example. [Figure 14] FIG. 1 is a schematic diagram showing a display in 3Dof (Degree of Freedom). DETAILED DESCRIPTION OF THE INVENTION

[0010] A. Embodiment A-1. System Configuration Fig. 1 is a diagram schematically illustrating a state in which a wearable device 10 according to an embodiment is used. As shown in Fig. 1, the wearable device 10 is a see-through head-mounted display (AR glasses) worn on the head of a user U. The wearable device 10 is an example of a see-through wearable device.

[0011] In this embodiment, the user U can use another information processing terminal while wearing the wearable device 10. FIG. 1 shows a personal computer 20 and a smartphone 30 as examples of other information processing terminals. Generally, an information processing terminal is provided with a display for displaying information. For example, the personal computer 20 includes a fixed display 21 installed on a desk D. The smartphone 30 includes a mobile display 31 on its housing. It is assumed that the fixed display 21 will be used while remaining in the position where the fixed display 21 is installed. It is assumed that the mobile display 31 will be carried by the user U and used at any position.

[0012] The shape of the wearable device 10 is similar to that of ordinary eyeglasses, for example. The wearable device 10 has lenses L and a frame F that supports the lenses L. The lenses L include a left lens disposed in front of the left eye of the user U and a right lens disposed in front of the right eye of the user U. The left and right lenses L each have a half mirror. The half mirrors of the left and right lenses L transmit light representing real space R, thereby guiding the light representing real space R to the eyes of the user U. The half mirrors of the left and right lenses L also reflect light representing a virtual object VO, which is guided by an optical member, toward the eyes of the user U. The light of real space R that has passed through the half mirror and the light representing the virtual object VO that has been reflected by the half mirror are superimposed and enter the eyes of the user U, allowing the user U to simultaneously view the real space R and the virtual object VO.

[0013] In general, the virtual object VO displayed on the wearable device 10 is either a position-responsive object associated with a specific position in the real space R, or an instruction-responsive object displayed on the wearable device 10 based on an instruction from the user U. A position-responsive object is displayed at a specific position relative to a marker, such as a two-dimensional code, when the marker is read by a camera. Alternatively, a position-responsive object is associated with position information such as latitude and longitude, and is displayed when the position information of the wearable device 10 detected by a global positioning system (GPS) or the like matches the position information associated with the position-responsive object.

[0014] A position-based object is associated with a specific position in real space R. Therefore, a position-based object is displayed based on a real-space coordinate system that is based on a predetermined position in real space R. In other words, a position-based object is displayed using 6 DoF. 6 DoF corresponds to a total of six degrees of freedom: three axes that define the direction of movement around the user U's neck when moving their head, and three axes that define the direction of movement when the user U moves their body. In a display using 6 DoF, a virtual object VO is fixed to a specific position in real space R, so that the user U can, for example, get behind the virtual object VO or move the virtual object VO to another location. In this embodiment, displaying a virtual object VO using 6 DoF is referred to as a "second display mode."

[0015] On the other hand, the instruction-responsive object is, for example, each application screen when using a message application, a video viewing application, an internet browser, or an accessory application such as a calendar or calculator on the wearable device 10.

[0016] The instruction-responsive object is displayed, for example, with 0DoF or 3DoF. 0DoF (no degrees of freedom) indicates that the display position of the virtual object VO on the wearable device 10 is fixed regardless of the movement of the user U. In a 0DoF display, a specific virtual object VO is displayed at a specific location within the user U's field of view, and the position of the virtual object VO within the field of view is always maintained even if the user U moves their head or moves around. Content displayed with 0DoF is generally limited to small content to ensure visibility within the field of view. Specifically, this includes, for example, notification icons, clocks, or shortcuts to frequently used apps.

[0017] Figure 14 is a schematic diagram showing a display in 3Dof. 3Dof corresponds to three degrees of freedom on three axes that define the direction of movement around the neck when a user U moves their head. As shown in Figure 14, 3DoF is imagined as a multi-layered spherical display SD centered on the user U. The display SD does not necessarily have to be spherical, but it is often spherical to make the content easier to view. A virtual object VO is fixed to one of the layers of the multiple spheres. When the user U moves their body, the display SD to which the virtual object VO is fixed appears to follow them. Furthermore, when the user U moves their head, the virtual object VO positioned on the sphere in the direction of head movement is visible. Furthermore, a virtual object VO in the direction the user U was originally facing will disappear if it goes out of their field of view.

[0018] In the present embodiment, displaying a virtual object VO using 0DoF or 3DoF is referred to as a "first display mode." The first display mode may be referred to as a display mode in which the virtual object VO is displayed based on a user coordinate system based on the user's position. Also, 0DoF may be referred to as a display mode in which the virtual object VO is displayed based on a device coordinate system based on the position of the wearable device 10. In the present embodiment, since the positional relationship between the user U and the wearable device 10 does not change while the user U is wearing the wearable device 10, 0DoF and 3DoF are referred to as first display modes in which the virtual object VO is displayed based on a user coordinate system based on the user's position. Furthermore, in the present embodiment, unless otherwise specified, the virtual object VO displayed on the wearable device 10 is assumed to be an instruction-responsive object. That is, in the present embodiment, the virtual object VO is usually displayed using 0DoF or 3DoF (first display mode).

[0019] A-2. Configuration of Wearable Device 10 FIG. 2 is a block diagram showing the configuration of the wearable device 10. The wearable device 10 includes a projection device 101, a speaker 102, a microphone 103, an input device 104, a communication device 105, a first camera 106A, a second camera 106B, a storage device 107, a processing device 108, and a bus 110. Each component shown in FIG. 2 is stored in, for example, a frame F. The projection device 101, the speaker 102, the microphone 103, the input device 104, the communication device 105, the first camera 106A, the second camera 106B, the storage device 107, and the processing device 108 are interconnected by a bus 110 for communicating information. The bus 110 may be a single bus or may be multiple buses that are different for each element of the device.

[0020] The projection device 101 includes left and right lenses L, a display panel, and optical members. A pair of display panels and optical members may be provided, one on each side, corresponding to the left and right lenses L. The projection device 101 displays a projection image corresponding to a virtual object VO on the display panel based on control from the processing device 108. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The optical members guide light emitted from the display panel to the left and right lenses L.

[0021] The speaker 102 plays back audio data and outputs audio corresponding to the audio data. The audio data played back by the speaker 102 indicates, for example, the speech of a call partner when a user U makes a call using a voice call application. The microphone 103 collects surrounding audio and generates audio data. The audio collected by the microphone 103 is, for example, the speech of the user U when the user U makes a call using a voice call application. The speaker 102 and the microphone 103 may not be included in the wearable device 10 and may be separate from the wearable device 10. In this case, the wearable device 10 is connected to the speaker 102 and the microphone 103 by an interface not shown.

[0022] The input device 104 is an input device (for example, a keyboard, a mouse, a switch, a button, or a sensor) that accepts an input from the outside. In particular, in the wearable device 10, the input device 104 may be, for example, a sensor that detects the movement of the user U. For example, when the sensor detects that the user U has made a specific gesture, it may be recognized that a specific operation has been performed on the wearable device 10.

[0023] The communication device 105 includes a communication interface for communicating with other information processing devices. The communication device 105 connects to a communication network using wireless or wired communication, and communicates with other information processing devices via the communication network.

[0024] First camera 106A and second camera 106B generate images showing real space R. First camera 106A captures a first image that shows the range of real space R that is included in the field of view of user U. In this embodiment, for the sake of convenience, it is assumed that the range of real space R that is included in the field of view of user U matches the range that is shown in the first image. The first image is used to identify the range of real space R that is included in the field of view of user U. Furthermore, second camera 106B captures a second image that shows user U's eyes. The second image is used to estimate the direction of user U's gaze.

[0025] As described above, the first image captured by the first camera 106A includes a range of real space R that is included in the field of view of the user U. The user U views the real space R through the lens L. Therefore, the real space R within the range captured in the first image and the virtual object VO projected onto the lens L are visually recognized by the user U as overlapping. The correspondence between each pixel of the first image captured by the first camera 106A and each pixel of the transmissive display realized by the left and right lenses L is calibrated in advance. In other words, the position of an object captured in the first image on the left and right lenses L is known when viewed from the user U wearing the wearable device 10. Therefore, the display control unit 112, which will be described later, can project the virtual object VO, whose display position is determined based on the first image, onto the left and right lenses L.

[0026] As described above, the second image captured by the second camera 106B captures the eyes of the user U. The second image is used to detect the direction of the user U's gaze. Specifically, for example, a "reference point" that does not move even when the gaze moves and a "moving point" that moves when the gaze moves are detected from the image of the user U's eyes. The reference point may be, for example, the inner corner of the eye, and the moving point may be, for example, the iris. The gaze can be detected based on the position of the iris relative to the inner corner of the eye. For example, if the iris of the left eye is far from the inner corner of the eye, the user U is detected as looking left. If the inner corner of the left eye and the iris are close to each other, the user U is detected as looking right. Furthermore, for example, the correspondence between the gaze direction of the user U and the position on the lens L may be calibrated in advance. Specifically, for example, before starting work, a small moving virtual object VO is displayed on the lens L, and the user U is instructed to follow it with their eyes. A second image captured during this time (more specifically, the positional relationship between the reference point and the moving point in the second image) is associated with the display position of the virtual object VO on the lens L. This makes it possible to use the second image to accurately detect which part of the lens L the user U is looking at. In addition, the method for detecting the direction of the user U's line of sight may be a corneal reflex method using an infrared camera and an infrared LED.

[0027] The first camera 106A and the second camera 106B each have an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens. The imaging lens of the first camera 106A is arranged, for example, on the bridge described above, facing the outside world toward which the face of the user U is facing. The imaging lens of the second camera 106B is arranged, for example, near the eyes of the user U in the frame F described above, facing the direction of the eyes of the user U. Note that the second camera 106B may have two imaging lenses, one for each of the left and right eyes of the user U.

[0028] The imaging optical system may have various optical elements such as a prism, or may have a zoom lens, a focus lens, etc. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0029] The storage device 107 is a storage medium readable by the processing device 108. The storage device 107 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM). The storage device 107 stores a program PG1. The program PG1 is a program for operating the wearable device 10.

[0030] The processing device 108 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0031] The processing device 108 reads the program PG1 from the storage device 107. By executing the program PG1, the processing device 108 functions as an object determination unit 111, a display control unit 112, and a gaze determination unit 113. That is, by executing the program PG1, the processing device 108 functions as a display control device.

[0032] At least one of the object determination unit 111 and the display control unit 112 may be configured by a circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0033] A-3. Details of the processing device 108 Next, the object determination unit 111 and the display control unit 112, which are realized by the processing unit 108 executing the program PG1, will be described.

[0034] The object determination unit 111 determines whether or not a first object is present within the field of view of a user U wearing the wearable device 10. In this embodiment, the first object is placed at a predetermined position in real space R. "An object is placed at a predetermined position" means, for example, that the object is intended to be used while remaining at the predetermined position. "An object is placed at a predetermined position" may mean that the object is fixed so that it cannot be moved. "An object is placed at a predetermined position" may also mean that the object can be moved by applying a certain amount of force, but is not intended to be carried and used by the user U.

[0035] In this embodiment, the first object is a planar fixed display 21 installed at a predetermined position in real space R. More specifically, in this embodiment, the first object is the fixed display 21 (see FIG. 1) of a desktop personal computer 20. The fixed display 21 may be said to be an example of an electronic device.

[0036] The object determination unit 111 performs image analysis processing on the first image, for example, and determines that the first object is present within the field of view of the user U when the fixed display 21 appears in the first image.

[0037] Furthermore, in addition to image analysis processing, the object determination unit 111 may determine the presence or absence of a first object based on the movement of the user U. The movement of the user U mainly refers to whether the user U is moving. For example, when a tablet terminal is in the user U's field of view, there are cases where the user U uses the tablet terminal while keeping it in a predetermined position, such as by placing it on a desk D or using a stand, and cases where the user U uses the tablet terminal while holding it in their hand and moving. When the tablet terminal is in the user U's field of view and the user U is not moving (more specifically, when the state in which the user U is not moving continues for a predetermined time or more), it is considered that the user U uses the tablet terminal while keeping it in a predetermined position, and the tablet terminal can be determined to be the first object. On the other hand, when the tablet terminal is in the user U's field of view and the user U is moving, it is considered that the user U uses the tablet terminal while moving, and it can be determined that the tablet terminal is not the first object (it is the second object, which will be described later).

[0038] The movement of the user U can be determined, for example, by analyzing the first image. Specifically, if the amount of change per unit time in the range of the real space R shown in the first image is less than a predetermined amount, it may be determined that the user U is not moving, and if the amount of change is equal to or greater than the predetermined amount, it may be determined that the user U is moving. Furthermore, if the input device 104 is equipped with a sensor that detects the movement of the user U, it may be determined whether or not the user U is moving based on the detection result of the sensor.

[0039] Although a tablet terminal has been given as an example above, it may also be the case that a smartphone 30 or a laptop computer is used in the same way, and it may be determined whether the device is used while being placed in a predetermined position or whether the user U holds it in their hand and uses it while moving around.

[0040] Furthermore, the object determination unit 111 determines whether or not a second object held by the user U is within the field of view of the user U. The second object being held by the user U means, for example, that the user U is holding the second object in his / her hand. The second object is an object that is expected to be used by the user U while being held in his / her hand. In this embodiment, the second object is the movable display 31 (see FIG. 1 ), which is the display of the smartphone 30. The object determination unit 111 performs, for example, image analysis processing on the first image, and determines that the second object is within the field of view of the user U when the movable display 31 appears in the first image.

[0041] The display control unit 112 controls the display mode of the virtual object VO projected onto the wearable device 10 based on the type of object in the field of view of the user U. The display control unit 112 switches the display mode of the virtual object VO projected onto the wearable device 10 between a first display mode and a second display mode, for example, based on the presence or absence of a first object in the field of view of the user U. As described above, the first display mode displays the virtual object VO based on a user coordinate system based on the position of the user U. Furthermore, the second display mode displays the virtual object VO based on a real space coordinate system based on a predetermined position in the real space R. The predetermined position is, for example, a position where the fixed display 21 is installed. More specifically, the display control unit 112 displays the virtual object VO in the first display mode when the fixed display 21, which is the first object, is not in the field of view of the user U, and displays the virtual object VO in the second display mode when the fixed display 21 is in the field of view of the user U.

[0042] 3 to 5 are schematic diagrams showing the field of view SR of the user U when there is no fixed display 21. FIG. 3 shows the field of view SR1 when the user U is sitting on a chair C (see FIG. 1) and facing the wall WL1. The field of view SR1 includes a real space R including a desk D and the like, and virtual objects VO1 to VO3. The virtual object VO1 is a video viewing screen using a video viewing application. The virtual object VO2 is a message sending / receiving screen using a message application. The virtual object VO3 is a calendar image displayed by a calendar application. It is assumed that the virtual objects VO1 to VO3 are displayed in 3DoF in the first display mode.

[0043] In this embodiment, a reference plane is set for the virtual object VO. Since the virtual objects VO1 to VO3 illustrated in FIG. 3 and other figures are two-dimensional images (planar images), the plane that constitutes the virtual objects VO1 to VO3 itself serves as the reference plane. Furthermore, if the virtual object VO is a three-dimensional object, the front of the three-dimensional object is defined. The reference plane of the three-dimensional object may be, for example, a plane that is perpendicular to the direction of the front of the three-dimensional object and passes through the center point (center of gravity) of the three-dimensional object. In FIG. 3 and other figures, the outer edges of the virtual objects VO1 to VO3 are drawn with dotted lines in order to distinguish the virtual object VO from real objects placed in the real space R on the drawing.

[0044] 3, virtual objects VO1 to VO3 are placed at positions based on instructions from user U, for example. The positions based on instructions from user U are positions designated by user U in consideration of ease of use of each application, for example. More specifically, when sending and receiving messages using a messaging application, user U places the message sending and receiving screen (virtual object VO1) near the center of the field of view SR to improve the visibility of the messages. On the other hand, a calendar (virtual object VO3) displayed by a calendar application, for example, is static information that does not change, so user U places it at the edge of field of view SR and refers to it when necessary.

[0045] Alternatively, the virtual objects VO1 to VO3 may be displayed at positions designated by an application that is the source of displaying the virtual objects VO1 to VO3.

[0046] Hereinafter, the display positions of the virtual objects VO1 to VO3 shown in Fig. 3 will be referred to as "standard positions." The standard positions are, for example, positions based on an instruction from the user U, or positions specified by an application that is the display source of the virtual object VO.

[0047] As described above, when there is no fixed display 21, the virtual object VO is displayed in the first display mode (e.g., 3DoF). For example, if the user U moves only his / her head to the right from the state shown in FIG. 3, the range of the real space R that falls within the field of view of the user U changes to the right, as shown in the field of view SR2 in FIG. 4. Specifically, the wall WL2 that is adjacent to the wall WL1 also falls within the field of view SR2. When the user U moves only his / her head in 3DoF, the positional relationship between the user U and the spherical display SD does not change, and therefore the range of the virtual object VO that falls within the field of view of the user U changes. Specifically, in the field of view SR2 shown in FIG. 4, parts of the virtual objects VO1 and VO3 are not visible.

[0048] Furthermore, for example, if user U rotates chair C clockwise from the state shown in Fig. 3 and turns his / her entire body to the right without changing the position of his / her head, the range of real space R that falls within user U's field of view will be as shown by field of view range SR3 in Fig. 5. In Fig. 5, unlike Fig. 4, user U has rotated his / her entire body to the right, and therefore the spherical display SD also rotates in accordance with this rotation. Comparing Fig. 3 and Fig. 5, the positional relationship between user U and virtual objects VO1 to VO3 does not change, but the superimposition state of the objects in real space R and virtual object VO has changed.

[0049] When the virtual objects VO1 to VO3 are displayed with 0 DoF, the positions of the virtual objects VO1 to VO3 relative to the outer edge of the field of view SR do not change. Therefore, for example, when the user U moves only his / her head to the right from the state in which the virtual objects VO1 to VO3 are displayed as shown in Fig. 3, or when the user U rotates the chair C clockwise and turns his / her entire body to the right, the virtual objects VO1 to VO3 are displayed as shown in Fig. 5.

[0050] 6 to 8 are schematic diagrams showing the visual field range SR of a user U when there is a fixed display 21. As shown in Fig. 6, the fixed display 21 of a personal computer 20 is placed on a desk D. Like Fig. 3, Fig. 6 shows the visual field range SR4 when the user U is sitting on a chair C (see Fig. 1) and facing the wall WL1.

[0051] As described above, when the fixed display 21 is present, the virtual object VO is displayed in the second display mode (6DoF). As shown in Fig. 6, the display control unit 112 places the reference planes of the virtual objects VO1 to VO3 on the same plane as the display surface of the fixed display 21. This causes the information displayed on the fixed display 21 and the information indicated by the virtual objects VO1 to VO3 to be arranged on the same plane as in a multi-display, maintaining the visibility of both.

[0052] FIG. 7 is a diagram schematically illustrating a display area of ​​a virtual object VO relative to the fixed display 21. In this embodiment, the display control unit 112 places the virtual object VO in an area RD1 that is above a bottom edge L1 of the display surface of the fixed display 21 and does not overlap with the fixed display 21. In FIG. 7, the area RD1 is shown shaded. This improves the visibility of the virtual object VO compared to when the virtual object VO is displayed below the display surface of the fixed display 21 or when the virtual object VO is displayed overlapping with the display surface of the fixed display 21. Note that, for convenience of illustration, in FIG. 7, the bottom edge L1 of the display surface of the fixed display 21 and the bottom edge L2 of the area RD1 are slightly offset from each other, but in reality, the bottom edges L1 and L2 are at the same position.

[0053] Furthermore, the display control unit 112 may place the virtual object VO, for example, above the bottom edge L3 of the bezel of the fixed display 21 and in an area that does not overlap with the fixed display 21. This is because the difference between the position of the bottom edge L1 of the display surface and the position of the bottom edge L3 of the bezel makes virtually no difference in terms of visual recognition of the virtual object VO. Furthermore, in FIG. 7, the top, right, and left edges of the area RD1 are aligned with the outer edge of the field of view SR of the user U, but at least one of the top, right, and left edges of the area RD1 may be located inside the outer edge of the field of view SR.

[0054] Furthermore, depending on the setting of the user U (specifically, the setting of the distance between the user U and the virtual object VO in the first display mode), the virtual object VO may be placed farther away in the second display mode than in the first display mode, which may result in poor visibility. In this case, for example, the display size (font size, etc.) of the virtual object VO may be adjusted by the display control unit 112.

[0055] For example, when the user U rotates the chair C clockwise from the state shown in Fig. 6, the range of real space R that is within the field of view of the user U changes to the right, as shown in field of view SR5 in Fig. 8. On the other hand, in the second display mode, the positions of the virtual objects VO1 to VO3 relative to the objects in real space R do not change. Therefore, for example, the virtual object VO2 that is located to the left of the fixed display 21 in field of view SR4 shown in Fig. 6 does not fall within field of view SR5 and is not visible to the user U.

[0056] For example, when the fixed display 21 disappears from the field of view of the user U due to the user U moving (or when a predetermined time has passed since the fixed display 21 disappeared from the field of view of the user U), the display control unit 112 returns the display mode of the virtual objects VO1 to VO3 to the first display mode, and returns the display positions of the virtual objects VO1 to VO3 to the standard positions shown in FIG. 3.

[0057] Next, a description will be given of a case where the movable display 31 (see FIG. 1) is within the field of view of the user U. When the movable display 31 is within the field of view of the user U, the display control unit 112 maintains the first display mode and displays the virtual object VO at a position in the field of view of the user U that does not overlap with the object on the movable display 31.

[0058] The display mode of the virtual object VO when the movable display 31 is not present in the field of view SR of the user U is the same as the display mode when the fixed display 21 is not present in the field of view SR of the user U shown in Fig. 3. For example, in Fig. 3, the virtual object VO3 is displayed overlapping with the note N placed on the desk D.

[0059] 9 to 11 are schematic diagrams showing the visual field range SR of the user U when a moving display 31 is present. For example, assume that the user U takes out the smartphone 30 and starts using it, as shown in the visual field range SR5 in FIG. 9. The position of the moving display 31 of the smartphone 30 overlaps with the positions of the virtual objects VO1 and VO3 in FIG. 3. In this case, the display control unit 112 moves the virtual objects VO1 and VO3 to positions where they do not overlap with the moving display 31. In the example of FIG. 9, the virtual objects VO1 and VO3 have moved to the left compared to FIG. 3. Furthermore, if the position of another virtual object VO2 needs to be adjusted in conjunction with the movement of the virtual objects VO1 and VO3, the position of the other virtual object VO2 may also be moved.

[0060] When the movable display 31 is within the field of view of the user U, the first display form is maintained because the user U may carry the movable display 31 while using it. When the movable display 31 is carried and used, the position of the movable display 31 in the real space R changes. Therefore, it is not appropriate to display the virtual object VO in the second display form, but it is appropriate to display it in the first display form based on the user U.

[0061] FIG. 10 is a diagram schematically illustrating a display area of ​​a virtual object VO on the fixed display 21. The display control unit 112 places the virtual object VO in an area RD2 that does not overlap with the display surface of the movable display 31. In FIG. 10, the area RD2 is shown shaded. This improves the visibility of the virtual object VO compared to when the virtual object VO is displayed overlapping with the display surface of the movable display 31. Note that in FIG. 10, the top, bottom, right, and left edges (outer edges) of the area RD2 are aligned with the outer edge of the visual field range SR of the user U, but at least one of the top, bottom, right, and left edges of the area RD2 may be located inside the outer edge of the visual field range SR.

[0062] As described above, when the moving display 31 is within the field of view of the user U, the virtual object VO1 continues to be displayed in the first display mode. For example, when the user U turns his / her face to the right from the state shown in FIG. 9, the range of real space R in the field of view of the user U changes to the right, as shown in the field of view SR7 in FIG. 11. The display control unit 112 changes the display positions of the virtual objects VO1 to VO3 on the spherical display SD in accordance with the change in the position of the moving display 31 in the field of view of the user U. For example, in FIG. 11, the moving display 31 is located near the left end of the field of view SR7. The virtual objects VO1 to VO3 are displayed from the right side of the field of view SR7 to near the center, avoiding the moving display 31.

[0063] Alternatively, when the moving display 31 is within the field of view of the user U, the display control unit 112 may maintain the first display mode and hide the virtual object VO at a position that overlaps the moving display 31 in the field of view of the user U. For example, the position of the moving display 31 of the smartphone 30 in Fig. 9 overlaps the positions of the virtual objects VO1 and VO3 in Fig. 3. In this case, the display control unit 112 may hide the virtual objects VO1 and VO3.

[0064] Alternatively, depending on the number of virtual objects VO within the visual field range SR or the proportion of the moving display 31 occupying the visual field range SR, for example, when the virtual object VO is moved to a position where it does not overlap with an object on the moving display 31, the virtual object VO may not be fully displayed (the display may overlap, or the display may need to be reduced to fit). In such cases, the display control unit 112 may hide the display of part of the virtual object VO.

[0065] For example, when the user U stops using the smartphone 30 and the moving display 31 disappears from the user U's field of view (or when a predetermined time has passed since the moving display 31 disappeared from the user U's field of view), the display control unit 112 returns the display positions of the virtual objects VO1 to VO3 to the standard positions shown in Figure 3.

[0066] When the movable display 31 is not within the field of view of the user U, the display control unit 112 switches the display mode of the virtual object VO projected onto the wearable device 10 based on the presence or absence of an object on the fixed display 21 within the field of view of the user U. For example, when neither the movable display 31 nor the fixed display 21 is within the field of view of the user U, the display control unit 112 displays the virtual object VO in the first display mode.

[0067] A-4. Flowchart Fig. 12 is a flowchart showing the operation of the processing device 108. The processing shown in Fig. 12 is performed continuously while the user U is using the wearable device 10. It is assumed that the projection device 101 of the wearable device 10 is projecting at least one virtual object VO.

[0068] The processing device 108 acquires a first image from the first camera 106A (step S100). The processing device 108 functions as an object determination unit 111 and determines whether the fixed display 21 is captured in the first image, that is, whether the fixed display 21 is within the field of view of the user U (step S102).

[0069] If the fixed display 21 is within the field of view of the user U (step S102: YES), the processing device 108 functions as the display control unit 112 and displays the virtual object VO projected by the projection device 101 in the second display form and at a position that does not overlap with the fixed display 21 (indicated as "non-overlapping position" in the figure) (step S104), then returns to step S100 and repeats the subsequent processing.

[0070] If the fixed display 21 is not within the field of view of the user U (step S102: NO), the processing device 108 functions as an object determination unit 111 and determines whether the movable display 31 is present in the first image, i.e., whether the movable display 31 is within the field of view of the user U (step S106).

[0071] If the movable display 31 is within the field of view of the user U (step S106: YES), the processing device 108 functions as the display control unit 112, displays the virtual object VO projected by the projection device 101 in the first display form and at a position that does not overlap with the movable display 31 (indicated as "non-overlapping position" in the figure) (step S108), returns to step S100, and repeats the subsequent processing.

[0072] If the movable display 31 is not within the field of view of the user U (step S106: NO), the processing device 108 functions as the display control unit 112, displays the virtual object VO projected by the projection device 101 in the first display form and at the standard position (step S108), returns to step S100, and repeats the subsequent processing.

[0073] A-5. Summary of implementation As described above, the wearable device 10 according to the embodiment switches the display mode of the virtual object VO projected on the wearable device 10 between the first display mode and the second display mode based on whether the fixed display 21 is present in the field of view of the user U. In other words, the display mode of the virtual object VO is switched according to the object in the field of view of the user U. This improves visibility in the wearable device 10, where the user U simultaneously visually recognizes the object in the real space R and the virtual object VO. In particular, when the user U uses the wearable device 10 in combination with another information processing device (such as the personal computer 20 or the smartphone 30), the user U can perform work or the like without worrying about differences in the display modes of the information displayed on each device.

[0074] Furthermore, the wearable device 10 displays the virtual object VO in the first display mode when the fixed display 21 is not within the field of view of the user U, and displays the virtual object VO in the second display mode when the fixed display 21 is within the field of view of the user U. Therefore, the virtual object VO can be displayed based on the position of the fixed display 21, thereby improving the visibility of the fixed display 21 and the virtual object VO.

[0075] Furthermore, the wearable device 10 places the reference plane of the virtual object VO on the same plane as the display surface of the fixed display 21. This makes it easier to simultaneously view the information displayed on the fixed display 21 and the virtual object VO.

[0076] Furthermore, the wearable device 10 places the virtual object VO above the bottom edge L1 of the display surface of the fixed display 21 and in an area that does not overlap with the fixed display 21. This allows the range that the user U should pay attention to to be within a certain range, allowing for efficient information collection.

[0077] Furthermore, when the movable display 31 is within the field of view of the user U, the wearable device 10 maintains the first display mode and displays the virtual object VO at a position in the field of view of the user U that does not overlap with the movable display 31. Therefore, when the user U uses the wearable device 10 and the smartphone 30 together, the virtual object VO displayed on the wearable device 10 and the information displayed on the movable display 31 of the smartphone 30 can be viewed without interference.

[0078] Furthermore, when the movable display 31 is within the field of view of the user U, the wearable device 10 maintains the first display mode and hides the virtual object VO that overlaps with the movable display 31 in the field of view of the user U. Therefore, when the user U uses the wearable device 10 and the smartphone 30 together, the information displayed on the movable display 31 of the smartphone 30 can be seen without being hidden by the virtual object VO displayed on the wearable device 10.

[0079] B: Modified example The following are variations of the above-described embodiment. Two or more variations arbitrarily selected from the following variations may be combined as appropriate within the scope of not mutually contradicting each other.

[0080] B1: First modified example 13 is a block diagram showing the configuration of wearable device 10A according to Modification 1. In Modification 1, processing device 108 executes program PG2 stored in storage device 107, thereby functioning as gaze determination unit 113 and operation determination unit 114 in addition to object determination unit 111 and display control unit 112 described above.

[0081] When the fixed display 21 is within the field of view of the user U, the gaze determination unit 113 determines whether the user U is gazing at the fixed display 21. The gaze determination unit 113 detects the direction of the user U's gaze using, for example, the second image captured by the second camera 106B. When the direction of the user U's gaze is directed toward the fixed display 21 (when the user U's point of gaze is on the fixed display 21), the gaze determination unit 113 determines that the user U is gazing at the fixed display 21. Furthermore, when the direction of the user U's gaze is not directed toward the fixed display 21 (when the user U's point of gaze is not on the fixed display 21), the gaze determination unit 113 determines that the user U is not gazing at the fixed display 21.

[0082] When the user U is not gazing at the fixed display 21, the display control unit 112 maintains the first display mode even if the fixed display 21 is within the field of view of the user U. In other words, even if the fixed display 21 is within the field of view of the user U, if the user U is not gazing at the fixed display 21, it is considered that the user U is not using the fixed display 21. In this case, the display control unit 112 displays the virtual object VO in the same way as when the fixed display 21 is not within the field of view of the user U. This prevents the display mode of the virtual object VO from being changed unnecessarily, and improves the visibility of the virtual object VO.

[0083] Note that the gaze determination unit 113 may similarly determine whether or not the user U is gazing at the moving display 31 when the moving display 31 is within the field of view of the user U. In this case, when the user U is not gazing at the moving display 31, the display control unit 112 allows the virtual object VO to be displayed at a position overlapping the moving display 31.

[0084] The operation determination unit 114 determines whether the fixed display 21, which is an electronic device, is in an operating state or a stopped state. For example, the operation determination unit 114 performs image analysis on the first image, and determines that the fixed display 21 is in an operating state if the brightness of the display surface of the fixed display 21 is equal to or greater than a predetermined value, and determines that the fixed display 21 is in a stopped state if the brightness is less than the predetermined value. Furthermore, when the fixed display 21 and the wearable device 10 can communicate with each other, the operation determination unit 114 may receive a signal indicating whether the fixed display 21 is in an operating state from the fixed display 21 via communication.

[0085] When the fixed display 21 is in a stopped state, the display control unit 112 maintains the first display mode even if the fixed display 21 is within the field of view of the user U. In other words, even if the fixed display 21 is within the field of view of the user U, if the fixed display 21 is stopped, it is considered that the user U is not using the fixed display 21. In this case, the display control unit 112 displays the virtual object VO in the same way as when the fixed display 21 is not within the field of view of the user U. This prevents the display mode of the virtual object VO from being changed unnecessarily, and improves the visibility of the virtual object VO.

[0086] The gaze determination unit 113 may also determine whether the moving display 31 is in an operating state or a stopped state when the moving display 31 is within the field of view of the user U. In this case, when the moving display 31 is in a stopped state, the display control unit 112 allows the virtual object VO to be displayed at a position overlapping the moving display 31.

[0087] According to the first modification, when the user U is not paying attention to the fixed display 21, the display mode of the virtual object VO is not switched even if the fixed display 21 is within the field of view of the user U. This prevents unnecessary switching of the display mode and improves the visibility of the virtual object VO.

[0088] In the above description, the processing device 108 functions as both the gaze determination unit 113 and the operation determination unit 114, but the processing device 108 may function as either the gaze determination unit 113 or the operation determination unit 114.

[0089] B2: Second variant In the above description, the virtual object VO is a two-dimensional image. However, the virtual object VO may be a three-dimensional object. If the fixed display 21 is within the field of view of the user U, for example, the reference plane of the three-dimensional object may be located on the same plane as the fixed display 21.

[0090] C:Other (1) In the above-described embodiment, ROM and RAM are exemplified as storage device 107, but storage device 107 may also be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disc), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium.

[0091] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0092] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0093] (4) In the above-described embodiment, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0094] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0095] (6) Each function illustrated in FIG. 2 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0096] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0097] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0098] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0099] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0100] (10) In the above-described embodiments, the portable device may be a mobile station (MS). Those skilled in the art may also refer to a mobile station as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term. In this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0101] (11) In the above-described embodiments, the terms "connected," "coupled," or any variation thereof refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0102] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0103] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" can include regarding some action as having been "judgment" or "decision." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0104] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0105] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0106] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0107] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information). [Explanation of symbols]

[0108] 10, 10A...wearable device, 20...personal computer, 21...fixed display, 30...smartphone, 31...mobile display, 101...projection device, 102...speaker, 103...microphone, 104...input device, 105...communication device, 106A...first camera, 106B...second camera, 107...storage device, 108...processing device, 111...object determination unit, 112...display control unit, 113...gaze determination unit, 114...operation determination unit, R...real space, U...user, VO (VO1 to VO3)...virtual object.

Claims

1. an object determination unit that determines whether a first object is present within a visual field of a user wearing the see-through wearable device; a display control unit that switches a display mode of a virtual object projected onto the see-through wearable device between a first display mode in which the virtual object is displayed based on a user coordinate system based on a position of the user, and a second display mode in which the virtual object is displayed based on a real space coordinate system based on a predetermined position in real space, based on the presence or absence of the first object within the user's field of view; A display control device comprising:

2. the first object is placed at the predetermined position in the real space, the display control unit displays the virtual object in the first display form when the first object is not within the user's field of view, and displays the virtual object in the second display form when the first object is within the user's field of view. The display control device according to claim 1.

3. the object determination unit determines whether a second object held by the user is within a visual field of the user; the display control unit maintains the first display mode when the second object is present within the user's field of view, and displays the virtual object at a position in the user's field of view that does not overlap with the second object.

3. The display control device according to claim 2.

4. the object determination unit determines whether a second object held by the user is within a visual field of the user; the display control unit maintains the first display mode when the second object is present within the user's field of view, and hides the virtual object at a position overlapping the second object in the user's field of view.

3. The display control device according to claim 2.

5. a gaze determination unit that determines whether the user is gazing at the first object when the first object is within the user's visual field, the display control unit maintains the first display mode when the user is not gazing at the first object even if the first object is within the user's field of view. The display control device according to claim 1.

6. the first object is an electronic device; further comprising an operation determination unit that determines whether the electronic device is in an operating state or a stopped state; the display control unit maintains the first display mode when the electronic device is in a stopped state even if the first object is within the user's field of view. The display control device according to claim 1.

7. the first object is a flat, fixed display installed at the predetermined position in the real space, a reference plane is set for the virtual object, the display control unit places the reference surface of the virtual object on the same plane as a display surface of the fixed display; The display control device according to claim 1.

8. the display control unit places the virtual object in a region above a lower end of the display surface of the fixed display and not overlapping with the fixed display; The display control device according to claim 7.

9. The virtual object is displayed on the see-through wearable device based on an instruction from the user. The display control device according to claim 1.

10. determining whether a first object is present within a visual field of a user wearing the see-through wearable device; based on the presence or absence of the first object within the user's field of view, switching a display mode of the virtual object projected onto the see-through wearable device between a first display mode in which the virtual object is displayed based on a user coordinate system with respect to the position of the user, and a second display mode in which the virtual object is displayed based on a real space coordinate system with respect to a predetermined position in real space; Display control method.

Images