Display control device
The display control device addresses the physical burden and accuracy issues in existing HMD technologies by using gaze information to determine virtual object selection, allowing users to choose objects with reduced physical strain and improved precision.
Patent Information
- Application Number
- JP2023161511
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-09-25
- Publication Date
- 2025-04-07
AI Technical Summary
Existing HMD technologies, such as those described in Patent Document 1, impose a heavy physical burden on users due to the need to maintain head orientation for extended periods and suffer from inaccurate virtual object selection when head movement is paused.
A display control device comprising a display control unit, a first acquisition unit for gaze information, a second acquisition unit for input device instructions, and a decision unit that determines virtual object selection based on the relationship between gaze information and virtual object positions, reducing the need for prolonged head fixation and improving selection accuracy.
The proposed solution reduces the physical burden on users while enabling accurate selection of virtual objects, as users can select objects by shifting their gaze rather than maintaining head orientation or relying on potentially misaligned ray operations.
Smart Images
Figure 2025052678000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a display control device. [Background technology]
[0002] Head-mounted display (HMD) devices worn on a person's head are widely used. One example of an HMD device is a glass device such as an augmented reality (AR) glass. In the AR glass, an image of a virtual object that does not exist in real space is displayed superimposed on the real space. A user of the glass device can obtain information corresponding to a selected virtual object by selecting a virtual object displayed on the glass device. Patent Document 1 discloses a technology for moving a pointer for selecting a virtual object according to the movement of the head of a user wearing a glass device on the head. In this technology, a virtual object on which the pointer has stayed for a predetermined time or more is selected as an operation target.
[0003] Ray manipulation is also known as an operation for selecting a virtual object. In ray manipulation, a ray is displayed in a virtual space and aimed at a virtual object to be selected. One example of ray manipulation in a glass device is a method of aiming a ray at a virtual object to be selected by changing the direction of the ray according to the attitude of a smartphone connected to the glass device, and confirming the selection in this state. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2023-87303 A Summary of the Invention [Problem to be solved by the invention]
[0005] The technology disclosed in Patent Document 1 requires the user to fix the direction of the head for a predetermined period of time or more since the pointer needs to remain on the virtual object to be selected. Therefore, the technology disclosed in Patent Document 1 has a problem that the physical burden on the user is large. In addition, when the user stops moving his / her head for a predetermined period of time, the technology disclosed in Patent Document 1 selects a virtual object that the pointer happens to be overlapping with, regardless of the user's intention. Therefore, the technology disclosed in Patent Document 1 has a problem that the virtual object cannot be accurately selected.
[0006] In addition, ray manipulation using a smartphone has the problem that the smartphone is too heavy to be used as a pointing device, which places a large physical burden on the user. In addition, when performing ray manipulation using a smartphone with one hand, the direction of the light beam may shift when confirming a selection. For this reason, ray manipulation using a smartphone has the problem that virtual objects cannot be selected accurately. [Means for solving the problem]
[0007] A display control device according to a preferred embodiment of the present disclosure includes a display control unit, a first acquisition unit, a second acquisition unit, and a determination unit. The display control unit displays a virtual space including a plurality of virtual objects on a display disposed in front of a user's eyeball. The first acquisition unit acquires gaze information related to the user's gaze. The second acquisition unit acquires first instruction information indicating an instruction from an input device that accepts an instruction from the user to change a selected virtual object. When the determination unit acquires the first instruction information, the determination unit determines a virtual object to be selected by the user based on a relationship between the gaze information and each position of the plurality of virtual objects. Effect of the Invention
[0008] According to the present disclosure, it is possible to reduce the physical burden on a user while accurately selecting a virtual object. [Brief description of the drawings]
[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of a display system 1 including a mobile device 10 according to an embodiment of a display control device of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of a virtual space in the present embodiment. [Diagram 3] 10A to 10C are diagrams for explaining changes in the display image of the glass device 20 caused by the user changing the direction of his or her head. [Figure 4] 1 is a diagram showing an example of the configuration of a mobile device 10. FIG. [Diagram 5] 11 is a diagram for explaining a first position SP and a mark M indicating the first position SP. FIG. [Figure 6] 11A to 11C are diagrams illustrating a change in the display mode of a virtual object determined as an operation target. [Figure 7] 10 is a flowchart showing the flow of a display control method executed by the processing device 140 of the mobile device 10 in accordance with the program PRA. [Figure 8] FIG. 13 is a diagram for explaining a fourth modified example. [Figure 9] FIG. 13 is a diagram for explaining a modified example (5). [Figure 10A] FIG. 13 is a diagram for explaining a sixth modification. [Figure 10B] FIG. 13 is a diagram for explaining another example of the modified example (6). [Figure 11] FIG. 13 is a diagram for explaining a modified example (7). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] (A. Embodiment) FIG. 1 is a block diagram showing a configuration example of a display system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the display system 1 includes a mobile device 10, a glass device 20, and an input device 30. The glass device 20 is connected to the mobile device 10 by wire. The mobile device 10 and the glass device 20 may be connected wirelessly. The input device 30 is connected to the mobile device 10 by wireless. The mobile device 10 communicates with a management device 40 via a communication line NW. The management device 40 is a server device that provides a virtual space service. In the virtual space service in this embodiment, a virtual space in which a plurality of virtual objects are arranged is displayed on the glass device 20 in a superimposed manner on a real space, and information is provided according to a virtual object selected by a user wearing the glass device 20.
[0011] Fig. 2 is a schematic diagram of a virtual space displayed by the glass device 20. As shown in Fig. 2, virtual objects VO1 to VO5 representing various contents such as a browser, a cloud service, an image, and a video are arranged in this virtual space. In this embodiment, each of the virtual objects VO1 to VO5 is arranged on a spherical surface R of a predetermined radius centered on the position of the user's head.
[0012] The management device 40 manages virtual objects and position information for arranging the virtual objects in the virtual space for each user who uses the virtual space service. When the management device 40 receives a login request to the virtual space service from the mobile device 10, it transmits the virtual object and position information corresponding to the user to the mobile device 10. The position information indicates the initial position of the virtual object in the virtual space. The initial position of the virtual object in the virtual space is a position on a sphere R of a predetermined radius centered on the position of the user's head.
[0013] The glass device 20 is an HMD device worn on the head of a user. The glass device 20 in this embodiment is an AR glass that does not block the field of view of a user wearing the glass device 20 on the head. As shown in FIG. 1, the appearance of the glass device 20 is similar to that of general glasses, and includes temples 91 and 92, a bridge 93, body parts 94 and 95, a touch sensor 96, and lenses 41L and 41R. Each of the lenses 41L and 41R is provided with a half mirror. The body part 94 is provided with a liquid crystal panel or an organic EL panel for the left eye (hereinafter, collectively referred to as a display panel), a touch sensor 96 that accepts a touch operation by a user, and an optical member that guides light emitted from the display panel for the left eye to the lens 41L. The half mirror provided on the lens 41L transmits light from the outside world to guide it to the left eye, and reflects the light guided by the optical member to make it enter the left eye. The body 95 is provided with a display panel for the right eye and an optical member that guides light emitted from the display panel for the right eye to the lens 41R. The half mirror provided on the lens 41R transmits external light and guides it to the right eye, and reflects the light guided by the optical member so that it enters the right eye. The lenses 41L and 41R are an example of a display placed in front of the user's eyeballs.
[0014] A user wearing the glass device 20 on his / her head can observe an image on the display panel in a see-through state in which the image is superimposed on the outside world. In addition, in the glass device 20, among the binocular images with parallax, an image for the left eye is displayed on the display panel for the left eye, and an image for the right eye is displayed on the display panel for the right eye, so that the user can perceive the displayed image as if it has depth and a three-dimensional effect.
[0015] Although detailed illustration is omitted in FIG. 1, the glass device 20 includes an inertial sensor and a gaze detection device. The inertial sensor outputs attitude information indicating the orientation of the glass device 20 to the mobile device 10. The gaze detection device outputs gaze information indicating the direction of the gaze of the user wearing the glass device 20 on his / her head to the mobile device 10. The gaze detection device in this embodiment is a device that detects the direction of the user's gaze by eye gaze. In eye gaze, the direction in which the user is looking, i.e., the direction of the gaze, is detected by eye tracking that tracks the movement of the user's left and right eyes.
[0016] The mobile device 10 is, for example, a smartphone. The mobile device 10 is attached to the user's body in the same manner as the glass device 20. For example, the mobile device 10 is attached to the user's body by hanging it from the neck using a strap or the like. When the mobile device 10 is instructed to start using the virtual space service, the mobile device 10 communicates with the management device 40 to acquire virtual object and position information. In addition, the mobile device 10 communicates with the glass device 20 to acquire posture information and line of sight information. The mobile device 10 generates an image representing a virtual space corresponding to the user's field of view based on the virtual object and position information acquired from the management device 40 and the posture information acquired from the glass device 20, and transmits the generated image data to the glass device 20. In addition, the mobile device 10 updates the image data based on the posture information acquired from the glass device 20. For example, when the user turns his head to the left by about 60 degrees, the image data generated by the mobile device 10 is updated from the image data representing the virtual space shown in FIG. 2 to the image data representing the virtual space shown in FIG. 3.
[0017] The input device 30 is worn on the body of the user of the glass device 20. In this embodiment, the input device 30 is worn on the finger or arm of the user of the glass device 20. The input device 30 in this embodiment is a ring mouse. The input device 30 has an operator 300 for receiving an input operation by the user's finger. By pressing the operator 300, the user can perform a start operation to instruct the start of use of the virtual space service, an end operation to instruct the end of use of the virtual space service, and a decision operation to instruct the decision of a virtual object to be selected. An example of the start operation is an operation of pressing the operator 300 twice in succession. An example of the end operation is a long press operation to continue pressing the operator 300 for a predetermined time. An example of the decision operation is an operation of pressing the operator 300 once. The input device 30 transmits operation content data indicating the operation performed on the operator 300 to the mobile device 10. A confirmation operation performed in a state in which any virtual object is already selected is an operation to instruct a change of the selected virtual object. The operation content data indicating the confirmation operation is an example of first instruction information in the present disclosure.
[0018] Fig. 4 is a block diagram showing a configuration example of a mobile device 10 according to an embodiment of the present disclosure. As shown in Fig. 4, the mobile device 10 includes a first communication device 100, a second communication device 110, a third communication device 120, a storage device 130, a processing device 140, and a bus 150. The first communication device 100, the second communication device 110, the third communication device 120, the storage device 130, and the processing device 140 are connected to each other by the bus 150 that mediates data transmission and reception. The bus 150 may be configured using a single bus, or may be configured using different buses between each element.
[0019] The first communication device 100 is hardware (transmission / reception device) for communicating with the management device 40 via the communication line NW. The first communication device 100 is also called, for example, a network device, a network controller, a network card, or a communication module.
[0020] The second communication device 110 is a communication module for communicating with the glass device 20 via a wired connection. The second communication device 110 supplies the posture information and gaze information received from the glass device 20 to the processing device 140. The second communication device 110 also transmits image data provided by the processing device 140 to the glass device 20.
[0021] The third communication device 120 is a communication module for wirelessly communicating with the input device 30. The third communication device 120 supplies the operation content data received from the input device 30 to the processing device 140. Note that the glass device 20 may be modified to communicate with the third communication device 120 wirelessly, and in this embodiment, the second communication device 110 may be omitted. Similarly, the input device 30 may be modified to communicate with the second communication device 110 by wire, and in this embodiment, the third communication device 120 may be omitted.
[0022] The storage device 130 is a recording medium readable by the processing device 140. The storage device 130 may be configured with at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. A program PRA is pre-stored in the storage device 130.
[0023] The processing device 140 includes one or more central processing units (CPUs). When the processing device 140 receives operation content data indicating a start operation from the input device 30, the processing device 140 reads the program PRA from the storage device 130 and starts executing the program PRA. The processing device 140 operating according to the program PRA functions as the display control unit 140a, the first acquisition unit 140b, the second acquisition unit 140c, the calculation unit 140d, and the determination unit 140e shown in FIG. 4. The processing device 140 is an example of a display control device of the present disclosure. The display control unit 140a, the first acquisition unit 140b, the second acquisition unit 140c, the calculation unit 140d, and the determination unit 140e shown in FIG. 4 are software modules realized by operating a computer such as a CPU according to software such as a program. The functions of the display control unit 140a, the first acquisition unit 140b, the second acquisition unit 140c, the calculation unit 140d, and the determination unit 140e are as follows.
[0024] The first acquisition unit 140b acquires posture information and gaze information by communicating with the glass device 20 using the second communication device 110. The second acquisition unit 140c acquires operation content data by communicating with the input device 30 using the third communication device 120.
[0025] Calculation unit 140d calculates a first position SP, which is the position of the intersection between a straight line extending from the position of the user's head in the direction of the line of sight indicated by the line of sight information and spherical surface R, based on the line of sight information.
[0026] The display control unit 140a acquires virtual objects and position information by communicating with the management device 40 using the first communication device 100. The display control unit 140a generates image data representing an image of a virtual space based on the virtual objects, position information, and posture information. In this embodiment, the display control unit 140a generates image data representing an image in which a mark M indicating the first position SP is placed at the first position SP1 in addition to the virtual objects located within the field of view, as shown in FIG. 5. Then, the display control unit 140a transmits the generated image data to the glass device 20 using the second communication device 110, thereby displaying the image of the virtual space represented by the image data on the lenses 41R and 41L of the glass device 20.
[0027] When the determination unit 140e acquires operation content data indicating a confirm operation using the third communication device 120, the determination unit 140e determines a virtual object to be selected by the user based on the relationship between the line of sight of the user indicated by the line of sight information and the positions of the multiple virtual objects arranged in the virtual space. In the present embodiment, the determination unit 140e calculates the distance from the first position SP for each of the multiple virtual objects. Then, the determination unit 140e determines the virtual object having the shortest distance from the first position SP1 as the virtual object to be selected by the user. For example, as shown in FIG. 5, when the operation content data indicating a confirm operation is acquired under a situation in which the virtual object having the shortest distance from the first position SP among the virtual objects VO1 to VO5 is the virtual object VO3, the determination unit 140e determines the virtual object to be selected by the user as the virtual object VO3. Note that when the virtual object to be selected by the user is determined by the determination unit 140e, the display control unit 140a may change the display mode of the determined virtual object from that before the determination. Specifically, as shown in Fig. 6, a frame CL may be added to the determined virtual object VO3, and the display color of the virtual object VO3 may be changed from that before the determination. Alternatively, the size of the virtual object VO3 may be increased, or the virtual object VO3 may be deformed. In this way, by changing the display mode of the selected virtual object before and after the determination, the user can recognize that the selected virtual object has been changed.
[0028] According to the above operations, when a user wants to change a selected virtual object, the user only needs to direct his / her gaze in the direction of the target virtual object and press down the operator 300 of the input device 30. Moreover, the gaze does not need to be aimed at the target virtual object, but only needs to be positioned in the vicinity of the target virtual object. Therefore, unlike ray operation, it is not necessary to execute an operation to confirm the selected virtual object while aiming a ray at the target virtual object. Therefore, according to this embodiment, it is possible to accurately select a virtual object while reducing the physical burden on the user.
[0029] Moreover, the processing device 140 operating according to the program PRA executes a display control method shown in Fig. 7. As shown in Fig. 7, this display control method includes each process of steps SA100 to SA160. The process contents of each process of steps SA100 to SA160 are as follows.
[0030] In step SA100, the processing device 140 functions as a display control unit 140a. In step SA100, the processing device 140 communicates with the management device 40 to obtain virtual objects and position information.
[0031] In step SA110 following step SA100, the processing device 140 functions as the first acquisition unit 140b. In step SA110, the processing device 140 communicates with the glass device 20 to acquire posture information and gaze information.
[0032] In step SA120 following step SA110, the processing device 140 functions as the calculation section 140d. In step SA110, the processing device 140 calculates the first position SP based on the line-of-sight information.
[0033] In step SA130 following step SA120, the processing device 140 functions as a display control unit 140a. In step SA130, the processing device 140 generates image data representing a virtual space corresponding to the user's field of view based on the virtual object and position information acquired in step SA100 and the posture information and line of sight information acquired in step SA110. Then, the processing device 140 displays the image of the virtual space on the lenses 41R and 41L of the glass device 20 by transmitting the generated image data to the glass device 20 via the second communication device 110.
[0034] In step SA140 following step SA130, the processing device 140 functions as the second acquisition unit 140c and acquires operation content data. Then, in step SA130, the processing device 140 judges whether the acquired operation content data indicates a decision operation. If operation content data indicating a decision operation is acquired, the judgment result of step SA140 is "Yes". If operation content data other than operation content data indicating a decision operation (for example, operation content data indicating an end operation) is acquired, the judgment result of step SA140 is "No". As shown in FIG. 7, if the judgment result of step SA140 is "Yes", the processing device 140 executes the process of step SA150, and then executes the processes after step SA110 again. If the judgment result of step SA140 is "No", the processing device 140 executes the process of step SA160.
[0035] In step SA150, the processing device 140 functions as the determination unit 140e. In step SA140, the processing device 140 determines a virtual object to be selected by the user from among the multiple virtual objects displayed on the glass device 20 based on the line-of-sight information acquired in step SA110. The processing device 140 determines one virtual object based on the relationship between each position of the multiple virtual objects displayed on the glass device 20 and the line of sight indicated by the line-of-sight information. Specifically, the processing device 140 determines the virtual object having the shortest distance from the first position SP calculated based on the line-of-sight information to be the virtual object to be selected by the user. The distance is the length from the point of the virtual object closest to the first position SP to the first position SP. Note that the distance may be the length from the center of the virtual object to the first position SP.
[0036] In step SA160, the processing device 140 functions as the second acquisition unit 140c and acquires operation content data. Then, in step SA150, the processing device 140 judges whether the acquired operation content data indicates a termination operation. If operation content data indicating a termination operation is acquired, the judgment result of step SA160 is "Yes". If the judgment result of step SA160 is "Yes", the processing device 140 terminates this display control method. If the judgment result of step SA160 is "No", the processing device 140 executes the process of step SA110 again.
[0037] Since the input device 30 is smaller and lighter than the portable device 10, according to this embodiment, the physical burden on the user is reduced compared to the aspect of performing ray operation using the portable device 10. Moreover, this embodiment does not require displaying a ray in the virtual space to select a virtual object, and is suitable for use outdoors. Moreover, according to this embodiment, at the time when the user performs a determination operation on the input device 30, the virtual object that is the shortest distance from the first position SP determined according to the line of sight information is determined as the operation target, so that the virtual object is not determined regardless of the user's intention. According to this embodiment, it is possible to accurately select a virtual object while reducing the physical burden on the user. In addition, it is sufficient for the input device 30 to detect the touch operation of the user. Therefore, the input device 30 can be configured simply and inexpensively compared to an input device equipped with an inertial measurement device to support ray operation.
[0038] (B: Transformation) The present disclosure is not limited to the above-described exemplary embodiments. Specific modified aspects are as follows. Two or more aspects selected from the following examples may be combined. (B-1: Variation 1) The glass device 20 in the above embodiment is an AR glass, but may be a goggle used to realize VR (Virtual Reality). Hereinafter, the goggle used to realize VR is referred to as a VR goggle. The VR goggle is different from the AR glass in that it blocks the user's field of vision. However, the VR goggle is common to the AR glass in that it has a display placed in front of the user's eyeball and displays a virtual space in which a plurality of virtual objects are placed on the display placed in front of the user's eyeball. Therefore, by applying the display control device of the present disclosure to the display control of the VR glass, the same effect as the above embodiment is achieved. The VR goggle may be a video see-through type.
[0039] (B-2: Variation 2) The position where the virtual object is placed is not limited to the spherical surface R, and may be any position in the virtual space. In an aspect where each of the multiple virtual objects is placed at an arbitrary position in the virtual space, the position of the intersection of a perpendicular line from the virtual object to a straight line along the line of sight indicated by the line of sight information and the straight line may be set as the first position SP. Therefore, the first position SP may be different for each virtual object. Note that, in an aspect where each of the multiple virtual objects is placed at an arbitrary position in the virtual space, the length of a perpendicular line from the virtual object to a straight line along the line of sight indicated by the line of sight information is the distance between the virtual object and the first position SP.
[0040] (B-3: Variation 3) The input device 30 in the above embodiment may have four operators corresponding to the up, down, left and right directions in addition to or instead of the operator 300, and the selected virtual object may be changed in response to pressing of these four operators.
[0041] (B-4: Variation 4) When the selected virtual object is outside the user's field of view, the display control unit 140a may display guidance information indicating the direction of the selected virtual object as seen by the user on the glass device 20. For example, assume that the virtual space shown in FIG. 3 is displayed on the glass device 20 in response to the user turning his head to the left while the virtual object VO5 is selected. In this case, the display control unit 140a may guide the user in the direction of the selected virtual object by displaying an image YC of an arrow indicating that the selected virtual object is on the right side in the virtual space as shown in FIG. 8. The guidance of the direction of the selected virtual object may be performed by displaying a statement indicating the direction or by outputting a voice reading out the statement. Alternatively, the position of the sound source may be set as the position where the selected virtual object is placed.
[0042] (B-5: Variation 5) When the distance of the closest virtual object from the first position SP is equal to or greater than a predetermined threshold, the determination unit 140e may not determine the virtual object as the virtual object selected by the user even if the determination unit 140e acquires the first instruction information. In addition, the display control unit 140a may change the display mode of the mark M indicating the first position SP between the case where the distance from the first position SP of all of the multiple virtual objects displayed on the glass device 20 is equal to or greater than the threshold and the case where the distance from the first position SP of at least one virtual object is less than the threshold. As shown in FIG. 9, it is assumed that the virtual objects VO1, VO3, and VO4 are displayed on the glass device 20, and the distance from the first position SP of each of the virtual objects VO1, VO3, and VO4 is equal to or greater than the threshold. In this case, the display control unit 140a displays a star-shaped mark M at the first position SP as shown in FIG. 9. On the other hand, when the distance from the first position SP to at least one of the virtual objects VO1, VO3, and VO4 is less than the threshold, the display control unit 140a displays a round mark M at the first position SP as shown in FIG. 8. According to this embodiment, it is possible to indicate to the user through the display mode of the mark M indicating the position of the first position SP that the virtual object closest to the first position SP can be determined as the operation target by the determination operation. In addition, the display control unit 140a may change the display mode of one virtual object between when the distance from the first position SP to all of the multiple virtual objects displayed on the glass device 20 is equal to or greater than the threshold and when the distance from the first position SP to any one virtual object is less than the threshold. Specifically, when the distance from the first position SP to any one virtual object is less than the threshold, the display control unit 140a may invert the background color and foreground color when displaying the one virtual object, or may add a border to the one virtual object.According to this aspect as well, it is possible to suggest to the user through the display aspect of the virtual object that it is now possible to determine, as the operation target, the virtual object that is closest to the first position SP by a determination operation.
[0043] (B-6: Variation 6) The multiple virtual objects arranged in the virtual space may include a first virtual object and a second virtual object arranged inside a preset area in the virtual space, and a third virtual object arranged outside the area, and when the first position SP is a position within the above-mentioned area, the determination unit 140e may determine the first virtual object or the second virtual object that is closer to the first position as the virtual object to be selected by the user. This is because, since the determination operation was performed when the first position SP was a position within the above-mentioned area, it is considered that the user intends to select a virtual object arranged within the area.
[0044] For example, in the example shown in FIG. 10A, the virtual panel VP provided in the virtual space is an example of the above-mentioned predetermined region. The virtual object VO2 arranged on the virtual panel VP in FIG. 10A is an example of the first virtual object. Similarly, the virtual object VO5 arranged on the virtual panel VP in FIG. 10A is an example of the second virtual object. Also, the virtual object VO1 and the virtual object VO4 in FIG. 10A are examples of the third virtual object. In the example shown in FIG. 10A, among the virtual objects VO1, VO2, VO2, and VO4, the virtual object having the shortest distance from the first position SP is the virtual object VO4. Therefore, in the above embodiment, the virtual object VO4 is determined as the virtual object selected by the user. In contrast, in this modified example, since the first position SP is a position within the virtual panel VP, the determination unit 140e determines the one of the virtual objects VO2 and VO5 having the shortest distance from the first position SP as the virtual object selected by the user. 10A, since the virtual object VO5 is closer to the first position SP, the determination unit 140e determines the virtual object VO5 as the virtual object to be selected by the user. According to this embodiment, when the user intends to select the virtual object VO5, it is not necessary to change the direction of the line of sight until the distance between the first position SP and the virtual object VO5 becomes the shortest, and it becomes possible to quickly select the virtual object VO5.
[0045] Also, as shown in FIG. 10B, the virtual panel VP includes a display area MA of the moving image, a virtual button VB1 for instructing to start playing the moving image, a virtual button VB2 for instructing to stop playing the moving image, and a virtual button VB3 for instructing to play the moving image in reverse, and each of the virtual buttons VB1, VB2, and VB3 may be a virtual object. The display area MA is an example of a region set in advance in the virtual space. In the example shown in FIG. 10B, the virtual object with the shortest distance from the first position SP is the virtual object VO4, but the first position SP is a position within the virtual panel VP. Since the virtual object with the shortest distance from the first position SP among the virtual buttons VB1, VB2, and VB3 arranged on the virtual panel VP is the virtual button VB1, when a determination operation is performed in the example shown in FIG. 10B, the determination unit 140e determines the virtual button VB1 as the virtual object selected by the user.
[0046] (B-7: Variation 7) Although the gaze detection device in the above embodiment detects the gaze of the user by eye gaze, the gaze of the user may be detected by head gaze. In head gaze, the gaze of the user is detected based on the direction of the head of the user wearing the glass device 20 on the head. In this case, the gaze of the user is detected by an inertial sensor provided in the glass device 20, and the posture information output from the inertial sensor functions as gaze information. This is because the direction in which the user wearing the glass device 20 on the head is looking roughly coincides with the direction of the head of the user. When the gaze of the user is detected by head gaze, the gaze information (posture information) indicates the direction from the origin of the virtual space (i.e., the position of the user's head) toward the position on the spherical surface R corresponding to the center of the image displayed on the glass device 20. Therefore, when the gaze of the user is detected by head gaze, the mark M indicating the first position SP is always positioned at the center of the image of the virtual space displayed on the glass device 20, as shown in FIG. 11. According to the conventional method using head gaze, the mark M needs to be aligned with a small virtual object, so the user needs to move his / her head finely. In addition, the user needs to execute the decision operation while maintaining the virtual object and the mark M overlapping each other. This places a large physical burden on the user. In contrast, according to the headgaze technique of the seventh modification, the user only needs to roughly move his / her head and execute the decision operation when the virtual object and the mark M are close to each other, so that the user can accurately select the virtual object while reducing the physical burden on the user.
[0047] (B-8: Variation 8) The display control unit 140a may not display the mark M indicating the first position SP until a predetermined time has elapsed since the determination unit 140e has determined the virtual object selected by the user. This is because the user may feel annoyed if the display of the mark M indicating the first position SP continues even after the virtual object selected by the user has been determined. The display control unit 140a may also hide the mark M indicating the first position SP1 when the distance from the first position SP of all of the multiple virtual objects displayed on the glass device 20 is equal to or greater than the threshold, and display the mark M when the distance from the first position SP of any one of the virtual objects is less than the threshold. This embodiment can also reduce the annoyance of the user caused by the mark M being continuously displayed. The input device 30 may also receive a user's instruction to switch between displaying and hiding the mark M. In this case, the second acquisition unit 140c may acquire second instruction information indicating an instruction to switch from the input device 30. Then, the display control unit 140a may switch between displaying and hiding the mark M indicating the first position SP based on the second instruction information.
[0048] (B-9: Variation 9) The input device 30 may receive a user's instruction to switch between enabling and disabling the selection of a virtual object based on the gaze information. In this case, the second acquisition unit 140c may acquire third instruction information indicating an instruction to switch between enable and disable from the input device 30. Then, when the second acquisition unit 140c acquires third instruction information indicating a switch to disable, the determination unit 140e may not determine a virtual object based on the gaze information until the second acquisition unit 140c acquires third instruction information indicating a switch to enable.
[0049] (B-10: Variation 10) In the above embodiment, the glass device 20 and the mobile device 10 are separate devices, but the glass device 20 and the mobile device 10 may be integrated. When the glass device 20 and the mobile device 10 are integrated, the processing device of the glass device 20 plays the role of the display control device of the present disclosure. In addition, the glass device 20 and the input device 30 can be integrated by making the touch sensor 96 of the glass device 20 play the role of the operation element 300. Therefore, the glass device 20, the mobile device 10, and the input device 30 may be integrated.
[0050] (B-11: Variation 11) In the above embodiment, the program PRA is stored in the storage device 130 of the mobile device 10, but the program PRA may be manufactured or sold separately. When selling the program PRA, the program PRA may be provided to a purchaser by writing the program PRA to a computer-readable recording medium such as a flash ROM and distributing it, or by downloading it via a telecommunications line. Similarly, the program PRB may be manufactured or sold separately.
[0051] (B-12: Variation 12) In the above embodiment, the display control unit 140a, the first acquisition unit 140b, the second acquisition unit 140c, the calculation unit 140d, and the determination unit 140e are all software modules. However, any one, any two, any three, any four, or all of the display control unit 140a, the first acquisition unit 140b, the second acquisition unit 140c, the calculation unit 140d, and the determination unit 140e may be hardware modules. Specific examples of hardware modules include DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), and FPGAs (Field Programmable Gate Arrays). The same effects as those of the above embodiment can be achieved even if any one, any two, any three, any four, or all of the display control unit 140a, the first acquisition unit 140b, the second acquisition unit 140c, the calculation unit 140d, and the determination unit 140e are hardware modules.
[0052] (C:Other) (1) In the above-described embodiment, ROM, RAM, etc. are given as examples of storage device 130, but storage device 130 may also be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disc), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium.
[0053] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0054] (3) In the above-described embodiment, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.
[0055] (4) In the above-described embodiments, the determination may be made based on a value represented by 1 bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., comparison with a predetermined value).
[0056] (5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed without causing any inconsistency. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0057] (6) Each function illustrated in Fig. 4 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by using wires, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.
[0058] (7) The programs exemplified in the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.
[0059] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.
[0060] (8) In each of the above embodiments, the terms “system” and “network” are used interchangeably.
[0061] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or corresponding other information.
[0062] (10) In the above-described embodiments, the portable device may be a mobile station (MS). A mobile station may be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term. In this disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.
[0063] (11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "accessed". As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.
[0064] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0065] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.
[0066] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this disclosure is not intended to be an exclusive or.
[0067] (15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.
[0068] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."
[0069] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).
[0070] (D: Aspects understood from the above-mentioned forms or modifications) Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented as modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. The following aspects can be understood from at least one of the above-mentioned embodiments or modifications.
[0071] A display control device according to a first aspect of the present disclosure includes a display control unit, a first acquisition unit, a second acquisition unit, and a determination unit. The display control unit displays a virtual space including a plurality of virtual objects on a display disposed in front of the user's eyeball. The first acquisition unit acquires line-of-sight information related to the line of sight of the user. The second acquisition unit acquires first instruction information indicating an instruction from an input device that accepts an instruction from the user to change a selected virtual object. When the determination unit acquires the first instruction information, the determination unit determines a virtual object to be selected by the user based on a relationship between the line-of-sight information and each position of the plurality of virtual objects. The display control device of the first aspect can reduce a physical burden on the user while accurately selecting a virtual object.
[0072] A display control device according to a second aspect (an example of the first aspect) of the present disclosure further includes a calculation unit that calculates a first position in the virtual space based on the line-of-sight information. A determination unit in the display control device of the second aspect determines a virtual object located at a position closest to the first position as a virtual object selected by the user. According to the second aspect, the virtual object located at the closest distance from the first position calculated based on the line-of-sight information can be determined as a virtual object selected by the user.
[0073] A display control device according to a third aspect (an example of the first aspect) of the present disclosure further includes a calculation unit that calculates a first position in the virtual space based on the line-of-sight information. The plurality of virtual objects in the third aspect include a first virtual object and a second virtual object that are arranged inside a region that is preset in the virtual space, and a third virtual object that is arranged outside the region. When the first position is a position within the region, the determination unit in the display control device of the third aspect determines the one of the first virtual object and the second virtual object that is closer to the first position as the virtual object to be selected by the user. According to the third aspect, when the first position calculated based on the line-of-sight information is a position within a region that is preset in the virtual space, the one of the first virtual object and the second virtual object that is closer to the first position can be determined as the virtual object to be selected by the user.
[0074] A display control unit in a display control device according to a fourth aspect (an example of the second or third aspect) of the present disclosure displays a mark indicating the first position on the display. According to the fourth aspect, since the mark indicating the first position is displayed on the display, the user can grasp the first position.
[0075] A display control unit in a display control device according to a fifth aspect (an example of the fourth aspect) of the present disclosure does not display the mark until a predetermined time has elapsed since the determination unit has determined a virtual object selected by a user. According to the fifth aspect, it is possible to reduce annoyance caused by the mark indicating the first position being constantly displayed on the display.
[0076] An input device in a sixth aspect of the present disclosure (an example of the fourth or fifth aspect) receives an instruction from the user to switch between displaying and hiding the mark. A second acquisition unit in the display control device of the sixth aspect acquires second instruction information indicating the instruction to switch from the input device. A display control unit in the display control device of the sixth aspect switches between displaying and hiding the mark based on the second instruction information. According to the sixth aspect, a user can instruct switching between displaying and hiding the mark indicating a first position.
[0077] The input device in a seventh aspect (example of the first aspect, the second aspect, the third aspect, the fourth aspect, the fifth aspect, or the sixth aspect) of the present disclosure receives an instruction from the user to switch between enabling and disabling the selection of a virtual object based on the gaze information. The second acquisition unit in the display control device of the seventh aspect acquires third instruction information indicating an instruction to switch between enabling and disabling from the input device. When the third instruction information indicating a switch to disabling is acquired by the second acquisition unit, the determination unit in the display control device of the seventh aspect does not determine a virtual object to be selected by the user based on the gaze information until the third instruction information indicating a switch to enabling is acquired by the second acquisition unit. According to the seventh aspect, a user can instruct whether to enable or disable the selection of a virtual object based on gaze information. [Explanation of symbols]
[0078] 1...display system, 10...mobile device, 100...first communication device, 110...second communication device, 120...third communication device, 130...storage device, 140...processing device, 140a...display control unit, 140b...first acquisition unit, 140c...second acquisition unit, 140d...calculation unit, 140e...determination unit, 150...bus, PRA...program, 20...glass device, 30...input device, 40...management device, NW...communication line.
Claims
1. a display control unit that displays a virtual space including a plurality of virtual objects on a display placed in front of the user's eyeball; A first acquisition unit that acquires line-of-sight information regarding the user's line of sight; a second acquisition unit that acquires, from an input device that accepts a user instruction to change a selected virtual object, first instruction information indicating the instruction; a determination unit that determines a virtual object to be selected by the user based on a relationship between the line of sight information and each position of the plurality of virtual objects when the first instruction information is acquired; A display control device comprising:
2. A calculation unit that calculates a first position in the virtual space based on the line of sight information, The determination unit is determining a virtual object that is located at a position that is the shortest distance from the first position as a virtual object selected by the user; The display control device according to claim 1 .
3. A calculation unit that calculates a first position in the virtual space based on the line of sight information, the plurality of virtual objects include a first virtual object and a second virtual object that are arranged inside a preset area in the virtual space, and a third virtual object that is arranged outside the area; The determination unit is When the first position is a position within the area, one of the first virtual object and the second virtual object that is closer to the first position is determined to be the virtual object selected by the user. The display control device according to claim 1 .
4. The display control unit displays a mark indicating the first position on the display. The display control device according to claim 2 .
5. the display control unit does not display the mark until a predetermined time has elapsed since the virtual object selected by the user is determined by the determination unit, The display control device according to claim 4.
6. the input device accepts an instruction from the user to switch between displaying and hiding the mark; The second acquisition unit acquires second instruction information indicating an instruction to switch from the input device, The display control unit switches between displaying and hiding the mark based on the second instruction information. The display control device according to claim 4.
7. the input device receives an instruction from the user to switch between enabling and disabling selection of a virtual object based on the line of sight information; The second acquisition unit acquires third instruction information indicating an instruction to switch between enable and disable from the input device, When the third instruction information indicating a change to invalid is acquired by the second acquisition unit, the determination unit does not determine a virtual object to be selected by the user based on the line of sight information until the third instruction information indicating a change to valid is acquired by the second acquisition unit. The display control device according to claim 1 .
Citation Information
Patent Citations
Ultrasound diagnostic apparatus and control method of ultrasound diagnostic apparatus
JP2023087303A