Information processing device

Abstract

To provide an information processing device which improves operability for a user concerning a virtual object.SOLUTION: A terminal device 20 comprises a display control unit 211, a recognition unit 212, a determination unit 213, and a virtual object control unit 214. The determination unit 213, after a position of a hand is located at a region corresponding to an icon in a virtual space, and it is recognized that a type of gesture of the hand is a first type, executes first determination which determines whether or not it has been recognized that the position of the hand moves to a first virtual object, in a state where the first type gesture is maintained, and whether or not the type of the gesture of the hand has changed from the first type to a second type in the region corresponding to the first virtual object. The virtual object control unit 214, when the determination result of the first determination by the determination unit 213 is affirmative, executes processing associated with the icon on the first virtual object.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Patent Document 1 discloses a device that executes a process to close a tab when a user's gesture of putting both fists together is detected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2014-503903 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when a plurality of virtual objects exist in an augmented reality space or a mixed reality space, if some processing is to be executed in response to a gesture, the user needs to perform an operation to designate a target virtual object.

[0005] The present disclosure aims to provide an information processing device that improves user operability regarding virtual objects. [Means for solving the problem]

[0006] The information processing device according to the present disclosure includes: a display control unit that causes a display device viewed by a user to display a portion of a virtual space in which a first virtual object and an icon are superimposed on real space; a recognition unit that recognizes the position of the user's hand and a type of hand gesture in the virtual space based on an image captured by capturing an image of the outside world; a determination unit that performs a first determination to determine whether, after the position of the hand is located in an area corresponding to the icon in the virtual space and the type of the hand gesture is recognized to be a first type, the position of the hand moves toward the first virtual object while the first type of gesture is maintained, and the type of the hand gesture in the area corresponding to the first virtual object is recognized to have changed from the first type to a second type; and a virtual object control unit that, when a result of the first determination by the determination unit is positive, executes processing associated with the icon on the first virtual object. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to improve the user's operability regarding virtual objects. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the overall configuration of an information processing system 1. [Figure 2] FIG. 2 is an explanatory diagram showing an example of a virtual space VS in which a user U is located. [Figure 3] 3 is a schematic diagram of a visual field image IMG that is visually recognized when the user U of FIG. 2 looks ahead through the XR glasses 30. FIG. [Figure 4] FIG. 2 is a perspective view showing an example of the appearance of XR glasses 30. [Figure 5] FIG. 2 is a block diagram showing an example of the electrical configuration of the XR glasses 30. [Figure 6] FIG. 2 is a block diagram showing an example of the electrical configuration of a terminal device 20. [Figure 7] 3 is a diagram showing an example of display of an icon ICN in the virtual space VS of FIG. 2. [Figure 8]An explanatory diagram for fixing the position of the icon ICN. [Figure 9] FIG. 4 is an explanatory diagram of a series of operations determined by a determination unit 213. [Figure 10] 4 is a flowchart showing an example of the operation of the terminal device 20. [Figure 11] 10 is a flowchart showing an example of the operation of the terminal device 20 according to the first modification. [Figure 12] FIG. 10 is a diagram showing an example of displaying a line VOE along an extension line Le in a virtual space VS. [Figure 13] FIG. 11 is a block diagram showing an example of the electrical configuration of a terminal device 20 according to a third modification. [Figure 14] 10A and 10B are diagrams showing examples of displaying a first icon ICN_A and a fourth virtual object VO4 corresponding to an application in the virtual space VS. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1: Embodiment The information processing system 1 will be described below with reference to FIGS.

[0010] 1.1: Configuration of the embodiment FIG. 1 is a block diagram showing the overall configuration of an information processing system 1. As shown in FIG. 1, the information processing system 1 includes a user device 10 and a management server 40. The user device 10 is a device used by a user U. In the information processing system 1, the number of user devices 10 is not limited to one, but may be two or more. The management server 40 includes a position management server 40A that manages the position of the user device 10, and an object management server 40B that manages virtual objects.

[0011] In the information processing system 1, the location management server 40A and the user device 10 communicate with each other via a communication network NW. The object management server 40B and the user device 10 communicate with each other via the communication network NW. The user device 10 also includes a terminal device 20 and XR glasses 30. The terminal device 20 and the XR glasses 30 communicate with each other using wireless communication or wired communication.

[0012] The terminal device 20 is an example of an information processing device that has at least the function of controlling the display of the XR glasses 30. The terminal device 20 also functions as a relay device that relays communication between the XR glasses 30 and the position management server 40A and the object management server 40B. The terminal device 20 is, for example, a portable information processing device such as a smartphone or a tablet terminal.

[0013] The XR glasses 30 are worn on the head of the user U. The XR glasses 30 are see-through glasses that allow virtual objects to be viewed. The user U views the virtual objects while viewing the real space through the XR glasses 30. The virtual objects are virtual objects that are placed in the virtual space VS, and are placed in the virtual space VS at positions that correspond to the coordinate system of the real space. By using the XR glasses 30, the user U perceives an augmented reality space or mixed reality space in which the real space and the virtual space VS are combined. The augmented reality space refers to a virtual space VS in which a virtual object can be displayed superimposed on a real object RO, which is an object that actually exists in the real space. The mixed reality space refers to a virtual space VS in which a real object RO in the real space and a virtual object are interrelated. In the augmented reality space, a virtual object is simply displayed superimposed on the real object RO. In contrast, in the mixed reality space, the interaction between the virtual object and the real object RO can be expressed by using a physical model of the virtual object. For example, when the user U throws a virtual ball at the wall of the real object RO, the ball is expressed bouncing off the wall. Hereinafter, the augmented reality space will be described as the virtual space vs.

[0014] The XR glasses 30 shown in FIG. 1 include an imaging device 36 (see FIG. 4) that captures an image of the outside world. The captured image Gk captured by the imaging device 36 is transmitted to the location management server 40A via the terminal device 20. The location management server 40A receives the captured image Gk transmitted from the terminal device 20. The location management server 40A identifies glass position information Pk and direction information Dk based on the captured image Gk, and returns the glass position information Pk and direction information Dk to the terminal device 20. The glass position information Pk is information indicating the position of the XR glasses 30 in real space. The direction information Dk is information indicating the pointing direction of the XR glasses 30 in real space.

[0015] The location management server 40A stores a feature point map M. The feature point map M is data that represents a plurality of feature points using a three-dimensional global coordinate system. The feature point map M is generated, for example, by extracting a plurality of feature points from an image captured using a stereo camera. Positions in real space are represented using the global coordinate system.

[0016] The location management server 40A extracts a plurality of feature points from the captured image Gk and identifies the location where the captured image Gk was captured and the direction in which it was captured by comparing the extracted feature points with the feature points stored in the feature point map M. The location management server 40A regards the location where the image was captured as glass position information Pk and the direction in which the image was captured as direction information Dk.

[0017] The XR glasses 30 periodically transmit captured images Gk to the position management server 40A, thereby periodically acquiring glasses position information Pk and direction information Dk from the position management server 40A. The XR glasses 30 perform real-time tracking of local coordinates and correct their own position and direction of orientation in real time using the glasses position information Pk and direction information Dk acquired from the position management server 40A. This correction allows the XR glasses 30 to recognize their own position and direction of orientation expressed in the global coordinate system in real time. In the following description, the glasses position information Pk corrected by the XR glasses 30 will be referred to as corrected glasses position information Pck, and the direction information Dk corrected by the XR glasses 30 will be referred to as corrected direction information Dck.

[0018] When the object management server 40B receives the corrected glasses position information Pck and the corrected direction information Dck from the user device 10, the object management server 40B performs rendering of a virtual object based on the corrected glasses position information Pck and the corrected direction information Dck. The object management server 40B transmits the rendered virtual object to the terminal device 20. The virtual object is, for example, image data that resembles an arbitrary object.

[0019] In the following, a window that displays information will be described as an example of a virtual object. A window is an area used to display information in a computer's GUI (Graphical User Interface). The shape of this area is typically rectangular in a planar view. The information includes at least text, still images, and moving images. At least one virtual object is placed in the virtual space VS. The following describes a case where three virtual objects, a first virtual object VO1, a second virtual object VO2, and a third virtual object VO3, are placed in the virtual space VS.

[0020] FIG. 2 is an explanatory diagram showing an example of a virtual space VS in which a user U is located. The virtual space VS shown in the figure is an augmented reality space. In the figure, solid lines indicate real objects RO that exist in real space, and dashed lines indicate virtual objects (a first virtual object VO1, a second virtual object VO2, and a third virtual object VO3). In addition, in the figure, the orthogonal coordinate axes of the X-axis, Y-axis, and Z-axis are shown for convenience to make it easier to understand the space, and do not accurately represent the global coordinate system and local coordinate system that indicate the real space. In the following description, the user U is used as the base point, and the direction in which the imaging direction Dp points is defined as forward, and the direction opposite to this forward is defined as backward. In the figure, the imaging direction Dp is parallel to the X-axis. 2 shows a virtual space VS in which one real object RO, a first virtual object VO1, and a second virtual object VO2 are located in front of a user U, and a third virtual object VO3 is located behind the user U. In addition, in the virtual space VS, the X coordinate values ​​of the user U, the real object RO, the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 are X_U, X_RO, X_VO1, X_VO2, and X_VO3, respectively. Of these X coordinate values, X_VO1 is larger than X_VO2, and the first virtual object VO1 is located farther from the user U as the base point than the second virtual object VO2.

[0021] FIG. 3 is a schematic diagram of a visual field image IMG that the user U of FIG. 2 visually recognizes when looking ahead through the XR glasses 30. As shown in FIG. 3, in the virtual space VS shown in FIG. 2, the user U views a space in front of the user U that corresponds to the imaging direction Dp (i.e., a part of the virtual space VS). Specifically, of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 arranged in the virtual space VS, the user U views the first virtual object VO1 and the second virtual object VO2. Also, as shown in FIG. 2, the first virtual object VO1 is located farther from the user U along the imaging direction Dp than the second virtual object VO2, and therefore, as shown in FIG. 3, the first virtual object VO1 is displayed smaller than the second virtual object VO2. This display creates a sense of perspective.

[0022] FIG. 4 is a perspective view showing an example of the appearance of the XR glasses 30. As shown in FIG. The XR glasses 30 include temples 301 and 302, a bridge 303, frames 304 and 305, rims 306 and 307, lenses 310A and 310B, and an imaging lens LEN, which are visible from the outside.

[0023] Bridge 303 is a member that extends in one direction and connects rims 306 and 307. Imaging lens LEN is provided in approximately the center of bridge 303 in the extending direction. Imaging lens LEN is a member included in imaging device 36 shown in FIG. 5.

[0024] The frame 304 is a rod-shaped member connecting the temple 301, which is hooked onto the left ear of the user U, and the bridge 303. The frame 304 includes a display panel for the left eye and an optical member for the left eye. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display panel for the left eye displays projected images corresponding to virtual objects (first virtual object VO1, second virtual object VO2, and third virtual object VO3) based on control from the terminal device 20. The optical member for the left eye is an optical member that guides light emitted from the display panel for the left eye to the lens 310A.

[0025] The frame 305 is a rod-shaped member connecting a temple 302, which is hooked onto the right ear of the user U, and a bridge 303. The frame 305 is equipped with a display panel for the right eye and an optical member for the right eye. The display panel for the right eye displays projected images corresponding to the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 based on control from the terminal device 20. The optical member for the right eye is an optical member that guides light emitted from the display panel for the right eye to the lens 310B. The rim 306 holds the lens 310A for the left eye. The rim 307 holds the lens 310B for the right eye.

[0026] Each of the lenses 310A and 310B includes a half mirror. The half mirror transmits light from the real space, thereby guiding the light to the eyes of the user U. The half mirror also reflects light representing the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3, which is guided by the optical member, toward the eyes of the user U. The light from the real space that has passed through the half mirror and the light representing the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 that has been reflected by the half mirror are superimposed and enter the eyes of the user U, causing the user U to perceive the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 as being located in real space. In other words, the lenses 310A and 310B function as a transmissive display device disposed in front of the eyeball.

[0027] FIG. 5 is a block diagram showing an example of the electrical configuration of the XR glasses 30. The XR glasses 30 include a first processing device 31, a storage device 32, a detection device 35, an imaging device 36, a communication device 37, and a projection device 38. The first processing device 31, the storage device 32, the detection device 35, the imaging device 36, the communication device 37, and the projection device 38 are connected via a bus so that they can send and receive data to and from each other.

[0028] The first processing device 31 includes one or more central processing units (CPUs). The CPU includes an interface with peripheral devices, an arithmetic unit, a register, etc. The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer. In the first processing device 31, circuits such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA) may be used instead of or in addition to the CPU.

[0029] The storage device 32 is a recording medium readable by the first processing device 31. The storage device 32 includes a non-volatile memory and a volatile memory. The non-volatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), and an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Random Access Memory). The storage device 32 stores a first control program PR1. The first control program PR1 is a program that controls the operation of the XR glasses 30.

[0030] The detection device 35 detects the state of the XR glasses 30. Examples of the detection device 35 include inertial sensors such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration, and a geomagnetic sensor that detects orientation. The acceleration sensor detects acceleration on orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around the X-, Y-, and Z-axes as central axes of rotation. The detection device 35 generates orientation information indicating the orientation of the XR glasses 30 based on the output information of the gyro sensor. The detection device 35 also generates motion information related to the movement of the XR glasses 30. The motion information includes acceleration information indicating the acceleration on each of the three axes and angular acceleration information indicating the angular acceleration on each of the three axes. The detection device 35 outputs this orientation information, motion information, and orientation information indicating the orientation of the XR glasses 30 to the first processing device 31.

[0031] An example configuration of the imaging device 36 is a configuration including an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens LEN (see FIG. 4). The imaging optical system may include various optical elements such as a prism in addition to the imaging lens LEN, or may include a zoom lens or a focus lens. The imaging element is a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary MOS) image sensor, or the like.

[0032] The communication device 37 includes a communication interface that enables the XR glasses 30 to communicate with other devices. The communication device 37 communicates with the terminal device 20 using wireless communication or wired communication. In this embodiment, the communication device 37 communicates with the communication device 23 (see FIG. 6) of the terminal device 20 using short-range wireless communication such as Bluetooth (registered trademark).

[0033] The projection device 38 includes lenses 310A and 310B, a display panel for the left eye, an optical member for the left eye, a display panel for the right eye, and an optical member for the right eye. As described above, the lenses 310A and 310B function as displays disposed in front of the eyes of the user U. Based on control from the terminal device 20, the projection device 38 displays projection images corresponding to the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 on the display panel for the left eye and the display panel for the right eye.

[0034] FIG. 6 is a block diagram showing an example of the electrical configuration of the terminal device 20. As shown in FIG. The terminal device 20 includes a second processing device 21, a storage device 22, a communication device 23, and a touch display 24. The elements of the terminal device 20 are connected to each other by one or more buses for communicating information.

[0035] The second processing device 21 includes one or more CPUs that control the entire terminal device 20. The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer. The second processing device 21 may be implemented by using circuits such as a DSP, ASIC, PLD, and FPGA instead of or in combination with the CPU. The second processing device 21 executes various processes in parallel or sequentially.

[0036] The storage device 22 includes a recording medium that can be read from and written to by the second processing device 21. The storage device 22 also stores a plurality of programs. These programs include at least a second control program PR2 executed by the second processing device 21. The storage device 22 also functions as a work area for the second processing device 21.

[0037] The communication device 23 is hardware that serves as a transmitting / receiving device for the terminal device 20 to communicate with other devices. The communication device 23 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 23 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 23 may also include a wireless communication interface.

[0038] The touch display 24 is a device that displays images and text information, and also includes an input device that accepts input by a user U's touch operation.

[0039] The second processing device 21 reads and executes the second control program PR2 from the storage device 22. By executing the second control program PR2, the second processing device 21 functions as a display control unit 211, a recognition unit 212, a determination unit 213, and a virtual object control unit 214.

[0040] The display control unit 211 controls the projection device 38 of the XR glasses 30 to cause the lenses 310A and 310B of the XR glasses 30 to display the first virtual object VO1 and the second virtual object VO2 (i.e., part of the virtual space VS) of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 that are located within the field of view of the user U. As described above, the lenses 310A and 310B of the XR glasses 30 are transmissive display devices that are visually recognized by the user U. By displaying the first virtual object VO1 and the second virtual object VO2, the user U visually recognizes the virtual space VS in which the first virtual object VO1 and the second virtual object VO2 are superimposed on the real space in front of the user U's eyes.

[0041] The display control unit 211 also controls the projection device 38 of the XR glasses 30 to display an icon ICN on the lenses 310A and 310B of the XR glasses 30. The icon ICN is used by the user U to selectively input a command to execute a predetermined process to any one of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3, which are windows. In the following description, the predetermined process is a process including at least one of the following first to fifth processes. The first process is a process of minimizing the display of a window selected by the user U. The second process is a process of hiding the window selected by the user U. The third process is a process of saving information displayed in the window selected by the user U. The destination for saving the information is arbitrary. The fourth process is a process of starting playback of a video in the window selected by the user U. The fifth process is a process of stopping the video being played in the window selected by the user U.

[0042] FIG. 7 is a diagram showing an example of display of the icon ICN in the virtual space VS of FIG. 7, when the display control unit 211 displays the icon ICN, the display control unit 211 places the icon ICN in a position (X coordinate value: X_ICN) in front of the user U, a predetermined distance in the X-axis direction from the position of the user U (X coordinate value: X_U). Specifically, when the X coordinate value of the user U is X_U and the separation distance between the user U and the icon ICN is Q, the icon ICN is placed in a position where the X coordinate value X_ICN is X_U+Q. The value of the separation distance Q is arbitrary as long as it is within the reach of the user U.

[0043] Furthermore, in display control of the icon ICN, the display control unit 211 displays the icon ICN at a fixed position within the display area of ​​the lenses 310A and 310B, regardless of the direction of the XR glasses 30 (the pointing direction of the imaging device 36). Due to this control by the display control unit 211, as shown in Fig. 8, even if the user U turns his head left or right and changes the imaging direction Dp to a left-front or right-front oblique direction, the icon ICN is always fixed at a fixed position in the visual field image IMG of the user U.

[0044] The recognition unit 212 recognizes the hand position Ph and the type of hand gesture of the user U in the virtual space VS based on the captured image Gk captured by the imaging device 36. The terminal device 20 periodically acquires the captured image Gk from the imaging device 36 of the XR glasses 30, and the recognition unit 212 performs image recognition processing on the captured image Gk to identify the hand position Ph and the type of gesture. Note that if the XR glasses 30 are equipped with a depth sensor, the recognition unit 212 may identify the hand position Ph and the type of gesture using the detection value of the depth sensor in addition to the captured image Gk.

[0045] A hand gesture refers to an action in which the user U moves the hand itself or one or more fingers in a predetermined manner to indicate an operation on the icon ICN. The types of hand gestures include at least the following first and second types. The first type of gesture is a hand movement indicating an operation by the user U to select the icon ICN. The second type is a hand movement indicating that the user U is instructing any one of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 to execute a predetermined process associated with the icon ICN.

[0046] In this description, pinch-in is pre-assigned to the first type of gesture, and pinch-out is pre-assigned to the second type of gesture. Pinch-in is a hand movement in which the user U narrows the distance between two fingers so as to pinch an icon ICN with two fingers. Pinch-out is a hand movement in which the user U widens the distance between two fingers so as to separate the icon ICN that is being pinched with two fingers. Note that any gesture other than pinch-in and pinch-out may be assigned to the first type and the second type of gesture.

[0047] The determination unit 213 performs a first determination based on the recognition result of the recognition unit 212. The first determination is a determination as to whether or not the user U's hand has made the following series of movements in the virtual space VS. The series of movements to be determined broadly includes three movements: an icon selection movement, a target designation movement, and an execution instruction movement, as shown in FIG. The icon selection action is a movement by the user U that clearly indicates the selection of the icon ICN, and is a movement in which the hand position Ph is located in the corresponding area of ​​the icon ICN in the virtual space VS and the type of hand gesture indicates the first type. The corresponding area of ​​the icon ICN is an area in which it is determined that the hand gesture has been made with respect to the icon ICN, and an area of ​​a predetermined size that includes the icon ICN is set in advance in the virtual space VS.

[0048] The target designation action is an action in which the user U explicitly designates a target (target) from among the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 on which to execute a process associated with the icon ICN. In this description, the target designation action is a movement in which, while maintaining a first type of gesture, the hand position Ph moves toward any one of the corresponding areas of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3, and the hand position Ph reaches any one of the corresponding areas. The corresponding areas of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 are areas in which it is determined that the corresponding virtual object has been designated by the user U, and areas of a predetermined size that include the corresponding virtual object are set in advance in the virtual space VS.

[0049] The execution instruction action is an action in which the user U explicitly instructs the virtual object designated by the target designation action to execute a process associated with the icon ICN. In this description, the execution instruction action is a movement in which the type of hand gesture changes from the first type to the second type when the hand position Ph is located in any one of the corresponding areas of the first virtual object VO1, the corresponding area of ​​the second virtual object VO2, and the corresponding area of ​​the third virtual object VO3.

[0050] The determination unit 213 performs a first determination based on the recognition result of the recognition unit 212 regarding the hand position Ph and the type of hand gesture, and the setting values ​​of the corresponding areas of the icon ICN, the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3. The setting values ​​of the corresponding areas are stored in advance in the storage device 22, for example.

[0051] If the judgment result of the first judgment by the judgment unit 213 is positive, the virtual object control unit 214 executes processing associated with the icon ICN for the virtual object explicitly indicated by the user U by moving the hand position Ph, among the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3.

[0052] 1.2: Operation of the embodiment The operation of the terminal device 20 will now be described. FIG. 10 is a flowchart showing an example of the operation of the terminal device 20.

[0053] In step S10, the second processing device 21 displays a part of the virtual space VS on the XR glasses 30. The virtual space VS is a virtual space (augmented reality space or mixed reality space) in which a first virtual object VO1, a second virtual object VO2, a third virtual object VO3, and an icon ICN are superimposed on real space. Through the processing of step S10, the user U visually recognizes the virtual space VS.

[0054] In step S11, the second processing device 21 recognizes the hand position Ph and the type of hand gesture of the user U in the virtual space VS based on the captured image Gk. In step S12, the second processing device 21 determines whether the hand position Ph of the user U is located in the corresponding area of ​​the icon ICN and whether the type of hand gesture is the first type. If the determination result is affirmative, the second processing device 21 proceeds to step S13. By the processing of step S12, it is determined whether the icon ICN has been selected by the user U.

[0055] In step S13, the second processing device 21 determines whether the hand position Ph of the user U has moved to the corresponding area of ​​any of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 while maintaining the first type of gesture. If the determination result is affirmative, the second processing device 21 proceeds to step S14. Through the processing of step S13, it is determined whether any one of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 has been selected by the user U as the target for executing the process corresponding to the icon ICN.

[0056] In step S14, the second processing device 21 determines whether the gesture type has changed from the first type to the second type in the corresponding area of ​​any of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3. If the determination result is affirmative, the second processing device 21 proceeds to step S15. By the processing of step S14, it is determined whether the execution of the processing has been instructed by the user U.

[0057] In step S15, the second processing device 21 executes processing associated with the icon ICN for the virtual object corresponding to the corresponding region including the hand position Ph, among the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3.

[0058] 10, the second processing device 21 functions as a display control unit 211 in step S10. It also functions as a recognition unit 212 in step S11. It also functions as a determination unit 213 in steps S12, S13, and S14. It also functions as a virtual object control unit 214 in step S15.

[0059] 1.3: Effects of the embodiment In the above description, the terminal device 20 is an example of an information processing device that controls the see-through type XR glasses 30. The terminal device 20 includes a display control unit 211, a recognition unit 212, a determination unit 213, and a virtual object control unit 214. The display control unit 211 displays, on the XR glasses 30 viewed by the user U, a portion of the virtual space VS in which a first virtual object VO1 and an icon ICN are superimposed on real space. The recognition unit 212 recognizes the position Ph of the user U's hand and the type of hand gesture in the virtual space VS based on a captured image Gk obtained by capturing an image of the outside world. The determination unit 213 performs a first determination to determine whether the recognition unit 212 has recognized that the hand position Ph is located in an area corresponding to the icon ICN in the virtual space VS and that the type of hand gesture is a first type, and then whether the recognition unit 212 has recognized that the hand position Ph has moved to the first virtual object VO1 while maintaining the first type of gesture, and that the type of hand gesture in the area corresponding to the first virtual object VO1 has changed from the first type to the second type. If the determination result of the first determination by the determination unit 213 is positive, the virtual object control unit 214 executes the process associated with the icon ICN on the first virtual object VO1.

[0060] The terminal device 20 has the above configuration, which improves the operability of the user U. More specifically, when an operation icon is provided on a virtual object in the virtual space VS, the farther the virtual object is located from the user U, the smaller the operation icon becomes along with the virtual object, making it more difficult for the user U to operate it with their hand. On the other hand, according to the present embodiment, the icon ICN is placed independently of the first virtual object VO1 in the virtual space VS, so the operability of the icon ICN is not affected by the distance from the user U to the first virtual object VO1. Furthermore, when the user U moves the hand position Ph to an area corresponding to the first virtual object VO1, the target object of the processing associated with the icon ICN is specified. Therefore, compared to a configuration in which an operation icon provided on a virtual object is operated, the target to which the hand position Ph is moved is larger, which makes it possible to prevent erroneous operations and improves operability.

[0061] Furthermore, the display control unit 211 fixes and arranges the icon ICN at a predetermined position in the virtual space VS. Because the terminal device 20 has the above configuration, even if the user U moves his / her head and the viewing direction (imaging direction Dp) of the user U changes, the icon ICN is fixed at a predetermined position in the virtual space VS. Since the icon ICN is fixed, the user U does not need to search for the icon ICN even if the viewing direction changes in the augmented reality space or mixed reality space, improving operability.

[0062] In addition, the first virtual object VO1 is a window that displays information, and the processes associated with the icon ICN include at least one of a first process for minimizing the display of the window, a second process for hiding the window, a third process for saving the information displayed in the window, a fourth process for starting playback of a video in the window, and a fifth process for stopping the video being played in the window. The terminal device 20 has the above configuration, which improves the operability of the user U with respect to the window or the information on the window.

[0063] Furthermore, the display control unit 211 displays a first virtual object VO1 and a second virtual object VO2 on the XR glasses 30. The first determination by the determination unit 213 is a determination as to whether, after the hand position Ph is located in an area corresponding to the icon ICN in the virtual space VS and the type of the hand gesture is recognized as a first type, the hand position Ph moves to an area corresponding to the first virtual object VO1 or an area corresponding to the second virtual object VO2 while the first type of gesture is maintained and the type of the hand gesture has changed from the first type to the second type. When the determination result of the first determination by the determination unit 213 is positive, the virtual object control unit 214 executes processing associated with the icon ICN on the virtual object corresponding to the area including the hand position Ph, out of the first virtual object VO1 and the second virtual object VO2. Because the terminal device 20 has the above configuration, even when a plurality of virtual objects including a first virtual object VO1 and a second virtual object VO2 are arranged in the virtual space VS, the user U can selectively operate either the first virtual object VO1 or the second virtual object VO2 using an icon ICN common to the first virtual object VO1 and the second virtual object VO2. Furthermore, the user U can operate the different first virtual object VO1 and the second virtual object VO2 using a unified operation system, improving operability.

[0064] 2: Variation The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected arbitrarily from the following examples may be combined.

[0065] 2.1: Variation 1 The above-described terminal device 20 identifies a processing target associated with the icon ICN when the hand position Ph moves to any one of the corresponding regions of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3. However, the present disclosure is not limited to the embodiment. In the present disclosure, the terminal device 20 may identify a processing target based on one of the corresponding regions of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3 that is located in the movement direction of the hand position Ph.

[0066] Fig. 11 is a flowchart showing an example of the operation of the terminal device 20 of Modification 1. In Fig. 11, the same processes as those in Fig. 9 described in the embodiment are denoted by the same reference numerals, and the description thereof will be omitted. 11, in the operation example of Modification 1, steps S13a and S14a are different from steps S13 and S14 of the operation example of the embodiment. Steps S13a and S14a are steps of identifying a target for processing associated with an icon ICN from among the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3, and identifying an instruction by the user U to execute processing for the identified virtual object. Specifically, in step S13a, the second processing device 21 of the terminal device 20 determines whether or not the hand position Ph of the user U has moved while the first type of gesture is maintained. If the determination result is affirmative, the second processing device 21 proceeds to step S14a. In step S14a, the second processing device 21 determines whether the type of hand gesture is recognized as having changed from the first type to the second type when the corresponding area of ​​any one of the corresponding areas of the first virtual object VO1, the corresponding area of ​​the second virtual object VO2, and the corresponding area of ​​the third virtual object VO3 is located on an extension line of the movement direction of the hand position Ph (second determination). Through the processing of steps S13a and S14a, an execution target of the processing associated with the icon ICN and an instruction to execute the processing on the identified virtual object are identified.

[0067] In the processing of the flowchart shown in FIG. 11, the second processing device 21 functions as the determination unit 213 in steps S12, S13a, and S14a.

[0068] According to the operation example of the terminal device 20 of the first modification example, the user U can select a virtual object to be processed by the movement direction of the hand position Ph without having to move the hand position Ph to any one of the corresponding regions of the first virtual object VO1, the second virtual object VO2, and the third virtual object VO3. Therefore, even if the virtual object to be processed is located far from the user U, the user U can select the virtual object as the processing target without having to stretch out his / her arm far, and the user U does not feel tired in his / her arm.

[0069] 2.2: Variation 2 In the above-described first modification, when the hand position Ph of the user U moves while maintaining the first type of gesture, the terminal device 20 may display a line VOE extending along an extension line Le of the movement of the hand position Ph on the XR glasses 30 (display device), as shown in Fig. 12. The display of this line VOE allows the user U to easily recognize the virtual object on which processing is to be executed.

[0070] 2.3: Variation 3 In the above-described embodiment, the third virtual object VO3 in the virtual space VS is located behind the user U. Therefore, when the user U faces forward, the third virtual object VO3 is not displayed on the XR glasses 30 and is not visible to the user U. Furthermore, for example, if an application that displays email or a social networking service (SNS) is associated with the third virtual object VO3, even if the state of this application transitions from a normal state (first state) to a state that notifies the user U of the receipt of a new message (second state), the user U has difficulty understanding the transition of the application state because the third virtual object VO3 is located behind the user U. Therefore, the terminal device 20 of the third modification example has a configuration that makes it easier for the user U to understand the transition of the application state. Note that the second state of the application is not limited to a state that notifies the user U and is arbitrary.

[0071] Fig. 13 is a block diagram showing an example of the electrical configuration of the terminal device 20 of Modification 3. In Fig. 13, the same processes as those in Fig. 6 described in the embodiment are denoted by the same reference numerals, and the description thereof will be omitted. As shown in FIG. 13 , the terminal device 20 of the third modification example differs from the terminal device 20 of the embodiment in that it includes a state detection unit 215. The state detection unit 215 detects that the state of an application associated with a third virtual object VO3 has transitioned from a first state to a second state. When the state detection unit 215 detects a transition from the first state to the second state, the display control unit 211 causes the XR glasses 30 to display a first icon ICN_A corresponding to the application in front of the user U (in the imaging direction Dp) in the virtual space VS, as shown in FIG. 14 . The first icon ICN_A depicts an image that allows the user U to identify the application. After displaying the first icon ICN_A, the display control unit 211 causes the XR glasses 30 to display a fourth virtual object VO4. The fourth virtual object VO4 is a virtual object that creates a visual effect of moving from the position of the first icon ICN_A toward the position of the third virtual object VO3, thereby inducing a gaze guidance effect for the user U. Specific examples of the fourth virtual object VO4 include a bullet, a rocket, and a ray of light. Furthermore, when displaying the fourth virtual object VO4, the display control unit 211 controls the movement speed of the fourth virtual object VO4 or the time from the display timing of the first icon ICN_A to the display timing of the fourth virtual object VO4 in accordance with the movement of the head or the movement of the line of sight of the user U. Through this control, the movement speed of the fourth virtual object VO4 or the display timing of the fourth virtual object VO4 is delayed or advanced in accordance with the movement of the head or the movement of the line of sight of the user U so that the user U does not lose sight of the display of the fourth virtual object VO4, thereby improving the recognition rate of the fourth virtual object VO4.

[0072] With the terminal device 20 of variant example 3, the user U can quickly grasp the state transition of the application corresponding to the third virtual object VO3 located outside the display range of the XR glasses 30 by displaying the first icon ICN_A, and can intuitively grasp the position of the third virtual object VO3 by displaying the fourth virtual object VO4.

[0073] 3:Other (1) In the above-described embodiment, the storage device 22 and the storage device 32 are exemplified by ROM and RAM, but they may also be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. The programs may also be transmitted from a network via telecommunications lines. The programs may also be transmitted from a communications network NW via telecommunications lines.

[0074] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0075] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0076] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0077] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0078] (6) Each function illustrated in Figures 1 to 14 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0079] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0080] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0081] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0082] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0083] (10) In the above-described embodiments, the user device 10, the terminal device 20, and the XR glasses 30 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term. In this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0084] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0085] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0086] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0087] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0088] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0089] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0090] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0091] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0092] 1...information processing system, 10...user device, 21...second processing device, 31...first processing device, 20...terminal device, 30...XR glasses, 36...imaging device, 211...display control unit, 212...recognition unit, 213...determination unit, 214...virtual object control unit, 215...state detection unit, 310A, 310B...lens, Gk...captured image, ICN...icon, ICN_A...first icon, IMG...field of view image, Le...extension line, RO...real object, VO1...first virtual object, VO2...second virtual object, VO3...third virtual object, VO4...fourth virtual object, VOE...line, VS...virtual space.

Claims

1. a display control unit that displays, on a display device visually recognized by a user, a portion of a virtual space in which the first virtual object and the icon are superimposed on a real space; a recognition unit that recognizes the position of the user's hand and the type of gesture of the hand in the virtual space based on a captured image obtained by capturing an image of the outside world; a determination unit that executes a first determination to determine whether, after the position of the hand is located in an area corresponding to the icon in the virtual space and the type of the hand gesture is recognized as a first type, the position of the hand moves toward the first virtual object while the first type of gesture is maintained, and whether the type of the hand gesture in the area corresponding to the first virtual object is recognized as having changed from the first type to a second type; a virtual object control unit that executes a process associated with the icon on the first virtual object when a determination result of the first determination by the determination unit is affirmative; An information processing device comprising:

2. The display control unit The icon is fixedly arranged at a predetermined position in the virtual space. The information processing device according to claim 1 .

3. The determination unit a second determination is performed to determine whether the type of the hand gesture has been recognized as having changed from the first type to a second type in a state in which, in the virtual space, the position of the hand is located in an area corresponding to the icon and the type of the hand gesture is recognized as being a first type, and then the position of the hand moves while the first type of gesture is maintained and an area corresponding to the first virtual object is located on an extension line of the moving direction of the hand position; The virtual object control unit When a determination result of the second determination by the determination unit is positive, the process is executed on the first virtual object. The information processing device according to claim 1 .

4. The display control unit displaying a line along the extension line on the display device in the virtual space; The information processing device according to claim 3 .

5. The first virtual object is It is a window that displays information, The process comprises: a first process of minimizing the display of the window; a second process of hiding the window; a third process for saving the information displayed in the window; a fourth process of starting playback of the video in the window; and A fifth process of stopping the video being played in the window. The method includes at least one process of The information processing device according to claim 1 .

6. The display control unit displaying the first virtual object and the second virtual object on the display device; The first determination by the determination unit is a determination as to whether, in the virtual space, after the position of the hand is located in an area corresponding to the icon and the type of the hand gesture is recognized as a first type, the position of the hand moves to an area corresponding to the first virtual object or an area corresponding to the second virtual object while the gesture of the first type is maintained, and the type of the hand gesture is recognized as having changed from the first type to the second type; The virtual object control unit When the determination result of the first determination by the determination unit is positive, the process associated with the icon is executed for one of the first virtual object and the second virtual object that corresponds to a region including the position of the hand. The information processing device according to claim 1 .

7. the virtual space includes a third virtual object that is located at a position outside the part of the virtual space displayed on the display device and that is associated with an application; a state detection unit that detects that a state of the application of the third virtual object has transitioned from a first state to a second state; The display control unit when the state detection unit detects a transition from the first state to the second state, displaying a first icon corresponding to the application on the display device, and then displaying a fourth virtual object on the display device that produces a visual effect of moving from a position of the first icon toward a position of the third virtual object; The information processing device according to claim 2 .

8. The display control unit controlling a moving speed of the fourth virtual object in accordance with a movement of the user's head or a movement of the user's line of sight; The information processing device according to claim 7 .

Images