Information processing device, control method of information processing device, and program

Abstract

To provide an information processing device for making it possible to perform an operation relating to a virtually displayed image such as a menu without feeling of strangeness, and to provide a control method of the information processing device, and a program.SOLUTION: An information processing device obtains information of a position of each fingertip of a user in a real space, and determines contact between a virtual object set within a virtual space and a finger of the user. The information processing device sets the virtual object in a partly deformed state such that a part of the virtual object, the part corresponding to a position of the finger determined to be in contact with the object among the fingers of the user, is located more to a far side from a user side than the finger, and displays the virtual object having a shape set thereto as an image in the virtual space on a display device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, devices that are worn on a user's head and display information, such as head-mounted displays, have become widespread. Some of these devices present virtual images seen from different positions to the user's left and right eyes, allowing the user to view a stereoscopic image.

[0003] The device also presents a menu to the user as a virtual image and accepts selections from the user. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional menu selection, the user must use a separate device equipped with physical buttons, such as a game controller, to select a menu. It is also possible to detect the tilt and position of the device, and move a cursor displayed as a virtual image according to the tilt, movement direction, and movement amount of the device, thereby allowing the user to select a menu. However, in either case, the user cannot directly touch the displayed virtual menu object, and therefore cannot operate it comfortably.

[0005] The present invention has been made in consideration of the above-mentioned situation, and one of its objects is to provide an information processing device, a control method for an information processing device, and a program that enable operations related to virtually displayed images, such as menus, to be performed without any sense of discomfort. [Means for solving the problem]

[0006] The present invention, which solves the problems of the above-mentioned conventional examples, is an information processing device connected to a display device that presents an image of a virtual space to a user, and includes an acquisition means for acquiring information on the position of each of the user's fingertips in real space, an object setting means for setting a position and an image of a virtual object in the virtual space, a determination means for determining contact between the virtual object set in the virtual space and the user's finger, and a shape setting means for setting the virtual object to a partially deformed state so that a portion of the virtual object corresponding to the position of the user's finger that is determined to be in contact with the object is located further back than the finger toward the user, and the virtual object with the shape set is displayed as an image of the virtual space on the display device. Effect of the Invention

[0007] According to the present invention, it becomes possible to perform operations relating to virtually displayed images such as menus without feeling uncomfortable. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing device according to an embodiment of the present invention. [Diagram 2] 1 is a configuration block diagram illustrating an example of a display device connected to an information processing device according to an embodiment of the present invention. [Diagram 3] 1 is a functional block diagram illustrating an example of an information processing device according to an embodiment of the present invention. [Figure 4] FIG. 2 is an explanatory diagram illustrating an example of a coordinate system used by the information processing device according to the embodiment of the present invention. [Diagram 5] 1 is an explanatory diagram illustrating an example of a virtual object set by an information processing device according to an embodiment of the present invention. [Figure 6] FIG. 2 is an explanatory diagram illustrating an example of an image displayed by an information processing device according to an embodiment of the present invention. [Figure 7] 11 is an explanatory diagram illustrating another example of a virtual object set by the information processing device according to the embodiment of the present invention. FIG. [Figure 8] FIG. 11 is an explanatory diagram illustrating yet another example of a virtual object set by the information processing device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An embodiment of the present invention will be described with reference to the drawings. As shown in Fig. 1, an information processing device 1 according to the embodiment of the present invention includes a control unit 11, a storage unit 12, and a communication unit 13. The information processing device 1 is also connected to a display device 2 such as a head mounted display (HMD) worn by a user on the head so as to be able to communicate with the display device 2.

[0010] The display device 2 may be, for example, a transparent HMD that presents an image of a real space and a virtual space to a user wearing the display device 2 by superimposing them (in a semi-transparent composite state), or may be a non-transparent HMD that displays an image of a virtual space. An example of the display device 2 is a display device that is worn by a user on the head, and includes a control unit 21, a communication unit 22, an imaging unit 23, a depth camera 24, and a display unit 25, as illustrated in FIG. 2. In this example, the control unit 21 of the display device 2 is a program-controlled device such as a microcomputer, and operates according to a program stored in a memory (not shown) such as a built-in storage unit, and displays an image corresponding to information input from the information processing device 1 via the communication unit 22 on the display unit 25, allowing the user to view the image.

[0011] The communication unit 22 is connected to the information processing device 1 in a wired or wireless manner so as to be able to communicate with the information processing device 1. The communication unit 22 transmits, to the information processing device 1, image data captured by an imaging unit 23 (described later) and depth information output by a depth camera 24. The communication unit 22 also outputs information transmitted from the information processing device 1 to the display device 2 to the control unit 21.

[0012] The imaging unit 23 is a camera or the like, and repeatedly captures images of real space within a predetermined field of view in front of the user (in front of the head), and transmits the captured image data to the information processing device 1 via the communication unit 22.

[0013] The depth camera 24 detects the distance to an object imaged within each pixel included in an image within the field of view captured by the imaging unit 23, and outputs depth information (depth map) indicating the detected distance for each pixel. Specifically, the depth camera 24 can be a well-known depth camera that projects infrared rays to measure the distance to an object.

[0014] The display unit 25 displays images corresponding to the right and left eyes of the user in front of each of the eyes. The display unit 25 includes a display element such as an organic EL display panel or a liquid crystal display panel. The display element displays images according to instructions input from the control unit 21. The display element may be a single display element that displays an image for the left eye and an image for the right eye in a line, or a pair of display elements that display an image for the left eye and an image for the right eye independently. In this embodiment, the display device 2 is a transmissive display device that allows the user to view the outside world.

[0015] Here, the control unit 11 is a program control device such as a CPU, and executes a program stored in the storage unit 12. In this embodiment, the control unit 11 executes an application program such as a game application, and also executes the following processes.

[0016] That is, based on image data captured by the imaging unit 23 and output by the display device 2, the control unit 11 detects the position of each of the user's fingertips in real space and the direction of each finger (whether each finger is facing away from the user or towards the front or back), and obtains information representing these.

[0017] The control unit 11 also sets information about the virtual space. Specifically, the control unit 11 sets the position and image of a virtual object in a predetermined virtual space, and determines whether the virtual object set in the virtual space is in contact with the user's finger. The control unit 11 further sets the virtual object in a partially deformed state so that a part of the virtual object corresponding to the position of the user's finger that is turned away and determined to be in contact with the object is located on the farther side of the user's finger.

[0018] A detailed description will be given later of the operation of the control unit 11. In the following example, the virtual object that is displayed here and that is deformed by determining contact with the user's finger is called a target virtual object in order to distinguish it from other virtual objects (for example, virtual objects that are placed corresponding to objects in the real space and that are not deformed by determining contact with the user's finger).

[0019] The storage unit 12 is a memory device such as a RAM or a disk device, and stores the program executed by the control unit 11. The storage unit 12 also operates as a work memory for the control unit 11, and stores data used by the control unit 11 in the process of executing the program. The program may be provided by being stored in a computer-readable and non-transitory recording medium, and may be stored in the storage unit 12.

[0020] The communication unit 13 is connected to the user's display device 2 via wired or wireless communication. The communication unit 13 receives image data output by the display device 2 and sends it to the control unit 11. The communication unit 13 also accepts information including image data sent from the control unit 11 to the display device 2 and outputs the information to the display device 2.

[0021] Here, a description will be given of the operation of the control unit 11 of the information processing device 1. As shown in Fig. 3 as an example, the control unit 11 of the present embodiment includes a finger detection unit 31, an object setting unit 32, a determination processing unit 33, an object control unit 34, an output unit 35, and a process control unit 36.

[0022] Here, the finger detection unit 31 acquires information on the position and orientation of each fingertip of the user in the real space. As an example, the finger detection unit 31 of the present embodiment estimates the finger position from among the objects captured in the image data based on the image data and depth information received from the display device 2, and determines whether the finger is facing back (nail side) or ventral side as seen from the user. Specifically, the detection process of the finger position from the image by the finger detection unit 31 can adopt various widely known methods in addition to the method shown in VIPavlovic, et.al., “Visual interpretation of hand gestures for human-computer interaction: a review,” IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 19, No.7, pp. 677-695, 1997. In this embodiment, the information processing device 1 expresses the position of the finger in the real space using an orthogonal coordinate system in which the X-axis (the right side of the user is the positive direction) is an axis that is in a plane parallel to the coronal plane of the user when the user is standing upright and is parallel to the floor, the Y-axis (the vertical upward direction is the positive direction) is an axis that is in a plane parallel to the coronal plane of the user and is parallel to the normal to the floor, and the Z-axis (the forward direction of the user is the positive direction) is the normal direction of the coronal plane of the user, as illustrated in Fig. 4. In this coordinate system, each point on the coordinates (x, y, ζ) (ζ is an arbitrary value) parallel to the Z-axis is visually recognized on a point T(x, y) on the X, Y orthogonal coordinate system in the user's field of view (two-dimensional field of view).

[0023] As an example of a process for determining whether a finger is facing back (nail side) or ventral side, a process for detecting the nail part of a finger (for example, the process shown in Noriaki Fujishima, et. al., “Fingernail Detection Method from Hand Images including Palm,” Proceedings of IAPR International Conference on Machine Vision Applications, pp. 117-120, 2013, etc.) is performed, and among the fingers at the detected position, a finger whose corresponding image part is not recognized as a nail is determined to be facing the ventral side of the finger, and a finger whose corresponding image part is recognized as a nail is determined to be facing the back side. In this embodiment, this finger detection unit 31 realizes the acquisition means of the present invention.

[0024] The object setting unit 32 sets the position and image of the target virtual object in the virtual space. In one example of the present embodiment, the object setting unit 32 arranges a sheet-like or polyhedral target virtual object in the virtual space according to an instruction output by the control unit 11 based on the processing of the application program while the application program is being executed. The object setting unit 32 also sets the surface of the target virtual object displayed on the user side to display an image instructed based on the processing of the application program according to the instruction. Specifically, the instruction based on the application program includes information such as the shape of a user interface component (such as a virtual button), a character string to be displayed in the button, and the position of the button. The object setting unit 32 generates an image representing a user interface according to the instruction, and sets the texture of the surface of the target virtual object so that the image is displayed on the surface of the target virtual object. Since the process of setting an image on the surface of the target virtual object in this manner is widely known, a detailed description thereof will be omitted here.

[0025] The object setting unit 32 also sets the placement position of the target virtual object with an image set on its surface in the virtual space. In this embodiment, the information processing device 1 sets the virtual space using an orthogonal coordinate system in which the X-axis (the right side of the user is the positive direction) is an axis that is in a plane parallel to the coronal plane of the user when the user stands upright and is parallel to the floor surface, the Y-axis (the positive direction is vertically upward) is an axis that is in a plane parallel to the coronal plane of the user and is parallel to the normal to the floor surface, and the Z-axis (the positive direction is in front of the user) is the normal direction to the coronal plane of the user, as in the coordinate system of the real space illustrated in FIG. 4. In order to match the coordinate values ​​of the real space and the virtual space, the ratio α of the X-axis direction, the ratio β of the Y-axis direction, and the ratio γ of the Z-axis direction of the coordinate system are obtained in advance by calibration. As an example, a point is displayed at known coordinates in the virtual space, and the user is instructed to touch this point with his / her fingertip. The process of obtaining the coordinate value in real space detected by finger detection unit 31 at the time when the user touches the point is performed for multiple coordinate points in the virtual space, and the ratio of the distance between at least one specified pair of points in the virtual space to the distance between the corresponding pair of points in real space is calculated.

[0026] The object setting unit 32 sets the position and shape of the target virtual object (FIG. 4: V) using the coordinate values ​​in this coordinate system. Since the setting of the arrangement and shape of such a virtual three-dimensional object is also widely known, a detailed description thereof will be omitted here.

[0027] The determination processing unit 33 determines contact between a target virtual object set in the virtual space and the user's finger. In the present embodiment, the determination of contact between the target virtual object and the user's finger is to determine whether or not there is virtual contact between the user's own finger in the real space visually recognized by the user through the display device 2 and the target virtual object in the virtual space displayed superimposed thereon. As a specific example, the determination processing unit 33 sets a virtual three-dimensional area including an area of ​​the outline of the target virtual object in the virtual space set by the object setting unit 32, and determines that the target virtual object has come into contact with the user's finger when the coordinate value in the virtual space corresponding to the position of the finger in the real space detected by the finger detection unit 31 is within the three-dimensional area set here.

[0028] If there is a finger F that the determination processing unit 33 has determined to be one of the user's fingers facing away and in contact with the target virtual object set by the object setting unit 32, the object control unit 34 determines whether a portion of the target virtual object corresponding to the finger F is on the user side or on the back side of the finger F. If the portion of the target virtual object corresponding to the finger F is on the user side of the finger F, the object control unit 34 sets the target virtual object to a partially deformed state so that the portion is located on the back side of the finger F (so that the portion is concave on the back side).

[0029] As a specific example, it is assumed that the object setting unit 32 places a target virtual object J such that the normal of the surface of the object J is parallel to the Z axis, as illustrated in Fig. 5. It is assumed that this object is placed at the position Z = Zj, that its shape in the XY plane is rectangular, and that the coordinates of its upper left corner as seen from the user's side are (Xtl, Ytl) and the coordinates of its lower right corner are (Xrb, Yrb). It is also assumed that the thickness is substantially "0", and that the object J is substantially sheet-like.

[0030] The determination processing unit 33 sets a virtual three-dimensional region R including the region of the outline of the virtual sheet-like object J in the virtual space set by the object setting unit 32. As an example, the region R is a hexahedral region surrounded by a rectangular proximal side with diagonals of (Xtl, Ytl, Zj-ΔZ) and (Xrb, Yrb, Zj-ΔZ) and a rectangular distal side with diagonals of (Xtl, Ytl, Zj+ΔZ) and (Xrb, Yrb, Zj+ΔZ).

[0031] Here, when the corresponding position coordinates of the user's finger F in the virtual space are (Xf, Yf, Zf) (the value in the Z-axis direction can be obtained from depth information from the depth camera 24) and the finger F is facing away from the user (nail side), the judgment processing unit 33 checks whether the corresponding position coordinates (Xf, Yf, Zf) in the virtual space of the finger F that is determined to be facing away from the user (nail side) is included in the above-set area R.

[0032] Then, while the determination processing unit 33 determines that the corresponding position coordinates (Xf, Yf, Zf) of the finger F in the virtual space are not included in the set region R, the object control unit 34 does not perform any processing. On the other hand, when the determination processing unit 33 determines that the corresponding position coordinates (Xf, Yf, Zf) of the finger F in the virtual space are included in the set region R, the Z-axis value (here, Zj regardless of the X and Y coordinate values) of the virtual sheet-like object J at the position of the position coordinates (Xf, Yf) of the finger F as seen from the user side is compared with the Z-axis value Zf of the position coordinates of the finger F. Here, if Zf>Zj, the finger F is located farther away than the virtual sheet-like object J, and in this case, the object control unit 34 partially deforms the virtual sheet-like object J so that the region Rf in the X and Y planes in which the finger F of the virtual sheet-like object J is displayed is located on the distal side (farther away from the user) than the corresponding position of the finger F. This transformation can be performed by a known method using, for example, height mapping, and therefore a detailed description thereof will be omitted here.

[0033] The output unit 35 renders the objects in the virtual space that have been set by the object setting unit 32 (and further transformed by the object control unit 34 under certain conditions) as images from a virtual camera virtually placed at the position of the user's eyes (for example, the position of the imaging unit 23 of the display device 2), and sends image data obtained as a result of the rendering to the display device 2.

[0034] The processing control unit 36 ​​executes a predetermined process based on the image displayed on the surface of the target virtual object and the information on the position of the user's finger. Specifically, when the determination processing unit 33 changes from a state in which it has determined that the finger F, which is determined to be facing its back, is in contact with the target virtual object to a state in which it has determined that the finger F, which is determined to be facing its back, is not in contact with the target virtual object, the processing control unit 36 ​​executes a process (e.g., starting a program) that has been set in advance for the user interface component, assuming that a user interface component (such as a virtual button) in an image portion displayed on the surface of the target virtual object at the position of the finger F has been pressed.

[0035] Furthermore, in this embodiment, when the user performs an operation of moving the finger while maintaining the state in which the determination processing unit 33 has determined that the finger F, which is determined to be facing away, is in contact with the target virtual object, that is, when an operation such as a so-called swipe or pinch-in / pinch-out (when an operation is performed to change the distance between the fingers in a state in which it is determined that two or more fingers are in contact) is performed, the process control unit 36 ​​moves the position that is concave toward the back to the position of the fingers after the movement, and executes processing that has been specified in advance by an application program or the like in response to the swipe or pinch-in / pinch-out operation.

[0036] [Operation] An example of this embodiment has the above configuration and operates as follows: That is, the information processing device 1 of this embodiment executes an application program such as a game, and during the execution, performs processing to display a menu (an image in which interface components of selection items are arranged) in a virtual space.

[0037] At this time, the information processing device 1 of this embodiment generates an image representing the user interface of the menu in accordance with instructions from the application program, and generates a surface texture for a virtual sheet-like object with the image set as the surface texture.

[0038] Then, the information processing device 1 places a virtual sheet-like object J in a coordinate system similar to that shown in FIG. 4, as shown in FIG. 5, such that the normal to the surface of the object J is parallel to the Z axis. That is, the object J is placed at the position Z=Zj. The shape of the object J in the XY plane is rectangular. The coordinates of the upper left corner as seen from the user's side are (Xtl, Ytl) and the coordinates of the lower right corner are (Xrb, Yrb). The thickness in the Z axis direction is substantially "0".

[0039] The information processing device 1 then renders object J in the virtual space as an image from a virtual camera virtually placed at the position of the user's eyes (e.g., the position of the imaging unit 23 of the display device 2), and transmits the image data obtained as a result of the rendering to the display device 2.

[0040] The display device 2 then superimposes (transparently) the result of this rendering on the image in real space in the same direction, allowing the user to visually recognize an image such as that shown in Fig. 6 (S1).

[0041] If the user extends his / her hand so that his / her index finger F reaches the position Z=Zj in the virtual space, the user will also see the hand as an image in the real space. In this state, the user's index finger F is facing its back (nail side) toward the user.

[0042] In this state, when the user extends the finger F to the button B on the object J displayed by the display device 2, the information processing device 1 recognizes the position (position of the fingertip of the finger F) and the orientation (whether the back side or the ventral side is facing the user) of the user's finger F in the real space. Here, the information processing device 1 determines that the orientation of the finger F is "back side."

[0043] The information processing device 1 then checks whether a virtual three-dimensional region R including the outline of a predetermined virtual sheet-like object J (this region R is a hexahedral region surrounded by a rectangular proximal side with diagonals at (Xtl, Ytl, Zj-ΔZ) and (Xrb, Yrb, Zj-ΔZ) and a rectangular distal side with diagonals at (Xtl, Ytl, Zj+ΔZ) and (Xrb, Yrb, Zj+ΔZ)) contains coordinates in the virtual space corresponding to the position of the fingertip of the user's finger F in real space.

[0044] Specifically, when the coordinates in the virtual space corresponding to the position of the tip of the user's finger F in the real space are (Xf, Yf, Zf), the information processing device 1 calculates: Xtl≦Xf≦Xrb, and Ytl≦Yf≦Yrb, and Zj-ΔZ≦Zf≦Zj+ΔZ It is judged whether the above condition (called the contact condition) is satisfied.

[0045] If it is determined that the coordinates of the fingertip of the dorsal finger F satisfy the above contact conditions, the information processing device 1 compares the Z-axis value (here, Zj regardless of the X and Y coordinate values) of a virtual sheet-like object J located at the position coordinates (Xf, Yf) of the finger F as seen from the user's side with the Z-axis value Zf of the position coordinates of the finger F.

[0046] If Zf>Zj, the finger F is located farther away than the virtual sheet-like object J, so the information processing device 1 partially deforms the virtual sheet-like object J so that the region Rf in the X, Y plane where the finger F of the virtual sheet-like object J is displayed is located distal (farther away from the user) than the corresponding position of the finger F. In other words, the Z-axis value of each point of the object J located at a position of a radius r equal to or less than a predetermined R with the position (Xf, Yf) as the center is Zj+z0(Zf-Zj)·exp(-r 2 / R 2 ), where z0 is an empirically determined constant equal to or greater than 1. In this way, the part of object J that is centered on the position (Xf, Yf) and has a radius R or less is deformed as if it were pushed in further from the user's perspective than the position of finger F (S2). Here, a bell-shaped function is used as the shape of this deformation, but the shape is not limited to this.

[0047] During this process, the information processing device 1 renders the object J in the virtual space as an image from a virtual camera virtually placed at the position of the user's eyes (for example, the position of the imaging unit 23 of the display device 2), and transmits image data obtained as a result of the rendering to the display device 2. The display device 2 then displays the result of this rendering by superimposing it (transparently) on an image in real space in the same direction.

[0048] At this time, the information processing device 1 may perform control so as to mask and not draw the portion of the virtual sheet-like object J that overlaps with the user's fingers. This allows the user to clearly see that an interface component such as a button on the object J is being operated, and prevents the finger from penetrating through the object J and resulting in unnatural drawing.

[0049] The information processing device 1 further identifies an interface component, such as a button, on the object J that is located at a position corresponding to the (X, Y) coordinates of the fingertip of the finger F when the coordinates of the fingertip of the finger F no longer satisfy the contact condition. If the identified interface component is found, the information processing device 1 executes a process that is predetermined on the application program side as a process to be performed when the user operates the interface component.

[0050] [Example of moving an object] In the explanation so far, in order to prevent the user's finger from passing through the target virtual object displayed by the display device 2, a part of the target virtual object corresponding to the position of the finger is deformed so that the part of the target virtual object is positioned farther away than the position of the user's finger, but the present embodiment is not limited to this.

[0051] For example, if there is a finger F that is facing away from the user and that the determination processing unit 33 has determined to be in contact with the target virtual object set by the object setting unit 32, the object control unit 34 determines whether the part of the target virtual object corresponding to the finger F is on the user side or further back than the finger F. When it is determined that the part of the target virtual object corresponding to the finger F is on the user side of the finger F, the position of the entire target virtual object may be translated in the Z axis direction to be set farther away than the finger F so that the part of the target virtual object corresponding to the finger F is further back than the finger F.

[0052] Furthermore, in one example of this embodiment, when it is determined that the coordinates of the fingertip of the back finger F satisfy the above-mentioned contact condition, the information processing device 1 compares the Z-axis value (Zj(Xf, Yf)) of the target virtual object J located at the position of the position coordinates (Xf, Yf) of the finger F as seen from the user side with the Z-axis value Zf of the position coordinates of the finger F, Zth>Zf-Zj(Xf,Yf)>0 If so, deform a part of the target virtual object so that a portion of the target virtual object corresponding to the position of the finger is located farther away than the position of the user's finger; Zf-Zj(Xf,Yf)≧Zth In this case, the position of the entire target virtual object may be translated in the Z-axis direction to set the part of the target virtual object corresponding to the finger F further away than the finger F. Note that Zth here is a preset threshold value (Zth>0).

[0053] In this example, while the user's finger is less than Zth behind the original position of object J on the Z axis, a part of object J is deformed and displayed as if it is partially pressed in, and when the user continues to move the finger beyond Zth, the whole of object J moves, making it possible to perform a movement operation in the Z axis direction of object J. When the whole of object J is moved in this way, the information processing device 1 may be configured not to execute an operation on the interface component at the corresponding position even if the coordinates of the user's fingertip no longer satisfy the contact condition.

[0054] [Mask Object] Furthermore, in this embodiment, the information processing device 1 refers to the depth information output by the display device 2, and when the user's finger is in front of point (x, y, Zj(x, y)) on the target virtual object J, that is, when the Z-axis value Zv in the virtual space corresponding to the distance on the Z-axis to the target in real space represented by the depth information at point (x, y) in the field of view is Zj(x, y)>Zv, the image at the position (x, y) of the object J may be masked and not drawn. This makes it possible to express a state in which the object is hidden by an object in the foreground.

[0055] [Ventral finger] Furthermore, the information processing device 1 of this embodiment may perform control so as not to mask the rendering of the object J at a position corresponding to the user's finger F when the user's finger F facing its ventral side (opposite the nail) is located behind the point (x, y, Zj(x, y)) on the target virtual object J, that is, when the Z-axis direction component Zf of the position of the fingertip of the finger F is Zf>Zj(x, y). In this case, the image of the user's own finger with its ventral side facing the user is semi-transparently composited with the corresponding part of the object J by the display device 2 and presented to the user.

[0056] In this process, when determining whether the Z-axis component Zf of the position of the fingertip of finger F is Zf>Zj(x,y), the point (x,y,Zj(x,y)) on the target virtual object J may use the coordinate value before deformation, even if the target virtual object J is displayed partially deformed according to the position of another finger that is facing away.

[0057] [Multiple Finger Recognition] Furthermore, in this embodiment, the information processing device 1 recognizes the positions and orientations of the user's fingers. Furthermore, the information processing device 1 in this embodiment may obtain information on the positions and orientations (whether the back side or the ventral side faces the user) of the fingers of each of the user's left and right hands.

[0058] In this example, the information processing device 1 sequentially acquires coordinate values ​​in the XYZ coordinate system of the virtual space corresponding to the position in the real space of each finger FL1 to FL5 of the user's left hand, and information on the orientation of each finger. Similarly, the information processing device 1 sequentially acquires coordinate values ​​in the XYZ coordinate system of the virtual space corresponding to the position in the real space of each finger FR1 to FR5 of the user's right hand, and information on the orientation of each finger. Then, the information processing device 1 determines whether each finger is in contact with the target virtual object.

[0059] Furthermore, for example, when a first finger F1 determined to be facing its ventral side and a second finger F2 determined to be facing its dorsal side are both in contact with the target virtual object, and the distance between the coordinate value (x1, y1, z1) of the fingertip of the finger F1 in the XYZ coordinate system of the virtual space and the coordinate value (x2, y2, z2) of the fingertip of the finger F2 in the XYZ coordinate system of the virtual space is below a predetermined threshold (at this time, the finger F2 corresponding to the dorsal side of the target virtual object is determined to be facing its ventral side), part is deformed so as to be deeper than the finger F2, but the position in the Z-axis direction with the finger F1 facing ventral side is compared using the coordinate values ​​before deformation, so that the finger F1 is drawn in a semi-transparent composite), the information processing device 1 may determine that the target virtual object is virtually sandwiched between the fingers F1 and F2 at the positions of the fingers F1 and F2 (a predetermined range including (x1, y1) and (x2, y2), for example, the range of the smallest circle including these), and execute a predetermined process.

[0060] For example, when the user moves the fingers F1 and F2 in a state where it is determined that the fingers are virtually sandwiched, the entire target virtual object may be translated in the virtual space according to the amount of movement. As an example, when it is recognized that the finger F1 has moved from (x1, y1, z1) to (x1+Δx1, y1+Δy1, z1+Δz1), the information processing device 1 translates the placement position of the target virtual object in the virtual space by Δx1 in the X-axis direction, Δy1 in the Y-axis direction, and Δz1 in the Z-axis direction.

[0061] In addition, in a state where a target virtual object is determined to be pinched at a plurality of positions P1 and P2 (usually, at one position P1, it is determined to be pinched by a plurality of fingers among FL1 to FL5 of the left hand, and at the other position P2, it is determined to be pinched by a plurality of fingers among FR1 to FR5 of the right hand), if the distance between any of the fingers Fp1 determined to virtually pinch the target virtual object at position P1 and any of the fingers Fp2 determined to virtually pinch the target virtual object at position P2 varies, the size of the target virtual object in the virtual space may be changed according to the variation in the distance. For example, when it is recognized that the distance has changed by L+ΔL from the original distance L, the information processing device 1 multiplies the size of the target virtual object by (L+ΔL) / L.

[0062] In this example, as shown in FIG. 7, when both ends of a target virtual object are held with both hands and the hands are moved apart (in the direction of the arrows in the figure), the information processing device 1 recognizes the hand movement and controls the target virtual object to be enlarged in the virtual space.

[0063] Furthermore, when there are multiple fingers that are determined to be in contact with the target virtual object and all of them are facing the same direction (either dorsal or ventral), the information processing device 1 may process the target virtual object as being moved in the direction of the movement of the multiple fingers. Also, when the distance between the multiple fingers is expanding or contracting, the information processing device 1 may execute the corresponding process by assuming that an operation of expanding or contracting an image displayed on the surface of the target virtual object (a texture set on the surface of the target virtual object) has been performed.

[0064] [Attachment to objects in real space] Furthermore, the information processing device 1 of this embodiment may recognize the position and range of not only the user's fingers but also the surfaces of walls, desks, etc. arranged in the real space. Such a plane recognition process can employ a widely known method such as a process of determining using depth information output by the display device 2.

[0065] When the information processing device 1 determines that a target virtual object has been moved and is in a state of touching one of the recognized planes, it may determine that the target virtual object is “pasted” at the position of that plane and may stop moving the target virtual object at that point.

[0066] In this example, when the information processing device 1 determines that a movement operation of the target virtual object has been performed and that the target virtual object has come into contact with one of the recognized planes, the information processing device 1 may stop moving the target virtual object at that point, determining that the target virtual object is “pasted” at the position of the plane only when the entire target virtual object in the field of view is contained within the recognized plane (the entire object overlaps with the recognized plane).

[0067] In this way, the target virtual object is prevented from being "pasted" onto a plane smaller than the target virtual object.

[0068] In this case, for example, when a user moves a finger from the periphery of the target virtual object inward, an animation of the target virtual object peeling off may be displayed as a "peeled off" state. After that, the display of the target virtual object may be stopped, or the target virtual object may be controlled so that a moving operation is performed again from the peeled off position.

[0069] When the target virtual object is pasted, the information processing device 1 stops the partial transformation process of the target virtual object, regardless of the position of the user's finger (whether or not it is determined that the target virtual object is in contact with the object). This prevents the target virtual object from being transformed beyond the plane on which the target virtual object is pasted.

[0070] [Finger side recognition] The information processing device 1 of this embodiment may further recognize the side of the user's finger from image data captured by the display device 2. When recognizing the side of the user's finger, if a region including the side of the finger is in contact with any side of the periphery of a target virtual object and moving toward the center of the target virtual object (for example, when the region including the side of the finger is in contact with the left side of the target virtual object and further moving to the right), the information processing device 1 may translate the target virtual object in accordance with the movement of the region including the side of the finger.

[0071] As an example, in this case, the information processing device 1 repeatedly acquires the center of gravity of the area including the side of the finger while the area including the side of the finger is in contact with one of the edges of the outer periphery of the target virtual object and moving toward the center of the target virtual object, and when the center of gravity moves by (Δx, Δy) in the XY plane, translates the target virtual object that is in contact with the area including the side of the finger by (Δx, Δy) in the XY plane.

[0072] Furthermore, in one example of this embodiment, the magnitude (ΣΔx) of the accumulated movement amount (ΣΔx, ΣΔy) from the time when it is determined that the area including the side of the finger has come into contact with the outer periphery of the target virtual object is 2 +(ΣΔy) 2 exceeds a predetermined threshold, the display of the target virtual object may be stopped.

[0073] [Object display start instruction] Furthermore, the information processing device 1 of this embodiment may control whether to display or stop displaying a target virtual object, or control the display mode, depending on the pose of the user's hand (the state of each finger).

[0074] As an example, when the information processing device 1 determines that the user has clenched his / her hand, it starts displaying a predetermined target virtual object. At this time, the information processing device 1 displays the target virtual object within a predetermined distance range from an area in which the user can visually recognize the clenched hand. Therefore, in this case, when the user moves the clenched hand, the information processing device 1 moves the position of the target virtual object in the virtual space by an amount of movement in the virtual space corresponding to the amount of movement (amount of movement in the real space).

[0075] Furthermore, the information processing device 1 may detect the strength of the grip of the hand. Specifically, in this case, a device (not shown) that recognizes the grip strength of the user's hand may be worn, an output signal from the device may be wirelessly received via the communication unit 13, and the strength may be detected based on the received output signal. In this case, the information processing device 1 may perform a process such as increasing the size of the target virtual object to be displayed as the grip strength of the hand increases.

[0076] Furthermore, in this example, the information processing device 1 may display the target virtual object only while the user is fisting his / her hand. As already described, when a user interface component is arranged on the surface of the target virtual object, in this example, an instruction can be input using the user interface only when the user is fisting his / her hand.

[0077] Furthermore, in this case, after a target virtual object is displayed with the user's hand closed, when the user virtually pinches the target virtual object between his or her fingers and moves it, the display of the target virtual object may not be interrupted even if the user then opens his or her hand.

[0078] In this process, while the target virtual object is displayed with the hand closed, the target virtual object is displayed in the vicinity of the user's predetermined part (here, for example, the arm) in "movement mode", and the information processing device 1 moves the target virtual object according to the movement of the arm, which is an example of the user's predetermined part. When the target virtual object displayed in "movement mode" is virtually pinched and moved between the fingers, the target virtual object is displayed and stays at the moved position, and the information processing device 1 does not interrupt the display of the target virtual object even if it is detected that the user has opened his / her hand. In this "air mode", the information processing device 1 may control the position of the target virtual object in the virtual space not to move, without following the movement of the arm, which is an example of the user's predetermined part.

[0079] Furthermore, as already described, the target virtual object in this "air mode" may be movable by the user virtually pressing its side with his / her fingers or virtually pinching it with his / her fingers through processing by the information processing device 1. When moved in this way, for example, when the target virtual object is moved to a flat surface such as a desk in the real space, the target virtual object may be displayed as being virtually pasted onto the flat surface (paste mode).

[0080] [Instruction to stay in the air] Furthermore, in one example of this embodiment, when a user performs an operation that changes the relative positions of multiple target virtual objects (for example, an operation that moves one target virtual object to the position of the other target virtual object), the information processing device 1 may execute a predetermined process regarding the multiple target virtual objects.

[0081] For example, when the information processing device 1 displays a virtual push pin object Jp in a virtual space as a target virtual object, and on the other hand displays a virtual sheet-like object J as another target virtual object, when the user moves the push pin object Jp to a position (a position in the virtual space) where it overlaps with the sheet-like object J and performs an operation to complete the movement there (for example, an operation to push the push pin object Jp away from the user), the information processing device 1 may control the virtual sheet-like object J so that it does not move from the position where the operation was performed by a subsequent operation.

[0082] This control can be realized, for example, by causing the information processing device 1 not to perform any processing for an operation performed by a user to move a virtual sheet-like object J.

[0083] In this case, when the user moves the push pin object Jp from a position where it overlaps with the sheet-like object J to a position where it does not overlap with the sheet-like object J, the virtual sheet-like object J may then be controlled to be movable by the user's operation.

[0084] [Pop-up window] Furthermore, in one example of this embodiment, the information processing device 1 may generate and display another target virtual object by superimposing it on the target virtual object.

[0085] In this case, for example, while a user's finger is pressing a button displayed on a certain target virtual object (at this time, the button portion of the target virtual object is controlled to be deformed and displayed so as to be concave toward the back), another target virtual object may be displayed superimposed on the target virtual object like a so-called pop-up window.

[0086] In this case, the information processing device 1 may perform control so as not to move the other target virtual object behind the user's finger even if the user's finger overlaps the other target virtual object until the user's finger moves once to the outside of the other target virtual object (to a position where the other target virtual object does not overlap in the rendered field of view). Such control can be realized by associating a flag indicating whether the user's finger has moved once to a position where the other target virtual object does not overlap with the other target virtual object with the newly generated target virtual object, and performing control so as not to move the other target virtual object behind the user's finger while the flag is not in a state indicating that the user's finger has moved once to a position where the other target virtual object does not overlap with the other target virtual object.

[0087] In addition, if the user does not move the finger outside the other target virtual object, but moves the finger in the real space to a position in front of the position in the virtual space where the other target virtual object is displayed, an animation may be displayed that deforms the other target virtual object so as to avoid the finger. For example, if the other target virtual object is a sheet-like object, an animation of the sheet being rolled up may be displayed to show that the user's finger has moved the target virtual object to the front while lifting it. In this case, when the user moves the finger in front of the target virtual object, the user's finger may be assumed to have moved once to a position where it does not overlap, and subsequent processing may be performed.

[0088] [Virtual object shape] Furthermore, the shape of the target virtual object displayed by the information processing device 1 is not limited to a sheet or a rectangular parallelepiped. For example, the target virtual object J may be cylindrical, and images of user interface components may be continuously arranged on the cylindrical surface, as illustrated in FIG.

[0089] In this example, when it is determined that the user has virtually stroked the surface of the target virtual object J with, for example, two fingers in one direction, the information processing device 1 may rotate and move the target virtual object J in the direction of the stroke, with the center of the cylinder as the center of rotation, in the direction of the arrow in Fig. 8. In this way, the target virtual object of the cylinder is rotated, and the user can access the user interface components that were arranged in the part that became visible to the user as a result of the rotation.

[0090] [Contact status display] Furthermore, the information processing device 1 may set the display mode of the target virtual object so that the display mode of the target virtual object in the virtual space differs between a state in which the target virtual object is in contact with the user's finger and a state in which the target virtual object is not in contact with the user's finger.

[0091] As an example, the information processing device 1 changes the color of the outer periphery (contour line) of the target virtual object depending on whether the target virtual object is in contact with the user's finger or not.

[0092] In addition, when a user is wearing a device capable of presenting a haptic sensation on a hand or the like, the information processing device 1 may instruct the device to present a predetermined haptic sensation when a target virtual object in the virtual space is in contact with the user's finger or when the contact state between the target virtual object and the user's finger is maintained.

[0093] This allows the user to receive haptic feedback, for example, when the target virtual object in the virtual space comes into contact with the user's finger.

[0094] [Restrict movement] Furthermore, in one example of this embodiment, the information processing device 1 may identify the position and shape of an object (real object) such as a wall or a desk in the real space (this method is widely known, so a detailed explanation will be omitted here), and place a corresponding virtual object (corresponding object) at a corresponding position in the virtual space (placed to match the relative position from the user).

[0095] In this case, while the user is moving the target virtual object, contact between the moving target virtual object and a corresponding object in the virtual space may be determined, and the target virtual object may be controlled not to move to a position where contact is determined to occur.

[0096] In this example, the shape of the corresponding object does not necessarily have to conform to the shape of the corresponding real object in real space. For example, the circumscribed rectangular solid of the real object may be the shape of the corresponding object in virtual space. [Explanation of symbols]

[0097] 1 information processing device, 2 display device, 11 control unit, 12 storage unit, 13 communication unit, 21 control unit, 22 communication unit, 23 imaging unit, 24 depth camera, 25 display unit, 31 finger detection unit, 32 object setting unit, 33 determination processing unit, 34 object control unit, 35 output unit, 36 processing control unit.

Claims

1. a display device that presents an image of the virtual space to a user; An acquisition means for acquiring information on the position of each fingertip of a user in a real space; an object setting means for setting a position and an image of a virtual object in a virtual space; a determination means for determining whether a virtual object set in the virtual space is in contact with a user's finger; a shape setting means for setting the virtual object in a partially deformed state so that a portion of the virtual object corresponding to a position of the user's finger that is determined to be in contact with the object is positioned on a more inward side of the user than the finger; having An information processing device that displays the virtual object, the shape of which has been set, as an image in a virtual space on the display device.

2. 2. The information processing device according to claim 1, The object setting means is an information processing device that sets the position of the virtual object in a virtual space so that the virtual object is located behind a finger of the user that is determined to be in contact with the object.

3. 3. The information processing device according to claim 1, The object setting means sets the virtual object so that the display manner of the virtual object differs when the determination means determines that the virtual object is in contact with one of the user's fingers and when the determination means determines that neither of the fingers is in contact with the virtual object.

4. 4. The information processing device according to claim 1, The object setting means is an information processing device that performs processing of a movement mode in which, in response to an instruction from the user, the position of the virtual object is moved by an amount of movement in the virtual space corresponding to the amount of movement of a specified part of the user in the real space.

5. 5. The information processing device according to claim 1, The object setting means is an information processing device that performs processing in an aerial mode in which the position of the virtual object is not moved, regardless of movement of a predetermined part of the user in real space, in response to an instruction from the user.

6. 6. The information processing device according to claim 1, means for recognizing a plane in real space; The object setting means is an information processing device that performs processing of a pasting mode in which the position of the virtual object is set to a position pasted on the plane in response to an instruction from the user.

7. 7. The information processing device according to claim 4, The object setting means executes processing in a plurality of modes including the movement mode, and when a predetermined operation is performed in each mode, the object setting means switches to processing in another mode and executes processing.

8. The information processing device according to claim 1 , The acquiring means further acquires directional information indicating whether each finger of the user is facing away from the user or facing toward the user's belly; The shape setting means is an information processing device that sets the virtual object to a partially deformed state so that a portion of the virtual object corresponding to the position of a finger of the user that is turned away and is determined to be in contact with the object is positioned further back from the finger toward the user.

9. 9. The information processing device according to claim 8, The information processing device further draws the virtual object by superimposing the virtual object on one of the user's fingers, the pad of which is facing the virtual object.

10. using a computer connected to a display device that presents an image of a virtual space to a user; Acquire information on the position of each fingertip of the user in the real space, Setting the position and image of a virtual object in a virtual space; determining whether a virtual object set in the virtual space is in contact with a user's finger; setting the virtual object to a partially deformed state such that a portion of the virtual object corresponding to a position of the user's finger that is determined to be in contact with the object is positioned on a deeper side of the user than the finger; A control method for an information processing device that causes the display device to display the virtual object, the shape of which has been set, as an image in a virtual space.

11. A computer connected to a display device that presents an image of a virtual space to a user, An acquisition means for acquiring information on the position of each fingertip of a user in a real space; an object setting means for setting a position and an image of a virtual object in a virtual space; a determination means for determining whether a virtual object set in the virtual space is in contact with a user's finger; a shape setting means for setting the virtual object in a partially deformed state so that a portion of the virtual object corresponding to a position of the user's finger that is determined to be in contact with the object is positioned on a more inward side of the user than the finger; a means for displaying the virtual object, the shape of which has been set, as an image in a virtual space on the display device; A program that functions as a

Images