Information processing device, information processing method, program

Abstract

The present invention provides an information processing device that allows users to input geometric shape information into a display space with greater accuracy. [Solution] A planar operation palette used for inputting geometric information into the display space is set in the display space of the display. A definition plane is set which defines a predetermined interval distance that serves as a reference when inputting geometric information into the plane of the display space including the surface of the operation palette.

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] A technique for displaying a virtual object within the field of view of a simulated environment on a display screen is disclosed in Patent Document 1.

[0003] For example, in paragraphs 0080 and 0082 of Patent Document 1, it is described that a handheld controller is used to generate a rectangular parallelepiped within a simulated environment. Also, in this technique, the first 3D point within the simulated environment is selected as the first vertex of the rectangular parallelepiped, and the rectangular parallelepiped is "dragged" to a location in the target 3D space with a single gesture, and the final position of the handheld controller corresponds to the vertex of the second diagonal of the rectangular parallelepiped, and it is disclosed that the user moves the handheld controller to drag the end point corresponding to the second vertex of the rectangular parallelepiped, etc.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a technique such as Patent Document 1, there is a need for a technique that allows a user to input geometric shape information into a display space with higher accuracy.

[0006] An object of this disclosure is to provide an information processing apparatus, an information processing method, and a program that solve the above problems.

Means for Solving the Problems

[0007] An information processing device according to one aspect of the present disclosure includes setting a planar operation palette, which is used for inputting geometric information into the display space, in the display space of a display, and setting a definition plane on the plane of the display space, which includes the surface of the operation palette, that defines a predetermined interval distance that serves as a reference when inputting the geometric information.

[0008] An information processing method according to one aspect of the present disclosure includes setting a planar operation palette, which is used for inputting geometric information into the display space, in the display space of a display, and setting a definition plane on the plane of the display space, which includes the surface of the operation palette, that defines a predetermined interval distance that serves as a reference when inputting the geometric information.

[0009] A program according to one aspect of the present disclosure causes the computer of an information processing device to function as means for setting a planar operation palette, which is used for inputting geometric information into the display space, in the display space of a display, and means for setting a defining plane that defines a predetermined interval distance that serves as a reference when inputting the geometric information into the plane of the display space, which includes the surface of the operation palette. [Effects of the Invention]

[0010] According to one embodiment described above, it is possible to provide an information processing device that allows users to input geometric shape information into the display space with greater accuracy. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of an information processing system according to one aspect of this disclosure. [Figure 2] This figure shows the hardware configuration of an HMD according to one aspect of the present disclosure. [Figure 3] This figure shows an example of the display on an HMD according to one aspect of the present disclosure. [Figure 4] This is a functional block diagram of an HMD according to one aspect of the present disclosure. [Figure 5]It is a diagram showing an example of display of an operation palette according to an aspect of the present disclosure. [Figure 6] It is a diagram showing an example of a posture set for an operation palette according to an aspect of the present disclosure. [Figure 7] It is a diagram showing an example of the interval between grids of a grid surface according to an aspect of the present disclosure. [Figure 8] It is a first diagram showing an example of display when a line, which is one of the geometric shapes according to an aspect of the present disclosure, is input. [Figure 9] It is a second diagram showing an example of display when a line, which is one of the geometric shapes according to an aspect of the present disclosure, is input. [Figure 10] It is a diagram showing an example of movement of a display space according to an aspect of the present disclosure. [Figure 11] It is a first diagram showing a user's instruction gesture according to an aspect of the present disclosure. [Figure 12] It is a second diagram showing a user's instruction gesture according to an aspect of the present disclosure. [Figure 13] It is a first diagram showing an example of a user's operation of inputting a geometric shape according to an embodiment of the present disclosure. [Figure 14] It is a second diagram showing an example of a user's operation of inputting a geometric shape according to an embodiment of the present disclosure. [Figure 15] It is a diagram showing a processing flow of an HMD according to an embodiment of the present disclosure. [Figure 16] It is a diagram showing a display range of virtual information according to an embodiment of the present disclosure. [Figure 17] It is a functional block diagram in another example of an information processing apparatus according to an aspect of the present disclosure. [Figure 18] It is a diagram showing a processing flow in another example of an information processing apparatus according to an aspect of the present disclosure.

MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, the information processing apparatus of the present disclosure will be described with reference to the drawings. FIG. 1 is a diagram showing an example of an information processing system including a head-mounted display, which is one aspect of the information processing apparatus of the present disclosure. As shown in FIG. 1, the information processing system 100 includes at least a head-mounted display 1, which is one aspect of the information processing apparatus of the present disclosure. Hereinafter, the head-mounted display 1 will be referred to as HMD1. The information processing system 100 may be configured such that HMD1 and a server are communicatively connected.

[0013] FIG. 2 is a diagram showing the hardware configuration of the HMD according to an embodiment of the present disclosure. As shown in FIG. 2, HMD1 is a computer including hardware such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage device 104, a communication module 105, a camera 106, a display 107, a speaker 108, a vibrator 109, and the like.

[0014] HMD1 is worn by the user on their head. HMD1 has a display 107 on the inside (body side) when worn by the user. The user can see the information displayed on the display 107 at eye level when wearing HMD1 on their head. A camera 106 on HMD1 takes pictures of the user's surroundings. Camera 106 may have the function of a distance sensor that can measure the distance to an object located in real space. HMD1 may display the images of the user's surroundings generated by camera 106 on the display 107. The display 107 of HMD1 may be light-transmitting, in which case the user can see various objects around the user through the display using ambient light that has passed through the display 107. The display 107 can display information generated by HMD1. The user can see the information (geometric information, etc.) generated by HMD1 and displayed on the display 107. Furthermore, the user can simultaneously view the information (such as geometric information) generated by the HMD1 and displayed on the display 107, as well as the surrounding objects. Note that the HMD1 displays images of the surrounding objects on the display 107, and there may be cases where the user cannot directly view the surrounding objects through the display 107.

[0015] Based on user input, the HMD1 can display a planar shape operation palette 2, used for inputting geometric shape information into the display space, in the display space of the display, as shown in Figure 1. The display space may be defined as a virtual space, an augmented reality space, or a mixed reality space.

[0016] Figure 3 shows an example of what is displayed on the HMD's display. As an example, suppose an object 3, such as a shelf as shown in Figure 3, is placed in the real space where the user is located. The user can see the object 3 through the display 107. The user can input geometric information such as lines and rectangles for the object 3 into the display space of the display 107 to, for example, design a room layout. In order to display such geometric information on the display 107, the user inputs a display instruction from the operation palette 2 to the HMD1. As a result, the HMD1 displays the operation palette 2 on the display. The operation palette 2 displays a definition plane that defines predetermined intervals that serve as a reference when inputting geometric information into the plane of the display space, which includes the plane of the operation palette 2. Specifically, the definition plane is a grid plane 21 with lines drawn in a grid pattern as shown in Figure 3. Hereafter, the definition plane will be referred to as the grid plane 21.

[0017] Here, the display orientation of the operation palette 2 in the display space including the grid surface 21 is based on setting information that is set in advance by the user and stored in the storage device 104 of the HMD1. Based on the setting information, the HMD1 determines and displays the display orientation of the operation palette 2 in the display space having the grid surface 21.

[0018] Figure 4 is a functional block diagram of the HMD. The HMD1 starts a pre-programmed device. This allows the HMD1 to perform the functions of the control unit 11, detection unit 12, display unit 13, and notification unit 14. The control unit 11 controls other functional units. The detection unit 12 detects real pointers and objects 3 in real space. The display unit 13 performs processing that includes control related to display, such as generating and displaying information to be displayed on the display 107. The notification unit 14 notifies the user of notification information using the speaker 108 and the transducer 109.

[0019] The HMD1 of this disclosure sets the operation palette 2 in the display space of the display 107 based on setting information, and sets the grid plane 21 on the plane of the display space that includes the surface of the operation palette 2.

[0020] HMD1 may identify a position in the display space corresponding to the position of a predetermined physical pointer detected in real space, and then display a virtual pointer. HMD1 may again identify a position in the display space corresponding to the movement position of the physical pointer in real space, display a virtual pointer at that identified position, and display geometric shape information based on the grid of the grid surface 21 displayed in the display space according to the movement of the physical and virtual pointers and the orientation of the operation palette 2.

[0021] Here, as an example, the physical pointer is the fingertip of either the left or right hand of the user detected in real space. The physical pointer may be an object other than a human finger. The virtual pointer may be a fingertip of a hand displayed in display space. The virtual pointer may also be an object other than a human fingertip (such as a pen). HMD1 can change the display orientation of the operation palette 2 in the display space based on changes in setting information made by the user. HMD1 may display multiple operation palettes 2 in the display space. The user may input multiple geometric shapes into the display space by switching between multiple operation palettes 2 displayed by HMD1 in the display space.

[0022] The HMD1 displays lines representing a part of a geometric shape on the grid surface 21 based on the movement of a virtual pointer, and outputs notification information according to the increase or decrease in the displayed length of the lines at predetermined intervals. The notification information may be sound or vibration. When the notification information is output as sound, the HMD1 uses speaker 108 to output a sound as notification information indicating that the length of the line has increased or decreased in grid units (predetermined intervals). When the notification information is notified as vibration, the HMD1 uses vibrator 109 to output a vibration as notification information indicating that the length of the line has increased or decreased in grid units (predetermined intervals). Because the user can detect that the length of the line has increased or decreased in grid units by sound or vibration, they can easily grasp the length of the lines needed to input geometric shapes.

[0023] In the following example, the real pointer is the fingertip of either the left or right hand detected in real space, and the virtual pointer is the fingertip of the hand displayed in display space (or the end point of a line displayed in display space, etc.). HMD1 identifies the type of geometric shape based on the shape detected by the other hand, which is different from the first hand. Based on the movement of the other hand and the position of the real or virtual pointer, HMD1 detects multiple coordinates in display space and displays information (geometric shape) based on the type of geometric shape based on these multiple coordinates.

[0024] HMD1 identifies the start and end coordinates of the geometric shape from among the multiple coordinates described above based on different movements of the other hand. HMD1 also identifies the intermediate coordinates of the geometric shape from among the multiple coordinates described above based on different movements of the other hand. The geometric shape may include at least one or more of the following: lines, rectangles, circles, ellipses, continuous lines, polygons, and any other shape.

[0025] Figure 5 shows an example of the operation palette display. For example, suppose the detection unit 12 of the HMD1 detects a point cloud of object 3 based on the distance to object 3 contained in each pixel of the captured image. The multiple points in Figure 5 represent the point cloud of the surface of object 3. A plane can be formed based on at least three points outside this point cloud of object 3. Based on the output instruction of the operation palette 2, the HMD1 may set and display the operation palette 2 on the plane of object 3 based on this point cloud of the surface of object 3. Alternatively, the HMD1 may identify the orientation of object 3 relative to the plane (reference plane) of object 3 based on the angle indicated by the output instruction and setting information of the operation palette 2, and set or display the operation palette 2 in the display space based on that orientation. The reference plane may be a plane with a pre-set inclination in the display space of the display 107, independent of object 3. As an example, the reference plane may be defined as a plane whose normal direction coincides with the user's line of sight and which is perpendicular in the display space.

[0026] Figure 6 shows an example of the posture set for the operation palette. The settings information stored by HMD1 may define a first angle θ1 and a second angle θ2 in three axial directions (X, Y, and Z directions) with respect to an origin set on a predetermined plane (reference plane). The first angle θ1 may be the angle between one side of the rectangular plane of the operation palette 2 and the axis in the X direction. The second angle θ2 may be the angle between the rectangular plane of the operation palette 2 and the reference plane. The user can input the first angle θ1 and the second angle θ2 stored as settings information by HMD1 into HMD1.

[0027] After setting or displaying the operation palette 2 in the display space based on user instructions, the HMD1 may change the set or displayed posture based on instructions from the user to make minor adjustments to their posture. For example, the user reaches out and touches the operation palette 2 displayed on the HMD1's display. The detection unit 12 detects from the image that the position of the operation palette 2 displayed on the HMD1's display matches the position of multiple fingers of the hand in real space (real pointer) or the position of multiple fingers of a virtual hand displayed in the display space (virtual pointer), and that the posture of the plane composed of points of multiple fingertips has changed, for example, by the user rotating their wrist. The display unit 13 may change the posture of the operation palette 2 and display it so that the surface of the operation palette 2 follows the plane composed of points of multiple fingertips (for example, the thumb, index finger, and middle finger of the right hand) detected by the detection unit 12. Alternatively, the HMD1 may display change buttons or value input fields for first angle θ1, second angle θ2, etc., on the display 107, and change the posture of the operation palette 2 based on the user's input of that information.

[0028] Figure 7 shows an example of the spacing between the grid cells on a grid surface. The HMD1 can change the distance between the grid cells on the grid surface 21 and display it based on the scale specified by the user as setting information. Specifically, if the reference object is an actual object 3 such as a desk, and the scale of the object 3 relative to the user is 1 / 1, the HMD1 may set the distance between the grid cells on the grid surface 21 to 10 cm and display a square grid surface 21 with sides of 1 m (meter). Alternatively, if the reference object is an object (for example, a building) virtually scaled down on the display 107, and the scale of that virtual object relative to the user is 1 / 10, the HMD1 may set the distance between the grid cells on the grid surface 21 to 1 m and display a square grid surface 21 with sides of 10 m (meter).

[0029] Figure 8 is the first example showing the display when a line, which is one of the geometric shapes, is entered. After the user gives the HMD1 an instruction to input geometric shape information, they can input a line constrained to the grid surface 21 and virtually display it on the display 107 by moving a pre-set finger of either the left or right hand from a starting point to an arbitrary ending point. In this case, as shown in Figure 8, when the finger is moved to the right from a grid point p0 corresponding to a certain starting point coordinate G0, the display unit 13 displays on the display 107 a line that is constrained to the grid surface 21 and extends over a distance corresponding to the movement of the finger.

[0030] In this process, the display unit 13 identifies the grid point G0 on the grid plane 21 that is closest to the spatial coordinates of the first detected right hand position p1, which corresponds to the fingertip of the right hand (for example, the fingertip of the index finger (second finger) of the right hand) that the detection unit 12 detected as the actual pointer, as the starting point coordinate (x0, y0). The user moves their right hand fingertip to input a virtual line on the display 107. The display unit 13 detects the first intersection point G1 (or second intersection point G2) that is closest to the point where a perpendicular line drawn from the spatial coordinates of the second right hand position p2 (or third position p3), which is the position of the right hand fingertip after movement detected by the detection unit 12, intersects with the grid plane 21. As the user moves their right hand fingertip, the position of the first intersection point G1 (or second intersection point G2) on the grid plane 21 corresponding to the right hand fingertip changes further.

[0031] Assuming a coordinate system with grid point p0 as the origin (x0, y0), and defining the rightward direction of the grid plane 21 as +x, the leftward direction as -x, the upward direction as +y, and the downward direction as -y, the display unit 13 compares the distance (Lx or Ly) from the first intersection G1 (or second intersection G2) to the left, right, front, and back extension lines (+x direction extension line L1, -y direction extension line L2, -x direction extension line L3, +y direction extension line L4) that extend along the edges of each grid on the grid plane 21, using the grid point p0 at the starting coordinates (x0, y0) as a reference. Based on this comparison, the display unit 13 displays the +x direction extension line L1 that is close in distance from the first intersection G1 (or second intersection G2) and whose length to the grid intersection is close to the value indicated by the x-coordinate of the first intersection G1 (or second intersection G2) as geometric shape information on the display 107.

[0032] In other words, when the fingertips of the right hand move from the first position p1 (start coordinate G0) to the second position p2, the display unit 13 displays the extension line in the +x direction that is close to the second position p2 and extends to the grid point (x3, y0) on the grid plane 21 that is close to the coordinates of the second position p2 as geometric shape information on the display 107. Similarly, when the fingertips of the right hand move from the first position p1 (start coordinate G0) to the third position p3, the display unit 13 displays the extension line in the +x direction that is close to the second position p3 and extends to the grid point (x3, y0) on the grid plane 21 that is close to the coordinates of the third position p3 as geometric shape information on the display 107. Furthermore, based on the movement of the spatial coordinates of the right fingertip of the detection unit 12, the display unit 13 similarly calculates the distance of one of the lines extending in the +x direction L1, the -y direction L2, the -x direction L3, or the +y direction L4, starting from grid point G0, and displays it on the display 107 as a geometric line constrained to the grid surface 21.

[0033] The fingertip, which is the actual pointer detected by the detection unit 12, does not need to be in a corresponding real-space position on the grid surface 21. When the detection unit 12 detects the spatial coordinates of the fingertip, which is the actual pointer, it identifies the grid point closest to the intersection of the grid surface 21 and a perpendicular line drawn from the coordinates in the display space corresponding to those spatial coordinates, as the first intersection point G1 and the second intersection point G2 shown in Figure 8. The detection unit then identifies the grid point closest to the intersection of these perpendicular lines and the grid surface 21 from among the four extension lines in the up, down, left, and right directions, starting from the starting point coordinate G0, and displays the geometric shape lines in the same manner as described above. If the geometric shape is set to be a circle, the display unit 13 displays a circle as the geometric shape on the display 107, with the diameter being the line connecting the grid point G0, which is the starting point coordinate, and the grid point on the grid surface 21 in the display space identified based on the spatial coordinates of the fingertip, which is the current actual pointer (coordinates (x3, y0) if the hand is in the second position p2). Alternatively, if the geometric shape is set to be a rectangle, the display unit 13 displays a rectangle as a geometric shape on the display 107, with the vertices set to be diagonally opposite each other, using the grid point G0, which is the starting point coordinate, and the grid point of the fingertip in the display space identified based on the spatial coordinates of the fingertip, which is the current physical pointer (coordinates (x3, y-1) if the hand is in the third position p3). Similarly, the display unit 13 displays a rectangular shape on the display 107 based on the starting point coordinates and current coordinates of the fingertip, which is the physical pointer, and the specified type of rectangular shape.

[0034] The processing of the detection unit 12 and the display unit 13 shown in Figure 8 is one aspect of the process of setting a virtual pointer (grid point on the grid plane 21) at a position in the display space corresponding to the position of a predetermined real pointer (fingertip of the right hand) detected in real space, setting the virtual pointer in the display space according to the movement position of the real pointer in real space, and displaying geometric shape information based on grids at predetermined intervals on the definition plane (grid plane 21) based on the movement of the virtual pointer.

[0035] Figure 9 is the second figure, showing an example of the display when a line, which is one of the geometric shapes, is entered. The detection unit 12 of the HMD1 can detect the movement speed of the fingertip, which is a real pointer, and output a grid plane 21 with different grid point spacings to the display unit 13. If the fingertip movement speed is above a predetermined threshold, the display unit 13 displays a grid plane 21-1 with the grid spacing distance according to the initially output scale. If the fingertip movement speed falls below the predetermined threshold (if the fingertip is moving slowly), the display unit 13 displays a grid plane 21-2 with a shorter grid spacing distance, such as one-tenth of the initially output scale. For example, when inputting a line to a certain position, the initial first grid plane 21-1 with a grid spacing of 10 cm is displayed, and if the fingertip movement speed slows down, it is changed to a second grid plane 21-2 with a grid spacing of 1 cm and output to the display 107. By changing the grid spacing of the grid plane in this way, the user can input geometric shapes of the desired dimensions with high accuracy while looking at the grid spacing of the grid. Further grid planes 21 with different grid spacings may be displayed based on the movement speed and the scale of the object displayed in the display space. For example, if the movement speed of the fingertip, which is the actual pointer, is even slower, the display unit 13 may change the grid plane 21 to one with a grid spacing of 1 mm and output it to the display 107. Also, if the scale of the object for the user is 1 / 1000, the display unit 13 may change the grid plane 21 to one with a grid spacing of 10 m (meters) and output it to the display 107.

[0036] Figure 10 shows an example of movement within the display space. The user can move through the display space by, for example, clenching both hands with both thumbs pointing upwards (in the positive vertical direction in space) and moving their hands left, right, forward, and backward while maintaining a predetermined distance between them. For example, if both hands are moved slightly forward from the state shown in Figure 10, the detection unit 12 detects the forward movement of an object shaped like both hands, and the detection unit 12 instructs the display unit 13 to move forward, and the display unit 13 displays the movement forward in the 3D display space. Similarly, if both hands are moved slightly to the right from the state shown in Figure 10, the detection unit 12 detects this movement, and the detection unit 12 instructs the display unit 13 to move to the right, and the display unit 13 displays the movement to the right in the 3D display space. Furthermore, if both hands are moved slightly to the left from the state shown in Figure 10, the detection unit 12 detects this movement, and the detection unit 12 instructs the display unit 13 to move to the left, and the display unit 13 displays the movement to the left in the 3D display space. Furthermore, if the user moves both hands slightly forward from the state shown in Figure 10, the detection unit 12 detects this movement and instructs the display unit 13 to move backward. The display unit 13 then displays the movement in the 3D display space in the backward direction. The user can input such hand gestures to the HMD1 and move to a predetermined position in the 3D display space shown on the display 107.

[0037] Figure 11 is the first diagram showing the user's instruction gestures. Figure 12 is a second diagram showing the user's instruction gestures.

[0038] The user can set which hand (left or right) is used for auxiliary input and which hand (right or left) is used for geometric shape input. For example, the user may designate their non-dominant hand (e.g., left hand) as the auxiliary input hand, and their dominant hand (e.g., right hand) as the geometric shape input hand. The user pre-inputs information into the HMD1 indicating which hand (left or right) is used for geometric shape input, and information indicating which hand (left or right) is used for auxiliary input. This input can be done using, for example, a desired user interface. In this disclosure, it will be assumed that the user inputs their non-dominant left hand as the auxiliary input hand and their dominant right hand as the geometric shape input hand.

[0039] Figure 11 shows the types of hand gestures used for auxiliary input. Starting from the left side of Figure 11, the gestures shown, as an example, are: gesture A1 for selecting the geometric shape "line", gesture A2 for selecting the geometric shape "circle", gesture A3 for selecting the geometric shape "ellipse", gesture A4 for selecting the geometric shape "rectangle", and gesture A5 for selecting the geometric shape "continuous line". The two left-hand gestures on the right side of Figure 11 show, as an example, the keyboard Enter key B1 (open palm) and the confirmation gesture (clenched fist) B2. The examples of auxiliary input gestures shown in Figure 11 are just examples, and other types of auxiliary input may be identified by other gestures. Auxiliary input may be defined as input for information other than geometric shapes.

[0040] As shown in Figure 11, in this disclosure, the gesture A1 for selecting the type of geometric shape "line" is a gesture in which the index finger of the left hand is raised upward in space. Furthermore, in this disclosure, gesture A2 for selecting the type of geometric shape "circle" is a gesture in which the index finger (second finger) and middle finger (third finger) of the left hand are raised upwards in space. Furthermore, in this disclosure, gesture A3 for selecting the type of geometric shape "ellipse" is a gesture in which the index finger (second finger), middle finger (third finger), and ring finger (fourth finger) of the left hand are raised upward in space. Furthermore, in this disclosure, gesture A4 for selecting the type of geometric shape "rectangle" is a gesture in which the four fingers of the left hand—the index finger (second finger), middle finger (third finger), ring finger (fourth finger), and little finger (fifth finger)—are raised upward in space. Furthermore, in this disclosure, gesture A5 for selecting the type of geometric shape "continuous line" is a gesture of making a circle with the index finger (second finger) and thumb (first finger) of the left hand.

[0041] Figure 12 shows the types of hand gestures used to input geometric shapes. Starting from the left in Figure 12, the gestures are C1 for inputting a geometric shape, C2 for grasping and stretching the inputted geometric shape, and D1 for canceling the input of a geometric shape while an input operation is in progress. The examples of geometric shape input gestures shown in Figure 12 are just examples, and geometric shapes may be input using other gestures.

[0042] As shown in Figure 12, the gesture C1 used to input geometric shapes in this disclosure is a gesture in which the right hand is clenched and only the index finger (second finger) is raised. Furthermore, the gesture C2 used in this disclosure to grasp and stretch the input geometric shape is a gesture that forms a ring with the index finger (second finger) and thumb (first finger) of the right hand. Furthermore, in this disclosure, the gesture D1 used to cancel the input of a geometric shape during an input operation is a gesture of opening the palm of the right hand.

[0043] Figure 13 is a first figure illustrating an example of a user's geometric shape input operation according to one embodiment of the present disclosure. Figure 14 is a second figure showing an example of a user's geometric shape input operation according to one embodiment of the present disclosure. Figure 15 shows the processing flow of an HMD according to one embodiment of the present disclosure. Next, the details of the HMD1 processing in this disclosure will be explained in order using Figures 13, 14, and 15.

[0044] Based on user operations on the user interface (such as buttons on the HMD1 or operation icons displayed on the display 107), the detection unit 12 of the HMD1 acquires information on whether the left or right hand is performing geometric shape input, and whether the left or right hand is performing auxiliary input (step S101). In this disclosure, it is assumed that the user inputs to the HMD1 that the right hand is the hand performing geometric shape input, and that the left hand is the hand performing auxiliary input. As a result, the HMD1 stores setting information indicating that the right hand is the hand performing geometric shape input and the left hand is the hand performing auxiliary input.

[0045] The user then uses a predetermined user interface (such as buttons on the HMD1 or operation icons displayed on the display 107) to instruct the HMD1 to begin inputting a geometric shape. The HMD1 detects the start of geometric shape input (step S102). Next, the user forms their left hand, which will perform the auxiliary input, into a shape that corresponds to the geometric shape selection gesture. For example, if the user wants to input a line as a geometric shape, they form their hand into a shape that corresponds to the geometric shape selection gesture A1, with their fist clenched and their index finger pointing upwards (Figure 13a). With the left hand in the shape corresponding to the geometric shape selection gesture, the user changes the shape of their left hand from an open palm (paper) gesture B1 to a clenched fist (rock) gesture B2 (Figure 13a). The user's action of changing the shape of their left hand from an open palm (paper) gesture B1 to a clenched fist (rock) gesture B2 indicates a confirmed input.

[0046] The detection unit 12 detects the left hand performing the auxiliary input based on the image acquired from the camera 106 and detects gesture A1, which indicates that the selection gesture for the type of geometric shape is "line" based on its shape (step S103). The detection unit 12 also detects from the image that the shape of the left hand has changed from gesture B1, where the palm is open (paper), to gesture B2, where the hand is clenched (rock), and determines that the selection gesture for the geometric shape is confirmed to be "line" (step S104). As a result, the display unit 13 enters a state of waiting for input of the geometric shape "line".

[0047] The user then forms their hands into a movement instruction gesture, as shown in Figure 13b, with both hands clenched and both thumbs pointing upwards (in the vertically positive direction in space), and moves their hands left, right, forward, and backward while maintaining a predetermined distance between them. The detection unit 12 detects the movement instruction gesture and whether the hands were moved left, right, forward, or backward from the image, and outputs the corresponding movement instruction to the display unit 13. The display unit 13 performs display processing to move the display space shown on the display 107 (step S105). When the user moves to the predetermined position in the display space, they release the movement instruction gesture.

[0048] The user forms a gesture C1 for inputting a geometric shape with their right hand by raising the index finger of their right hand in three-dimensional space at a predetermined position, thereby instructing the user to begin inputting the geometric shape (Figure 13c). The detection unit 12 detects the gesture C1 for inputting the geometric shape based on the acquired image (step S106). While the user's right hand is in the shape of the gesture C1 for inputting the geometric shape, the user changes the shape of their left hand from an open palm (paper) gesture B1 to a clenched fist (rock) gesture B2 (Figure 13c). When the detection unit 12 detects that the shape of the left hand has changed from an open palm (paper) gesture B1 to a clenched fist (rock) gesture B2, it determines that this is a confirmed input to start inputting the geometric shape (step S107). Based on the detection of the right-hand geometric shape input gesture C1 and the left-hand confirmation input, the detection unit 12 instructs the display unit 13 to start inputting the geometric shape.

[0049] As shown in Figure 13d, when the display unit 13 detects an instruction to start inputting a geometric shape, it displays the operation palette 2 and the grid surface 21 at the position in the display space within the display 107 corresponding to the coordinates of the actual pointer of the right index finger (step S108). At this point, the user may again form both hands to make a movement instruction gesture (Figure 9) and move within the display space together with the operation palette 2 and the grid surface 21 (Figure 14e).

[0050] Then, when the user maintains the shape of their right hand in the geometric input gesture C1 and performs confirmation input gestures (B1, B2) with their left hand (Figure 14f), the detection unit 12 detects the spatial coordinates of the tip of the right index finger, which is the actual pointer (step S109), and outputs them to the display unit 13. The display unit 13 identifies the display spatial coordinates corresponding to the spatial coordinates of the tip of the right index finger and identifies the starting point coordinates G0 of the grid plane 21 corresponding to those display spatial coordinates (step S110). The detection unit 12 also detects the spatial coordinates of the tip of the right index finger at predetermined intervals such as 10 milliseconds and identifies the grid point coordinates of the grid plane 21 in the display space corresponding to the spatial coordinates of the moving tip of the right index finger based on the process described in Figure 8 (step S111). The display unit 13 displays a line on the display 107 based on the process described in Figure 8, using the starting point coordinate G0 and the corresponding grid point coordinates of the grid plane 21 after the movement of the tip of the right index finger (the first intersection G1 and the second intersection G2 in Figure 8). (Step S112)

[0051] This type of processing involves a real pointer being the fingertip of either the left or right hand detected in real space, a virtual pointer being the coordinates set in the display space corresponding to the fingertip (coordinates of the grid points on the grid plane 21), identifying the type of geometric shape based on the shape detected by the other hand which is different from the first hand, detecting multiple coordinates in the display space of the virtual pointer based on the movement of the other hand and the position of the virtual pointer in the display space, and displaying information based on the type of geometric shape based on those multiple coordinates.

[0052] The user moves the shape of their right hand to the desired position while maintaining the geometric shape input gesture C1. This draws lines on the display 107 based on the position of the fingertips of the right hand. If the user changes the speed of movement of their right hand, the display unit 13 displays the initial first grid surface 21-1 with a grid spacing of 10 cm if the movement speed is above a predetermined threshold, and changes to the second grid surface 21-2 with a grid spacing of 1 cm and outputs it to the display 107 if the fingertip movement speed slows down. The grid spacing may vary depending on the scale of the object being displayed and the speed of fingertip movement.

[0053] Furthermore, each time the user moves their right index finger, changing the coordinates of the fingertip, and the corresponding grid point (the position of the virtual pointer) on the grid plane 21 moves, causing the displayed line to extend (increase) by the spacing of the grid points, or each time the displayed line shrinks (decreases) by the spacing of the grid points, the notification unit 14 outputs notification information to the user. For example, the notification information is sound, and each time the line extends by the spacing of the grid points, the notification unit 14 outputs a predetermined sound from the speaker 108. Alternatively, the notification information is vibration, and each time the line extends by the spacing of the grid points, the notification unit 14 controls the vibrator 109 to emit a predetermined vibration. This allows the user to intuitively grasp the extension state of the line extending by the grid spacing, enabling them to input geometric shapes to a highly accurate position.

[0054] This process is an example of a process in which the HMD1 displays lines representing a part of a geometric shape on grids at predetermined intervals on the grid surface 21 based on the movement of a virtual pointer, and outputs notification information in response to an increase or decrease in the display of such lines at predetermined intervals.

[0055] When the user draws a line to the desired position, they decide to end the line input at that point. In this case, the user confirms the input by changing the shape of their left hand from an open palm (palm) gesture B1 to a clenched fist (rock) gesture B2 (Figure 14g). The detection unit 12 detects the confirmation input (step S113). As a result, the display 107 confirms that a line has been drawn, and the geometric shape input is temporarily completed.

[0056] Furthermore, the detection unit 12 determines whether it has detected a gesture D1 (Figure 14h) that cancels the geometric shape input during the input operation, while it has not detected a confirmed input (Figure 14g) (step S114). If so, the detection unit 12 outputs a cancel instruction to the display unit 13. The display unit 13 deletes the drawn line in the display space and terminates the geometric shape input.

[0057] The example above shows the processing flow when the user inputs a line as a geometric shape, but similarly, the user can input geometric shapes (circles, ellipses, rectangles, etc.) in response to gestures.

[0058] Furthermore, in the process described above, based on the user's settings, the HMD1 can display two or more operation palettes 2 in the display space of the display 107. In this case, the display unit 13 of the HMD1 displays a grid surface 21 corresponding to each operation palette 2.

[0059] Figure 16 shows the display range of virtual information according to one embodiment of the present disclosure. The display unit 13 of the HMD1 may detect markers in real space and display geometric shape information in a display space area 61 at a position corresponding to a predetermined area on the marker. For example, the display unit 13 detects markers attached to the four corners of a desk in the input image. The display unit 13 identifies a display space area 61 on the display 107 to display information about virtual objects or geometric shapes. The display unit 13 displays information about virtual objects or geometric shapes at the position identified based on the above processing.

[0060] Although one embodiment of the present disclosure has been described above, the display unit 13 may simply set the operation palette 2 and grid plane 21 virtually without actually displaying them on the display 107. In this case, the HMD1 may, as an internal process, set the orientation of the operation palette 2 and the grid plane 21 in the display space on the display 107, and similarly display geometric information based on user operations using the grid plane 21 that is equivalent to the set operation palette 2 plane.

[0061] The HMD1 processing described above provides an information processing device that allows users to input geometric shape information into the display space with greater accuracy. Specifically, when a user inputs geometric shape information into the display space, such as when specifying coordinates in the display space, for example (1) when specifying a reference coordinate in the display space, (2) when specifying the coordinate of the starting point of the geometric shape, or (3) when specifying the coordinates of intermediate or ending points of the geometric shape, the display space coordinates corresponding to the position specified by the user with a real pointer in real space can be automatically corrected to assist the user in inputting more accurate geometric shape information in the display space. Furthermore, by displaying the operation palette 2 and grid surface 21, the information processing device allows the user to accurately input the angle and direction of the geometric shape information relative to the reference in the display space, and to input geometric shapes in which the line length and size of the geometric shape are maintained in threshold units.

[0062] Although this disclosure describes the information processing device as an HMD1, the information processing device in this disclosure may be a user terminal such as a PC (Personal Computer) or a tablet device. In that case, the PC or tablet device may detect the shape of the user's hand and the selection of fingers in the image it acquires and perform a similar geometric shape display process on the display.

[0063] Figure 17 shows a functional block diagram of another example of an information processing device. Figure 18 shows a processing flow for another example of an information processing device. The HMD1 and other information processing devices include at least an information processing unit 111. The information processing unit 111 sets a planar operation palette, which is used for inputting geometric information into the display space, in the display space of the display (step S201). The information processing unit 111 sets a definition plane that defines a predetermined interval distance that serves as a reference when inputting geometric information into the plane of the display space, which includes the surface of the operation palette (step S202).

[0064] The information processing device may be located in the HMD1, or in a cloud server connected to the HMD1 via communication. Alternatively, for example, a PC used by the user may be the information processing device disclosed herein.

[0065] Furthermore, the HMD1 or other information processing device may detect an object detected in real space as a real pointer, identify the coordinates of a virtual pointer calculated in the display space corresponding to that object, and identify the type of geometric shape based on sound such as the user's voice. In that case, the HMD1 or other information processing device may detect multiple coordinates of the virtual pointer in the display space based on the type of sound and the position of the object in the display space, and display information based on the type of geometric shape based on those multiple coordinates.

[0066] While the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure may be made that can be understood by those skilled in the art within the scope of the present disclosure.

[0067] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0068] (Note 1) A planar shape manipulation palette, used for inputting geometric shape information into the display space, is set in the display space of the display. A definition plane is set on the plane of the display space, including the surface of the operation palette, that defines a predetermined distance interval which serves as a reference when inputting the geometric shape information. Information processing device.

[0069] (Note 2) The following settings information is obtained to set the display orientation of the operation palette in the display space, Based on the aforementioned setting information, the operation palette is set in the display space. The information processing device described in Appendix 1.

[0070] (Note 3) A virtual pointer is set at a position in the display space corresponding to the position of a predetermined physical pointer detected in real space. The virtual pointer is set in the display space according to the movement position of the real pointer in the real space, and the geometric shape information is displayed based on a grid at predetermined intervals on the definition surface based on the movement of the virtual pointer. The information processing device described in Appendix 1 or Appendix 2.

[0071] (Note 4) The display orientation of the operation palette in the display space is changed based on the change in the aforementioned setting information. The information processing device described in Appendix 2.

[0072] (Note 5) In the aforementioned display space, a plurality of operation palettes are set. An information processing device described in any one of the appendices 1 through 4.

[0073] (Note 6) Based on the movement of the virtual pointer, the geometric shape information is displayed in the grid at predetermined intervals on the definition surface, and notification information is output based on the increase or decrease in the display of the geometric shape at the predetermined intervals. The information processing device described in Appendix 3.

[0074] (Note 7) The size of the predetermined interval on the defining plane is changed according to the size of the object to be displayed per unit distance in the display space. An information processing device described in any one of the appendices 1 through 6.

[0075] (Note 8) The aforementioned physical pointer is the fingertip of either the left or right hand detected in the physical space. The virtual pointer is a coordinate set in the display space corresponding to the fingertip of the hand, Based on the shape detected by the other hand, which is different from the first hand, the type of geometric shape is identified. Based on the movement of the other hand and the position of the virtual pointer in the display space, multiple coordinates of the virtual pointer in the display space are detected. Based on the aforementioned multiple coordinates, information is displayed that is based on the type of geometric shape. The information processing device described in Appendix 3 or Appendix 6.

[0076] (Note 9) The starting and ending coordinates of the geometric shape among the plurality of coordinates are identified based on the movement of the other hand, which is different from the other hand. The information processing device described in Appendix 8.

[0077] (Note 10) The intermediate coordinate of the geometric shape among the plurality of coordinates is identified based on the movement of the other hand. The information processing device described in Appendix 9.

[0078] (Note 11) The aforementioned geometric shape includes at least one or more of the following: lines, rectangles, circles, ellipses, continuous lines, and polygons. An information processing device described in any one of the appendices 1 through 10.

[0079] (Note 12) A planar shape manipulation palette, used for inputting geometric shape information into the display space, is set in the display space of the display. A definition plane is set on the plane of the display space, including the surface of the operation palette, that defines a predetermined distance interval which serves as a reference when inputting the geometric shape information. Information processing methods.

[0080] (Note 13) The following settings information is obtained to set the display orientation of the operation palette in the display space, Based on the aforementioned setting information, the operation palette is set in the display space. The information processing method described in Appendix 12.

[0081] (Note 14) A virtual pointer is set at a position in the display space corresponding to the position of a predetermined physical pointer detected in real space. The virtual pointer is set in the display space according to the movement position of the real pointer in the real space, and the geometric shape information is displayed based on a grid at predetermined intervals on the definition surface based on the movement of the virtual pointer. The information processing method described in Appendix 12 or Appendix 13.

[0082] (Note 15) The display orientation of the operation palette in the display space is changed based on the change in the aforementioned setting information. The information processing method described in Appendix 13.

[0083] (Note 16) In the aforementioned display space, a plurality of operation palettes are set. The information processing method described in any one of the appendices 12 to 15.

[0084] (Note 17) Based on the movement of the virtual pointer, the geometric shape information is displayed in the grid at predetermined intervals on the definition surface, and notification information is output based on the increase or decrease in the display of the geometric shape at the predetermined intervals. The information processing method described in Appendix 14.

[0085] (Note 18) The size of the predetermined interval on the defining plane is changed according to the size of the object to be displayed per unit distance in the display space. The information processing method described in any one of the appendices 12 to 17.

[0086] (Note 19) The aforementioned physical pointer is the fingertip of either the left or right hand detected in the physical space. The virtual pointer is a coordinate set in the display space corresponding to the fingertip of the hand, Based on the shape detected by the other hand, which is different from the first hand, the type of geometric shape is identified. Based on the movement of the other hand and the position of the virtual pointer in the display space, multiple coordinates of the virtual pointer in the display space are detected. Based on the aforementioned multiple coordinates, information is displayed that is based on the type of geometric shape. The information processing method described in Appendix 14 or Appendix 17.

[0087] (Note 20) The starting and ending coordinates of the geometric shape among the plurality of coordinates are identified based on the movement of the other hand, which is different from the other hand. The information processing method described in Appendix 19.

[0088] (Note 21) The intermediate coordinate of the geometric shape among the plurality of coordinates is identified based on the movement of the other hand. The information processing method described in Appendix 20.

[0089] (Note 22) The aforementioned geometric shape includes at least one or more of the following: lines, rectangles, circles, ellipses, continuous lines, and polygons. The information processing method described in any one of the appendices 12 to 21.

[0090] (Note 23) The computer of the information processing device, Means for setting a planar operation palette, which is used for inputting at least geometric information into the display space, in the display space of the display, Means for setting a definition plane that defines a predetermined interval distance that serves as a reference when inputting geometric shape information into the plane of the display space including the surface of the operation palette, A program that makes it function as such.

[0091] (Note 24) moreover, means for acquiring setting information that sets at least the display orientation of the operation palette in the display space, means for setting the operation palette in the display space based on the aforementioned setting information, The program described in Appendix 23 is used to make it function as follows.

[0092] (Note 25) moreover, A virtual pointer is set at a position in the display space corresponding to the position of a predetermined physical pointer detected in real space. The virtual pointer is set in the display space according to the movement position of the real pointer in the real space, and the geometric shape information is displayed based on a grid at predetermined intervals on the definition surface based on the movement of the virtual pointer. The program described in Appendix 23 or Appendix 24 that functions as such.

[0093] (Note 26) moreover, Means for changing the display orientation of the operation palette in the display space based on the change in the setting information, The program described in Appendix 24 is used to make it function as follows.

[0094] (Note 27) moreover, means for setting up a plurality of operation palettes in the display space, The program described in any one of the appendices 23 to 26 that functions as such.

[0095] (Note 28) moreover, A means for displaying information about the geometric shape in a grid at predetermined intervals on the defining surface based on the movement of the virtual pointer, and for outputting notification information based on an increase or decrease in the display of the geometric shape at predetermined intervals. The program described in Appendix 25 is used to make it function as follows.

[0096] (Note 29) moreover, Means for changing the size of the predetermined interval on the defining plane according to the size of the object to be displayed per unit distance in the display space, The program described in any one of the appendices 23 to 28 that functions as such.

[0097] (Note 30) moreover, The aforementioned physical pointer is the fingertip of either the left or right hand detected in the physical space. The virtual pointer is a coordinate set in the display space corresponding to the fingertip of the hand, Means for identifying the type of geometric shape based on the shape detected by the other hand, which is different from the one hand; Means for detecting a plurality of coordinates of the virtual pointer in the display space based on the movement of the other hand and the position of the virtual pointer in the display space, Means for displaying information based on the type of geometric shape based on the plurality of coordinates, The program described in Appendix 25 or Appendix 28 that functions as such.

[0098] (Note 31) moreover, Means for identifying the starting and ending coordinates of the geometric shape among the plurality of coordinates based on the movement of the other hand, The program described in Appendix 30 is used to make it function as follows.

[0099] (Note 32) Means for identifying the intermediate coordinate of the geometric shape among the plurality of coordinates based on the movement of the other hand, The program described in Appendix 31 is used to make it function as follows.

[0100] (Note 33) The aforementioned geometric shape includes at least one or more of the following: lines, rectangles, circles, ellipses, continuous lines, and polygons. The program described in any one of the appendices 23 to 32.

[0101] (Note 34) The aforementioned physical pointer is an object detected in the physical space, The virtual pointer is a coordinate set in the display space corresponding to the object, The type of geometric shape is identified based on a predetermined sound, Based on the type of sound and the position of the object in the display space, multiple coordinates of the virtual pointer in the display space are detected. Based on the aforementioned multiple coordinates, information is displayed that is based on the type of geometric shape. The information processing device described in Appendix 3 or Appendix 6.

[0102] (Note 35) The aforementioned physical pointer is an object detected in the physical space, The virtual pointer is a coordinate set in the display space corresponding to the object, The type of geometric shape is identified based on a predetermined sound, Based on the type of sound and the position of the object in the display space, multiple coordinates of the virtual pointer in the display space are detected. Based on the aforementioned multiple coordinates, information is displayed that is based on the type of geometric shape. The information processing method described in Appendix 14 or Appendix 17.

[0103] (Note 36) moreover, The aforementioned physical pointer is an object detected in the physical space, The virtual pointer is a coordinate set in the display space corresponding to the object, Means for identifying the type of geometric shape based on a predetermined sound, Means for detecting a plurality of coordinates of the virtual pointer in the display space based on the type of sound and the position of the object in the display space, Means for displaying information based on the type of geometric shape based on the plurality of coordinates The program described in Appendix 25 or Appendix 28 that functions as such. [Explanation of Symbols]

[0104] 1. Head-mounted display (HMD) 2. Operation Palette 3. Object 11. Control Unit 12. Detection Unit 13...Display section 14...Notification section 21...Grid surface

Claims

1. A planar shape manipulation palette, used for inputting geometric shape information into the display space, is set in the display space of the display. A definition plane is set on the plane of the display space, including the surface of the operation palette, that defines a predetermined distance interval which serves as a reference when inputting the geometric shape information. Information processing device.

2. The following settings information is obtained to set the display orientation of the operation palette in the display space, Based on the aforementioned setting information, the operation palette is set in the display space. The information processing apparatus according to claim 1.

3. A virtual pointer is set at a position in the display space corresponding to the position of a predetermined physical pointer detected in real space. The virtual pointer is set in the display space according to the movement position of the real pointer in the real space, and the geometric shape information is displayed based on a grid at predetermined intervals on the definition surface based on the movement of the virtual pointer. The information processing apparatus according to claim 2.

4. The display orientation of the operation palette in the display space is changed based on the change in the aforementioned setting information. The information processing apparatus according to claim 3.

5. In the aforementioned display space, a plurality of operation palettes are set. The information processing apparatus according to claim 4.

6. Based on the movement of the virtual pointer, the geometric shape information is displayed in the grid at predetermined intervals on the definition surface, and notification information is output based on the increase or decrease in the display of the geometric shape at the predetermined intervals. The information processing apparatus according to claim 5.

7. The size of the predetermined interval on the defining plane is changed according to the size of the object to be displayed per unit distance in the display space. The information processing apparatus according to claim 6.

8. The aforementioned physical pointer is the fingertip of either the left or right hand detected in the physical space. The virtual pointer is a coordinate set in the display space corresponding to the fingertip of the hand, Based on the shape detected by the other hand, which is different from the first hand, the type of geometric shape is identified. Based on the movement of the other hand and the position of the virtual pointer in the display space, a plurality of coordinates of the virtual pointer in the display space are detected. Based on the aforementioned multiple coordinates, information is displayed that is based on the type of geometric shape. The information processing apparatus according to claim 7.

9. A planar shape manipulation palette, used for inputting geometric shape information into the display space, is set in the display space of the display. A definition plane is set on the plane of the display space, including the surface of the operation palette, that defines a predetermined distance interval which serves as a reference when inputting the geometric shape information. Information processing methods.

10. The computer of the information processing device, Means for setting a planar operation palette, which is used for inputting at least geometric information into the display space, in the display space of the display, Means for setting a definition plane that defines a predetermined interval distance that serves as a reference when inputting geometric shape information into the plane of the display space including the surface of the operation palette, A program that makes it function as such.

Images