Information processing device, information processing method, and program

Abstract

To provide a technique for preventing erroneous operation for a virtual object.SOLUTION: An information processing device displays an augmented reality space or a mixed reality space where at least a user-operable virtual object is arranged. The information processing device includes: a specification unit configured to specify an operation position on the virtual object by using information about the user; a calculation unit configured to calculate an angle indicating an operation direction when the user operates the virtual object, by using both a reference point indicating a position for the information about the user and the operation position; and a changing unit configured to change an operation determination area for determining whether operation has been performed for the virtual object, by using the angle.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Technologies called Augmented Reality (AR) and Mixed Reality (MR) have been known for some time. These technologies allow virtual objects (hereinafter also referred to as virtual objects) to be displayed superimposed on real space, and allow users to operate the objects. For example, Patent Documents 1 to 3 and the like are known as technologies related to operations on virtual objects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-15553 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-56462 [Patent Document 3] Patent Publication No. 2021-15637 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, depending on the orientation of the user with respect to the virtual object to be operated, there is a possibility that the user may erroneously operate another virtual object adjacent to the virtual object.

[0005] The present disclosure has been made in consideration of the above points, and provides a technology for preventing erroneous operations on virtual objects. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure is an information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object that can be operated by a user is arranged, and includes: an identification unit configured to identify an operation position on the virtual object using information about the user; a calculation unit configured to calculate an angle representing the operation direction when the user operates the virtual object using a reference point representing the position of the information about the user and the operation position; and a modification unit configured to change an operation determination area for determining whether an operation has been performed on the virtual object using the angle. [Effects of the Invention]

[0007] A technique is provided for preventing erroneous operations on virtual objects. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram illustrating an example of an operation image arranged in a mixed reality space or an augmented reality space. [Figure 2] FIG. 10 is a diagram illustrating an example of operation determination. [Figure 3] FIG. 1 is a diagram for explaining a problem with the conventional technology. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a device according to the embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a device according to the present embodiment. [Figure 6] 10 is a flowchart illustrating an example of an operation position monitoring process according to the present embodiment. [Figure 7] 10 is a flowchart illustrating an example of a manipulation determination region change process according to the present embodiment. [Figure 8] FIG. 10 is a diagram (part 1) showing an example of calculation of an incident angle in the yz plane. [Figure 9] FIG. 10 is a diagram (part 2) showing an example of calculation of the angle of incidence in the yz plane. [Figure 10] FIG. 10 is a diagram (part 1) showing an example of calculation of an incident angle in the xz plane. [Figure 11] FIG. 10 is a diagram (part 2) showing an example of calculation of the angle of incidence in the xz plane. [Figure 12] FIG. 10 is a diagram (part 1) showing an example of changing the manipulation determination region. [Figure 13] FIG. 10 is a diagram (part 2) showing an example of changing the manipulation determination region. [Figure 14] 10 is a flowchart illustrating an example of an operation execution process according to the present embodiment. [Figure 15] FIG. 10 is a diagram (part 1) showing a modified example of calculation of the angle of incidence in the yz plane. [Figure 16] FIG. 10 is a diagram (part 2) showing a modified example of calculation of the angle of incidence in the yz plane. [Figure 17] FIG. 10 is a diagram (part 1) showing a modified example of calculation of the angle of incidence in the xz plane. [Figure 18] FIG. 10 is a diagram (part 2) showing a modified example of calculation of the angle of incidence in the xz plane. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment of the present invention will be described below.

[0010] <Operations on virtual objects> A virtual object is an image such as 2D or 3D CG (Computer Graphics) placed in an augmented reality space (hereinafter also referred to as AR space) or a mixed reality space (hereinafter also referred to as MR space). There are virtual objects that can be operated by the user and those that cannot be operated by the user. Below, we will mainly refer to virtual objects that can be operated by the user as "operation images." Furthermore, we will refer to operation images that are the object of operation by the user as "operation target images," and other operation target images as "non-operation target images."

[0011] In the following, as an example, operation image 1000 shown in FIG. 1 will be mainly considered. Operation image 1000 shown in FIG. 1 is arranged on an xy plane in an AR space or MR space in which a three-dimensional xyz coordinate system is set (i.e., arranged on an xy plane (x, y, z0) represented by z = z0), and is composed of operation images 1001 to 1009. For example, operation images 1001 to 1006 are images for displaying the statuses of facilities A to F, respectively, through a user's operation. Furthermore, operation images 1007 and 1009 are images for displaying an image representing a previous screen and an image representing a next screen, respectively, through a user's operation. For example, operation image 1008 is an image for displaying an image representing a setting screen through a user's operation.

[0012] Generally, an operation determination area for determining an operation for each of the operation images 1001 to 1006 is set on the operation image. For example, as shown in Fig. 2, an operation determination area 1103 is set on the operation image 1003, an operation determination area 1106 is set on the operation image 1006, and an operation determination area 1109 is set on the operation image 1009. Note that Fig. 2 shows operation images 1003, 1006, and 1009 that constitute the operation image 1000 on a certain yz plane, and operation determination areas 1103, 1106, and 1109 that are set on the operation images 1003, 1006, and 1009, respectively.

[0013] When a user U wishes to operate the operation image 1006, the user U makes a gesture of pressing the operation image 1006 with, for example, a finger or the like. At this time, if the user U's finger or the like moves to the position of a certain threshold D in the operation determination area 1109, it is determined that the operation image 1006 has been operated. Note that in the example shown in FIG. 2, the threshold D is set in the vertical direction (z-axis direction) of the operation image, but this is just one example, and the direction in which the threshold D is set is not limited to the vertical direction, and the threshold D can be set in various directions. For example, the threshold D may be set in any direction in any xy plane horizontal to the operation image, or the threshold D may be set in both the z-axis direction and any direction in the xy plane.

[0014] In this way, in general, in many AR or MR technologies, an operation determination area is set on an operation image, and whether or not the operation image has been operated is determined based on whether or not a finger or the like has moved within that operation determination area to a certain threshold position.

[0015] However, with the conventional technology, there is a possibility that an image that is not the target of operation and that is adjacent to the target image of operation may be mistakenly operated.

[0016] 3, when the operation image 1000 is located diagonally above and in front of the user U, the user U makes a gesture of pressing the operation image 1006 with his / her finger from diagonally below the operation image 1000. Note that P is the pointer position, and indicates the position in the operation image 1006 that the user U is about to press.

[0017] In such a case, the user U's finger or the like will move within the operation judgment area 1109 of the operation image 1009 located below the operation image 1006, and as a result, the user U's finger or the like may move to the position of the threshold D of the operation judgment area 1109, and it may be determined that the operation image 1009 has been operated.

[0018] As described above, in the conventional technology, depending on the orientation of the user relative to the operation target image, the user may accidentally operate a non-operation target image adjacent to the operation target image. This is particularly likely to occur when the user attempts to operate the operation target image in a small space or while crouching. Furthermore, the more densely multiple operation images are arranged, the more likely this is to occur. For this reason, it is considered that, for example, when performing work requiring many operations, such as maintenance and inspection work or repair work on equipment, facilities, plants, etc., using an AR / MR device, it is likely that the non-operation target image will be accidentally operated.

[0019] Therefore, the following describes a device 10 that can prevent erroneous operations on a non-operation target image by changing the orientation of an operation determination region set in an operation image in an AR space or an MR space. Here, any information processing device that can arrange an operation image in real space using AR or MR technology can be used as the device 10. However, the following mainly assumes a head-mounted display (HMD). However, the device 10 is not limited to a head-mounted display and may be, for example, a smartphone, a tablet terminal, a wearable device, or the like. Furthermore, the device 10 is communicably connected to an information processing device such as a PC (personal computer), and some of the components of the device 10 described below may be included in the information processing device such as a PC, or some of the processing performed by the device 10 may be performed by the information processing device such as a PC.

[0020] 2 and 3 are merely examples. The shapes of the manipulation determination regions shown in FIGS. 2 and 3 are assumed to be cubes, but are not limited thereto. For example, the shapes may be polyhedrons, cubes with curved faces on some or all of their faces, or three-dimensional regions corresponding to the shape of a virtual object (for example, a three-dimensional region surrounding a virtual object). Furthermore, for example, the manipulation determination region is not limited to a three-dimensional region, but may be a two-dimensional region such as a plane or a curved surface. Regardless of the shape of the manipulation determination region, erroneous manipulation of other virtual objects adjacent to the virtual object to be manipulated may occur depending on the orientation of the virtual object to be manipulated and the user, and the arrangement density of multiple virtual objects.

[0021] Furthermore, the determination of whether a virtual object has been operated is not limited to whether a finger or the like has moved to a threshold position within an operation determination area set for the virtual object. For example, the determination may be based on whether a finger or the like has touched the operation determination area, or whether a finger or the like has passed through the operation determination area. Using a finger is just one example, and the present invention is not limited to this. For example, a pen-type input device or some other input device may be used instead of a finger.

[0022] For simplicity, it is assumed below that the shape of the manipulation determination region is a cubic region set on the manipulation image, and that the user performs manipulations on the manipulation image with his / her finger.

[0023] <Device 10 hardware configuration example> An example of the hardware configuration of device 10 according to this embodiment is shown in Fig. 4. As shown in Fig. 4, device 10 according to this embodiment includes an input device 11, a display device 12, an external I / F 13, a communication I / F 14, a sensor 15, a memory device 16, and a processor 17. These pieces of hardware are communicatively connected via a bus 18.

[0024] The input device 11 is, for example, various physical buttons, etc. If the device 10 is a smartphone, a tablet terminal, etc., the input device 11 may include a touch panel, etc.

[0025] The display device 12 is, for example, a display, a display panel, etc. The AR space or MR space and the virtual objects arranged in the space are displayed on the display device 12.

[0026] The external I / F 13 is an interface with an external device such as a recording medium 13a, etc. Examples of the recording medium 13a include an SD memory card (Secure Digital memory card) and a USB (Universal Serial Bus) memory card.

[0027] The communication I / F 14 is an interface for connecting the device 10 to a communication network. The sensor 15 is, for example, various sensors including a depth sensor. In addition to the depth sensor, the sensor 15 may also include, for example, an acceleration sensor, a gyro sensor, a visible light camera, an infrared camera, etc.

[0028] The memory device 16 is, for example, one of various storage devices such as a flash memory, a solid state drive (SSD), etc. The processor 17 is, for example, one of various arithmetic devices such as a central processing unit (CPU), a micro processing unit (MPU), etc.

[0029] 4 is an example, and the device 10 may have other hardware configurations. For example, the device 10 may have multiple memory devices 16 or multiple processors 17, or may have various types of hardware other than the hardware shown in the figure.

[0030] <Example of functional configuration of Device 10> An example of the functional configuration of the device 10 according to this embodiment is shown in Fig. 5. As shown in Fig. 5, the device 10 according to this embodiment has an operation position monitoring processing unit 101, an operation determination area change processing unit 102, and an operation execution processing unit 103. Each of these units is realized, for example, by a process in which one or more programs installed in the device 10 are executed by the processor 17. The device 10 according to this embodiment also has a storage unit 104. The storage unit 104 is realized, for example, by the memory device 16 or the like.

[0031] The operation position monitoring processing unit 101 executes an operation position monitoring process for monitoring an operation position (for example, pointer position P shown in FIG. 3) indicating a position where a user is about to perform an operation. In the following, as an example, the operation position is a pointer position, and the pointer position is a position indicating the intersection of a vector indicating a direction where a user points with a specific finger (for example, the index finger of the right hand) and some object (not limited to a virtual object, but including an actual object existing in the real space). Here, the operation position monitoring processing unit 101 includes an operator information acquisition unit 111, an operation position identification unit 112, and an operation position determination unit 113.

[0032] The operator information acquisition unit 111 acquires operator information for identifying an operation position from the storage unit 104. Hereinafter, the operator information is assumed to be hand mesh data representing mesh data related to a user's hand. The hand mesh data is data created by known image recognition processing (e.g., object recognition processing using deep learning, etc.) from information obtained by photographing or measuring the user's hand using a camera, a depth sensor, etc. The hand mesh data makes it possible to easily calculate the coordinates of feature points of the user's hand (e.g., finger joints, tips, etc.) in the AR or MR space. Note that since both the process of creating the hand mesh data and the process of calculating the coordinates of the feature points are known processes, hereinafter, it is assumed that the hand mesh data has already been created and that the coordinates of the feature points are all known.

[0033] The operation position identification unit 112 identifies the pointer position from the hand mesh data acquired by the operator information acquisition unit 111. For example, the operation position identification unit 112 calculates a three-dimensional vector representing the direction pointed by the index finger of the user's right hand from the hand mesh data, and identifies the intersection of the three-dimensional vector with some object as the pointer position.

[0034] The operation position determination unit 113 determines whether or not the pointer position identified by the operation position identification unit 112 exists on the operation image.

[0035] When the operation position determination unit 113 of the operation position monitoring processing unit 101 determines that the pointer position is on the operation image, the operation determination area change processing unit 102 executes an operation determination area change process to change the operation determination area in accordance with the orientation of the user with respect to the operation image. Here, the operation determination area change processing unit 102 includes a distance calculation unit 121, an angle calculation unit 122, and an operation determination area change unit 123.

[0036] The distance calculation unit 121 calculates, in a yz plane including a predetermined reference point, a distance a representing the length of a perpendicular line drawn from the reference point perpendicularly to the operation image, and a distance b representing the distance from the intersection of the perpendicular line and the operation image to a point where the pointer position is projected onto the yz plane. The reference point is a point representing the position of the user operating the operation image. For example, when pointing to the pointer position with a finger or the like, the tip of the finger or the like (e.g., the tip of the index finger of the right hand) can be used as the reference point.

[0037] In addition, the distance calculation unit 121 calculates a distance a' representing the length of a perpendicular line drawn from the reference point perpendicularly to the operation image in the xz plane that includes the reference point, and a distance b' from the intersection of the perpendicular line and the operation image to the point where the pointer position is projected onto the xz plane.

[0038] The angle calculation unit 122 calculates angles θ and γ (hereinafter also referred to as incident angles θ and γ) that respectively represent the user's orientation in the yz plane and the xz plane relative to the operation image, using the distances calculated by the distance calculation unit 121. The incident angle θ represents the user's operation direction in the yz plane, and similarly, the incident angle γ represents the user's operation direction in the xz plane.

[0039] The manipulation determination area modification unit 123 changes the orientation of the manipulation determination area using the incident angles θ and γ calculated by the angle calculation unit 122. That is, the manipulation determination area modification unit 123 changes the orientation of the manipulation determination area by −θ in the yz plane and by −γ in the xz plane. This causes the manipulation determination area to face the front direction of the user, thereby preventing erroneous manipulations on the manipulation image.

[0040] The operation execution processing unit 103 determines whether an operation has been performed on the operation image and executes an operation execution process to perform the operation according to the determination result. Here, the operation execution processing unit 103 includes a threshold determination unit 131 and an operation execution unit 132.

[0041] The threshold determination unit 131 determines whether or not the user's finger has moved to the position of the manipulation determination area equal to the threshold D. This means that the user has pressed the manipulation determination area with their finger a distance equal to the threshold D.

[0042] When the threshold determination unit 131 determines that the user's finger has moved to the position of the threshold D of the operation determination area, the operation execution unit 132 executes an operation process corresponding to the operation image in which the operation determination area is set.

[0043] The storage unit 104 stores various information (for example, hand mesh data, operation images, threshold D, etc.). Note that the coordinates of the operation images placed in the AR space or MR space can be calculated using known techniques, and therefore, in the following, it is assumed that the coordinates of each operation image are known.

[0044] <Operation position monitoring process> The operation position monitoring process according to this embodiment will be described below with reference to Fig. 6. The operation position monitoring process shown in Fig. 6 is repeatedly executed at predetermined time intervals (for example, at time intervals of several milliseconds to several tens of milliseconds).

[0045] The operator information acquisition unit 111 of the operation position monitoring processing unit 101 acquires hand mesh data from the storage unit 104 (step S101).

[0046] Next, the operation position identification unit 112 of the operation position monitoring processing unit 101 identifies the pointer position from the hand mesh data acquired in the above step S101 (step S102).

[0047] Next, the operation position determination unit 113 of the operation position monitoring processing unit 101 determines whether or not the pointer position identified in the above step S103 exists on the operation image (step S103).

[0048] If it is determined in step S103 that the pointer position is not on the operation image, the operation position monitoring processing unit 101 ends the operation position monitoring process. On the other hand, if it is determined in step S103 that the pointer position is on the operation image, the operation determination area change processing unit 102 executes operation determination area change processing (step S104).

[0049] <Operation detection area change processing> Hereinafter, the manipulation determination region change process according to this embodiment will be described with reference to FIG.

[0050] The distance calculation unit 121 of the manipulation determination area change processing unit 102 calculates the distance a, the distance b, the distance a', and the distance b' (step S201).

[0051] Next, the angle calculation unit 122 of the operation judgment area change processing unit 102 calculates an incident angle θ representing the orientation of the user in the yz plane and an incident angle γ representing the orientation in the xz plane relative to the operation image, using the distance a, distance b, distance a', and distance b' calculated in the above step S201 (step S202).

[0052] As an example, consider a case where an operation image 1000 is present diagonally above and in front of a user U, and the user wishes to operate operation image 1006 among operation images 1001 to 1009 that make up operation image 1000, as shown in FIG.

[0053] In this case, in the yz plane including the reference point Q, if the intersection point between the perpendicular line drawn from the reference point Q to the operation image 1000 and the operation image 1000 is R, and the point where the pointer position P is projected onto the yz plane is P1, then as shown in Figure 8, the distance a is the length of the line segment with Q and R as its endpoints, and the distance b is the length of the line segment with R and P1 as its endpoints. Therefore, the angle of incidence θ is given by θ=tan -1 This is calculated as (b / a). This state can be represented on the yz plane as shown in Figure 9. The coordinates of P, P1, and R can be calculated from the coordinates of the reference point Q and the coordinates of each operation image.

[0054] Similarly, in the xz plane including the reference point Q, if the intersection point between the perpendicular line drawn from the reference point Q to the operation image 1000 and the operation image 1000 is R' and the point where the pointer position P is projected onto the xz plane is P2, then as shown in Figure 10, the distance a' is the length of the line segment with Q and R' as its endpoints, and the distance b' is the length of the line segment with R' and P2 as its endpoints. Therefore, the angle of incidence γ is expressed as γ=tan -1 This is calculated as (b' / a'). This state can be represented on the xz plane as shown in Figure 11. Note that P, P2, and R' can be calculated from the coordinates of the reference point Q and the coordinates of each operation image.

[0055] Returning to the explanation of Fig. 7, following step S202, the manipulation determination area modification unit 123 of the manipulation determination area modification processing unit 102 changes the orientation of the manipulation determination area by using the incident angles θ and γ calculated in step S202 (step S203). That is, the manipulation determination area modification unit 123 changes the orientation of the manipulation determination area by -θ in the yz plane and by -γ in the xz plane.

[0056] 12, within the yz plane, the manipulation determination region 1106 is rotated by −θ around one of the sides of the top or bottom surface of the manipulation determination region 1106 that is in contact with the manipulation image 1006 as an axis so as to face the direction of the user U. The example shown in FIG. 12 shows a case where the manipulation determination region 1106 is rotated by −θ around the side of the bottom surface of the manipulation determination region 1106 that is in contact with the manipulation image 1006 as an axis.

[0057] 13, for example, the manipulation determination region 1106 is rotated by −γ around one of the sides on the left or right side of the manipulation determination region 1106 that is in contact with the manipulation image 1006 as an axis within the xz plane so as to face the direction of the user U. The example shown in FIG. 13 represents a case where the manipulation determination region 1106 is rotated by −γ around the side on the right side of the manipulation determination region 1106 that is in contact with the manipulation image 1006 as an axis.

[0058] This allows each operation determination area to face the front of the user's operation direction, preventing accidental operation of adjacent operation images. Moreover, since the operation image itself does not change, the user can operate the operation image without feeling uncomfortable.

[0059] <Operation execution process> The operation execution process according to this embodiment will be described below with reference to FIG.

[0060] The threshold determination unit 131 of the operation execution processing unit 103 determines whether or not the user's finger has moved to the position of the threshold D in the operation determination area (step S301).

[0061] If it is not determined in step S302 that the user's finger has moved to the position of threshold D of the operation determination area, the operation execution processing unit 103 ends the operation execution processing. On the other hand, if it is determined in step S302 that the user's finger has moved to the position of threshold D of the operation determination area, the operation execution unit 132 of the operation execution processing unit 103 executes operation processing corresponding to the operation image in which the operation determination area is set (step S302). As a result, the operation processing may, for example, transition to another screen, display some information, or perform some other information processing.

[0062] <Modification> A modification of this embodiment will now be described.

[0063] <<Variation 1>> In the above embodiment, the intersection of the direction pointed by a finger or the like and some object is used as the pointer position, but for example, by using known eye tracking technology, the line of sight may be used instead of the direction pointed by a finger or the like. In this case, the operator information is an eyeball image obtained by photographing the user's eyeballs, and the reference point is the point between the user's eyes (for example, the midpoint of the line segment connecting the center of the left eyeball and the center of the right eyeball).

[0064] <<Variation 2>> In the above embodiment, the intersection of the direction pointed by a finger or the like and some object is taken as the pointer position, but for example, if the device 10 is a head-mounted display, the tip of the finger or the like is taken as the reference point, but instead, for example, the position of the head-mounted display (in other words, the position of the user's head) may be taken as the reference point. In this case, the operator information is information indicating the position of the device 10.

[0065] As an example, consider a case where an operation image 2000 is located diagonally above and in front of a user wearing device 10, which is a head-mounted display, as shown in Fig. 15, and the user wishes to operate this operation image 2000. For simplicity's sake, the position (reference point) of device 10 is set to origin O. Furthermore, operation image 2000 is placed on an xy plane (x, y, z0) represented by z = z0, and the pointer position is set to P = (x0, y0, z0). The pointer position is the intersection of the user's line of sight and some object.

[0066] In this case, in the yz plane including the reference point (origin O), if the intersection point between the operation image 2000 and a perpendicular line drawn from the reference point (origin O) to the operation image 2000 is R=(x0,0,z1), and the point where the pointer position P is projected onto the yz plane is P1, then as shown in Figure 15, the distance a is the length of the line segment with O and R as its endpoints, and the distance b is the length of the line segment with R and P1 as its endpoints. Therefore, the angle of incidence θ is expressed as θ=tan -1 (b / a)=tan -1This is calculated as (y0 / x0). This state can be represented on the yz plane as shown in Figure 16. The coordinates of P, P1, and R can be calculated from the coordinates of each operation image.

[0067] Similarly, in the xz plane containing the reference point (origin O), the intersection point between the perpendicular line drawn from the reference point (origin O) to the operation image 2000 and the operation image 2000 is defined as R', and the point where the pointer position P is projected onto the xz plane is defined as P2. Also, it is assumed that the pointer position P exists on the xz plane. In this case, as shown in FIG. 17, R'=P1 and P2=P, and the distance a' is the length of the line segment with O and R' as endpoints, and the distance b' is the length of the line segment with R' and P2 as endpoints. Therefore, the angle of incidence γ is given by γ=tan -1 (b' / a')=tan -1 (z0-z1 / √(x0 2 +y0 2 )) is calculated. This state can be represented on the xz plane as shown in Figure 18. The coordinates of P, P2, and R can be calculated from the coordinates of each operation image.

[0068] <<Variation 3>> In the above embodiment, after calculating the incident angles θ and γ, the orientation of the operation judgment region is changed by −θ in the yz plane and by −γ in the xz plane. However, for example, threshold values ​​may be set for the incident angles θ and γ, and the orientation of the operation judgment region may be changed only when the threshold values ​​are exceeded.

[0069] Specifically, the orientation of the manipulation judgment region may be changed by -θ in the yz plane only when the incident angle θ exceeds a predetermined threshold th1, and may be changed by -γ in the xz plane only when the incident angle γ exceeds a predetermined threshold th2. Note that th1 may be equal to th2 or th1 may not be equal to th2.

[0070] <<Variation 4>> In the third modification, the orientation of the manipulation determination region may be changed by a fixed value only when the incident angles θ and γ exceed threshold values.

[0071] Specifically, the orientation of the manipulation judgment region in the yz plane may be changed by a constant value c1 toward the front of the user only when the incident angle θ exceeds a predetermined threshold value th1, and the orientation of the manipulation judgment region in the xz plane may be changed by a constant value c2 toward the front of the user only when the incident angle γ exceeds a predetermined threshold value th2. Note that either c1 = c2 or c1 ≠ c2 may be satisfied.

[0072] <<Variation 5>> In the above embodiment, the orientation of the operation image is not changed, but it may be changed as well. That is, the orientation of the operation image may be changed by −θ in the yz plane and by −γ in the xz plane.

[0073] However, frequently changing the orientation of the operation image may cause discomfort to the user, so threshold values ​​may be set for the incident angles θ and γ, and the orientation of the operation image may be changed only when the threshold values ​​are exceeded.

[0074] Specifically, the orientation of the control image may be changed by −θ in the yz plane only when the incident angle θ exceeds a predetermined threshold th3, and may be changed by −γ in the xz plane only when the incident angle γ exceeds a predetermined threshold th4. Note that th3 may be equal to th4 or th3 may not be equal to th4.

[0075] Furthermore, when the above thresholds are set, the orientation of the operation image may be changed by a fixed value only when the incident angles θ and γ exceed the thresholds.

[0076] Specifically, the orientation of the operation image may be changed by a constant value c3 in the yz plane toward the front of the user only when the incident angle θ exceeds a predetermined threshold th3, and may be changed by a constant value c4 in the xz plane toward the front of the user only when the incident angle γ exceeds a predetermined threshold th4. Note that either c3 = c4 or c3 ≠ c4 may be satisfied.

[0077] <Summary> As described above, when a user operates a virtual object in an AR space or an MR space, the device 10 according to the present embodiment changes the orientation of the operation determination region set for the virtual object to face the user. This causes the operation determination region to be in the front direction of the user, so that even when the user operates an operation image from an oblique direction, for example, it is possible to prevent the user from erroneously operating another operation image.

[0078] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims. [Explanation of symbols]

[0079] 10 devices 11 Input Devices 12 Display device 13 External I / F 13a Recording media 14 Communication I / F 15 sensors 16 Memory Device 17 processors 18 Bus 101 Operation position monitoring processing unit 102 Operation determination area change processing unit 103 Operation execution processing unit 104 Storage section 111 Operator information acquisition unit 112 Operation position identification section 113 Operation position determination section 121 Distance calculation unit 122 Angle calculation unit 123 Operation determination area change unit 131 Threshold judgment unit 132 Operation execution unit

Claims

1. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object operable by a user is arranged, an identification unit configured to identify an operation position on the virtual object using information about the user; a calculation unit configured to calculate an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change unit configured to change an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; and and the information about the user is hand mesh data representing a hand of the user; The identification unit an information processing device configured to specify, using the hand mesh data, an intersection of a three-dimensional vector representing a direction in which a specific finger of the hand is pointing and the virtual object as the operation position;

2. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object that can be operated by a user is arranged, an identification unit configured to identify an operation position on the virtual object using information about the user; a calculation unit configured to calculate an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change unit configured to change an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; and and the information about the user is information indicating a position of a head-mounted display worn by the user, The identification unit an information processing device configured to specify, as the operation position, an intersection of a three-dimensional vector representing the user's line of sight and the virtual object, using information indicating the position of the head-mounted display.

3. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object that can be operated by a user is arranged, an identification unit configured to identify an operation position on the virtual object using information about the user; a calculation unit configured to calculate an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change unit configured to change an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; and and The change unit The information processing device is configured to change the operation determination region when the angle exceeds a predetermined threshold.

4. The change unit The information processing device according to claim 1 , wherein the direction of the operation determination region is changed by the angle in a front direction of the user.

5. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object that can be operated by a user is arranged, an identification unit configured to identify an operation position on the virtual object using information about the user; a calculation unit configured to calculate an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change unit configured to change an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; and and The calculation unit a first angle representing the operation direction in a vertical plane including the reference point, and a second angle representing the operation direction in a horizontal plane including the reference point, The change unit changing the orientation of the manipulation determination region by the first angle in a front direction of the user within the vertical plane; the information processing device is configured to change the orientation of the operation determination region by the second angle in a front direction of the user within the horizontal plane.

6. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object operable by a user is arranged, an identification step of identifying an operation position on the virtual object using information about the user; a calculation step of calculating an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change procedure for changing an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; Run the information about the user is hand mesh data representing a hand of the user; The identification procedure includes: An information processing method that uses the hand mesh data to identify, as the operation position, an intersection between a three-dimensional vector representing a direction in which a specific finger of the hand is pointing and the virtual object.

7. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object operable by a user is arranged, comprising: an identification step of identifying an operation position on the virtual object using information about the user; a calculation step of calculating an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change procedure for changing an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; Run the information about the user is information indicating a position of a head-mounted display worn by the user, The identification procedure includes: An information processing method that uses information indicating the position of the head-mounted display to identify the intersection of a three-dimensional vector representing the user's line of sight and the virtual object as the operation position.

8. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object operable by a user is arranged, comprising: an identification step of identifying an operation position on the virtual object using information about the user; a calculation step of calculating an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change procedure for changing an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; Run The change procedure is as follows: An information processing method for changing the operation determination area when the angle exceeds a predetermined threshold.

9. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object operable by a user is arranged, comprising: an identification step of identifying an operation position on the virtual object using information about the user; a calculation step of calculating an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change procedure for changing an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; Run The calculation procedure is as follows: calculating a first angle representing the operation direction in a vertical plane including the reference point and a second angle representing the operation direction in a horizontal plane including the reference point; The change procedure is as follows: changing the orientation of the manipulation determination region by the first angle in a front direction of the user within the vertical plane; an information processing method for changing the orientation of the manipulation determination area by the second angle in a front direction of the user within the horizontal plane;

10. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object operable by a user is arranged, an identification step of identifying an operation position on the virtual object using information about the user; a calculation step of calculating an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change procedure for changing an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; Execute the information about the user is hand mesh data representing a hand of the user; The identification procedure includes: a program that uses the hand mesh data to identify, as the operation position, an intersection between a three-dimensional vector representing a direction in which a specific finger of the hand is pointing and the virtual object;

11. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object that can be operated by a user is arranged, an identification step of identifying an operation position on the virtual object using information about the user; a calculation step of calculating an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change procedure for changing an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; Execute the information about the user is information indicating a position of a head-mounted display worn by the user, The identification procedure includes: A program that uses information indicating the position of the head-mounted display to identify the intersection of a three-dimensional vector representing the user's line of sight and the virtual object as the operation position.

12. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object that can be operated by a user is arranged, an identification step of identifying an operation position on the virtual object using information about the user; a calculation step of calculating an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change procedure for changing an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; Execute The change procedure is as follows: a program for changing the operation determination region when the angle exceeds a predetermined threshold value;

13. An information processing device that displays an augmented reality space or a mixed reality space in which at least a virtual object that can be operated by a user is arranged, an identification step of identifying an operation position on the virtual object using information about the user; a calculation step of calculating an angle representing an operation direction when the user operates the virtual object, using a reference point representing a position of information related to the user and the operation position; a change procedure for changing an operation determination area for determining whether an operation has been performed on the virtual object, using the angle; Execute The calculation procedure is as follows: calculating a first angle representing the operation direction in a vertical plane including the reference point and a second angle representing the operation direction in a horizontal plane including the reference point; The change procedure is as follows: changing the orientation of the manipulation determination region by the first angle in a front direction of the user within the vertical plane; a program for changing the orientation of the operation determination region by the second angle in a front direction of the user within the horizontal plane;

Images