Information processing device and information processing program

Abstract

To cause the sensitivity of a pointer to be reduced without requiring a user to perform any operation separately, when the user is pointing or likely to point to a specific position in a virtual three-dimensional space by a pointer that virtually emits a ray to the virtual three-dimensional space.SOLUTION: An image processing unit 26 perspectively projects a virtual three-dimensional space where a virtual object is located, on the basis of a visual line direction 42 that is set in the virtual three-dimensional space in accordance with the attitude of an information processing device 10 to generate a two-dimensional image and causes it to be displayed on a display 14. A ray control unit 28 sets a ray emission angle in the virtual three-dimensional space on the basis of the attitude of a pointer 12. When a sensitivity reduction condition is satisfied that the angle change amount of the visual line direction 42 in a prescribed time is less than or equal to a threshold visual line angle and the angle change amount of a ray emission angle in the prescribed time is less than or equal to a threshold emission angle, the ray control unit 28 reduces the sensitivity of the pointer 12, as compared with the case where the sensitivity reduction condition is not satisfied.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing program. [Background technology]

[0002] Patent Document 1 discloses a method for identifying a user's gaze point on a display and reducing the moving speed of a cursor moving on the display by mouse operation as the distance between the gaze point and the current position of the cursor decreases. Patent Document 2 discloses a control method for monitoring the vicinity of a cursor moving on the display by mouse operation and reducing the moving speed of the cursor to make operation easier when an object approaches the cursor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5807989 [Patent Document 2] Japanese Patent Application Publication No. 9-265353 Summary of the Invention [Problem to be solved by the invention]

[0004] There is known a device that sets a virtual viewpoint and a line of sight in a virtual three-dimensional space, and displays a two-dimensional image obtained by projecting the virtual three-dimensional space based on the virtual viewpoint and line of sight on a display. In such a device, consider a situation in which a user points to a specific position in the virtual three-dimensional space by virtually emitting a ray (virtual beam) into the virtual three-dimensional space from a pointer held by the user.

[0005] Here, if the pointer sensitivity is high, it may be difficult for the user to point to the desired position with the pointer (i.e., ray). Pointer sensitivity is the amount of angular change in the emission angle of the ray emitted from the pointer relative to the amount of angular change in the attitude (or orientation) of the pointer.

[0006] It is possible to make the pointer sensitivity switchable by user operation, but this would require a separate operation to switch the pointer sensitivity, which is troublesome for the user and may reduce the user's concentration.

[0007] The object of the present invention is to reduce the sensitivity of a pointer that virtually emits a ray into the virtual three-dimensional space when the user is pointing or attempting to point to a specific position in the virtual three-dimensional space in an apparatus that displays a two-dimensional image obtained by projecting a virtual three-dimensional space on a display based on a virtual viewpoint and line of sight set within the virtual three-dimensional space, without the user having to perform any additional operation. [Means for solving the problem]

[0008] The invention of claim 1 is an information processing device comprising a processor that sets a line of sight of a user in a virtual three-dimensional space based on the attitude of a device worn or held by the user, projects the virtual three-dimensional space to generate a two-dimensional image based on a virtual viewpoint and the line of sight set in the virtual three-dimensional space, and displays the two-dimensional image on a display, sets an emission angle of a ray virtually emitted into the virtual three-dimensional space based on the attitude of a pointer held by the user, and when a sensitivity reduction condition is satisfied that an angular change in the line of sight of the user within a predetermined time is equal to or less than a threshold line of sight angle and an angular change in the emission angle of the ray within a predetermined time is equal to or less than a threshold emission angle, reduces an angular change in the emission angle of the ray relative to an angular change in the attitude of the pointer compared to when the sensitivity reduction condition is not satisfied. The invention of claim 2 is the information processing device described in claim 1, characterized in that the processor detects the distance from the emission position of the ray in the virtual three-dimensional space to the irradiation point of the ray, and the sensitivity reduction condition further includes a condition that the distance from the emission position of the ray in the virtual three-dimensional space to the irradiation point of the ray is greater than or equal to a threshold distance. The invention of claim 3 is the information processing device described in claim 1, characterized in that the processor reduces the angular change in the emission angle of the ray relative to the angular change in the attitude of the pointer as the angular change in the emission angle of the ray within a specified time period becomes smaller. The invention of claim 4 is the information processing device described in claim 2, characterized in that the processor reduces the angular change in the emission angle of the ray relative to the angular change in the attitude of the pointer, the greater the distance from the emission position of the ray to the irradiation point of the ray in the virtual three-dimensional space. The invention of claim 5 is the information processing device according to any one of claims 1 to 4, characterized in that the processor generates an information display object in the virtual three-dimensional space that shows information about the virtual object pointed to by the ray, displays a two-dimensional image including a projected image of the information display object on the display, and maintains a state in which an angular change in the emission angle of the ray relative to an angular change in the attitude of the pointer is reduced when it is determined that the user is looking at the information display object based on the user's gaze direction after the change, even if the user's gaze direction has changed by a larger amount than the threshold gaze angle after the sensitivity reduction condition is satisfied. The invention of claim 6 is an information processing program that causes a computer to set a line of sight of a user in a virtual three-dimensional space based on the attitude of a device worn or held by the user, project the virtual three-dimensional space to generate a two-dimensional image based on the virtual viewpoint and the line of sight set in the virtual three-dimensional space, display the two-dimensional image on a display, set an emission angle of a ray virtually emitted into the virtual three-dimensional space based on the attitude of a pointer held by the user, and when a sensitivity reduction condition is satisfied that an angular change in the line of sight of the user within a predetermined time is equal to or less than a threshold line of sight angle and an angular change in the emission angle of the ray within a predetermined time is equal to or less than a threshold emission angle, reduce an angular change in the emission angle of the ray relative to an angular change in the attitude of the pointer compared to when the sensitivity reduction condition is not satisfied. [Effects of the Invention]

[0009] According to the invention of claim 1 or 6, in a device that displays a two-dimensional image obtained by projecting a virtual three-dimensional space on a display based on a virtual viewpoint and line of sight set within the virtual three-dimensional space, when a user is pointing or attempting to point to a specific position in the virtual three-dimensional space using a pointer that virtually emits a ray into the virtual three-dimensional space, the sensitivity of the pointer can be reduced without the user having to perform any additional operation. According to the invention of claim 2, it is further possible to reduce the sensitivity of the pointer based on the distance from the ray emission position to the ray irradiation point. According to the invention of claim 3, when the user is pointing or trying to point to a more accurate position with the pointer, the sensitivity of the pointer can be further reduced. According to the invention of claim 4, the farther the position at which the user is attempting to point with the pointer is from the emission position of the ray, the more the sensitivity of the pointer can be reduced. According to the invention of claim 5, even when the user changes the line of sight to view an information display object that shows information about a virtual object pointed to by a ray, the pointer sensitivity can be maintained at a low level. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating the configuration of an information processing device according to an embodiment of the present invention. [Figure 2] 1A and 1B are diagrams illustrating a virtual viewpoint, a line of sight direction, and a virtual screen. [Figure 3] FIG. 1 is a diagram illustrating an example of a two-dimensional image. [Figure 4] FIG. 10 is a diagram showing a two-dimensional image illustrating a virtual object being pointed at by a pointer. [Figure 5] 10 is a flowchart showing a flow of processing performed by the information processing device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1 is a schematic diagram of the configuration of an information processing device 10 according to this embodiment. The information processing device 10 is a device that displays a two-dimensional image generated by projecting a virtual three-dimensional space on a display 14. More specifically, the information processing device 10 is a device that displays a two-dimensional image generated by projecting a virtual object arranged in the virtual three-dimensional space on the display 14.

[0012] The information processing device 10 according to this embodiment is a VR device that realizes so-called VR (Virtual Reality) (also called virtual reality), which displays a two-dimensional image showing a virtual object placed in a virtual three-dimensional space together with the background of the virtual three-dimensional space. However, as will be described later, the information processing device 10 may also be an AR device that realizes so-called AR (Augmented Reality) (also called extended reality), which displays a two-dimensional image showing a virtual object together with the background of a real three-dimensional space. Furthermore, the information processing device 10 may be an MR device or an SR device that realizes MR (Mixed Reality) (also called mixed reality) or SR (Substitutional Reality), which are a combination of VR and AR. VR, AR, MR, SR, etc. are collectively referred to as XR (Extended Reality). That is, the information processing device 10 may be an XR device.

[0013] The information processing device 10 is a device worn or held by a user. For example, the information processing device 10 is an HMD (Head Mounted Display), smart glasses, a tablet terminal, or the like. Of course, the information processing device 10 is not limited to these devices.

[0014] In this embodiment, the virtual object to be placed in the virtual three-dimensional space is not particularly limited. For example, a network graph can be placed as a virtual object in the virtual three-dimensional space. The network graph is configured to include a plurality of nodes, each associated with predetermined information, and edges that indicate the relationship between two nodes (strictly speaking, the relationship between the information associated with the two nodes).

[0015] A user of the information processing device 10 uses a pointer 12 together with the information processing device 10. The pointer 12 is held and used by the user, and is used by the user to point to a desired position in the virtual three-dimensional space. Specifically, a ray (a virtual laser) is emitted from the pointer 12 into the virtual three-dimensional space. Similar to a real laser, when the ray hits a virtual object in the virtual three-dimensional space, the ray stops there and irradiates the virtual object. The user can point to a desired virtual object by irradiating the ray at the desired virtual object. A specific description of the ray will be given later.

[0016] The display 14 is configured by, for example, a liquid crystal panel, an organic EL (Electro Luminescence), a waveguide (light guide plate), etc. A two-dimensional image formed by a processor 22 (described later) is displayed on the display 14.

[0017] The acceleration sensor 16 is a sensor that detects the position and attitude of the information processing device 10. Specifically, the acceleration sensor 16 performs a calibration process when the information processing device 10 is at a predetermined position and in a predetermined attitude, and detects the displacement of the information processing device 10 from the calibration position in three mutually orthogonal axis directions defined in real three-dimensional space, and the rotation angle around each of the three axes from the calibration attitude as a central axis.

[0018] The input interface 18 is configured with buttons, a touch panel, etc. The input interface 18 is used when the user inputs instructions to the information processing device 10.

[0019] The memory 20 includes, for example, an embedded multi-media card (eMMC), a read-only memory (ROM), or a random access memory (RAM). The memory 20 stores an information processing program for operating each unit of the information processing device 10. The information processing program can also be stored in a computer-readable non-transitory storage medium such as a universal serial bus (USB) memory or a CD-ROM. The information processing device 10 can read and execute the information processing program from such a storage medium.

[0020] Furthermore, attribute information about one or more virtual objects placed in the virtual three-dimensional space is stored in the memory 20. The attribute information about the virtual object includes information indicating the form of the virtual object, information indicating the name of the virtual object, information indicating the placement position of the virtual object in the virtual three-dimensional space, information indicating the content of the virtual object, and the like.

[0021] The processor 22 refers to a processor in a broad sense and includes at least one of a general-purpose processor (e.g., a CPU (Central Processing Unit)) and a dedicated processing device (e.g., a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a programmable logic device). The processor 22 may not be a single processing device, but may be configured by the cooperation of multiple processing devices located in physically separate locations. As shown in FIG. 1, the processor 22 performs the functions of a line of sight direction setting unit 24, an image processing unit 26, and a ray control unit 28 in accordance with an information processing program stored in the memory 20.

[0022] The line-of-sight direction setting unit 24 sets the virtual viewpoint and line-of-sight direction of the user in the virtual three-dimensional space.V Axis, Y V axis and Z V Shown are a virtual viewpoint 40 and a line of sight direction 42 set in a virtual three-dimensional space indicated by axes. Specifically, the line of sight direction setting unit 24 sets the position of the virtual viewpoint 40 based on the position of the information processing device 10 in the real three-dimensional space, and sets the line of sight direction 42 based on the attitude of the information processing device 10 in the real three-dimensional space.

[0023] Specifically, when the acceleration sensor 16 performs calibration, a virtual viewpoint 40 is set at a predetermined position in the virtual three-dimensional space. From there, the position of the virtual viewpoint 40 in the virtual three-dimensional space is changed in accordance with the displacement of the information processing device 10 from the calibration position (displacement in real three-dimensional space) detected by the acceleration sensor 16. Furthermore, when the acceleration sensor 16 performs calibration, a predetermined direction in the virtual three-dimensional space is set as a line of sight direction 42. From there, the line of sight direction 42 in the virtual three-dimensional space is changed in accordance with the rotation angle from the calibration posture of the information processing device 10 (rotation angle in real three-dimensional space) detected by the acceleration sensor 16. Note that the field of view from the virtual viewpoint 40 has a predetermined field of view angle (e.g., 180°), and the line of sight direction 42 refers to the direction at the center of the field of view. In other words, the field of view from the virtual viewpoint 40 has a field of view angle centered on the line of sight direction 42.

[0024] Furthermore, the line of sight direction setting unit 24 may set an upward vector that indicates the upward direction of the field of view from the virtual viewpoint 40. The upward vector can also be set based on the orientation of the information processing device 10 in the real three-dimensional space. Note that the upward vector is set to the vertically upward direction (Z V It may be fixed to the positive side of the shaft.

[0025] In this specification, the line of sight direction 42 is expressed by a vector (θxu, θyu, θzu). θxu, θyu, and θzu are the X axis of the virtual three-dimensional space. V Rotation angle around axis, Y V Rotation angle around axis, Z VThe vector (θxu, θyu, θzu) is the angle of rotation around the axis. The vector (θxu, θyu, θzu) also indicates the orientation of the information processing device 10 in the real three-dimensional space.

[0026] The image processing unit 26 projects the virtual three-dimensional space based on the virtual viewpoint 40 and the line-of-sight direction 42 set by the line-of-sight direction setting unit 24, and generates a two-dimensional image.

[0027] First, the image processing unit 26 defines a virtual screen 44 in the virtual three-dimensional space, as shown in Fig. 2. The virtual screen 44 is a surface perpendicular to the line of sight 42, and may be large enough to cover the field of view from the virtual viewpoint 40. In this embodiment, the image processing unit 26 generates a two-dimensional image by perspectively projecting the virtual three-dimensional space (more specifically, a virtual object placed in the virtual three-dimensional space) onto the virtual screen 44.

[0028] The image processing unit 26 displays the generated two-dimensional image on the display 14. FIG. 3 shows an example of a two-dimensional image displayed on the display 14. The two-dimensional image includes a virtual object image 50 corresponding to a virtual object placed in the virtual three-dimensional space. Note that the two-dimensional image shown in FIG. 3 is a two-dimensional image generated by projecting virtual objects representing furniture placed in a virtual three-dimensional space simulating a room. However, the virtual objects placed in the virtual three-dimensional space are not limited to this. For example, as described above, a network graph may be placed in the virtual three-dimensional space.

[0029] When the user changes the position or posture of the information processing device 10, the position of the virtual viewpoint 40 or the viewing direction 42 set by the viewing direction setting unit 24 changes sequentially. The image processing unit 26 sequentially updates the two-dimensional image at a predetermined frame rate, generates a two-dimensional image according to the changing virtual viewpoint 40 or viewing direction 42, and displays it on the display 14.

[0030] The ray control unit 28 sets, in the virtual three-dimensional space, the emission position of a ray, which is a virtual laser emitted from the pointer 12, based on the position of the pointer 12 in the real three-dimensional space, and sets the emission direction of the ray (referred to as an emission angle in this specification) in the virtual three-dimensional space based on the orientation of the pointer 12 in the real three-dimensional space. Like the acceleration sensor 16 of the information processing device 10, the pointer 12 is equipped with an acceleration sensor that detects the position and orientation of the pointer 12 in the real three-dimensional space, and a detection signal of the acceleration sensor is transmitted to the information processing device 10. The ray control unit 28 detects the position and orientation of the pointer 12 based on the detection signal received from the acceleration sensor of the pointer 12, and sets the emission position and emission angle of the ray based on the position and orientation of the pointer 12.

[0031] Specifically, when the ray control unit 28 performs calibration, the emission position of the ray is set to a predetermined position in the virtual three-dimensional space. From there, the emission position of the ray in the virtual three-dimensional space is changed according to the displacement from the calibration position (displacement in real three-dimensional space) detected by the acceleration sensor of the pointer 12. Furthermore, when the ray control unit 28 performs calibration, a predetermined direction in the virtual three-dimensional space is set as the emission angle of the ray. From there, the emission angle of the ray in the virtual three-dimensional space is changed according to the rotation angle from the calibration posture (rotation angle in real three-dimensional space) detected by the acceleration sensor of the pointer 12.

[0032] In this specification, the emission angle of a ray is expressed as a vector (θxp, θyp, θzp). θxp, θyp, and θzp are the X axis of the virtual three-dimensional space. V Rotation angle around axis, Y V Rotation angle around axis, Z V The vector (θxp, θyp, θzp) also indicates the orientation of the pointer 12 in the real three-dimensional space.

[0033] 4 is a diagram showing a two-dimensional image representing a state in which a virtual object is pointed at by the pointer 12. When the user emits a ray within the field of view (in other words, within the field of view centered on the line of sight 42 when viewed from the virtual viewpoint 40), the image processing unit 26 generates a two-dimensional image including a pointer image 52 representing the pointer 12, a ray image 54 representing the ray emitted from the pointer 12, and an irradiation point image 56 representing the irradiation point at which the ray is irradiated on the virtual object, and displays the generated image on the display 14.

[0034] The ray image 54 can be obtained in the virtual three-dimensional space by projecting a ray extending from the emission position set as described above in the emission angle (i.e., the direction represented by the vector (θxp, θyp, θzp)) onto the virtual screen 44. The position of the illuminated point image 56 in the two-dimensional image can be obtained by projecting the intersection of the virtual object and the ray in the virtual three-dimensional space onto the virtual screen 44.

[0035] Furthermore, when the user points at a virtual object within the field of view with a ray, the processor 22 generates an information display object in the virtual three-dimensional space that displays attribute information of the virtual object or an enlarged view of the virtual object image 50 corresponding to the virtual object pointed at by the ray. The image processing unit 26 performs rear projection of the information display object to display a two-dimensional image including an information display frame 58 on the display 14. By displaying the information display frame 58, the user can grasp the attribute information of the virtual object pointed at with the ray (for example, the name and content of the virtual object) and details of the virtual object image 50. When the virtual object is no longer pointed at with the ray (when the irradiation point of the ray moves away from the virtual object), the processor 22 erases the information display object, and the image processing unit 26 erases the information display frame 58.

[0036] In this embodiment, rays are always emitted from the pointer 12, but the emission / stop of rays from the pointer 12 may be switched by a user instruction (for example, by pressing a button provided on the pointer 12).

[0037] When the user changes the orientation of the pointer 12, the ray control unit 28 changes the ray emission direction accordingly. By changing the ray emission direction, the user can point to a desired virtual object. The image processing unit 26 generates a two-dimensional image in which the ray image 54 and the illuminated point image 56 have been updated in response to the change in the ray emission direction, and displays the updated image on the display 14.

[0038] The ray control unit 28 performs control to change the sensitivity of the pointer 12 (which may also be referred to as the sensitivity of the ray emission angle). The sensitivity of the pointer 12 is the angular change in the ray emission angle relative to the angular change in the attitude of the pointer 12. Specifically, when a sensitivity reduction condition is satisfied, in which the angular change in the line of sight 42 within a predetermined time is equal to or less than a threshold line of sight angle and the angular change in the ray emission angle within a predetermined time is equal to or less than a threshold emission angle, the ray control unit 28 reduces the sensitivity of the pointer 12 compared to when the sensitivity reduction condition is not satisfied. In this specification, the operation mode of the information processing device 10 when the sensitivity reduction condition is not satisfied is referred to as a normal sensitivity mode, and the operation mode of the information processing device 10 when the sensitivity reduction condition is satisfied is referred to as a low sensitivity mode.

[0039] As described above, since the line of sight direction 42 is expressed by the vector (θxu, θyu, θzu), the amount of angular change in the line of sight direction 42 means the amount of angular change in the vector (θxu, θyu, θzu). The amount of angular change in the vector (θxu, θyu, θzu) is expressed, for example, as Δθxu+Δθyu+Δθzu. Here, Δθxu means the difference between θxu at a certain point in time and θxu after a predetermined time has elapsed since that point in time. The same applies to Δθyu and Δθzu. In this embodiment, the line of sight direction 42 is set based on the attitude of the information processing device 10, and therefore the amount of angular change in the line of sight direction 42 can also be said to mean the amount of angular change in the attitude of the information processing device 10.

[0040] Similarly, since the emission angle of a ray is expressed by the vector (θxp, θyp, θzp), the angular change amount of the emission angle of a ray means the angular change amount of the vector (θxp, θyp, θzp). The angular change amount of the vector (θxp, θyp, θzp) is expressed, for example, as Δθxp+Δθyp+Δθzp. Here, Δθxp means the difference between θxp at a certain point in time and θxp after a predetermined time has elapsed from that point in time. The same applies to Δθyp and Δθzp. In this embodiment, the emission angle of a ray is set based on the attitude of the pointer 12, so it can also be said that the angular change amount of the emission angle of a ray means the angular change amount of the attitude of the pointer 12.

[0041] The amount of change in the emission angle of the ray within a predetermined time can also be calculated based on the distance from the emission position of the ray to the irradiation point of the ray and the moving distance of the irradiation point of the ray within the predetermined time.

[0042] The ray control unit 28 sets the ray emission position based on the position of the pointer 12, and therefore the ray emission position in the virtual three-dimensional space is known to the ray control unit 28. The irradiation point of the ray is the surface position of the first virtual object located in the direction of the ray emission angle (the direction indicated by the vector (θxp, θyp, θzp)) from the ray emission position. The placement position (coordinates) of each virtual object in the virtual three-dimensional space is also known to the ray control unit 28. Therefore, the ray control unit 28 can calculate the position of the ray irradiation point based on the ray emission position, the ray emission angle, and the position of the virtual object. Then, the ray control unit 28 calculates the distance from the ray emission position to the ray irradiation point.

[0043] The irradiation point of the ray in the virtual 3D space is (Xp, Yp, Zp) (Xp, Yp, Zp are the X and Y coordinates of the virtual 3D space, respectively). V Axis coordinate, Y V Axis coordinate, Z V If we express the angles in terms of the x, y, and z coordinates, Δθxp, Δθyp, and Δθzp can be calculated using the following equations. Δθxp=tan-1 (ΔXp / L) Δθyp=tan -1 (ΔYp / L) Δθzp=tan -1 (ΔZp / L) In the above formula, ΔXp is the change in Xp within a specified time, ΔYp is the change in Yp within a specified time, ΔXp is the change in Zp within a specified time, and L represents the distance from the ray emission position to the ray irradiation point.

[0044] The "predetermined time" may be determined as appropriate by an administrator of the information processing device 10, for example. One method for determining the predetermined time is to determine the predetermined time based on the frame rate of the two-dimensional images displayed on the display 14 (the number of two-dimensional images displayed on the display 14 per unit time). For example, if the frame rate of the two-dimensional images is 60 fps (frames per second), the predetermined time may be set to 1 / 60 seconds. However, if the predetermined time is short, the influence of the body movement of the user wearing or holding the information processing device 10 may be present. Therefore, the angle change amount used for comparison with the threshold gaze angle may be an average of multiple angle changes in the gaze direction 42 over multiple predetermined times. Furthermore, if the predetermined time is short, the influence of the body movement of the user holding the pointer 12 may be present. Therefore, the angle change amount used for comparison with the threshold gaze angle may be an average of multiple angle changes in the emission angle of the ray over multiple predetermined times.

[0045] When the amount of change in the angle of gaze direction 42 within a predetermined time is equal to or less than the threshold gaze angle, it is highly likely that the user is gazing at a certain position (e.g., a certain virtual object) in the virtual three-dimensional space. Conversely, the threshold gaze angle is set in advance by an administrator of information processing device 10 or the like, depending on the amount of change in the angle of gaze direction 42 when the user is gazing at a certain position in the virtual three-dimensional space, and is stored in memory 20. For example, the threshold gaze angle is set to about 10 degrees.

[0046] Furthermore, if the amount of change in the emission angle of the ray within a predetermined time is equal to or less than the threshold emission angle, it can be said that there is a high possibility that the user is pointing or attempting to point to a certain position (e.g., a certain virtual object) in the virtual three-dimensional space with the pointer 12. Conversely, the threshold emission angle is set in advance by an administrator of the information processing device 10 or the like and stored in the memory 20 according to the amount of change in the emission angle of the ray when the user is pointing or attempting to point to a certain position in the virtual three-dimensional space with the pointer 12. For example, the threshold emission angle is set to about 5 degrees. The threshold emission angle may be an angle smaller than the threshold line-of-sight angle.

[0047] If the amount of angular change in the line of sight 42 within a predetermined time is equal to or less than the threshold line of sight angle and the amount of angular change in the emission angle of the ray within the predetermined time is equal to or less than the threshold emission angle, it is considered that the user needs to finely control the position of the irradiation point of the ray. If the sensitivity of the pointer 12 is high, the irradiation point of the ray will move significantly even with a small movement of the pointer 12, making it difficult for the user to align the irradiation point of the ray with the desired position. Therefore, in this embodiment, the ray control unit 28 reduces the sensitivity of the pointer 12 when the amount of angular change in the line of sight 42 within a predetermined time is equal to or less than the threshold line of sight angle and the amount of angular change in the emission angle of the ray within the predetermined time is equal to or less than the threshold emission angle, in other words, when it is highly likely that the user is gazing at a specific position in the virtual three-dimensional space and is pointing or attempting to point to a specific position in the virtual three-dimensional space with the pointer 12.

[0048] As described above, the sensitivity of the pointer 12 is the amount of change in the emission angle of the ray relative to the amount of change in the angle of the attitude of the pointer 12. Therefore, reducing the sensitivity of the pointer 12 means reducing the amount of change in the emission angle of the ray relative to the amount of change in the angle of the attitude of the pointer 12. For example, in the normal sensitivity mode, the ray control unit 28 reduces the amount of change in the emission angle of the ray relative to the amount of change in the angle of the attitude of the pointer 12 when the attitude of the pointer 12 is changed by 4 degrees in the X-axis direction in the real three-dimensional space. VIn the low sensitivity mode, when the attitude of the pointer 12 is changed by 4 degrees in the X-axis direction in the real three-dimensional space, the X-axis direction in the virtual three-dimensional space is changed by 4 degrees. V Change the ray's emission angle by 2 degrees in the axial direction.

[0049] Reducing the sensitivity of the pointer 12 has the effect of making it easier for the user to point to a desired position with the pointer 12. Furthermore, in this embodiment, the sensitivity of the pointer 12 is automatically reduced, so to speak, without the user having to perform a separate operation to switch the sensitivity. Therefore, the user does not have to perform a switch operation or the like to switch the sensitivity of the pointer 12.

[0050] Furthermore, it can generally be said that the closer the irradiation point of the ray is to the user's target position, or the smaller (narrower) the area of ​​three-dimensional space that the user is pointing to, the smaller the amount of change in the emission angle of the ray within a predetermined time. Therefore, the ray control unit 28 may be configured to reduce the sensitivity of the pointer 12 the smaller the amount of change in the emission angle of the ray within a predetermined time. This has the effect of making it easier for the user to point to a target position with the pointer 12.

[0051] In principle, the ray control unit 28 changes the emission angle of the ray in accordance with changes in the attitude of the pointer 12, but in the low sensitivity mode, the average attitude of the pointer 12 within a unit time (the average value of the angles around each axis of the pointer 12 within a unit time) may be set as the emission angle of the ray. This reduces the influence of the body movement of the user holding the pointer 12, allowing the user to point with the pointer 12 to a desired position with higher accuracy.

[0052] The ray control unit 28 maintains the operation mode of the information processing device 10 in the low sensitivity mode as long as the above-mentioned sensitivity reduction condition is satisfied, and then returns the operation mode of the information processing device 10 to the normal sensitivity mode when the above-mentioned sensitivity reduction condition is no longer satisfied.

[0053] In the virtual three-dimensional space, the greater the distance from the ray emission position to the ray irradiation point, the greater the movement amount of the ray irradiation point relative to the angular change in the ray emission angle (i.e., the angular change in the attitude of the pointer 12). In other words, if the distance in the virtual three-dimensional space from the ray emission position to the virtual object that the user wants to point to with the ray is large, it becomes more difficult for the user to point to the desired position with the ray.

[0054] Therefore, the ray control unit 28 may detect the distance from the ray emission position to the ray irradiation point in the virtual three-dimensional space, and if the sensitivity reduction conditions are met, that is, the amount of angular change in the line of sight direction 42 within a predetermined time is equal to or less than the threshold line of sight angle, the amount of angular change in the ray emission angle within a predetermined time is equal to or less than the threshold emission angle, and the distance from the ray emission position to the ray irradiation point in the virtual three-dimensional space is equal to or greater than the threshold distance, the ray control unit 28 may reduce the sensitivity of the pointer 12 compared to when the sensitivity reduction conditions are not met.

[0055] As described above, the greater the distance from the ray emission position to the ray irradiation point, the greater the movement of the ray irradiation point relative to the angular change in the ray emission angle. Therefore, the ray control unit 28 may be configured to reduce the sensitivity of the pointer 12 more as the distance from the ray emission position to the ray irradiation point in the virtual three-dimensional space increases.

[0056] Consider a case where the image processing unit 26 displays an information display frame 58 (see FIG. 4 ) on the display 14. In this case, first, the user sets the line of sight 42 in a certain direction to point to a desired virtual object with a ray, and points to the desired virtual object with the ray. During this process, the ray control unit 28 reduces the sensitivity of the pointer 12 (i.e., sets the operation mode of the information processing device 10 to a low-sensitivity mode). Thereafter, the image processing unit 26 displays, on the display 14, an information display frame 58 corresponding to an information display object that displays information about the virtual object pointed to with the ray. The user may turn the line of sight 42 toward the information display object to view the information display object generated in the virtual three-dimensional space. In this case, one of the sensitivity reduction conditions, that is, the amount of angular change in the line of sight 42 within a predetermined time is equal to or less than a threshold line of sight angle, may no longer be satisfied.

[0057] However, as described above, if the irradiation point of the ray moves away from the virtual object, the information display object and the information display frame 58 are erased, and therefore, while the user is viewing the information display object (information display frame 58), the user needs to maintain the emission angle of the ray so that the irradiation point of the ray does not move away from the virtual object. In this case, if the sensitivity of the pointer 12 returns to the normal sensitivity, it may become difficult to maintain the irradiation point of the ray so that it does not move away from the virtual object.

[0058] Therefore, even if the user's line of sight 42 changes by more than the threshold line of sight angle after the sensitivity reduction condition is once satisfied, the ray control unit 28 may maintain the low sensitivity mode if it determines that the user is looking at the information display object based on the user's line of sight 42 after the change. Whether the user is looking at the information display object can be determined based on the position of the virtual viewpoint 40, the line of sight direction 42, and the position of the information display object in the virtual three-dimensional space.

[0059] According to this, the sensitivity of the pointer 12 is maintained at a low level while the user is viewing the information display object, making it easier for the user to keep the irradiation point of the ray from straying from the virtual object.

[0060] The configuration of the information processing device 10 according to this embodiment is as described above. The flow of processing by the information processing device 10 according to this embodiment will be described below with reference to the flowchart shown in FIG.

[0061] In step S10, the line-of-sight direction setting unit 24 sets a virtual viewpoint 40 in the virtual three-dimensional space based on the position of the information processing device 10, and sets a line-of-sight direction 42 in the virtual three-dimensional space based on the attitude of the information processing device 10. Then, the image processing unit 26 projects the virtual three-dimensional space based on the set virtual viewpoint 40 and line-of-sight direction 42, generates a two-dimensional image, and displays it on the display 14. The image processing unit 26 sequentially updates the two-dimensional image at a predetermined frame rate, and displays the two-dimensional image on the display 14 according to the changing virtual viewpoint 40 or line-of-sight direction 42.

[0062] In step S12, the ray control unit 28 sets the emission position of the ray in the virtual three-dimensional space based on the position of the pointer 12 in the real three-dimensional space, and sets the emission angle of the ray in the virtual three-dimensional space based on the orientation of the pointer 12 in the real three-dimensional space. Then, the ray control unit 28 emits a ray in the virtual three-dimensional space from the set emission position in the direction of the emission angle. It is assumed that the operating mode of the information processing device 10 is the normal sensitivity mode when the information processing device 10 is started up.

[0063] In step S14, the ray control unit 28 determines whether the amount of angular change in the line of sight 42 within a predetermined time is equal to or less than the threshold line of sight angle. If the amount of angular change in the line of sight 42 within a predetermined time is equal to or less than the threshold line of sight angle, the process proceeds to step S16.

[0064] In step S16, the ray control unit 28 determines whether the amount of change in the emission angle of the ray within a predetermined time is equal to or less than the threshold emission angle. If the amount of change in the emission angle of the ray within a predetermined time is equal to or less than the threshold emission angle, the process proceeds to step S18.

[0065] In step S18, the ray control unit 28 detects the distance from the ray emission position to the ray irradiation point in the virtual three-dimensional space, and then determines whether the distance from the ray emission position to the ray irradiation point is equal to or greater than a threshold distance. If the distance from the ray emission position to the ray irradiation point is equal to or greater than the threshold distance, the process proceeds to step S22.

[0066] If the amount of angular change in the line of sight direction 42 within a predetermined time is greater than the threshold line of sight angle in step S14, if the amount of angular change in the ray emission angle within a predetermined time is greater than the threshold emission angle in step S16, or if the distance from the ray emission position to the ray irradiation point is less than the threshold distance in step S18, the process proceeds to step S20. In step S20, the ray control unit 28 changes the operation mode of the information processing device 10 to or maintains the normal sensitivity mode. That is, the sensitivity of the pointer 12 is changed to or maintained at the normal sensitivity.

[0067] In step S22, the ray control unit 28 changes the operation mode of the information processing device 10 to or maintains the low sensitivity mode. That is, the sensitivity of the pointer 12 is changed to or maintained at low sensitivity.

[0068] In step S24, the ray control unit 28 determines whether or not the operation of the information processing device 10 has ended. For example, if the information processing device 10 has been powered off, the determination in step S24 is "Yes," and the information processing device 10 ends the processing. If the operation of the information processing device 10 has not ended, the process returns to step S14.

[0069] In this way, while the information processing device 10 is operating, the ray control unit 28 repeats the operations of steps S14 to S24, and repeats the process of setting the pointer 12 to low sensitivity if the sensitivity reduction conditions shown in steps S14 to S18 are met, and setting the pointer 12 to normal sensitivity if the sensitivity reduction conditions are not met.

[0070] Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the present invention.

[0071] For example, each function of the processor 22 of the information processing device 10 may be implemented in a device (e.g., a server) separate from the device worn or held by the user. In this case, the device, such as the server, receives information indicating the position and attitude of the device from the acceleration sensor 16 of the device worn or held by the user and sets a virtual viewpoint 40 and a line of sight 42. A two-dimensional image generated by an image processing unit 26 of the device, such as the server, is transmitted to the device worn or held by the user, causing the image to be displayed on the display of the device. A ray control unit 28 of the device, such as the server, calculates an angular change in the user's line of sight 42 based on information indicating the attitude of the device obtained from the acceleration sensor 16 of the device worn or held by the user, calculates an angular change in the ray emission angle based on information indicating the attitude of the pointer 12 obtained from the pointer 12, and sets the sensitivity of the pointer 12 based on the angular change in the user's line of sight 42 and the angular change in the ray emission angle.

[0072] Although the information processing device 10 is a VR device in the present embodiment, the information processing device 10 may be an AR device, an MR device, or an SR device. In this case, the image processing unit 26 displays a 2D image generated by projecting a virtual 3D space onto an image of the real world captured by a camera (not shown) of the information processing device 10, and displays the 2D image on the display 14. Alternatively, the display 14 may be a transparent display. If the display 14 is a transparent display, the 2D image generated by the image processing unit 26 is displayed on the display 14, and the user is shown an image in which the real world transmitted through the display 14 and the 2D image generated by the image processing unit 26 are superimposed. Whether the information processing device 10 is an AR device, an MR device, or an SR device, the user can point to a virtual object using a ray emitted into the virtual 3D space. Even in this case, the ray control unit 28 can change the sensitivity of the pointer 12 by the above-described process. [Explanation of symbols]

[0073] 10 information processing device, 12 pointer, 14 display, 16 acceleration sensor, 18 input interface, 20 memory, 22 processor, 24 gaze direction setting unit, 26 image processing unit, 28 ray control unit, 40 virtual viewpoint, 42 gaze direction, 44 virtual screen, 50 virtual object image, 52 pointer image, 54 ray image, 56 irradiation point image, 58 information display frame.

Claims

1. a processor; The processor: setting a line of sight direction of the user in a virtual three-dimensional space based on the posture of a device worn or held by the user; generating a two-dimensional image by projecting the virtual three-dimensional space based on a virtual viewpoint and the line of sight set in the virtual three-dimensional space; Displaying the two-dimensional image on a display; setting an emission angle of a ray virtually emitted into the virtual three-dimensional space based on the orientation of the pointer held by the user; When a sensitivity reduction condition is satisfied, that is, the amount of angular change in the gaze direction of the user within a predetermined time is equal to or less than a threshold gaze angle and the amount of angular change in the emission angle of the ray within a predetermined time is equal to or less than a threshold emission angle, the amount of angular change in the emission angle of the ray relative to the amount of angular change in the attitude of the pointer is reduced compared to when the sensitivity reduction condition is not satisfied.

1. An information processing device comprising:

2. The processor: Detecting a distance from an emission position of the ray to an irradiation point of the ray in the virtual three-dimensional space; the sensitivity reduction condition further includes a condition that the distance from the emission position of the ray to the irradiation point of the ray in the virtual three-dimensional space is equal to or greater than a threshold distance.

2. The information processing apparatus according to claim 1, wherein:

3. The processor: the smaller the amount of change in the emission angle of the ray within a predetermined time, the more the amount of change in the emission angle of the ray relative to the amount of change in the angle of the attitude of the pointer is reduced; 2. The information processing apparatus according to claim 1, wherein:

4. The processor: the greater the distance from the emission position of the ray to the irradiation point of the ray in the virtual three-dimensional space, the more the angular change amount of the emission angle of the ray relative to the angular change amount of the attitude of the pointer is reduced; 3. The information processing apparatus according to claim 2, wherein:

5. The processor: generating an information display object in the virtual three-dimensional space that displays information about the virtual object pointed to by the ray; displaying a two-dimensional image including a projected image of the information display object on the display; Even if the user's line of sight direction changes by a larger amount than the threshold line of sight angle after the sensitivity reduction condition is satisfied, if it is determined that the user is looking at the information display object based on the user's line of sight direction after the change, the state in which the angular change amount of the emission angle of the ray relative to the angular change amount of the attitude of the pointer is maintained reduced.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. On the computer, setting a line of sight direction of the user in a virtual three-dimensional space based on the posture of a device worn or held by the user; projecting the virtual three-dimensional space based on a virtual viewpoint and a line-of-sight direction set in the virtual three-dimensional space to generate a two-dimensional image; Displaying the two-dimensional image on a display; setting an emission angle of a ray virtually emitted into the virtual three-dimensional space based on the orientation of the pointer held by the user; When a sensitivity reduction condition is satisfied, that is, the amount of angular change in the gaze direction of the user within a predetermined time is equal to or less than a threshold gaze angle and the amount of angular change in the emission angle of the ray within a predetermined time is equal to or less than a threshold emission angle, the amount of angular change in the emission angle of the ray relative to the amount of angular change in the attitude of the pointer is reduced compared to when the sensitivity reduction condition is not satisfied. An information processing program characterized by:

Images