Information processing device, information processing method and program

Abstract

To expand a range that can be indicated by a ray.SOLUTION: An information processing device that controls display of a ray extended from a starting point, includes: determination means that determines a position of the starting point of the ray on the basis of a position of a joint point of a hand of a user, which has been determined based on a captured image captured by an imaging device; posture determination means that determines a posture of the ray on the basis of the position of the starting point of the ray; and control means that controls display means so that the ray is displayed in a visual field of the user. The posture determination means determines the posture of the ray so that a second angle is larger than a first angle, when the first angle is formed by a straight line connecting a position of a first object as the imaging device or the eye of the user with the starting point of the ray and a flat surface including an axis indicating a center direction of the visual field of the first object, and a second angle is formed by the flat surface and a straight line corresponding to the ray.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In VR (Virtual Reality) and MR (Mixed Reality), various functions are provided that allow users to operate virtual objects and menus in a virtual space.

[0003] Rays (light rays) extending from the user's own hand or controller are used to operate virtual objects and menus in the virtual space. A common operation system is one in which objects close to the user are operated directly with the user's hand, and objects farther from the user are operated with rays.

[0004] For rays extending from the hand, methods have been proposed for determining the position and orientation of the ray that are easy for the user to operate. Patent Document 1 describes a method in which the position of the user's arm joint is identified based on the position of the display, and a ray projected from the arm joint position to a specific position on the hand is used to operate a virtual object. Patent Document 2 describes a method in which the reference position for calculating the orientation of the ray is changed based on the direction in which the user is facing or looking. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent Application Publication No. 2021 / 0383594 [Patent Document 2] U.S. Patent Application Publication No. 2021 / 0090331 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the techniques of Patent Documents 1 and 2, due to limitations imposed on the reference position for determining the position of the starting point of the ray and the orientation of the ray, there are areas within the field of view that cannot be pointed to by the ray.

[0007] Therefore, an object of the present invention is to provide a technique for expanding the range that can be pointed to by a ray. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. An information processing device that controls display of a ray extending from a starting point, a determination means for determining a position of the starting point of the ray based on a position of a joint point of the user's hand determined based on an image captured by an imaging device; an attitude determination means for determining an attitude of the ray based on the position of the starting point of the ray; a control means for controlling a display means to display the ray in the user's field of view; and the attitude determination means determines the attitude of the ray so that a second angle formed by a line connecting the position of a first object, which is the imaging device or the user's eye, to the starting point of the ray and a plane including an axis representing the center direction of the field of view of the first object, is larger than a first angle formed by the plane and a line corresponding to the ray. The information processing device is characterized by the above.

[0009] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. An information processing method for controlling display of a ray extending from a starting point, comprising: a determining step of determining a position of the start point of the ray based on a position of a joint point of the user's hand determined based on an image captured by an imaging device; an attitude determination step of determining an attitude of the ray based on the position of the starting point of the ray; a control step of controlling a display means to display the ray in the user's field of view; and In the attitude determination step, the attitude of the ray is determined so that a second angle formed by a plane and a straight line corresponding to the ray is larger than a first angle formed by a straight line connecting the position of a first object, which is the image capture device or the eye of the user, and the starting point of the ray, and a plane including an axis representing a center direction of a field of view of the first object. The information processing method is characterized by the above. [Effects of the Invention]

[0010] According to the present invention, it is possible to expand the range that can be pointed to by a ray. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a configuration diagram of a system according to a first embodiment. [Figure 2] 1 is a hardware configuration diagram of an information processing device according to a first embodiment. [Figure 3] 4A to 4C are diagrams illustrating calculation of a reference position and a ray orientation according to the first embodiment. [Figure 4] 10 is a flowchart of a process for calculating the orientation of a ray according to the first embodiment. [Figure 5] 10A and 10B are diagrams illustrating calculation of a reference position and a ray orientation according to Modification 1. [Figure 6] 10A and 10B are diagrams illustrating calculation of another reference position and ray orientation according to Modification 1. [Figure 7] 10A and 10B are diagrams illustrating calculation of the orientation of a ray according to Modification 2. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0013] <Embodiment 1> 1 is a block diagram showing an example of the configuration of a system according to embodiment 1. In the system according to embodiment 1, a display device 1000 and an information processing device 1100 are connected to each other. The connection between the display device 1000 and the information processing device 1100 may be realized either by wire or wirelessly. Furthermore, the connection between the display device 1000 and the information processing device 1100 may be realized by a combination of wire and wireless.

[0014] The display device 1000 outputs a mixed reality image in which an image of real space and an image of virtual space are combined. Note that in this embodiment, the display device 1000 does not have to be capable of displaying a mixed reality image as long as it is capable of displaying an image of virtual space. The display device 1000 may be a VR system that displays only an image of virtual space, or an AR (Augmented Reality) system that displays an image of virtual space on a real space that is visible through the real space.

[0015] The display device 1000 may be detachable from the user. For example, the display device 1000 may be a head mounted display (HMD) that can be worn on the user's head.

[0016] The display device 1000 includes an imaging unit 1010 and a display unit 1020 .

[0017] The imaging unit 1010 captures an image of the real space to obtain a real image (captured image). The imaging unit 1010 outputs the real image to the information processing device 1100.

[0018] The display unit 1020 displays an image. If the display device 1000 is an HMD, the display unit 1020 has a display arranged at a position corresponding to the user's left eye and a display arranged at a position corresponding to the user's right eye. An image corresponding to each eye (left and right eyes) is displayed on each display. This allows the display unit 1020 to display images, rays, etc. in the user's field of view. The display unit 1020 may also be a handheld display.

[0019] The information processing device 1100 includes an image acquisition unit 1110 , a position and orientation measurement unit 1120 , a joint point measurement unit 1130 , a starting point position determination unit 1140 , a reference determination unit 1150 , a ray orientation determination unit 1160 , a virtual image generation unit 1170 , and an image generation unit 1180 .

[0020] The image acquisition unit 1110 acquires an image of the real space captured by the imaging unit 1010 (real image; captured image).

[0021] The position and orientation measuring unit 1120 extracts characteristic information such as points and lines in the real image by performing image processing on the real image acquired by the image acquiring unit 1110. The position and orientation measuring unit 1120 calculates the position and orientation of the display device 1000 based on the characteristic information. The position and orientation measuring unit 1120 may measure the position and orientation of the display device 1000 using infrared light. The position and orientation measuring unit 1120 may also measure the position and orientation of the display device 1000 using an ultrasonic sensor, a magnetic sensor, or a depth sensor.

[0022] The joint point measuring unit 1130 measures (determines) the positions and orientations of the joint points of the user's hand based on the real image acquired by the image acquiring unit 1110. Here, the joint point measuring unit 1130 may measure the positions and orientations of some of the joint points of the hand as the positions and orientations of the joint points of the hand. Note that the joint point measuring unit 1130 may measure the positions and orientations of the joint points of the user's hand using a glove-type device.

[0023] The start point position determination unit 1140 calculates the start point position of the ray based on the positions and orientations of the joint points of the hand. The start point position of the ray may be, for example, the position of a specific joint point of the hand, or may be a position calculated from the positions and orientations of any two joint points.

[0024] The reference determination unit 1150 determines the position of a reference point (a reference point for determining the orientation of a ray) (hereinafter referred to as "reference position") based on the position and orientation of the display device 1000 and the starting point position of the ray. Note that the reference determination unit 1150 may determine the position of a point other than the starting point of the ray as the reference position based on the position and orientation of the joint points of the hand. The reference determination unit 1150 may also determine the reference position by referring to the position of a device worn on the hand.

[0025] The ray posture determination unit 1160 determines the posture of a ray based on the start position of the ray and the reference position. Specifically, in the first embodiment, the ray posture determination unit 1160 determines the posture of a line connecting the reference position and the start position of the ray as the posture of the ray. Note that the ray posture determination unit 1160 may calculate the position of a point different from the start point of the ray based on the positions and postures of the joint points of the hand, and determine the posture of the ray based on that position. Furthermore, the ray posture determination unit 1160 may determine the posture of the ray by referring to the position of a device worn on the hand.

[0026] A specific method for calculating the reference position and the ray orientation will be described using FIG. 3. FIG. 3 shows the change in the reference position when the hand is moved. Origin 301 indicates the position of the display device 1000 (= the position of the user's eyes). In the following, a coordinate system is considered in which arrow 302 is the Z axis and arrow 303 is the X axis. Here, the X axis is the axis (visual center) on a plane that passes through the center of the field of view of the display device 1000 (imaging unit 1010). Note that the display device Since the field of view of 1000 (imaging unit 1010) substantially coincides with the user's field of view, the "field of view of the display device 1000" can be read as the "user's field of view." The front direction of the display device 1000 is the positive direction of the X axis. The positive direction of the Z axis is the upward direction of the display device 1000. A user 304 is a person experiencing the mixed reality space. A field of view 305 represents the field of view that can be captured by the imaging unit 1010 of the display device 1000. When the start position of the ray calculated by the start position determination unit 1140 is position 306 on the X axis 303, the reference position is set to a preset initial position 307. In FIG. 3, the initial position 307 is the position of the origin of the XZ plane. However, the initial position 307 may be located anywhere as long as its position in the "three-dimensional coordinate system with the position of the display device 1000 as the origin" does not change during the experience.

[0027] At this time, the ray posture determination unit 1160 calculates a ray posture 308 based on the start position 306 of the ray and the initial position 307 of the reference point. Next, assume that the user 304 moves their hand, and the start position of the ray moves to position 309. At this time, the ray posture determination unit 1160 places the reference point at position 310 by moving from the initial position 307 by the amount of movement in the Z-axis direction from the start position 306 to position 309, in the direction opposite to that movement. In this case, the reference position moves only in the Z-axis direction, and does not move in the X-axis direction. A ray posture 311 is calculated based on position 309, which is the initial position after movement, and reference position 310 after movement.

[0028] The virtual image generation unit 1170 generates an image (hereinafter referred to as a "virtual image") representing an area of ​​the virtual space including the ray that is to be displayed on the display unit 1020, based on the position and orientation of the display device 1000, the starting position of the ray, and the orientation of the ray. For this reason, the virtual image generation unit 1170 can be said to be a display control unit that controls the display of the ray.

[0029] The image generation unit 1180 generates an image of the mixed reality space (mixed reality image; composite image) by combining the real image and the virtual image. The image generation unit 1180 outputs the mixed reality image to the display unit 1020 of the display device 1000.

[0030] The hardware configuration of the information processing device 1100 will be described with reference to Fig. 2. The hardware configuration shown in Fig. 2 realizes the processing of the functional configuration shown in Fig. 1. The information processing device 1100 has a path 200, a CPU 201, a storage device 202, a ROM 203, a RAM 204, an input interface 205, and an output interface 206.

[0031] The CPU 201 performs overall control of each component connected via the path 200. The CPU 201 reads out and executes programs stored in the ROM 203 and RAM 204.

[0032] The storage device 202 stores programs and data used by the information processing device 1100. The storage device 202 stores various programs according to the first embodiment.

[0033] The ROM 203 stores an operating system (OS), device drivers, and a boot program.

[0034] The RAM 204 temporarily stores the programs and data loaded from the storage device 202 and the ROM 203. The RAM 204 has a work area when the CPU 201 executes various processes as needed.

[0035] The input interface 205 acquires an input signal in a format that can be processed by the information processing device 1100 from an external device (such as the display device 1000).

[0036] The output interface 206 outputs data in a format that can be processed by an external device (such as the display device 1000). The output signal of the formula is output to an external device.

[0037] FIG. 4 is an example of a flowchart showing a process for calculating the orientation of a ray.

[0038] In step S400, the position and orientation measurement unit 1120 acquires the position and orientation of the display device 1000.

[0039] In step S401, the joint point measurement unit 1130 acquires the positions and orientations of the joint points of the hand based on a real image (captured image).

[0040] In step S402, the start point position determination unit 1140 calculates the start point position of the ray based on the positions and orientations of the joint points of the hand.

[0041] In step S403, the reference determination unit 1150 calculates a reference position based on the starting point position of the ray.

[0042] In step S404, the ray attitude determination unit 1160 calculates the attitude of the ray based on the starting position and reference position of the ray.

[0043] According to the first embodiment, when the starting position of the ray moves, the reference point also moves in accordance with the movement of the starting position. Therefore, in the first embodiment, the amount of change in the attitude of the ray is greater than when the reference point is fixed at a specific position, and the range that can be pointed at with the ray is expanded. Therefore, the range that can be operated with the ray on the display surface of the display unit 1020 can be expanded.

[0044] <Variation 1> In the first embodiment, the information processing device 1100 moves the reference point by the same amount as the movement amount of the start position of the ray in the Z-axis direction. In the first modification, the information processing device 1100 calculates the reference position based on the angle formed by the "straight line from the origin toward the start position of the ray" and the "X-axis" (or a plane parallel to the X-axis and Y-axis). Note that the angle formed by two straight lines can be either an acute angle or an obtuse angle, but the following description will be given assuming that it is an acute angle.

[0045] With reference to FIG. 5, a method for calculating the reference position and the orientation of a ray when using the angle formed by the "straight line from the origin toward the starting position of the ray" and the "X axis" will be described.

[0046] In FIG. 5, an origin 501 indicates the position of the display device 1000. Consider a coordinate system in which an arrow 502 represents the Z axis and an arrow 503 represents the X axis. A user 504 is a person experiencing the mixed reality space. A field of view 505 represents the field of view that can be captured by the imaging unit 1010 of the display device 1000. An arc 509 has its center at a point on the XZ plane (X coordinate of the initial position of the reference point, Z coordinate of the initial position of the reference point ±R) = (X coordinate of the initial position of the reference point = 0, R) or (X coordinate of the initial position of the reference point = 0, -R). The radius R of the arc 509 is a preset value. In FIG. 5, the initial position of the reference point is (0, 0), but the initial position may be any position. In other words, the center of the arc 509 may be expressed as (0, t0 + R) or (0, t0 - R) using the Y coordinate of the initial position of the reference point = t0.

[0047] When the starting point position of the ray is position 506, the "straight line from the origin toward the starting point position of the ray" and the "X axis" form an angle 507. The angle of angle 507 is defined as Θ (rad). Here, angle Θ is multiplied by a preset constant k (constant multiple) to obtain angle kΘ. Then, on the XZ plane, a half line 508 that forms angle kΘ with the X axis is drawn from the origin in the opposite direction to the direction in which the starting point position 506 of the ray exists. Here, constant k is greater than 1. An intersection 510 other than the origin between the ray 508 and the arc 509 is calculated, and the position of the intersection 510 is determined as a new reference position. Then, based on the starting position 506 of the ray and the intersection 510 which is the reference position, a ray orientation 511 is determined.

[0048] As a result, when the starting position of the ray moves, the reference point also moves in accordance with the movement of the starting position. Therefore, in the first embodiment, the amount of change in the attitude of the ray is greater than when the reference point is fixed at a specific position, and the range that can be pointed at with the ray is expanded. Therefore, the range that can be operated with the ray on the display surface of the display unit 1020 can be expanded.

[0049] Furthermore, by moving the reference point along the arc, the change in the ray's posture is small when the hand is moved around the center of the field of view, making it easy to manipulate the ray around the center of the field of view.

[0050] In this modification, the reference point is moved along an arc using an angle. However, for example, a quadratic function or a trigonometric function may be used to make the amount of movement of the reference point small when moving around the center of the field of view, and to make the amount of movement of the reference point larger the further away from the center of the field of view.

[0051] Furthermore, the optimal values ​​of the constant k and the radius R may be determined from the set conditions. A specific example of calculating the constant k and the radius R will be described below. Here, the vertical angle of view of the display unit 1020 of the display device 1000 is defined as "Θ0." The X coordinate of the hand position is defined as "x0," and the X coordinate of the reference position is defined as "x1." The angle formed by the "straight line from the origin to the starting position of the ray" and the "X axis" is defined as "Θ1." It is assumed that the display unit 1020 and the imaging unit 1010 of the display device 1000 have the same angle of view. In the XZ plane, a ray can be expressed by a straight line shown in Equation (1) using the inclination a. z=ax+b (1)

[0052] The line representing the upper limit of the angle of view can be expressed as in equation (2).

number

[0053] In order for a ray to point to the entire upper and lower areas of the display surface of display unit 1020, the slope a of the line representing the ray on the XZ plane needs to be greater than the slope of the line representing the upper limit of the angle of view. Therefore, the following equation (3) holds.

number

[0054] Furthermore, by calculating the reference position using the radius R, the X coordinate t0 of the initial position of the reference point, and the angle Θ1, and substituting this into (1), the following equation (4) is derived. In equation (4), both the left and right sides indicate the Z coordinate value of the reference point.

number

[0055] Then, if the X coordinate of the intersection of the line (1) representing the ray and the line (2) representing the upper limit of the angle of view is x2 (x2>x1), the following equation (5) is derived.

number

[0056] Using equations (4) and (5), the slope a is expressed by equation (6).

number

[0057] Therefore, from equation (3) and x1-x2<0, the radius R is expressed by equation (7).

number

[0058] If the value of x1 is set in advance, the optimal radius R is when the starting point of the ray is placed at the highest (or lowest) position on the display surface, and the position pointed to by the ray is the lowest (or highest) position on the display surface. In other words, the angle Θ1 in equation (7) must be at its maximum value. On the XZ plane, if the maximum height in the angle of view is z0 and the upper limit of the height of the ray starting point is z1, then the height z0 can be expressed by equation (8) using the constant u.

number

[0059] If this is rearranged so that angles Θ0 and Θ1 are used, the maximum value of angle Θ1 can be expressed as in equation (9).

number

[0060] The constant u may be set in advance or may be arbitrarily changeable depending on the size or position (three-dimensional position) of the user's hand. From the above, the optimal value of the radius R can be found. Therefore, the radius R may be determined based on the magnitude of the constant k, the size of the hand in the captured image, and the three-dimensional position of the hand.

[0061] The constant k is preferably set so that the following equation (10) is satisfied in order to move the reference point along the arc.

number

[0062] In addition to this example, after moving the reference point in the Z-axis direction along the arc, the reference point may not be moved in the X-axis direction. In other words, the X coordinate of the reference position may be set to a fixed value. Figure 6A shows a state in which a user 504 moves his / her hand parallel to the Z-axis 502 from the state shown in Figure 5, This shows the state in which the starting position of the ray has moved to position 601. At this time, the reference position is controlled to position 602, and the orientation of the ray is controlled to orientation 603.

[0063] With reference to FIG. 6B, the ray orientation 511 shown in FIG. 5 and the ray orientation 603 shown in FIG. 6A are compared on the YX plane. The origin is the position of the display device 1000, as in the XZ plane. Therefore, between the orientations 511 and 603, the difference in the Y coordinate direction of the position pointed to by the ray increases as the position pointed to by the ray becomes farther from the starting point. At this time, while the hand is moving parallel to the Z axis, the position pointed to by the ray moves in the Y direction. Therefore, this may result in movement that is different from the user's intention. However, this problem can be solved by setting the X coordinate of the reference position to a fixed value.

[0064] <Variation 2> In the first embodiment, the information processing device 1100 expands the range that can be pointed by a ray by moving the reference position. In the second modification, the information processing device 1100 expands the range that can be pointed by a ray by fixing the reference position at a specific position and changing the method of calculating the ray orientation.

[0065] A specific method for calculating the ray orientation will be described with reference to FIG. 7. Origin 701 indicates the position of the display device 1000 (user's eye). Consider a coordinate system in which arrow 702 represents the Z axis and arrow 703 represents the X axis. User 704 is a person experiencing the mixed reality space. Field of view 705 represents the field of view that can be captured by the imaging unit 1010 of the display device 1000. Position 706 indicates a fixed reference position. When the starting position of the ray is position 707, a line segment 711 from point 710, which is the foot of a perpendicular line dropped from position 707 perpendicular to the X axis (or a plane parallel to the X axis and Y axis), to position 706, and a line segment from position 707 to position 706 form an angle 708. The ray orientation determination unit 1160 assumes that the angle of angle 708 is angle Θ and calculates angle k'Θ by multiplying angle θ by a coefficient k' (k'>1). Then, the ray posture determination unit 1160 draws a ray 712 parallel to the line segment 711 from the position 707, and further draws a ray 709 from the position 707 that forms an angle k'Θ with the ray 712 on the XZ plane. The ray posture determination unit 1160 handles the posture of the ray 709 as the posture of the ray.

[0066] According to variant example 2, the amount of change in the posture of the ray is large when the position of the starting point of the ray is moved, so the range that can be pointed at with the ray can be expanded, and the range that can be operated with the ray on the display surface can be widened.

[0067] Furthermore, the coefficient k' may not be a constant, but may be a variable proportional to the absolute value of the z coordinate value of the hand position (= the distance between the plane containing the X and Y axes and the hand position). This reduces the change in the ray's posture when the hand is moved around the center of the field of view, making it easier to manipulate the ray around the center of the field of view.

[0068] In the above embodiment, a method applied to the Z-axis direction (the vertical direction of the display surface) has been described, but a similar method may also be applied to the Y-axis direction (the horizontal direction of the display surface). This allows the range that can be pointed to by the ray to be expanded to the left and right of the display surface, thereby widening the operable range. Therefore, the "Z-axis" in the above embodiment and modified examples may be read as the "Y-axis" or as the "axis indicating the direction perpendicular to the X-axis." Furthermore, the "angle formed by the X-axis and a line" may be read as the "angle formed by a line and a specific plane containing the X-axis." Note that the "semi-ray" and "line segment" in the above description may also be read as "line."

[0069] In this way, in any of the above-described embodiments and modifications, the angle formed by the line connecting the starting point of the ray and the origin (the position of the user's eye or the image capture device) and the plane including the X axis is smaller than the angle formed by the line connecting the starting point of the ray and the origin (the position of the user's eye or the image capture device) and the plane including the X axis. The angle formed by the straight line corresponding to the ray and the plane is larger than the angle formed by the ray itself. This makes it possible to point to a range that is larger than the field of view of the user or the imaging device.

[0070] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, unless a contradiction arises, "greater than or equal to A" may be read as "greater than (higher; longer; more) than A," and "less than or equal to A" may be read as "less than (lower; shorter; fewer) than A." Furthermore, "greater than (higher; longer; more) than A" may be read as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be read as "less than or equal to A."

[0071] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0072] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0073] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0074] In the above-described embodiment, the present invention has been described as being applied to an information processing device, but the present invention is not limited to this example and can be applied to any electronic device capable of controlling the display of rays. The electronic device may be a computer, a smartphone, a tablet terminal, a digital camera, or a home appliance.

[0075] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0076] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An information processing device that controls display of a ray extending from a starting point, Based on the positions of the joint points of the user's hand determined based on the captured image captured by the imaging device, determining means for determining the position of the starting point of the ray; an attitude determination means for determining an attitude of the ray based on the position of the starting point of the ray; a control means for controlling a display means to display the ray in the user's field of view; and the attitude determination means determines the attitude of the ray so that a second angle formed by a line connecting the position of a first object, which is the imaging device or the user's eye, to the starting point of the ray and a plane including an axis representing the center direction of the field of view of the first object, is larger than a first angle formed by the plane and a line corresponding to the ray. 1. An information processing device comprising: (Configuration 2) the attitude determination means determines the attitude of the ray based on a reference position and the position of the starting point of the ray. 2. The information processing device according to configuration 1, (Configuration 3) further comprising a position determination means for determining the reference position based on the position of the starting point; 3. The information processing device according to configuration 2. (Configuration 4) when the position of the starting point moves by a first amount in a first direction perpendicular to the plane, the position determining means moves the reference position by the first amount in a second direction opposite to the first direction; 4. The information processing device according to configuration 3. (Configuration 5) the position determination means determines the reference position based on the position of the first object and the position of the starting point. 4. The information processing device according to configuration 3. (Configuration 6) the position determination means determines the reference position so that an angle formed by a line connecting the position of the first object and the reference position and the plane is a constant multiple of the first angle; The constant is greater than 1. 6. The information processing device according to configuration 5. (Configuration 7) When the position determining means moves the reference position, the position determining means moves the reference position so as to move along a specific arc. 7. The information processing device according to configuration 6. (Configuration 8) the position determination means determines the radius of the specific arc based on the magnitude of the constant, the size of the hand in the captured image, and the three-dimensional position of the hand. 8. The information processing device according to configuration 7. (Configuration 9) Even when the position determination means changes the reference position, the position determination means does not move the reference position toward the center of the field of view of the first object. 6. The information processing device according to any one of configurations 3 to 5. (Configuration 10) The attitude determination means determining a third angle formed by a line connecting the reference position and the position of the starting point and a first line connecting the reference position and the foot of a perpendicular line drawn from the position of the starting point to the plane; determining, as the orientation of the ray, the orientation of a third line that forms a fourth angle between the second line and the third line, the fourth angle being obtained by multiplying the third angle by a coefficient greater than 1, when the second line is drawn from the position of the starting point and that line being parallel to the first line; 3. The information processing device according to configuration 2. (Configuration 11) the attitude determining means determines the coefficient to be proportional to the distance between the plane and the hand. 11. The information processing device according to configuration 10. (method) An information processing method for controlling display of a ray extending from a starting point, comprising: a determining step of determining a position of the start point of the ray based on a position of a joint point of the user's hand determined based on an image captured by an imaging device; an attitude determination step of determining an attitude of the ray based on the position of the starting point of the ray; a control step of controlling a display means to display the ray in the user's field of view; and In the attitude determination step, the attitude of the ray is determined so that a second angle formed by a plane and a straight line corresponding to the ray is larger than a first angle formed by a straight line connecting the position of a first object, which is the image capture device or the eye of the user, and the starting point of the ray, and a plane including an axis representing a center direction of a field of view of the first object. 1. An information processing method comprising: (program) 12. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0077] 1100: information processing device, 1010: imaging unit, 1140: starting point position determination unit, 1160: ray orientation determination unit, 1180: Image generation unit

Claims

1. An information processing device that controls display of a ray extending from a starting point, a determination means for determining a position of the starting point of the ray based on a position of a joint point of the user's hand determined based on an image captured by an imaging device; an attitude determination means for determining an attitude of the ray based on the position of the starting point of the ray; a control means for controlling a display means to display the ray in the user's field of view; and the attitude determination means determines the attitude of the ray so that a second angle formed by a line connecting the position of a first object, which is the imaging device or the eye of the user, to the starting point of the ray and a plane including an axis representing a center direction of a field of view of the first object, is larger than a first angle formed by the plane and a line corresponding to the ray.

1. An information processing device comprising:

2. the attitude determination means determines the attitude of the ray based on a reference position and the position of the starting point of the ray.

2. The information processing apparatus according to claim 1, wherein:

3. further comprising a position determination means for determining the reference position based on the position of the starting point; 3. The information processing apparatus according to claim 2, wherein:

4. when the position of the starting point moves by a first amount in a first direction perpendicular to the plane, the position determining means moves the reference position by the first amount in a second direction opposite to the first direction; 4. The information processing apparatus according to claim 3,

5. the position determination means determines the reference position based on the position of the first object and the position of the starting point.

4. The information processing apparatus according to claim 3,

6. the position determination means determines the reference position so that an angle formed by a line connecting the position of the first object and the reference position and the plane is a constant multiple of the first angle; The constant is greater than 1.

6. The information processing apparatus according to claim 5,

7. When the position determining means moves the reference position, the position determining means moves the reference position so as to move along a specific arc.

7. The information processing apparatus according to claim 6,

8. the position determination means determines the radius of the specific arc based on the magnitude of the constant, the size of the hand in the captured image, and the three-dimensional position of the hand.

8. The information processing apparatus according to claim 7,

9. the position determining means does not move the reference position toward the center of the field of view of the first object even when changing the reference position; 4. The information processing apparatus according to claim 3,

10. The attitude determination means determining a third angle formed by a straight line connecting the reference position and the position of the starting point and a first straight line connecting the foot of a perpendicular line drawn from the position of the starting point to the plane and the reference position; determining, as the orientation of the ray, the orientation of a third line that forms a fourth angle between the second line and the third line, the fourth angle being obtained by multiplying the third angle by a coefficient greater than 1, when the second line is drawn from the position of the starting point and that line being parallel to the first line; 3. The information processing apparatus according to claim 2, wherein:

11. the attitude determining means determines the coefficient to be proportional to the distance between the plane and the hand.

11. The information processing apparatus according to claim 10,

12. An information processing method for controlling display of a ray extending from a starting point, comprising: a determining step of determining a position of the start point of the ray based on a position of a joint point of the user's hand determined based on an image captured by an imaging device; an attitude determination step of determining an attitude of the ray based on the position of the starting point of the ray; a control step of controlling a display means to display the ray in the user's field of view; and In the attitude determination step, the attitude of the ray is determined so that a second angle formed by a plane and a straight line corresponding to the ray is larger than a first angle formed by a straight line connecting the position of a first object, which is the image capture device or the eye of the user, and the starting point of the ray, and a plane including an axis representing a center direction of a field of view of the first object.

1. An information processing method comprising:

13. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

Images