Information processing device, information processing system, information processing method, and information processing program

Abstract

An information processing device, an information processing system, an information processing method, and an information processing program that can improve user operability are provided. [Solution] An information processing device according to one aspect of the present disclosure includes an image acquisition unit, a coordinate calculation unit, a coordinate conversion unit, and a calculation unit. The image acquisition unit acquires an image captured by a camera, including a user's fingertip pointing at an object. The coordinate calculation unit calculates the three-dimensional coordinates of the fingertip from the camera's viewpoint from the image. The coordinate conversion unit converts the three-dimensional coordinates of the fingertip from the camera's viewpoint into three-dimensional coordinates of the fingertip from a viewpoint different from the camera's viewpoint. The calculation unit calculates the angle at which the fingertip points at the object based on the two-dimensional coordinates of the fingertip excluding the depth direction, which are identified from the three-dimensional coordinates of the fingertip from the different viewpoint.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, various devices detect the movement of a user's hand and control their operation in accordance with the detected hand movement (see, for example, Patent Document 1). For example, there are devices that change the content displayed on an operating device in accordance with the movement of the user's hand, without the user directly touching the operating device, for example, a display that is out of the user's reach or an operating device that allows the user to point to an object outside the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2024-34419 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described technologies, for example, the user may need to change their posture in order to input data to the display, or the switch operation may lack intuition, making operation difficult.

[0005] The present disclosure provides an information processing device, an information processing system, an information processing method, and an information processing program that can improve user operability. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes an image acquisition unit, a coordinate calculation unit, a coordinate conversion unit, and a calculation unit. The image acquisition unit acquires an image captured by a camera, including a user's fingertip pointing at an object. The coordinate calculation unit calculates the three-dimensional coordinates of the fingertip from the camera's viewpoint from the image. The coordinate conversion unit converts the three-dimensional coordinates of the fingertip from the camera's viewpoint into three-dimensional coordinates of the fingertip from a viewpoint different from the camera's viewpoint. The calculation unit calculates the angle at which the fingertip points at the object based on the two-dimensional coordinates of the fingertip excluding the depth direction, which are identified from the three-dimensional coordinates of the fingertip from the different viewpoint. [Effects of the Invention]

[0007] According to the information processing device of the present disclosure, it is possible to improve user operability. Note that the effects described herein are not necessarily limited to those described herein, and may be any of the effects described in this specification. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an information processing device. [Figure 2] FIG. 2 is a schematic diagram for explaining the processing of the information processing device according to the first comparative example. [Figure 3] FIG. 3 is a schematic diagram for explaining the processing of the information processing device according to the first comparative example. [Figure 4] FIG. 4 is a diagram showing a schematic configuration of an information system including a control device that is an information processing device according to the first embodiment. [Figure 5] FIG. 5 is a schematic diagram for explaining the processing of the control device according to the first embodiment. [Figure 6] FIG. 6 is a schematic diagram for explaining the processing of the control device according to the first embodiment. [Figure 7] FIG. 7 is a flowchart showing the processing procedure of the control device according to the first embodiment. [Figure 8] FIG. 8 is a schematic diagram for explaining the processing of the control device according to the first modified example. [Figure 9]FIG. 9 is a schematic diagram for explaining the processing of the information processing device according to the second modified example. [Figure 10] FIG. 10 is a schematic diagram for explaining the processing of the information processing device according to the second modification. [Figure 11] FIG. 11 is a diagram showing a schematic configuration of an information system including a control device that is an information processing device according to the second embodiment. [Figure 12] FIG. 12 is a schematic diagram for explaining the processing of the control device according to the second embodiment. [Figure 13] FIG. 13 is a flowchart showing a processing procedure of the control device according to the second embodiment. [Figure 14] FIG. 14 is a schematic diagram for explaining the processing of the control device according to the third modification. [Figure 15] FIG. 15 is a schematic diagram for explaining the processing of the control device according to the third modification. [Figure 16] FIG. 16 is a schematic diagram for explaining the processing of the information processing device according to the third comparative example. [Figure 17] FIG. 17 is a diagram showing a schematic configuration of an information system including a control device that is an information processing device according to the third embodiment. [Figure 18] FIG. 18 is a schematic diagram illustrating an example of second correction coefficient information according to the third embodiment. [Figure 19] FIG. 19 is a diagram showing a schematic configuration of an information system including a control device that is an information processing device according to the fourth embodiment. [Figure 20] FIG. 20 is a schematic diagram for explaining the processing of the information processing device according to the fourth embodiment. [Figure 21] FIG. 21 is a flowchart showing a processing procedure of the control device according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of an information processing device according to the present disclosure will be described with reference to the drawings.

[0010] (First embodiment) Before describing the information processing device according to the first embodiment, a device that operates based on the movement of a user's hand will be described.

[0011] FIG. 1 is a schematic diagram showing an example of an information processing device. FIG. 1 shows an example of an information processing device that operates based on the movement of a user's hand. The information processing device has a sensor 500 and an operation device 600. The sensor 500 is, for example, a camera. The sensor 500 captures an image of a user who will be the operator. The user has an arm. Here, the arm includes not only the upper arm and forearm but also the hand. The hand also includes a fingertip 72. The fingertip 72 is, for example, the first joint of the finger. The operation device 600 is, for example, a display device such as a display mounted on a vehicle. The information processing device controls the data displayed by the operation device 600 based on the position of the user's fingertip 72.

[0012] Furthermore, for example, when the user points at a building outside the vehicle from inside the vehicle, the information processing device controls operation device 600 to display the direction in which the target object is pointed, based on the position of user's fingertip 72. The target object pointed at by user's fingertip 72 includes objects such as operation device 600 and a building outside the vehicle.

[0013] The information processing device also calculates an angle for accepting an operation to the device. Specifically, the information processing device calculates an angle 900 indicating the angle between the vehicle's traveling direction M1 and an angle line 700 along which the user's fingertip 72 points toward the operation device 600 on the XZ axis plane.

[0014] 1 and the drawings relating to the operating device 600 described below, the mutually orthogonal X-axis, Y-axis, and Z-axis respectively refer to the left-right direction, up-down direction, and front-back direction of the operating device 600. Furthermore, in the following description, when simply referring to the X-direction, Y-direction, or Z-direction, it refers to the respective axial direction and includes the two opposite directions.

[0015] Furthermore, when the positive direction of the X axis is specified, it is a direction from left to right, when the positive direction of the Y axis is specified, it is a direction from bottom to top, and when the positive direction of the Z axis is specified, it is a direction from front to back, when the negative direction of the X axis is specified, it is a direction from right to left, when the negative direction of the Y axis is specified, it is a direction from top to bottom, and when the negative direction of the Z axis is specified, it is a direction from back to front.

[0016] 2 and 3 are schematic diagrams for explaining the processing of an information processing device according to a first comparative example. FIG. 2 shows a sensor 500 mounted on a vehicle and a user's fingertip 72. The sensor 500 is mounted on the vehicle so as to be located in the positive direction of the Y axis and the negative direction of the Z axis on a YZ-axis plane, with the user's fingertip 72 as the base point. The sensor 500 captures an image of the movement of the user's fingertip 72 in an imaging area at a field angle of a first angle AR1 and a second angle AR2.

[0017] 3 shows the result of extracting feature points of the user's hand in an image captured by sensor 500. The feature points of the user's hand are the joints of the user's five fingers and wrist. Here, it is assumed that the user's fingertip 721 and fingertip 722 are both pointing at the same azimuth angle relative to the manipulation device 600. The user's wrist 731 is positioned in the positive direction of the X axis and the negative direction of the Y axis relative to the user's wrist 732.

[0018] 3 is an angled line formed from a straight line that starts from the user's wrist 731 and connects the user's fingertip 721. Arrow 801 is an angled line formed from a straight line that starts from the user's wrist 732 and connects the user's fingertip 722. Arrow 702 is an angled line obtained by translating arrow 701 so that the user's wrist 732 is the base point.

[0019] Here, when comparing arrow 702 and arrow 801, a differential angle 901 occurs even though the user is pointing at the same azimuth angle. The reason for this differential angle 901 is that the tilt of the attitude of sensor 500 and the positional relationship between sensor 500 and the user's hand are not taken into consideration. Specifically, sensor 500, which is mounted on a vehicle so as to be positioned in the negative direction of the Z axis with user's fingertip 72 as the base point, captures an image of the user's fingertip 72, and the information processing device identifies the user's hand from the image captured by sensor 500 and calculates the angle on the XZ axis plane, resulting in differential angle 901.

[0020] As a result, the user may find it difficult to operate the operating device 600 because different directions are indicated even though the same azimuth angle is being pointed at the operating device 600. Therefore, this embodiment provides an information processing device that can improve the operability for the user when, for example, the sensor 500 is located in the positive direction of the Y axis and the negative direction of the Z axis with the user's fingertip 72 as the base point.

[0021] 4 is a diagram showing a schematic configuration of an information system including a control device which is an information processing device according to the first embodiment. The information processing device according to the first embodiment includes a sensor 500, an operation device 600, and a control device 1.

[0022] The sensor 500 is, for example, a camera device. The sensor 500 is, for example, a visible light camera. The sensor 500 captures an image of a user who is an operator and outputs the captured image to the control device 1. The sensor 500 is an example of an imaging unit. The sensor 500 continuously executes imaging processing and outputs the image to the control device 1. The sensor 500 is mounted in the interior of the vehicle and is located, for example, in front of the user as a reference and in the positive direction of the Y axis and the negative direction of the Z axis on the YZ axis plane. In other words, the sensor 500 is located diagonally above the user as a reference. The operation device 600 is a display unit that displays various data. The operation device 600 is, for example, a display device such as a display mounted in the vehicle.

[0023] Control device 1 processes the data displayed on manipulation device 600 in accordance with the movement of user's fingertip 72 detected via sensor 500.

[0024] The control device 1 includes a control unit 10 and a memory unit 30. The control unit 10 is configured as, for example, a CPU (Central Processing Unit) and controls the overall operation of each unit of the control device 1. The control device 1 according to this embodiment includes a ROM and a RAM (not shown). The ROM stores various programs. The RAM is a working area when the CPU executes the programs. The control device 1 includes, for example, a processor and a memory, and the processor executes the programs stored in the memory to realize the functions of the control unit 10 and each functional block of the control unit 10. The CPU is an example of a processor. The memory unit 30 is an example of a memory.

[0025] 4, the CPU uses the RAM as a working area and executes the program stored in the ROM to realize the image acquisition unit 11, the identification unit 12, the coordinate calculation unit 13, the coordinate conversion unit 14, and the calculation unit 15. This can also be said as the control device 1 including the image acquisition unit 11, the identification unit 12, the coordinate calculation unit 13, the coordinate conversion unit 14, and the calculation unit 15. The image acquisition unit 11, the identification unit 12, the coordinate calculation unit 13, the coordinate conversion unit 14, and the calculation unit 15 may be realized by different hardware.

[0026] The image acquisition unit 11 acquires an image of the user from the sensor 500. Furthermore, the image acquisition unit 11 acquires an image including a fingertip 72 of the user pointing at an object from the acquired image of the user.

[0027] The identification unit 12 identifies feature points corresponding to the fingertip 72 from an image including the fingertip 72 of the user pointing at an object. Specifically, the identification unit 12 identifies feature points corresponding to the fingertip 72 from an image including the fingertip 72 of the user pointing at an object, acquired by the image acquisition unit 11. Here, the feature points corresponding to the fingertip 72 are the fingertip 72 of the user's hand and each joint of the wrist. The identification unit 12 identifies feature points corresponding to the fingers of the user's hand and each joint of the wrist from an image including the fingertip 72 pointing at an object using a known method.

[0028] The coordinate calculation unit 13 calculates the three-dimensional coordinates of the fingertip 72 from the camera's viewpoint from an image including the fingertip 72 with which the user is pointing at an object. Specifically, the coordinate calculation unit 13 calculates the three-dimensional coordinates of the fingertip 72 from the camera's viewpoint based on feature points corresponding to the joints of the user's hand and wrist identified by the identification unit 12 from the image including the fingertip 72 with which the user is pointing at an object.

[0029] For example, the coordinate calculation unit 13 calculates three-dimensional coordinates of feature points corresponding to each joint of the user's hand and wrist in a coordinate system based on the coordinate positions of the X-axis and Y-axis in a pixel coordinate system of an image including the user's fingertip pointing at an object and the preset distance to the fingertip 72 in the depth direction of the sensor 500. The depth direction of the sensor 500 is, for example, a direction connecting a camera installed in a vehicle and the position of a given fingertip 72. In other words, the depth direction of the sensor 500 is the direction of the fingertip 72 located in the negative Y-axis direction and the positive Z-axis direction on the YZ-axis plane, when the camera installed in the vehicle is used as the base point.

[0030] The coordinate conversion unit 14 converts the three-dimensional coordinates of the fingertip 72 at the camera viewpoint into three-dimensional coordinates of the fingertip 72 at a viewpoint different from the camera viewpoint. Specifically, the coordinate conversion unit 14 converts the three-dimensional coordinates of the fingertip 72 at the camera viewpoint calculated by the coordinate calculation unit 13 into three-dimensional coordinates of the fingertip 72 at a viewpoint different from the camera viewpoint. Here, the contents of the coordinate conversion process performed by the coordinate conversion unit 14 will be described with reference to FIG. 5.

[0031] FIG. 5 is a schematic diagram illustrating the processing of the control device 1 according to the first embodiment. FIG. 5 illustrates a sensor 500 mounted on the vehicle shown in FIG. 2 and a viewpoint 501 different from that of the sensor 500. The viewpoint 501 is a viewpoint obtained by translating the coordinates of the sensor 500 on the Z axis and orthogonal to the ground surface, and viewing the user's fingertip 72 from directly above the vehicle. In other words, it is assumed that the viewpoint 501 is mounted on the vehicle so as to be positioned in the positive direction of the Y axis and point directly downward, with the user's fingertip 72 as the base point. The viewpoint 501 is a state in which an image of the movement of the user's fingertip 72 is captured in an imaging area at a field angle of a third angle AR11 and a fourth angle AR21.

[0032] As shown in FIG. 5, for example, the coordinate conversion unit 14 converts the three-dimensional coordinates of the fingertip 72 at the viewpoint of the sensor 500 calculated by the coordinate calculation unit 13 into three-dimensional coordinates of the fingertip 72 at a viewpoint 501 different from the viewpoint of the sensor 500 using a homogeneous transformation matrix.

[0033] Returning to Fig. 4, the explanation will be continued. The calculation unit 15 calculates the angle at which the fingertip 72 points to the target object based on the two-dimensional coordinates of the fingertip 72 excluding the depth direction, which are identified from the three-dimensional coordinates of the fingertip 72 at different viewpoints 501. Specifically, the calculation unit 15 calculates the angle at which the fingertip 72 points to the target object based on the two-dimensional coordinates of the fingertip 72 excluding the depth direction, which are identified from the three-dimensional coordinates of the fingertip 72 at different viewpoints 501 converted by the coordinate conversion unit 14. Here, the contents of the processing result processed by the calculation unit 15 will be explained using Fig. 6.

[0034] FIG. 6 is a schematic diagram for explaining the processing of the control device according to the first embodiment. FIG. 6 shows the contents of the processing result processed by the calculation unit 15. Here, it is assumed that the user's fingertip 723 and fingertip 724 are both pointing at the same azimuth angle with respect to the manipulation device 600. The user's wrist 733 is located in the positive direction of the X axis and the negative direction of the Y axis with respect to the user's wrist 734. It is assumed that the user's fingertip 723 is located at the same position as the user's fingertip 721 shown in FIG. 3 described above. Similarly, it is assumed that the user's fingertip 724 is located at the same position as the user's fingertip 722 shown in FIG. 3 described above.

[0035] 6 is an angle line formed from a straight line connecting the user's wrist 733 as a base point and the fingertip 723 of the user's left hand. Arrow 803 is an angle line formed from a straight line connecting the user's wrist 734 as a base point and the fingertip 724 of the user's left hand. Arrow 704 is an angle line obtained by translating arrow 703 so that the user's wrist 734 is the base point.

[0036] Here, when comparing arrow 704 and arrow 803, a differential angle 902 occurs even though the user is pointing at the same azimuth angle. However, when comparing differential angle 901 shown in Fig. 3 with differential angle 902 shown in Fig. 6, the magnitude of the difference between differential angle 902 and differential angle 901 shown in Fig. 3 is smaller, and the difference is reduced.

[0037] The reason why the magnitude of the difference is reduced is because the tilt of the attitude of the sensor 500 and the positional relationship between the sensor 500 and the user's hand are not taken into consideration. Specifically, this is because the calculation unit 15 calculates the angle at which the fingertip 72 points to the target object based on the two-dimensional coordinates of the fingertip 72 excluding the depth direction, based on the coordinates obtained by converting the three-dimensional coordinates of the fingertip 72 at the viewpoint of the sensor 500 by the coordinate conversion unit 14 into three-dimensional coordinates of the fingertip 72 at a viewpoint when the user's fingertip 72 is viewed from directly above the vehicle, which is perpendicular to the ground plane, unlike the sensor 500.

[0038] As a result, for example, even if the operator is pointing at the same azimuth angle to the operating device 600 but the position of the wrist is different, the angle calculated by the control device 1 can be made to point at the same azimuth angle to the operating device 600, thereby improving operability for the user.

[0039] Returning to FIG. 4 , the explanation will be continued. The memory unit 30 stores various types of information. The memory unit 30 is realized by hardware for storing information (in other words, data), such as a memory or storage. Specifically, the memory unit 30 stores first coordinate information 31, first image information 32, second image information 33, feature point information 34, second coordinate information 35, third coordinate information 36, and angle information 37.

[0040] The first coordinate information 31 includes three-dimensional coordinates of the installation position of the sensor 500, information on the mounting angle of the sensor 500, and three-dimensional coordinates of the position of the operation device 600. The first image information 32 is information including an image of the user captured by the sensor 500. The second image information 33 is information including an image acquired by the control device 1 that includes a fingertip 72 of the user pointing at an object. The first image information 32 and the second image information 33 include the date and time when the sensor 500 captured the image.

[0041] The feature point information 34 is information including the feature points corresponding to the fingertip 72 identified by the control device 1. The second coordinate information 35 is information including the three-dimensional coordinates of the fingertip 72 at the camera viewpoint calculated by the control device 1. The third coordinate information 36 is information including the three-dimensional coordinates of the fingertip 72 at the camera viewpoint converted by the control device 1 into the three-dimensional coordinates of the fingertip 72 at a viewpoint 501 different from the camera viewpoint. The angle information 37 is information including the angle at which the fingertip 72 points to an object calculated by the control device 1.

[0042] Here, the coordinate system for the first coordinate information 31 is absolute coordinates. The coordinate systems for the feature point information 34 and the second coordinate information 35 are pixel coordinates of the sensor 500. The third coordinate information 36 and angle information 37 are relative coordinates based on the sensor 500 viewing the fingertip 72 at a different viewpoint 501 from directly above the vehicle.

[0043] Next, the processing procedure of the control device 1 according to the first embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the processing procedure of the control device 1 according to the first embodiment. Note that this processing describes the flow of processing for calculating the angle at which the user's fingertip 72 points to an object.

[0044] First, the image acquiring unit 11 acquires an image of the user from the sensor 500 (step S71). Then, the image acquiring unit 11 acquires an image including the fingertip 72 of the user pointing at an object from the acquired image of the user (step S72). Next, the identifying unit 12 identifies a feature point corresponding to the fingertip 72 from the image including the fingertip 72 of the user pointing at an object acquired by the image acquiring unit 11 (step S73).

[0045] Next, the coordinate calculation unit 13 calculates the three-dimensional coordinates of the fingertip 72 at the camera viewpoint based on the feature points corresponding to the joints of the user's hand and wrist identified by the identification unit 12 from the image including the fingertip 72 of the user pointing at an object (step S74). The coordinate conversion unit 14 converts the three-dimensional coordinates of the fingertip 72 at the camera viewpoint calculated by the coordinate calculation unit 13 into three-dimensional coordinates of the fingertip 72 at a viewpoint 501 different from the camera viewpoint (step S75).

[0046] Next, the calculation unit 15 calculates the angle at which the fingertip 72 points to the object based on the two-dimensional coordinates of the fingertip 72 excluding the depth direction, which are identified from the three-dimensional coordinates of the fingertip 72 at different viewpoints 501 converted by the coordinate conversion unit 14 (step S76). When the process of step S76 ends, this process executed by the control device 1 ends.

[0047] As described above, the control device 1 according to an embodiment of the present disclosure acquires an image captured by a camera, including a user's fingertip pointing at an object. The control device 1 also calculates the three-dimensional coordinates of the fingertip 72 from the camera's viewpoint from the image. The control device 1 then converts the three-dimensional coordinates of the fingertip from the camera's viewpoint into three-dimensional coordinates of the fingertip 72 from a viewpoint 501 different from the camera's viewpoint. The control device 1 then calculates the angle at which the fingertip 72 points at the object, based on the two-dimensional coordinates of the fingertip 72 excluding the depth direction, which are determined from the three-dimensional coordinates of the fingertip 72 from the viewpoint 501 different from the camera's viewpoint.

[0048] As a result, even when the camera is positioned diagonally above the user, the control device 1 can operate the display without losing the user's posture by calculating the angle at which the fingertip 72 points to an object based on the two-dimensional coordinates of the fingertip 72 excluding the depth direction, which are identified from the three-dimensional coordinates of the fingertip 72 at a viewpoint 501 different from the camera viewpoint. Therefore, the control device 1 can improve operability for the user.

[0049] The above-described embodiment can be modified as needed by partially changing the configuration or functions of each of the above-described devices. Therefore, several modifications of the above-described embodiment will be described below as other embodiments. The following mainly focuses on differences from the above-described embodiment, and a detailed description of commonalities with the content already described will be omitted.

[0050] (First Modification) In the above-described embodiment, a configuration in which there is one manipulation device 600 has been described. In the first modified example, a configuration in which the coordinate calculation unit 13 of the control unit 10 of the control device 1 calculates three-dimensional coordinates taking into consideration the focal length of the sensor 500 will be described. For example, the coordinate calculation unit 13 of the first modified example calculates the three-dimensional coordinates of the fingertip 72 in the world coordinate system of the origin of the sensor 500 using a parameter of the focal length of the sensor 500 and the distance from the sensor 500 to the hand in the depth direction, based on feature points corresponding to the joints and wrist of the user's hand identified by the identification unit 12 from an image including the fingertip 72 with which the user is pointing at an object.

[0051] FIG. 8 is a schematic diagram for explaining the processing of the control device according to the first modification. The schematic diagram shown in FIG. 8 is a diagram showing an image viewed from the side. FIG. 8 shows a sensor 500, a user's fingertip 72, and a fingertip 725 included in an image of the user's fingertip 72 captured by the sensor 500. In the example of FIG. 8, the distance from the sensor 500 to the fingertip 72 is Zc [mm], and the distance from the fingertip 72 to the user's wrist is Yc [mm]. Note that Zc is a fixed value and can be set arbitrarily. Furthermore, the focal length from the focal point of the sensor 500 to the image is fy [px], and the distance from the fingertip 725 on the image to the user's wrist is u [px]. Here, the focal length fy [px] is also referred to as the focal length parameter of the sensor 500.

[0052] This makes it possible to find two similar triangles, and from the similarity relationship of u:fy=Yc:Zc, it is possible to find u=fy×Yc / Zc. By calculating the distance u to the user's wrist, and assuming that the value of the optical center of the sensor 500 is cy [px], the Y-axis coordinate y of the fingertip 72 can be calculated as y=u+cy=fy×Yc / Zc+cy.

[0053] Similarly, the X-axis coordinate x of the fingertip 72 can be calculated as x=v+cy=fx×Yc / Zc+cx, where v [px] is the distance from the fingertip 725 to the user's wrist on the image, and cx [px] is the value of the optical center of the sensor 500. Note that since the user is seated in a seat in the vehicle, the user's position is almost fixed. Therefore, the control device 1 can calculate absolute coordinates with the sensor 500 as the origin by setting Zc, the distance from the sensor 500 to the hand, as a fixed value.

[0054] (Second Modification) In the above-described embodiment, the sensor 500 is located in front of the user. However, the sensor 500 mounted on a vehicle may be attached to a location other than the front of the user, for example, to the back or side of the user. Therefore, in the second modified example, a location in which the sensor 500 is located other than the front of the user will be described.

[0055] 9 and 10 are schematic diagrams for explaining the processing of an information processing device according to a second modification. In Fig. 9, for example, sensor 500 is positioned in front of user's fingertip 72, and image 111 captured by sensor 500 is shown. In Fig. 10, for example, sensor 500 is positioned behind user's fingertip 72, and image 111 captured by sensor 500 is shown. Furthermore, it is assumed that user's fingertip 72 shown in Fig. 9 and 10 points at the same azimuth angle with respect to manipulation device 600.

[0056] The direction of the user's fingertip 72 shown in Fig. 9 is indicated by an arrow 705. The arrow Z1 shown in Fig. 9 is the viewpoint in the depth direction of the sensor 500. The direction of the user's fingertip 72 shown in Fig. 10 is indicated by an arrow 706. The arrow Z2 shown in Fig. 10 is the viewpoint in the depth direction of the sensor 500.

[0057] Here, angle 903 formed by arrow 705 and arrow Z1 shown in Fig. 9 and angle 904 formed by arrow 706 and arrow Z2 shown in Fig. 10 are different even though user's fingertip 72 is pointing at the same azimuth angle relative to manipulation device 600. This is because the mounting positions of sensor 500 on the vehicle are different between image 111 in Fig. 9 and image 112 in Fig. 10.

[0058] However, the control device 1 calculates the three-dimensional coordinates of the fingertip 72 at the camera viewpoint from the image, and converts the three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip 72 at a viewpoint 501 different from the camera viewpoint. Furthermore, the control device 1 calculates the angle at which the fingertip 72 points to an object based on the two-dimensional coordinates of the fingertip 72 excluding the depth direction, which are identified from the three-dimensional coordinates of the fingertip 72 at the different viewpoint 501, thereby allowing the user to operate the display without losing their posture. Therefore, the control device 1 can improve operability for the user.

[0059] (Second embodiment) Next, in the second embodiment, a process for calculating an angle in response to changes accompanying vertical movement of a user's fingertip 72 will be described. FIG. 11 is a diagram showing a schematic configuration of an information system including a control device 1, which is an information processing device according to the second embodiment. The information processing device of the second embodiment further implements a correction unit 16 in a control unit 10 of the control device 1, and first correction coefficient information 38 and first angle correction information 39 in a storage unit 30 of the control device 1, in addition to the information processing device of the first embodiment.

[0060] 12 is a schematic diagram for explaining the processing of the control device 1 according to the second embodiment. In the graph G1 shown in FIG. 12, the vertical axis represents angle d [deg] and the horizontal axis represents t [sec]. Here, it is assumed that the user's fingertip 72 points at the same azimuth angle relative to the manipulation device 600.

[0061] Graph G11 in Fig. 12 shows the angle in the left-right movement of the user's fingertip 72. Looking at graph G11, the angle value of graph G11 fluctuates up and down even though the same azimuth angle is indicated with respect to the manipulation device 600. This is because an error occurs due to the change accompanying the up-and-down movement of the user's fingertip 72.

[0062] Therefore, the correction unit 16 corrects the angle to a first correction angle in accordance with the movement of the fingertip 72 taking into account the depth direction of the sensor 500. Specifically, the correction unit 16 corrects the angle calculated by the calculation unit 15 to a first correction angle in accordance with the movement of the fingertip 72 taking into account the depth direction of the sensor 500.

[0063] For example, the correction unit 16 calculates the vertical movement corresponding to the two-dimensional coordinates in the left-right direction of the fingertip 72 excluding the depth direction from the three-dimensional coordinates of the fingertip 72 at different viewpoints 501, and calculates the vertical angles corresponding to the horizontal angles calculated by the calculation unit 15. Here, the calculated vertical angles, for example, in graph G12 of Fig. 12, are angles in the movement of the user's fingertip 72 in the vertical direction or the upward direction, which is positive.

[0064] Furthermore, the correction unit 16 generates first correction coefficient information indicating the coefficient of the correction value corresponding to the vertical angle, and stores it in the storage unit 30. The coefficient of the correction value corresponding to the vertical angle is, for example, a value that keeps the value of the left and right angle constant. Here, the first correction coefficient information is a table or the like that associates the left and right angles, the up and down angles, and the correction values ​​calculated by the calculation unit 15.

[0065] Then, the correction unit 16 corrects the angle calculated by the calculation unit 15 to a first corrected angle based on the first correction coefficient information. Graph G13 in FIG. 12 is the angle corrected from graph G11. Here, the first corrected angle is corrected by the correction unit 16 from graph G11, for example, as shown in graph G13 in FIG. 12. This allows the control device 1 to suppress errors caused by changes accompanying the up and down movement of the user's fingertip 72.

[0066] 11, the explanation will be continued. The first correction coefficient information 38 is information including a coefficient of a correction value corresponding to the vertical angle generated by the control device 1. The first angle correction information 39 is information including a first correction angle corrected by the control device 1.

[0067] Next, the processing procedure of the control device 1 according to the second embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the processing procedure of the control device 1 according to the second embodiment. Note that the processing from step S71 to step S76 in the flowchart shown in Fig. 13 is the same as the processing from step S71 to step S76 in the flowchart shown in Fig. 7, and therefore description thereof will be omitted.

[0068] In step S77, the correction unit 16 corrects the angle calculated by the calculation unit 15 (step S77). For example, in step S77, the correction unit 16 corrects the angle calculated by the calculation unit 15 to a first corrected angle in accordance with the movement of the fingertip 72 taking into account the depth direction of the sensor 500. When the process of step S77 ends, this process executed by the control device 1 ends.

[0069] As described above, the control device 1 according to an embodiment of the present disclosure corrects the angle to the first correction angle in accordance with the movement of the fingertip 72 taking into account the depth direction of the sensor 500. This allows the control device 1 to suppress errors caused by changes accompanying, for example, the up and down movement of the user's fingertip 72. Therefore, the control device 1 can improve the operability for the user.

[0070] (Third Modification) Next, the details of the process performed by the control device 1 to stabilize the deviation of the angle calculated by the user's fingertip 72 will be described. FIGS. 14 and 15 are schematic diagrams for explaining the process performed by the control device 1 according to the third modification. FIGS. 14 and 15 respectively show states in which the user's fingertip 72 points at the manipulation device 600. For example, as in state D1 shown in FIG. 14, when the direction in which the user's fingertip 72 points at an object is defined as an angle line 700 connecting the fingertip 72 and the second joint 74 of the finger, the deviation of the angle calculated by the control device 1 may become large if the deviation of the user's movement becomes large or if the deviation of the position identified by the identification unit 12 becomes large. In other words, if the distance between the fingertip 72 and the second joint 74 of the finger is short, the deviation of the angle calculated by the control device 1 may become large.

[0071] Therefore, the control device 1 arranges the start point and end point of the angle line 700 so that they are separated from each other, as in state D2 and state D3 shown in Fig. 15. For example, as in state D2 shown in Fig. 15, the start point of the angle line 700 is arranged at the wrist 741, and the end point of the angle line 700 is arranged at the fingertip 72. Also, for example, as in state D3 shown in Fig. 15, the start point of the angle line 700 is arranged at the elbow 742, and the end point of the angle line 700 is arranged at the fingertip 72.

[0072] This allows the control device 1 to suppress fluctuations in the calculated angle. The angle line 700 may be positioned so that the starting point is at the user's eye line and the end point is at the fingertip 72. The eye position may be set, for example, as a fixed value of three-dimensional coordinates at a viewpoint 501 different from the viewpoint of the sensor 500.

[0073] (Third embodiment) Next, in the third embodiment, a process of the control device 1 corresponding to the positional relationship between the sensor 500 and the user will be described. Here, the positional relationship between the sensor 500 and the user calculated by the control device 1 and the angle corresponding to the position of the sensor 500 and the fingertip 72 will be described with reference to FIG.

[0074] Fig. 16 is a schematic diagram for explaining the processing of an information processing device according to Comparative Example 3. In the graph schematic diagram G2 shown in Fig. 16, the vertical axis indicates the calculated angle [deg] calculated by the control device 1, and the horizontal axis indicates the instruction direction [deg].

[0075] Graph G21 in Fig. 16 represents ideal values ​​when the user's fingertip 72 points in a predetermined direction. Graph G21 is also called a true value. Graphs G22 to G25 in Fig. 16 represent the angle calculated by the control device 1 when the user's fingertip 72 points in a predetermined direction in a position and orientation different from that of graph G21.

[0076] Comparing graph G21 with graphs G22 to G25, the control device 1 calculates different angles. This result is due to the positional relationship between the sensor 500 and the user's fingertip 72, for example, whether the user is pointing with the right hand or the left hand. For example, even if the user is pointing in a direction of 0 [deg], the position and posture of the hand will be different, and therefore angle 900, which indicates the angle between the vehicle's traveling direction M1 and angle line 700 along which the user's fingertip 72 points to the object in the X-axis and Y-axis directions, will change.

[0077] In addition to the movement of the user's hand, the angle changes due to differences in the user's position (for example, the user's seat position in a vehicle) and differences in effectiveness depending on the user. As a result, the true value and the calculated angle may differ, as shown by the difference W1 in FIG. 16. Therefore, the control device 1 according to the third embodiment will be described below with regard to processing corresponding to the positional relationship between the sensor 500 and the user.

[0078] 17 is a diagram showing a schematic configuration of an information system including a control device 1 which is an information processing device according to the third embodiment. The information processing device of the third embodiment further realizes second correction coefficient information 40 and second angle correction information 41 in the storage unit 30 of the control device 1 compared to the information processing device of the second embodiment.

[0079] The correction unit 16 corrects the angle to a second correction angle according to the positions of the sensor 500 and the user and the positions of the sensor 500 and the fingertip 72. Specifically, the correction unit 16 corrects the angle calculated by the calculation unit 15 to the second correction angle according to the positions of the sensor 500 and the user and the positions of the sensor 500 and the fingertip 72. For example, the correction unit 16 refers to second correction coefficient information and corrects the angle calculated by the calculation unit 15 to the second correction angle. Here, the second correction coefficient information will be described with reference to FIG. 18 .

[0080] Fig. 18 is a schematic diagram showing an example of second correction coefficient information according to the third embodiment. Table T1 in the schematic diagram shown in Fig. 18 is a table that associates calculated values ​​calculated from angles starting from the user's wrist and ending at the user's fingertip 72 with true values ​​of the directions in which the user's fingertip 72 actually points toward an object. The correspondence between the calculated values ​​and true values ​​shown in Table T1 can be derived from, for example, the relationship between calculated value = approximate expression (true value) and true value = inverse function (calculated value).

[0081] 18, the correction unit 16 corresponds the angle calculated by the calculation unit 15 to the calculated value shown in Table T1, and corrects the corresponding calculated value shown in Table T1 to the true value shown in Table T1, thereby correcting the angle calculated by the calculation unit 15 to the second corrected angle. This allows the control device 1 to suppress errors caused by changes in the positional relationship between the sensor 500 and the user.

[0082] 17, the explanation will be continued. The second correction coefficient information 40 is information that associates a calculated value calculated from the angle starting from the user's wrist and ending at the user's fingertip 72, as shown in the above-mentioned table T1, with a true value of the direction in which the user's fingertip 72 actually points toward the target object.

[0083] As described above, the control device 1 according to an embodiment of the present disclosure corrects the angle to the second correction angle according to the positions of the sensor 500 and the user and the positions of the sensor 500 and the fingertip 72. This allows the control device 1 to suppress errors caused by changes in the positional relationship between the sensor 500 and the user. Therefore, the control device 1 can improve operability for the user.

[0084] (Fourth Modification) In the third embodiment described above, the control device 1 refers to the second correction coefficient information and corrects the calculated angle to the second correction angle. In the fourth modification, the correction unit 16 refers to the user's position information and corrects the calculated angle to the second correction angle. Here, the user's position information is information based on the sensor 500 and the user's position.

[0085] For example, the correction unit 16 refers to the user's position information and corrects the angle calculated by the calculation unit 15 to a second correction angle aligned with the center position of the user. Here, "aligned with the center position of the user" means, for example, moving the coordinates of the sensor 500 on the X axis to the coordinates of the user on the X axis. In other words, the viewpoint of the sensor 500 is the viewpoint when the user is viewed from directly above the vehicle.

[0086] For example, the correction unit 16 identifies the center position of the user from the image acquired by the image acquisition unit 11. Note that the correction unit 16 may also identify the positions of the user's right hand and left hand from the image acquired by the image acquisition unit 11. Note that the correction unit 16 may also identify the center position of the user from a TOF (Time of Flight) sensor mounted on the vehicle, for example.

[0087] The correction unit 16 may not only align with the center position of the user, but may also perform calibration by changing the correction value for each user, or may change the correction value for each user's right hand or left hand. The correction value is not limited to a predetermined fixed value, but may be variable using, for example, hand position information detected by AI (Artificial Intelligence) or a TOF sensor. This allows the control device 1 to correct the calculated angle based on the user's position, thereby suppressing errors due to changes in the positional relationship between the sensor 500 and the user. Therefore, the control device 1 can improve user operability.

[0088] (Fifth Modification) For example, depending on the direction of the sensor 500 and the direction of the hand, an occlusion may occur, indicating that the user's finger is hidden. In this case, the control device 1 may not be able to calculate the angle at which the user's fingertip 72 points to the target object, or the calculated angle may differ significantly.

[0089] Therefore, when the identification unit 12 of the control unit 10 of the control device 1 according to the fifth modified example cannot identify a feature point corresponding to the fingertip 72 from an image including the fingertip 72 of the user pointing at an object, the image acquisition unit 11 may identify feature points of the user's body from the acquired image of the user, and the coordinate calculation unit 13 may calculate the three-dimensional coordinates of the fingertip 72 from the camera viewpoint based on the feature points corresponding to the respective joints of the user's hand and wrist that correspond to the feature points of the user's body identified by the identification unit 12.

[0090] (Fourth embodiment) For example, a user pointing at an object outside the vehicle may want to know whether the position pointed by the user matches the angle information calculated by the control device 1. Therefore, in the fourth embodiment, a form in which the angle calculated by the control device 1 is output will be described.

[0091] FIG. 19 is a diagram showing a schematic configuration of an information system including a control device 1 that is an information processing device according to a fourth embodiment. The information processing device according to the fourth embodiment further realizes an output unit 17 of the control unit 10 of the control device 1 in addition to the components of the information processing device according to the first embodiment. The output unit 17 outputs angle-related information indicating information related to angles. Specifically, the output unit 17 outputs angle-related information indicating information related to the angle at which the fingertip 72 points to an object, calculated by the calculation unit 15.

[0092] Fig. 20 is a schematic diagram for explaining the processing of the information processing device according to the fourth embodiment. Fig. 20 shows a user UR, a fingertip 72 of the user UR, and an operation device 600. Fig. 20 also shows angle-related information 601 in which the output unit 17 of the control unit 10 of the control device 1 outputs the angle calculated by the calculation unit 15 to the operation device 600.

[0093] When the operation device 600 is a navigation device, the angle-related information 601 may be in the form of, for example, displaying a target pointed by a user as a marker on a map displayed on the navigation device. When the operation device 600 is a display mounted on the windshield of a vehicle, the angle-related information 601 may be in the form of, for example, displaying a target pointed by a user as a marker on the target using AR (Augmented Reality) technology or the like on the display.

[0094] Furthermore, when the operation device 600 is a light-adjusting panel mounted on the windshield of the vehicle, the angle-related information 601 may, for example, dim the area where the user is pointing. In the case of a light-adjusting panel mounted on the windshield of the vehicle, the area where the user is not pointing becomes transparent, thereby improving the user's visibility. Furthermore, the angle-related information 601 may, for example, be in the form of using an LED (Light Emitting Diode) mounted on the vehicle to illuminate a light-emitting element in the direction where the user is pointing. This allows the operator to know which position on the operation device 600 the operator is pointing at when actually pointing at the operation device 600.

[0095] Next, the processing procedure of the control device 1 according to the fourth embodiment will be described with reference to Fig. 21. Note that the processing from step S71 to step S76 in the flowchart shown in Fig. 21 is the same as the processing from step S71 to step S76 in the flowchart shown in Fig. 7, and therefore description thereof will be omitted.

[0096] In step S78, the output unit 17 outputs angle-related information indicating information related to the angle at which the fingertip 72 points to the target object, calculated by the calculation unit 15. When the process of step S78 ends, this process executed by the control device 1 ends.

[0097] As described above, the control device 1 according to one embodiment of the present disclosure outputs angle-related information indicating information related to the angle at which the fingertip 72 points at an object. This allows the user to understand whether the angle information calculated by the control device 1 when pointing at an object outside the vehicle is correct, for example. Therefore, the control device 1 can improve the operability for the user.

[0098] (Sixth Modification) The output unit 17 of the control unit 10 of the control device 1 of the fourth embodiment described above may output the first angle corrected by the correction unit 16 of the second embodiment and the second angle corrected by the correction unit 16 of the third embodiment to the operating device 600. This allows the user to, for example, determine whether the angle information calculated by the control device 1 when pointing at an object outside the vehicle is correct.

[0099] Although the embodiments of the present disclosure have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0100] Furthermore, the notation "... section" in the above-described embodiments may be replaced with other notations such as "... circuitry," "... assembly," "... device," "... unit," or "... module."

[0101] In each of the above embodiments, the present disclosure has been described as an example configured using hardware, but the present disclosure can also be realized by software in cooperation with hardware.

[0102] Furthermore, each functional block used in the description of each of the above embodiments is typically realized as an LSI, which is an integrated circuit. The integrated circuit controls each functional block used in the description of the above embodiments and may have input and output terminals. These may be individually integrated into single chips, or some or all of them may be integrated into a single chip. While the term LSI is used here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

[0103] Furthermore, the method of integration is not limited to LSI, but may be realized using dedicated circuits or general-purpose processors and memories. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, and reconfigurable processors, which allow the connections and settings of circuit cells within LSIs to be reconfigured, may also be used.

[0104] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology could be used to integrate functional blocks. The application of biotechnology, etc. is also a possibility.

[0105] Furthermore, the effects of the embodiments described in this specification are merely examples and are not limiting, and other effects may also be provided.

[0106] (Addendum) The above description of the embodiments discloses the following techniques. (Technology 1) an image acquisition unit that acquires an image captured by a camera, the image including the user's fingertip pointing at an object; a coordinate calculation unit that calculates three-dimensional coordinates of the fingertip from the camera viewpoint based on the image; a coordinate conversion unit that converts the three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip at a viewpoint different from the camera viewpoint; a calculation unit that calculates an angle at which the fingertip points to the object based on two-dimensional coordinates of the fingertip excluding a depth direction identified from the three-dimensional coordinates of the fingertip at the different viewpoints; An information processing device comprising: (Technology 2) a correction unit that corrects the angle to a first correction angle in accordance with the movement of the fingertip taking the depth direction into consideration; The information processing device according to the above technique 1. (Technology 3) a second correction unit that corrects the angle to a second correction angle according to positions of the camera and the user and positions of the camera and the fingertip; The information processing device according to the above technique 1. (Technology 4) an output unit that outputs angle-related information indicating information related to the angle; The information processing device according to any one of the above techniques 1 to 3. (Technology 5) An information processing system including a camera and an information processing device, The camera captures an image of a user who is an operator, The information processing device includes: an image acquisition unit that acquires an image captured by the camera, the image including the fingertip of the user pointing at an object; a coordinate calculation unit that calculates three-dimensional coordinates of the fingertip from the camera viewpoint based on the image; a coordinate conversion unit that converts the three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip at a viewpoint different from the camera viewpoint; a calculation unit that calculates an angle at which the fingertip points to the object based on two-dimensional coordinates of the fingertip excluding a depth direction identified from the three-dimensional coordinates of the fingertip at the different viewpoints; An information processing system comprising: (Technology 6) An information processing method executed by an information processing device, an image acquisition step of acquiring an image including a fingertip of a user pointing at an object, the image being captured by a camera; a coordinate calculation step of calculating three-dimensional coordinates of the fingertip from the camera viewpoint based on the image; a coordinate transformation step of transforming the three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip at a viewpoint different from the camera viewpoint; a calculation step of calculating an angle at which the fingertip points to the object based on two-dimensional coordinates of the fingertip excluding a depth direction identified from the three-dimensional coordinates of the fingertip at the different viewpoints; An information processing method including: (Technology 7) On the computer, an image acquisition step of acquiring an image including a fingertip of a user pointing at an object, the image being captured by a camera; a coordinate calculation step of calculating three-dimensional coordinates of the fingertip from the camera viewpoint based on the image; a coordinate transformation step of transforming the three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip at a viewpoint different from the camera viewpoint; a calculation step of calculating an angle at which the fingertip points to the object based on two-dimensional coordinates of the fingertip excluding a depth direction identified from the three-dimensional coordinates of the fingertip at the different viewpoints; An information processing program that executes the above. [Explanation of symbols]

[0107] 1. Control device 10 Control Unit 11 Image acquisition unit 12 Specific part 13 Coordinate calculation section 14 Coordinate conversion section 15 Calculation section 16 Correction unit 17 Output section 30 Storage section 500 sensors 600 Operating equipment

Claims

1. an image acquisition unit that acquires an image captured by a camera, the image including the user's fingertip pointing at an object; a coordinate calculation unit that calculates three-dimensional coordinates of the fingertip from the camera viewpoint based on the image; a coordinate conversion unit that converts the three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip at a viewpoint different from the camera viewpoint; a calculation unit that calculates an angle at which the fingertip points to the object based on two-dimensional coordinates of the fingertip excluding a depth direction identified from the three-dimensional coordinates of the fingertip at the different viewpoints; An information processing device comprising:

2. a correction unit that corrects the angle to a first correction angle in accordance with the movement of the fingertip taking the depth direction into consideration; The information processing device according to claim 1 .

3. a second correction unit that corrects the angle to a second correction angle according to positions of the camera and the user and positions of the camera and the fingertip; The information processing device according to claim 1 .

4. an output unit that outputs angle-related information indicating information related to the angle; The information processing device according to claim 1 .

5. An information processing system including a camera and an information processing device, The camera captures an image of a user who is an operator, The information processing device includes: an image acquisition unit that acquires an image captured by the camera, the image including the fingertip of the user pointing at an object; a coordinate calculation unit that calculates three-dimensional coordinates of the fingertip from the camera viewpoint based on the image; a coordinate conversion unit that converts the three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip at a viewpoint different from the camera viewpoint; a calculation unit that calculates an angle at which the fingertip points to the object based on two-dimensional coordinates of the fingertip excluding a depth direction identified from the three-dimensional coordinates of the fingertip at the different viewpoints; An information processing system comprising:

6. An information processing method executed by an information processing device, an image acquisition step of acquiring an image including a fingertip of a user pointing at an object, the image being captured by a camera; a coordinate calculation step of calculating three-dimensional coordinates of the fingertip from the camera viewpoint based on the image; a coordinate transformation step of transforming three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip at a viewpoint different from the camera viewpoint; a calculation step of calculating an angle at which the fingertip points to the object based on two-dimensional coordinates of the fingertip excluding a depth direction identified from the three-dimensional coordinates of the fingertip at the different viewpoints; An information processing method including:

7. On the computer, an image acquisition step of acquiring an image including a fingertip of a user pointing at an object, the image being captured by a camera; a coordinate calculation step of calculating three-dimensional coordinates of the fingertip from the camera viewpoint based on the image; a coordinate transformation step of transforming three-dimensional coordinates of the fingertip at the camera viewpoint into three-dimensional coordinates of the fingertip at a viewpoint different from the camera viewpoint; a calculation step of calculating an angle at which the fingertip points to the object based on two-dimensional coordinates of the fingertip excluding a depth direction identified from the three-dimensional coordinates of the fingertip at the different viewpoints; An information processing program that executes the above.

Images