Information processing apparatus, information processing method, and program

Abstract

It is desired to provide a technique that enables an instruction by a motion corresponding to a reference position to be more clearly conveyed.SOLUTION: There is provided an information processing apparatus including a detection unit configured to detect first skeletal information of a part of a body of a user on the basis of a first image obtained by imaging the part, a skeletal information correction unit configured to obtain second skeletal information from the first skeletal information by correcting a part or all of the first skeletal information with a predetermined reference position as a reference on the basis of detection of a predetermined state of the user, and a skeletal information output unit configured to output the second skeletal information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, due to a shortage of skilled engineers, skilled engineers are not always assigned to work sites, and remote work support has become increasingly important, in which skilled engineers located in locations other than the work site act as instructors and give instructions to workers performing work at the work site from a remote location. As examples of technologies related to remote work support, various remote work support systems are known.

[0003] For example, a remote work support system is known in which a worker terminal transmits video of the space where the worker is working (hereinafter also referred to as the "work space") and audio collected from the worker to an instructor terminal, and the instructor terminal presents the video and audio to the instructor, who then issues instructions to the worker while viewing the video and audio (see, for example, non-patent document 1).

[0004] More specifically, in such a remote work support system, the worker-side system superimposes a hand image showing the shape of the instructor's hand on a captured image obtained by capturing an image of the workspace, and displays the captured image with the hand image superimposed. Furthermore, the worker-side system transmits the captured image obtained by capturing an image of the workspace to the instructor-side system, and the instructor-side system superimposes the hand image on the captured image received from the worker-side system, and displays the captured image with the hand image superimposed.

[0005] This allows the instructor to understand how the instructor's hand and finger posture is being conveyed to the worker. Various techniques are known for detecting the instructor's hand and finger posture (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Shunsuke Ichihara, Yusuke Suzuki, "Field evaluation experiment of a remote work support system using hand gestures and line drawing functions," [online], March 2, 2017, Information Processing Society of Japan Interaction 2017, [Retrieved July 19, 2024], Internet<URL:http: / / www.interaction-ipsj.org / proceedings / 2017 / data / pdf / 1-502-13.pdf> [Non-patent document 2] Ultraleap's website, [online], [searched July 19, 2024], Internet<URL:https: / / www.ultraleap.com> Summary of the Invention [Problem to be solved by the invention]

[0007] Generally, there are cases where an object related to an instruction does not exist at the reference position of a motion for an instruction by an instructor. For example, even if an instructor tries to give instructions regarding an object that exists in the space where a worker exists, there is no object related to the instruction in the space where the instructor exists. Therefore, the instructor may set a reference position in the air and perform a motion based on the reference position set in the air.

[0008] However, there is no visual mark at the reference position set by the instructor in the air. This makes it difficult for the instructor to continue the motion using the same position as the reference position. This can lead to a discrepancy between the instructions the instructor actually gives and the instructions they want to convey to the worker. Furthermore, this discrepancy can lead to a situation where the clarity of the instructions conveyed from the instructor to the worker is reduced.

[0009] Therefore, it is desirable to provide a technology that enables instructions to be more clearly conveyed by motions according to a reference position. [Means for solving the problem]

[0010] In order to solve the above problem, according to one aspect of the present invention, there is provided an information processing device including: a detection unit that detects first skeletal information of a part of a user's body based on a first image obtained by capturing an image of the part; a skeletal information correction unit that obtains second skeletal information from the first skeletal information by correcting part or all of the first skeletal information based on the detection of a predetermined state of the user, using a predetermined reference position as a reference; and a skeletal information output unit that outputs the second skeletal information.

[0011] The information processing device may include a display control unit that generates a second image based on the second skeletal information and controls the second image to be displayed.

[0012] The skeletal information correcting unit may correct a part or all of the first skeletal information by rotating a part or all of the first skeletal information with respect to the predetermined reference position.

[0013] The predetermined state may be a state in which a part or all of the part remains stationary for a predetermined period of time or more.

[0014] The predetermined state may be a state in which the user has performed an input operation on a predetermined user interface.

[0015] The body part may be a finger of the user.

[0016] The fingers may include right and left fingers, the first skeletal information may include skeletal information of the right fingers and skeletal information of the left fingers, and when one of the right and left fingers remains stationary for a predetermined period of time or longer, the skeletal information correction unit may correct the skeletal information of the other by rotating the skeletal information of the other finger based on the predetermined reference position.

[0017] The user may be an instructor who gives instructions to a worker who performs a task.

[0018] The user may be a worker who receives instructions regarding a task from an instructor.

[0019] In order to solve the above problem, according to another aspect of the present invention, there is provided an information processing method executed by a computer, including: detecting first skeletal information of a part of a user's body based on a first image obtained by capturing an image of the part; obtaining second skeletal information from the first skeletal information by correcting part or all of the first skeletal information based on detection of a predetermined state of the user, and outputting the second skeletal information.

[0020] In addition, according to another aspect of the present invention, in order to solve the above-mentioned problem, there is provided a program that causes a computer to function as a detection unit that detects first skeletal information of a part of a user's body based on a first image obtained by capturing an image of the part; a skeletal information correction unit that obtains second skeletal information from the first skeletal information by correcting part or all of the first skeletal information based on the detection of a predetermined state of the user, using a predetermined reference position as a reference; and a skeletal information output unit that outputs the second skeletal information. [Effects of the Invention]

[0021] As described above, the present invention provides a technique that enables instructions to be more clearly conveyed by motions according to a reference position. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of a remote operation support system according to an embodiment of the present invention. [Figure 2] 3 is a block diagram showing an example of the functional configuration of an operator-side device 14. FIG. [Figure 3] 3 is a block diagram showing an example of the functional configuration of an instructor side device 24. FIG. [Figure 4] 10 is a diagram showing an example of a display of a work captured video on the instructor side display unit 26 before an instruction is started. FIG. [Figure 5] 10 is a diagram showing an example of a display of a work captured video on the instructor side display unit 26 after instruction has started. FIG. [Figure 6] 10 is a diagram showing an example of three-dimensional skeletal information of fingers detected by the finger detection unit 241. FIG. [Figure 7] FIG. 10 is a first diagram for explaining the correction process. [Figure 8] FIG. 10 is a second diagram for explaining the correction process. [Figure 9] 10 is a diagram showing an example of a display on the worker-side display unit 16 of a composite hand and finger image superimposed on a captured work image. FIG. [Figure 10] 10 is a flowchart showing an example of the operation of the worker-side system 10. [Figure 11] 10 is a flowchart showing an example of the operation of the instructor system 20. [Figure 12] FIG. 9 is a diagram showing a hardware configuration of an information processing device 900 as an example of the instructor system 20 according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0024] (1. Details of the embodiment) The details of the embodiment of the present invention will now be described.

[0025] (1-1. Configuration of remote work support system) First, an example of the configuration of a remote work support system according to an embodiment of the present invention will be described. FIG. 1 is a diagram showing an example of the functional configuration of the remote work support system according to an embodiment of the present invention. The remote work support system 1 is used by a worker who performs work while receiving instructions related to the work from an instructor, and an instructor who gives instructions to the worker. In other words, the worker and the instructor are each users of the remote work support system 1. As shown in FIG. 1, the remote work support system 1 has a worker-side system 10, an instructor-side system 20, and a network 30.

[0026] The worker-side system 10 is a system used by a worker in the remote work support system 1. The worker performs work at a location (i.e., a remote location) away from the instructor who issues instructions to the worker. On the other hand, the instructor-side system 20 is a system used by the instructor in the remote work support system 1. The worker-side system 10 and the instructor-side system 20 are connected to a network 30 and are configured to be able to communicate with each other via the network 30.

[0027] In remote work support, voice calls and the like are generally conducted between the instructor and the worker. However, the configuration necessary for voice calls and the like is not involved in the description of the configuration of the remote work support system 1 according to the embodiment of the present invention, and therefore a description of the configuration necessary for voice calls and the like will be omitted.

[0028] 1, the worker-side system 10 includes a work imaging unit 12, a worker-side device 14, and a worker-side display unit 16. On the other hand, as shown in FIG. 1, the instructor-side system 20 includes a hand imaging unit 22, a worker-side device 24, and a worker-side display unit 26.

[0029] (Work imaging unit 12) The work imaging unit 12 obtains an image (hereinafter also referred to as a "work imaging image") by imaging the work space, which is the space where work is performed by the worker. In the following description, it is mainly assumed that the work imaging image is a moving image made up of a plurality of frames captured continuously in chronological order. A moving image can also be referred to as a "video." Therefore, in the following description, the term "work imaging video" is used as an example of a work imaging image. However, the work imaging image may also be a still image.

[0030] The work imaging unit 12 is preferably provided so as to face the same direction as the worker's line of sight. Therefore, the work imaging unit 12 is preferably integrated with an AR (Augmented Reality) display. However, the work imaging unit 12 may also be configured as hardware separate from the AR display.

[0031] (Operator side device 14) The operator-side device 14 is realized by a computer and may correspond to an information processing device. An example of the functional configuration of the operator-side device 14 will be described with reference to FIG.

[0032] Fig. 2 is a block diagram showing an example of the functional configuration of the worker-side device 14. As shown in Fig. 2, the worker-side device 14 includes a control unit 140, a communication unit 148, and a storage unit 149. The functions of the control unit 140, the communication unit 148, and the storage unit 149 will be described below.

[0033] (control unit 140) The control unit 140 performs various types of arithmetic processing. For example, the control unit 140 includes an arithmetic device such as a CPU (Central Processing Unit), and its functions can be realized by the arithmetic device expanding a program stored in a ROM (Read Only Memory) into a RAM (Random Access Memory) and executing it. In this case, a computer-readable recording medium on which the program is recorded can also be provided. Alternatively, these blocks may be configured with dedicated hardware or a combination of multiple pieces of hardware.

[0034] Data necessary for calculations by the calculation device is appropriately stored in storage unit 149. As shown in Fig. 2, control unit 140 includes a transmission control unit 141, a skeletal information acquisition unit 142, and an operator-side display control unit 143. Details of the functions of each of skeletal information acquisition unit 142, transmission control unit 141, and operator-side display control unit 143 will be explained later.

[0035] (Communication Department 148) The communication unit 148 is configured by a communication interface, and communicates with the instructor device 24 via the network 30 .

[0036] (Storage unit 149) The storage unit 149 is a memory capable of storing programs and data for operating the control unit 140. The storage unit 149 can also temporarily store various data required in the course of operation of the control unit 140. For example, the storage unit 149 may be a non-volatile memory.

[0037] (Worker side display section 16) The worker-side display unit 16 has a function of displaying various types of information to the worker in accordance with control by the control unit 140. More specifically, the worker-side display unit 16 has a screen that is visually recognized by the worker (hereinafter also referred to as the "worker-side screen"), and displays various types of information on the worker-side screen in accordance with control by the control unit 140.

[0038] For example, the worker-side display unit 16 may be a liquid crystal display (LCD) device or an OLED (organic light emitting diode) device. Furthermore, the worker-side display unit 16 may be a head-mounted display worn on the worker's head, or may be a fixed display.

[0039] (Hand and finger imaging unit 22) The hand and finger imaging unit 22 obtains an image (hereinafter also referred to as a "hand and finger image") by capturing an image of the instructor's fingers. In the following description, it is mainly assumed that the hand and finger image is a moving image (video). Therefore, in the following description, the term "hand and finger image" is used as an example of a hand and finger image. However, the hand and finger image may also be a still image.

[0040] As will be explained later, the instructor's fingers may be an example of a body part of the instructor. The fingers may include the area of ​​the body from the wrist to the tips of the fingers that make up the hand. Furthermore, the finger imaging unit 22 may be replaced with another sensor. That is, the image of the fingers may be an example of sensor data obtained by a sensor. The image of the fingers may be a first image.

[0041] (Instructor side device 24) The explanation will be continued by returning to Fig. 1. The instructor side device 24 is realized by a computer and can correspond to an information processing device. An example of the functional configuration of the instructor side device 24 will be explained with reference to Fig. 3.

[0042] Fig. 3 is a block diagram showing an example of the functional configuration of the instructor device 24. As shown in Fig. 3, the instructor device 24 includes a control unit 240, a communication unit 248, and a storage unit 249. The functions of the control unit 240, the communication unit 248, and the storage unit 249 will be described below.

[0043] (control unit 240) The control unit 240 performs various types of arithmetic processing. For example, the control unit 240 includes an arithmetic device such as a CPU (Central Processing Unit), and its functions can be realized by the arithmetic device expanding a program stored in a ROM (Read Only Memory) into a RAM (Random Access Memory) and executing it. In this case, a computer-readable recording medium on which the program is recorded can also be provided. Alternatively, these blocks may be configured with dedicated hardware or a combination of multiple pieces of hardware.

[0044] Data necessary for calculations by the calculation device is appropriately stored in storage unit 249. As shown in Fig. 3, control unit 240 includes hand detection unit 241, skeletal information correction unit 242, skeletal information output unit 243, work captured image acquisition unit 244, and instructor side display control unit 246. Details of the functions of hand detection unit 241, skeletal information correction unit 242, skeletal information output unit 243, work captured image acquisition unit 244, and instructor side display control unit 246 will be described later.

[0045] (Communications Department 248) The communication unit 248 is configured by a communication interface, and communicates with the operator-side device 14 via the network 30 .

[0046] (Storage unit 249) The storage unit 249 is a memory capable of storing programs and data for operating the control unit 240. The storage unit 249 can also temporarily store various data required in the course of operation of the control unit 240. For example, the storage unit 249 may be a non-volatile memory.

[0047] (Instructor side display section 26) The instructor-side display unit 26 has a function of displaying various information to the instructor in accordance with control by the control unit 240. More specifically, the instructor-side display unit 26 has a screen that is visually recognized by the instructor (hereinafter also referred to as the "instructor-side screen"), and displays various information on the instructor-side screen in accordance with control by the control unit 240.

[0048] For example, the instructor-side display unit 26 may be a liquid crystal display (LCD) device or an OLED (organic light emitting diode) device. Furthermore, the instructor-side display unit 26 may be a head-mounted display worn on the instructor's head, or may be a fixed display.

[0049] (Network 30) The network 30 connects the worker-side system 10 and the instructor-side system 20. The network 30 can function as a communication path between the worker-side system 10 and the instructor-side system 20.

[0050] (Before the person giving instructions begins) The state before the instructor starts giving instructions will be described.

[0051] The transmission control unit 141 controls the communication unit 148 so that the work captured video obtained by the work capturing unit 12 is transmitted to the instructor-side device 24 via the network 30. In the instructor-side system 20, the work captured video acquisition unit 244 acquires the work captured video received by the communication unit 248, and the instructor-side display control unit 246 controls the instructor-side display unit 26 so that the work captured video is displayed. A display example of the work captured video will be described with reference to FIG. 4.

[0052] Fig. 4 is a diagram showing an example of a work captured video displayed by the instructor side display unit 26 before instruction is started. Referring to Fig. 4, the instructor side display unit 26 has an instructor side screen 261. The instructor side display control unit 246 controls the instructor side display unit 26 so that a work captured video M10 is displayed on the instructor side screen 261. The work captured video M10 shows a work space, and work objects 61 to 63 that are the targets of work are present in the work space.

[0053] Here, it is assumed that work object 61 is a part, work objects 62-63 are screws, and the instructor is going to use hand imaging unit 22 to give instructions regarding work to turn the screws that are parts 62-63 to fix the part that is work object 61 (hereinafter also referred to as "screwdriving work"). However, the type of instruction is not limited to instructions regarding such screwdriving work.

[0054] The instructor can grasp the situation in the work space by visually checking the work captured image M10, and can consider giving instructions according to the situation in the work space.

[0055] (After the person giving the instructions has started) Next, a description will be given of what happens after the instructor starts giving instructions. Fig. 5 is a diagram showing an example of the display of the work captured video on the instructor-side display unit 26 after the instructor starts giving instructions. Referring to Fig. 5, it can be seen that in the space where the instructor gives instructions (hereinafter also referred to as the "instruction space"), the instructor gives instructions by positioning the fingers L1 of his left hand and the fingers R1 of his right hand in a shape similar to that used when performing screwdriver work (in the example shown in Fig. 5, the left hand is fixed and the tip of the index finger of the right hand is rotated).

[0056] When the instructor's left fingers L1 and right fingers R1 enter the imaging range of finger imaging unit 22, the instructor's left fingers L1 and right fingers R1 are imaged by finger imaging unit 22, thereby obtaining a finger image. Work image M12, left finger composite image L2, and right finger composite image R2 will be described later.

[0057] The finger detection unit 241 acquires the finger image captured by the finger imaging unit 22. The finger detection unit 241 functions as an example of a detection unit that detects three-dimensional skeletal information of the left fingers L1 and the right fingers R1 based on the finger image. The finger detection unit 241 associates the three-dimensional skeletal information of the left fingers L1 and the right fingers R1 with a timestamp indicating the current time and stores the information in the storage unit 249.

[0058] The three-dimensional skeletal information of the left finger L1 or the right finger R1 may correspond to the first skeletal information. In the following description, the three-dimensional skeletal information of the left finger L1 and the right finger R1 is also simply referred to as "three-dimensional skeletal information of the fingers" or "three-dimensional skeletal information." The three-dimensional skeletal information of the fingers is also an example of skeletal information of the fingers. Therefore, the three-dimensional skeletal information of the fingers may be replaced with skeletal information of other dimensions (for example, two dimensions).

[0059] The three-dimensional skeletal information of the fingers may be detected in any manner. For example, the three-dimensional skeletal information of the fingers may be detected using Leap Motion (registered trademark) developed by Ultraleap, Inc. Leap Motion (registered trademark) is a technology that tracks hand movements based on the detection results of an infrared stereo camera using infrared light irradiated onto the hand by multiple light emitting diodes (LEDs).

[0060] Here, the three-dimensional skeletal information of the fingers will be described with reference to FIG.

[0061] FIG. 6 is a diagram showing an example of three-dimensional skeletal information of fingers detected by finger detection unit 241. Referring to FIG. 6, the left fingers L1 and right fingers R1 of the instructor are present in finger image F1 obtained by finger imaging unit 22. As shown in FIG. 6, based on finger image F1, finger detection unit 241 detects, as examples of three-dimensional skeletal information of the fingers, skeletal position vectors indicating the skeletal positions of left fingers L1 and right fingers R1 in the instruction space and skeletal direction vectors indicating the skeletal direction. Here, for ease of viewing the diagram, only the three-dimensional skeletal information of right finger R1 is labeled.

[0062] The skeleton position vector may include vectors indicating the fingertip positions, finger joint positions, wrist position, vertex positions of the polygons that make up the palm, and the center position of the palm.

[0063] 6 shows, as examples of skeletal positions, the tip position P1 of the index finger of the right hand R1, the center position N of the palm, and the wrist position W. In the following description, multiple skeletal position vectors detected by the finger detection unit 241 may be collectively referred to as the "skeletal position vector P*."

[0064] The skeleton direction vectors include a normal vector GN relative to the back of the hand, a finger direction vector D1 which is a vector from the palm to the fingertip, and a hand direction vector HD.

[0065] Note that the finger direction vector D1 may be a vector pointing from the wrist position W to the tip of the index finger, but vectors pointing from the wrist position W to the tips of other fingers may also be detected as finger direction vectors. Furthermore, the hand direction vector HD may be a vector pointing from the wrist position W to the base of the middle finger. These skeleton direction vectors may be acquired as normalized vectors (i.e., unit vectors). In the following description, multiple skeleton direction vectors detected by the hand and finger detection unit 241 may be collectively referred to as the "skeleton direction vector D*."

[0066] The skeletal information correcting unit 242 corrects part or all of the three-dimensional skeletal information of the fingers based on a predetermined reference position based on the detection of a predetermined state of the instructor (hereinafter also referred to as a "correction start trigger state"), thereby obtaining corrected three-dimensional skeletal information from the three-dimensional skeletal information of the fingers. The corrected three-dimensional skeletal information may correspond to second skeletal information. The skeletal information correcting unit 242 outputs the corrected three-dimensional skeletal information to the skeletal information output unit 243.

[0067] In the following description, it is mainly assumed that the correction start trigger state is a state in which some or all of the fingers remain in a predetermined posture for a predetermined period of time or more. However, as will be explained later, the correction start trigger state is not limited to this state.

[0068] More specifically, the skeletal information correction unit 242 may correct part or all of the three-dimensional skeletal information of the fingers by rotating part or all of the three-dimensional skeletal information of the fingers based on a predetermined reference position.

[0069] Here, the fingers may include both right and left fingers. Furthermore, the three-dimensional skeletal information of the fingers may include three-dimensional skeletal information of the right fingers and three-dimensional skeletal information of the left fingers. All of the three-dimensional skeletal information of the fingers may include both right and left fingers. Furthermore, part of the three-dimensional skeletal information of the fingers may be either right or left fingers.

[0070] In the following description, it is mainly assumed that when one of the fingers of the right hand or the fingers of the left hand remains stationary for a predetermined period of time or longer, the skeletal information correcting unit 242 corrects the other's three-dimensional skeletal information by rotating the other's skeletal information based on a predetermined reference position. Furthermore, in the following description, it is mainly assumed that the predetermined reference position is the position of the instructor's wrist. However, as will be explained later, the predetermined reference position may be other than the position of the instructor's wrist.

[0071] On the other hand, when the skeletal information correcting unit 242 does not detect a predetermined state of the instructor (hereinafter also referred to as a "correction start trigger state"), it outputs the three-dimensional skeletal information of the fingers to the skeletal information output unit 243 without correcting the three-dimensional skeletal information of the fingers. An example of a method for correcting the three-dimensional skeletal information of the fingers by the skeletal information correcting unit 242 will be described in detail below. Specifically, the following will be described in the order of "pre-processing," "maintenance frame number counting processing," "reference position determination processing," "intermediate processing," "correction processing," and "post-processing."

[0072] In the following description, the Euclidean norm of vector A will be written as ||A||, and the normalized vector of vector A will be written as normalize(A). In this case, ||normalize(A)|| = 1 holds, and normalize(A) × ||A|| = A holds. In the following description, the dot product of vector A and vector B will be written as dot(A,B), and the cross product of vector A and vector B will be written as cross(A,B).

[0073] (Pretreatment) First, the skeleton information correcting unit 242 executes preprocessing. Specifically, the skeleton information correcting unit 242 initializes and prepares each variable.

[0074] For example, the skeletal information correction unit 242 initializes the variable left hand reference skeletal information (hereinafter also referred to as "Lb") as undefined, and initializes the variable right hand reference skeletal information (hereinafter also referred to as "Rb") as undefined.

[0075] In addition, the skeletal information correction unit 242 initializes the variable left front frame skeletal information (hereinafter also referred to as "Lp") to be undefined, and initializes the variable right front frame skeletal information (hereinafter also referred to as "Rp") to be undefined.

[0076] In addition, the skeletal information correcting unit 242 initializes the variable number of left hand maintaining frames (hereinafter also referred to as "cL") to 0, and initializes the variable number of right hand maintaining frames (hereinafter also referred to as "cR") to 0.

[0077] Furthermore, the skeleton information corrector 242 initializes a reference fixed flag (hereinafter also referred to as "flag_b"), which is a variable, to false.

[0078] In addition, the skeletal information correction unit 242 prepares three-dimensional skeletal information of the fingers of the left hand detected based on the current frame of the finger image video as left finger skeletal information of the current frame (hereinafter also referred to as "Lc"), and prepares three-dimensional skeletal information of the fingers of the right hand detected based on the current frame of the finger image video as right finger skeletal information of the current frame (hereinafter also referred to as "Rc").

[0079] (Maintained frame count processing) Next, the skeleton information correcting unit 242 executes a maintaining frame number counting process. Specifically, the skeleton information correcting unit 242 assigns false to a correction execution flag (hereinafter also referred to as "flag_a"), which is a variable.

[0080] If the finger detection unit 241 does not detect left hand finger skeletal information from the current frame of the finger image capture video (i.e., if the left hand finger skeletal information Lc of the current frame is undefined), the skeletal information correction unit 242 assigns "undefined" to the left preceding frame skeletal information Lp and assigns 0 to the left hand maintenance frame count cL.

[0081] Alternatively, if the finger detection unit 241 detects left hand finger skeletal information from the current frame of the finger image video (i.e., if the left hand finger skeletal information Lc of the current frame is not undefined), and if the left preceding frame skeletal information Lp is undefined, the skeletal information correction unit 242 substitutes the left hand finger skeletal information Lc of the current frame for the left preceding frame skeletal information Lp, and substitutes 0 for the left hand maintenance frame count cL.

[0082] Alternatively, when the finger detection unit 241 detects left hand finger skeletal information from the current frame of the finger image capture video (i.e., when the left hand finger skeletal information Lc of the current frame is not undefined) and when the left preceding frame skeletal information Lp is not undefined, the skeletal information correction unit 242 calculates the sum of absolute values ​​of differences between corresponding skeletal position vectors in the left preceding frame skeletal information Lp and the left hand finger skeletal information Lc of the current frame as a difference sum (hereinafter also referred to as "err"). Here, with respect to a predetermined threshold (hereinafter also referred to as "thr_e"), satisfaction of difference sum err<threshold thr_e can mean that the left hand fingers are in a stationary state.

[0083] Therefore, if the difference sum err<threshold thr_e is satisfied, the skeletal information correcting unit 242 determines that the current state of the left fingers is stationary and increases the left hand maintaining frame count cL by 1. On the other hand, if the difference sum err<threshold thr_e is not satisfied, the skeletal information correcting unit 242 determines that the current state of the left fingers is not stationary and assigns 0 to the left hand maintaining frame count cL. Furthermore, regardless of whether the difference sum err<threshold thr_e is satisfied or not, the skeletal information correcting unit 242 assigns the left hand finger skeletal information Lc of the current frame to the previous left frame skeletal information Lp.

[0084] The above describes the counting process for the left hand maintaining frame number cL. However, similar to the counting process for the left hand maintaining frame number cL, the counting process for the right hand maintaining frame number cR may also be executed. This updates the right front frame skeletal information Rp and the right hand maintaining frame number cR.

[0085] (Reference position determination process) Next, when the left hand maintaining frame count cL>=threshold value thr_c and the right hand maintaining frame count cR>=threshold value thr_c are satisfied with respect to a predetermined threshold (hereinafter also referred to as "thr_c"), the skeletal information correcting unit 242 determines that the current states of the left and right fingers have both continued for a predetermined time or more, and assigns false to the correction execution flag flag_a and true to the reference fixation flag flag_b. Furthermore, the skeletal information correcting unit 242 assigns the left hand finger skeletal information Lc of the current frame to the left hand reference skeletal information Lb, and assigns the right hand finger skeletal information Rc of the current frame to the right hand reference skeletal information Rb, and ends the reference position determination process.

[0086] When the left hand maintenance frame count cL>=threshold value thr_start and the reference fixation flag flag_b==true are satisfied with respect to a predetermined threshold (hereinafter also referred to as "thr_start"), the skeletal information correcting unit 242 determines that the current state of the left hand fingers has continued for a predetermined time or more and that the reference position has been determined, and assigns true to the correction execution flag flag_a. For example, a state in which the current state of the left hand fingers has continued for a predetermined time or more may correspond to a state in which the left hand fingers are fixed and a motion in which the tips of the right hand fingers are rotating. In this case, the right hand finger skeletal information is corrected in the correction process described next.

[0087] The threshold value thr_start may be set to a value greater than the threshold value thr_c, which can prevent the correction execution flag flag_a from being set to true immediately after the reference fixing flag flag_b is set to true.

[0088] If the right-hand maintenance frame count cR>=threshold value thr_start and the reference fixation flag flag_b==true are satisfied, the skeletal information corrector 242 determines that the current state of the right-hand fingers has continued for a predetermined time or longer and that the reference position has been determined, and assigns true to the correction execution flag flag_a. For example, a state in which the current state of the right-hand fingers has continued for a predetermined time or longer may correspond to a state in which the right-hand fingers are fixed and the tips of the left-hand fingers are rotating. In this case, the left-hand finger skeletal information is corrected using the correction process described next.

[0089] If the left hand maintenance frame count cL<threshold value thr_start and the right hand maintenance frame count cR<threshold value thr_start are satisfied, the skeletal information correction unit 242 determines that neither the current state of the right hand fingers nor the left hand fingers has continued for a predetermined period of time or longer, and assigns false to the reference fixed flag flag_b.

[0090] (Intermediate processing) If the correction execution flag flag_a is false, the skeletal information correcting unit 242 skips the correction process described below and proceeds to post-processing. On the other hand, if the correction execution flag flag_a is true, the skeletal information correcting unit 242 proceeds to the correction process described below.

[0091] (correction processing) The skeletal information correcting unit 242 assigns the left hand finger skeletal information Lc of the current frame to the corrected left hand skeletal information La, and assigns the right hand finger skeletal information Rc of the current frame to the corrected right hand skeletal information Ra. The following description describes the processing executed when the left hand maintenance frame count cL >= threshold value thr_start is satisfied. However, when the right hand maintenance frame count cR >= threshold value thr_start is satisfied, the processing executed when the left hand maintenance frame count cL >= threshold value thr_start is satisfied may be reversed.

[0092] Fig. 7 is a first diagram for explaining the correction process. Referring to Fig. 7, a left finger L1 and a right finger R1 are shown. As shown in Fig. 7, when there is no distinction between the fingers of the right hand and the fingers of the left hand, the wrist position vector can be written as "W", the hand direction vector can be written as "HD", and the normal vector to the back of the hand can be written as "GN".

[0093] FIG. 8 is a second diagram for explaining the correction process. Referring to FIG. 8, a reference frame Fb and a current frame Fc are shown. Right hand reference skeletal information Rb is detected from the reference frame Fb. Right hand finger skeletal information Rc of the current frame is detected from the current frame Fc. A right wrist position vector Wb is shown in the reference frame Fb, and the right wrist position vector Wb is included in the right hand reference skeletal information Rb. Meanwhile, a right wrist position vector Wc is shown in the current frame Fc, and the right wrist position vector Wc is included in the right hand finger skeletal information Rc of the current frame.

[0094] Additionally, the reference frame Fb indicates a direction vector HDb of the right hand in the reference frame, and the direction vector HDb of the right hand in the reference frame is included in the right hand reference skeletal information Rb. Also, the reference frame Fb indicates a normal vector GNb to the back of the right hand in the reference frame, and the normal vector GNb to the back of the right hand in the reference frame is included in the right hand reference skeletal information Rb.

[0095] The current frame Fc indicates the orientation vector HDc of the right hand in the current frame, which is included in the right hand finger skeletal information Rc of the current frame. The current frame Fc also indicates the normal vector GNc to the back of the right hand in the current frame, which is included in the right hand finger skeletal information Rc of the current frame.

[0096] When the number of left hand maintenance frames cL>=threshold value thr_start is satisfied, the skeletal information correction unit 242 calculates the vector HXb in the reference frame using the following equation (1) based on the normal vector GNb to the back of the hand in the reference frame and the hand direction vector HDb in the reference frame.

[0097] HXb:=normalize(cross(GNb,HDb))...(1)

[0098] Similarly, when the number of left hand maintenance frames cL>=threshold value thr_start is satisfied, the skeletal information correction unit 242 calculates the vector HXc in the current frame using the following equation (2) based on the normal vector GNc to the back of the hand in the current frame and the hand direction vector HDc in the current frame.

[0099] HXc:=normalize(cross(GNc,HDc))...(2)

[0100] Furthermore, the skeleton information correcting unit 242 calculates a vector HDb' (not shown) in the reference frame using the following equation (3) based on the normal vector GNb relative to the back of the hand in the reference frame and the vector HXb in the reference frame.

[0101] HDb':=normalize(cross(HXb, GNb))...(3)

[0102] Similarly, the skeleton information corrector 242 calculates a vector HDc' (not shown) in the current frame using the following equation (4) based on the normal vector GNc relative to the back of the hand in the current frame and the vector HXc in the current frame.

[0103] HDc':=normalize(cross(HXc, GNc))...(4)

[0104] Here, the set of vectors (HXb, PNb, HDb') are basis vectors that are orthogonal to each other, and the set of vectors (HXc, PNc, HDc') are basis vectors that are orthogonal to each other, and a rotation matrix can be defined using these basis vectors.

[0105] If HDc==HDb is satisfied between the hand direction vector HDc in the current frame and the hand direction vector HDb in the reference frame (if HDc and HDb are equal), the skeletal information correcting unit 242 substitutes an identity matrix into the rotation matrix RM.

[0106] On the other hand, if HDc==-HDb is satisfied between the hand direction vector HDc in the current frame and the hand direction vector HDb in the reference frame (if the directions of HDc and HDb are opposite to each other), the skeletal information correction unit 242 substitutes a rotation matrix that rotates 180 degrees around the hand direction vector HXc in the current frame as an axis into the rotation matrix RM.

[0107] If the conditions HDc!=HDb and HDc!=-HDb (where "!=" indicates an inequality operator, returning true if the two values ​​are not equal, and false if the two values ​​are equal) are satisfied between the hand direction vector HDc in the current frame and the hand direction vector HDb in the reference frame, the skeletal information correcting unit 242 calculates the rotation angle θ as shown in the following equation (5), calculates the unit vector A as shown in the following equation (6), and substitutes the rotation matrix that rotates by θ around the unit vector A as an axis into the rotation matrix RM.

[0108] θ:=arccos(dot(HDc,HDb))...(5) A:=normalize(cross(HDc,HDb))...(6)

[0109] Note that HDb=RM·HDc holds. The skeletal information correcting unit 242 calculates the translation vector T by substituting the right wrist position vector Wc into the rotation center position vector (hereinafter also referred to as "OC") and subtracting the right wrist position vector Wc from the right wrist position vector Wb.

[0110] The skeleton information correcting unit 242 converts the skeleton position vector P* according to the following equation (7) using the rotation center position vector OC, rotation matrix RM, and translation vector T to obtain a converted skeleton position vector P*'. Furthermore, the skeleton information correcting unit 242 converts the skeleton direction vector D* according to the following equation (8) using the rotation matrix RM to obtain a converted skeleton direction vector D*'.

[0111] P*':=RM·(P*-OC)+OC+T···(7) D*':=RM·D*···(8)

[0112] The skeleton information correcting unit 242 substitutes the converted skeleton position vector P*' and the converted skeleton direction vector D*' into corrected right hand skeleton information Ra.

[0113] (Post-processing) The skeletal information correcting unit 242 outputs the corrected right hand skeletal information Ra and the corrected left hand skeletal information La as three-dimensional skeletal information to the skeletal information output unit 243. Note that the corrected right hand skeletal information Ra may be substituted with the right hand finger skeletal information Rc of the current frame without being corrected. Similarly, the corrected left hand skeletal information La may be substituted with the left hand finger skeletal information Lc of the current frame without being corrected.

[0114] The skeletal information output unit 243 acquires the three-dimensional skeletal information output from the skeletal information correction unit 242. Then, the skeletal information output unit 243 outputs the acquired three-dimensional skeletal information. More specifically, the skeletal information output unit 243 controls the communication unit 248 so that the acquired three-dimensional skeletal information is transmitted to the worker-side device 14 by the communication unit 248. This completes the post-processing.

[0115] (After post-processing) The skeleton information acquisition unit 142 acquires the three-dimensional skeleton information received by the communication unit 148 from the communication unit 148 .

[0116] The operator-side display control unit 143 generates a three-dimensional hand model based on the three-dimensional skeletal information acquired by the skeletal information acquisition unit 132, and generates a two-dimensional composite hand image by projecting the generated three-dimensional hand model using a predetermined projection formula. The two-dimensional composite hand image may correspond to the second image.

[0117] For example, the projection formula may be a formula indicating orthogonal projection (parallel projection). Then, the worker-side display control unit 143 superimposes the generated composite hand image on the work captured image obtained by the work capturing unit 12, and controls the worker-side display unit 16 so that the composite hand image superimposed on the work captured image is displayed on the worker-side display unit 16. The composite hand image includes a left hand composite image and a right hand composite image.

[0118] Fig. 9 is a diagram showing an example of a display of a finger composite video superimposed on a work captured video by the worker-side display unit 16. Referring to Fig. 9, the worker-side display unit 16 has an worker-side screen 161. As shown in Fig. 9, the worker-side display control unit 143 superimposes the left hand finger composite video L3 and right hand finger composite video R3 generated on the work captured video M12 obtained by the work capturing unit 12, and controls the worker-side display unit 16 so that the left hand finger composite video L3 and right hand finger composite video R3 superimposed on the work captured video M12 are displayed on the worker-side screen 161.

[0119] The instructor-side display control unit 246 displays the work captured video M12 on the instructor-side display unit 26. The instructor-side display control unit 246 also generates a three-dimensional hand model based on the corrected skeletal information obtained by the skeletal information correcting unit 242, and projects the generated three-dimensional hand model using a predetermined projection formula, thereby generating a two-dimensional composite hand video. The projection by the instructor-side display control unit 246 may be performed in the same manner as the projection by the worker-side display control unit 143.

[0120] Then, the instructor-side display control section 246 superimposes the generated composite hand and finger image on the work captured image M12, and causes the instructor-side display section 26 to display the composite hand and finger image superimposed on the work captured image M12.

[0121] 5, the instructor-side display control unit 246 superimposes the generated left hand finger composite video L2 and right hand finger composite video R2 on the work captured video M12, and causes the instructor-side display unit 26 to display the left hand finger composite video L2 and right hand finger composite video R2 superimposed on the work captured video M12. Note that the left hand finger composite video L3 displayed by the worker-side display unit 16 and the left hand finger composite video L2 displayed by the instructor-side display unit 26 may be the same video or different videos. Similarly, the right hand finger composite video R3 displayed by the worker-side display unit 16 and the right hand finger composite video R2 displayed by the instructor-side display unit 26 may be the same video or different videos.

[0122] The above has described an example of the configuration of the remote operation support system 1 according to an embodiment of the present invention.

[0123] (1-2. Operation of the remote work support system) Next, an operation example of the remote work support system 1 according to an embodiment of the present invention will be described with reference to Fig. 10 and Fig. 11 (also with appropriate reference to Figs. 1 to 9). First, an operation example of the worker-side system 10 will be described with reference to Fig. 10, and then an operation example of the instructor-side system 20 will be described with reference to Fig. 11.

[0124] 10 is a flowchart showing an example of the operation of the worker-side system 10. In the worker-side system 10, the work imaging unit 12 captures an image of a work space where work is performed by a worker to obtain a work image. The worker-side display control unit 143 acquires the work image obtained by the work imaging unit 12 (S101) and displays the work image on the worker-side display unit 16 (S102). The transmission control unit 141 controls the communication unit 148 to transmit the work image to the instructor-side device 24 via the network 30 (S103).

[0125] If the three-dimensional skeletal information is not received by the communication unit 148 ("NO" in S104), the operation proceeds to S101. On the other hand, if the three-dimensional skeletal information is received by the communication unit 148 ("YES" in S104), the skeletal information acquisition unit 142 acquires the three-dimensional skeletal information received by the communication unit 148 from the communication unit 148.

[0126] The worker-side display control unit 143 generates a three-dimensional hand model based on the three-dimensional skeletal information, and generates a two-dimensional composite hand image by projecting the generated three-dimensional hand model using a predetermined projection formula. The worker-side display control unit 143 then displays the generated composite hand image superimposed on the work captured image displayed on the worker-side display unit 16 (S105). Then, the operation proceeds to S101.

[0127] 11 is a flowchart showing an example of the operation of the instructor system 20. In the instructor system 20, the communication unit 248 receives the work captured video (S201), and the instructor display control unit 246 causes the instructor display unit 26 to display the work captured video (S202). Next, the hand and finger imaging unit 22 captures an image of the instructor's fingers to obtain the finger captured video. The hand and finger detection unit 241 acquires the hand and finger captured video obtained by the hand and finger imaging unit 22 (S203). The hand and finger detection unit 241 attempts to detect three-dimensional skeletal information of the fingers based on the hand and finger captured video.

[0128] If the finger detection unit 241 does not detect three-dimensional skeletal information of the fingers ("NO" in S204), the operation proceeds to S201. On the other hand, if the finger detection unit 241 detects three-dimensional skeletal information of the fingers ("YES" in S204), the skeletal information correction unit 242 determines whether or not correction of the three-dimensional skeletal information of the right hand is necessary (S221).

[0129] Here, if the skeletal information correcting unit 242 determines that correction of the three-dimensional skeletal information is necessary ("YES" in S221), it corrects the three-dimensional skeletal information (S222) and outputs the corrected three-dimensional skeletal information to the instructor-side display control unit 246 and the skeletal information output unit 243. On the other hand, if the skeletal information correcting unit 242 determines that correction of the three-dimensional skeletal information is not necessary ("NO" in S221), it outputs the three-dimensional skeletal information to the instructor-side display control unit 246 and the skeletal information output unit 243 without correcting the three-dimensional skeletal information.

[0130] The instructor-side display control unit 246 generates a three-dimensional hand model based on the three-dimensional skeletal information output from the skeletal information correcting unit 242, and generates a two-dimensional composite hand image by projecting the generated three-dimensional hand model using a predetermined projection formula. The instructor-side display control unit 246 then displays the generated composite hand image superimposed on the work captured image displayed on the instructor-side display unit 26 (S225).

[0131] The skeleton information output unit 243 acquires the three-dimensional skeleton information output from the skeleton information correcting unit 242. Then, the skeleton information output unit 243 controls the communication unit 248 so that the acquired three-dimensional skeleton information is transmitted to the worker-side device 14 by the communication unit 248 (S231). Then, the operation proceeds to S201.

[0132] An example of the operation of the remote operation support system 1 according to the embodiment of the present invention has been described above.

[0133] (1-3. Effects) As described above, according to an embodiment of the present invention, there is provided an instructor-side device 24 including a finger detection unit 241 that detects finger skeletal information based on an image of the instructor's fingers obtained by capturing an image of the instructor's fingers, a skeletal information correction unit 242 that obtains corrected skeletal information from the finger skeletal information by correcting part or all of the finger skeletal information based on a predetermined reference position when a predetermined state of the instructor is detected, and a skeletal information output unit 243 that outputs the corrected skeletal information.

[0134] According to this configuration, it is possible to more clearly convey instructions by means of motions according to the reference position.

[0135] The effects achieved by the remote operation support system 1 according to the embodiment of the present invention have been described above.

[0136] (2. Hardware configuration example) Next, an example of the hardware configuration of the instructor system 20 according to the embodiment of the present invention will be described.

[0137] An example of the hardware configuration of the information processing device 900 will be described below as an example of the hardware configuration of the instructor system 20 according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the instructor system 20. Therefore, the hardware configuration of the instructor system 20 may be such that unnecessary components are deleted from the hardware configuration of the information processing device 900 described below, or new components are added. Note that the hardware configuration of the worker system 10 may also be realized in the same manner as the hardware configuration of the instructor system 20.

[0138] 12 is a diagram showing a hardware configuration of an information processing device 900 as an example of the instructor system 20 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0139] The CPU 901 functions as an arithmetic processing unit and control unit, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, etc. These are interconnected by a host bus 904 that is composed of a CPU bus, etc.

[0140] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not necessary to configure the host bus 904, bridge 905, and external bus 906 separately, and these functions may be implemented on a single bus.

[0141] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers that allow the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. By operating this input device 908, the user operating the information processing device 900 can input various data to the information processing device 900 and instruct the information processing device 900 to perform processing operations.

[0142] The output device 909 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp, and an audio output device such as a speaker.

[0143] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

[0144] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network, etc. The communication device 911 may be compatible with either wireless communication or wired communication.

[0145] An example of the hardware configuration of the instructor system 20 according to the embodiment of the present invention has been described above.

[0146] (3. Summary) Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0147] For example, in the above description, the case where the instructor's fingers are used as an example of a body part of the instructor has been mainly considered. However, instead of the instructor's fingers, other body parts of the instructor may be used as the body part of the instructor. For example, the other body parts may be the instructor's upper limbs, feet, torso, or head. Alternatively, the other body parts may be the instructor's entire body.

[0148] In the above description, we have mainly considered the case where part or all of the 3D skeletal information of the instructor's fingers is corrected by rotating the 3D skeletal information of the instructor's fingers based on the position of the instructor's wrist. Rotating the 3D skeletal information can also include rotating the 3D skeletal information based on a virtual reference position that exists outside the instructor's fingers. For example, it is conceivable to rotate the 3D skeletal information of the instructor's fingers when the instructor performs a motion to open a lid or a motion to open a door.

[0149] In the above description, it is mainly assumed that three-dimensional skeletal information of the instructor's fingers is detected based on an image of the instructor's fingers, a composite image of the instructor's fingers is generated based on the three-dimensional skeletal information of the instructor's fingers, and the generated composite image of the instructor's fingers is presented to the worker. However, it is also possible that three-dimensional skeletal information of the worker's fingers is detected based on an image of the worker's fingers, a composite image of the worker's fingers is generated based on the three-dimensional skeletal information of the worker's fingers, and the generated composite image of the worker's fingers is presented to the instructor.

[0150] In the above description, the correction start trigger state is primarily considered to be a state in which some or all of the instructor's fingers remain in a predetermined posture for a predetermined period of time or longer. However, the correction start trigger state may also be another state of the instructor. For example, the correction start trigger state may be a state in which the instructor performs an input operation on a predetermined user interface. For example, the input operation on the predetermined user interface may be, for example, pressing a button. [Explanation of symbols]

[0151] 1. Remote work support system 10 Operator side system 12 Work imaging unit 14 Operator side device 140 Control Unit 141 Transmission control section 142 Skeleton information acquisition unit 143 Operator side display control unit 148 Communications Department 149 Storage section 16 Operator side display section 161 Operator's screen 20 Instructor system 22 Finger imaging unit 24 Instructor side device 240 Control Unit 241 Finger detection unit 242 Skeleton Information Correction Unit 243 Skeleton Information Output Unit 244 Work image acquisition unit 246 Instructor side display control unit 248 Communications Department 249 Storage section 26 Instructor side display section 261 Instructor side screen 30 Network

Claims

1. a detection unit that detects first skeletal information of a body part of a user based on a first image obtained by capturing the body part; a skeletal information correcting unit that corrects a part or all of the first skeletal information based on a predetermined reference position, based on the detection of a predetermined state of the user, to obtain second skeletal information from the first skeletal information; a skeleton information output unit that outputs the second skeleton information; An information processing device comprising:

2. The information processing device includes: a display control unit that generates a second image based on the second skeletal information and controls the second image to be displayed; The information processing device according to claim 1 .

3. the skeletal information correcting unit corrects a part or all of the first skeletal information by rotating a part or all of the first skeletal information with respect to the predetermined reference position; The information processing device according to claim 1 .

4. The predetermined state is a state in which a part or all of the part remains stationary for a predetermined period of time or more. The information processing device according to claim 1 .

5. the predetermined state is a state in which the user has performed an input operation on a predetermined user interface; The information processing device according to claim 1 .

6. The body part is a finger of the user. The information processing device according to claim 1 .

7. the fingers include right and left fingers, the first skeletal information includes skeletal information of fingers of the right hand and skeletal information of fingers of the left hand, when one of the fingers of the right hand and the fingers of the left hand remains stationary for a predetermined time or more, the skeletal information corrector corrects the skeletal information of the other by rotating the skeletal information of the other finger with respect to the predetermined reference position. The information processing device according to claim 6 .

8. The user is an instructor who gives instructions to a worker who performs a task. The information processing device according to claim 1 .

9. The user is a worker who receives instructions regarding a work from an instructor. The information processing device according to claim 1 .

10. Detecting first skeletal information of a body part of a user based on a first image obtained by capturing the body part; obtaining second skeletal information from the first skeletal information by correcting a part or all of the first skeletal information based on a predetermined reference position, based on the detection of a predetermined state of the user; outputting the second skeleton information; 2. A computer-implemented information processing method, comprising:

11. Computer, a detection unit that detects first skeletal information of a body part of a user based on a first image obtained by capturing the body part; a skeletal information correcting unit that corrects a part or all of the first skeletal information based on a predetermined reference position, based on the detection of a predetermined state of the user, to obtain second skeletal information from the first skeletal information; a skeleton information output unit that outputs the second skeleton information; A program that functions as a

Images