Motion learning support device, motion learning support method, and program

Abstract

To provide a motion learning support device, a motion learning support method, and a program capable of supporting a user to easily learn a motion that the user wants to learn.SOLUTION: The motion learning support device 100 includes an imaging unit 120, a skeleton estimation unit 112, a comparison unit 114, and a display unit 130. The imaging unit 120 captures an image of the user. The skeleton estimation unit 112 estimates a plurality of feature points indicating the skeleton of the user on the basis of the image of the user captured by the imaging unit 120. The comparison unit 114 calculates a deviation amount between a plurality of reference points indicating a skeleton of a reference motion stored in advance and feature points corresponding to the reference points. The display unit 130 displays the deviation amount calculated by the comparison unit 114.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a motion learning assistance device, a motion learning assistance method, and a program. [Background technology]

[0002] Patent Document 1 discloses a display device that displays an image that allows the user to grasp the state of the subject's posture. This display device displays an image that allows the user to grasp the state of the subject's posture by superimposing a captured image of the subject on a reference line and a reference point on the body that correspond to the subject's state. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-95555 Summary of the Invention [Problem to be solved by the invention]

[0004] The display device disclosed in Patent Document 1 can grasp the posture of the subject, but there is a problem in that it is difficult to recognize the extent to which the subject's user's movements during practice deviate from the movements they want to master.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide an action learning assistance device, an action learning assistance method, and a program that can assist users in easily learning the actions they want to learn. [Means for solving the problem]

[0006] In order to achieve the object of the present invention, one aspect of the movement learning assistance device according to the present invention comprises: an imaging unit that images a user; a skeleton estimation unit that estimates a plurality of feature points that indicate a skeleton of the user based on an image of the user captured by the imaging unit; a comparison unit that calculates the amount of deviation between a plurality of reference points that indicate a skeleton of a pre-stored reference movement and the plurality of feature points that correspond to the plurality of reference points, respectively; a display unit that displays a plurality of markers corresponding to the plurality of deviation amounts calculated by the comparison unit; The present invention is characterized by comprising:

[0007] In order to achieve the object of the present invention, one aspect of a movement learning assistance method according to the present invention includes: an imaging step of imaging a user; a skeleton estimation step of estimating a plurality of feature points indicating a skeleton of the user based on the image of the user captured in the imaging step; a comparison step of calculating deviation amounts between a plurality of reference points indicating a skeleton of a pre-stored reference movement and the plurality of feature points corresponding to the plurality of reference points, respectively; a display step of displaying a plurality of markers corresponding to the plurality of deviation amounts calculated in the comparison step; The present invention is characterized by comprising:

[0008] In order to achieve the object of the present invention, one aspect of the program according to the present invention is to a computer that controls an imaging unit that images a user and a display unit; a skeleton estimation unit that estimates a plurality of feature points that indicate a skeleton of the user based on an image of the user captured by the imaging unit; a comparison unit that calculates amounts of deviation between a plurality of reference points indicating a skeleton of a reference movement stored in advance and the plurality of feature points corresponding to the plurality of reference points, respectively, and causes the display unit to display signs corresponding to the calculated amounts of deviation; Function as. [Effects of the Invention]

[0009] According to the present invention, it is possible to assist a user in easily mastering the movements that the user wishes to learn. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing an operation learning assistance device according to an embodiment of the present invention; [Figure 2] 1 is a block diagram showing an operation learning assistance device according to an embodiment of the present invention; [Figure 3] 1 is a diagram showing an image of a user according to an embodiment of the present invention; [Figure 4] FIG. 4 is a diagram showing data indicating reference actions stored in a reference action DB according to the embodiment of the present invention. [Figure 5] 10A and 10B are diagrams illustrating an image projected onto a user according to an embodiment of the present invention. [Figure 6] 1 is a flowchart showing a movement learning assistance process according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an image projected onto the upper body of a user according to a modified example. [Figure 8] FIG. 10 is a diagram showing an image displayed on a display unit according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, a motion learning assistance device according to an embodiment of the present invention will be described with reference to the drawings.

[0012] 1, movement learning assistance device 100 according to this embodiment is a device that supports the user in easily learning a movement that the user wants to learn by capturing an image of user U, calculating the amount of deviation between a pre-stored reference movement and the movement of user U, and displaying the calculated amount of deviation in real time. Movement learning assistance device 100 includes control unit 110, imaging unit 120 that captures an image of user U, and display unit 130 that displays the amount of deviation.

[0013] The imaging unit 120 captures images of the movements of the user U and includes at least one of a two-dimensional camera that captures two-dimensional images, a three-dimensional camera that captures three-dimensional images, or LiDAR (Light Detection and Ranging). The imaging unit 120 also outputs image data indicating the captured image of the user U to the control unit 110. If the imaging unit 120 is a two-dimensional camera, the imaging unit 120 estimates the distance to the user U and outputs distance data indicating the estimated distance to the control unit 110.

[0014] The display unit 130 projects and displays the amount of deviation between a pre-stored reference motion and the motion of the user U onto the user U, and includes a projector that projects and displays an image. The display unit 130 also receives image data showing an image indicating the amount of deviation output from the control unit 110, and projects an image indicating the amount of deviation onto the user U based on the received image data.

[0015] As shown in FIG. 2, the movement learning assistance device 100 has an electrical configuration including the above-mentioned imaging unit 120 and display unit 130, as well as a control unit 110, an operation unit 140, a ROM (Read Only Memory) 150, a RAM (Random Access Memory) 160, and a reference movement DB (Data Base) 170.

[0016] The control unit 110 is configured with a CPU (Central Processing Unit) etc. The control unit 110 executes a program stored in the ROM 150 to function as an image acquisition unit 111, a skeleton estimation unit 112, a correction unit 113, and a comparison unit 114.

[0017] The image acquisition unit 111 acquires image data of the user U captured by the imaging unit 120. For example, the image acquisition unit 111 acquires image data of the entire body of the user U captured by the imaging unit 120. The image data includes two-dimensional or three-dimensional image data. If the image data is two-dimensional image data, distance data indicating the distance to the user U is further acquired. The acquired image data and distance data are stored in the RAM 160.

[0018] The skeleton estimation unit 112 estimates the three-dimensional skeleton of the user U based on the image data acquired by the image acquisition unit 111. As shown in FIG. 3, the skeleton estimation unit 112 detects feature points FP1 to FPN, such as the person's face and joints, from a two-dimensional image or a three-dimensional image, through clothing, to detect the person's posture, and estimates the three-dimensional skeleton including the three-dimensional position coordinates of the feature points FP1 to FPN. Specifically, the feature point FP1 indicates the position of the right wrist, the feature point FP2 indicates the position of the right elbow, and the feature point FP3 indicates the position of the right shoulder. Next, the skeleton estimation unit 112 estimates the length of the user U's skeleton. Specifically, the skeleton estimation unit 112 estimates the length between adjacent feature points FPn. For example, the skeleton estimation unit 112 estimates the length from the feature point FP1 to the feature point FP2. When the imaging unit 120 is a 2D camera that captures two-dimensional images, the skeleton estimation unit 112 estimates the length between adjacent feature points FPn based on the distance from the imaging unit 120 to the user U and the distance between adjacent feature points FPn on the image. When the imaging unit 120 is a 3D camera or LiDAR that captures three-dimensional images, the skeleton estimation unit 112 estimates the length between adjacent feature points FPn based on three-dimensional position coordinates included in the image data.

[0019] The correction unit 113 adjusts the skeletal length of the reference movement to match the skeletal length of the user U. Specifically, data indicating the reference movement is acquired from the reference movement DB 170. For example, the correction unit 113 acquires data indicating the reference movement shown in FIG. 4 stored in the reference movement DB 170 based on an instruction to select the type of reference movement input by the user to the operation unit 140. For example, a dance movement is acquired as the reference movement. Next, the correction unit 113 adjusts the length between the reference points RPn of the reference movement shown in FIG. 4 to match the skeletal length of the reference movement to that of the user U. This allows data indicating the adjusted reference movement to be obtained. For example, the length from the reference point RP1 to the reference point RP2 is adjusted to be the same as the length from the feature point FP1 to the feature point FP2. This makes it possible to appropriately compare the reference movement with the movement of the user U even if the skeletal length in the reference movement differs from that of the user U.

[0020] The comparison unit 114 calculates the amount of deviation between the reference motion stored in the reference motion DB 170 and the motion of the user U, and projects and displays a marker indicating the calculated amount of deviation onto the user U. Specifically, the comparison unit 114 superimposes feature points FPn in the motion of the user U on reference points RPn in the adjusted reference motion. Specifically, feature point FP1 indicating the right wrist, feature point FP2 indicating the right elbow, and feature point FP3 indicating the right shoulder of the user U are superimposed on reference points RP1 indicating the right wrist, reference point RP2 indicating the right elbow, and reference point RP3 indicating the right shoulder, respectively, in the reference motion. The other feature points FPn are superimposed on the corresponding reference points RPn. At this time, the sum of the differences between feature points FPn and reference points RPn for n=1 to N is calculated, and the feature points FPn and reference points RPn are superimposed so as to minimize this difference. In this way, it is possible to make feature points FPn that deviate from the reference point RPn stand out. Next, the comparison unit 114 calculates the amount of deviation between the feature point FPn in the user U's motion and the reference point RPn in the adjusted reference motion based on the three-dimensional position coordinates of the feature point FPn and the three-dimensional position coordinates of the reference point RPn in the adjusted reference motion. For example, the comparison unit 114 calculates the amount of deviation between the feature point FP1 representing the right wrist and the reference point RP1. If the amount of deviation is less than a first reference value, it is referred to as a first deviation amount. If the amount of deviation is less than a second reference value equal to or greater than the first reference value, it is referred to as a second deviation amount. If the amount of deviation is equal to or greater than the second reference value, it is referred to as a third deviation amount. Next, the comparison unit 114 controls the display unit 130 to project and display a marker indicating the calculated amount of deviation to the user U. In this example, the marker is displayed in a display color corresponding to the amount of deviation. As shown in FIG. 5, a display color corresponding to the amount of deviation is projected and displayed for each feature point of the user U by the projector of the display unit 130. Specifically, the display unit 130 projects a first marker 10 indicating a first amount of misalignment onto the right shoulder, a second marker 20 indicating a second amount of misalignment onto the right elbow, and a third marker 30 indicating a third amount of misalignment onto the right wrist. To make it easy to distinguish between the different markers, the display color of the first marker 10 is blue, the display color of the second marker 20 is yellow, and the display color of the third marker 30 is red. This makes it easy for the user U to understand the amount of misalignment from the colors. The size of the markers is set based on the magnitude of the amount of misalignment.Specifically, by making the size of the second marker 20 indicating a second amount of deviation larger than the first amount of deviation larger than the first marker 10, and making the size of the third marker 30 indicating a third amount of deviation larger than the second amount of deviation larger than the second marker 20, the markers are projected onto the user U even when projected onto an area with a large amount of deviation.

[0021] The operation unit 140 receives instructions to start and end the action learning assistance process based on user input. The operation unit 140 also receives instructions to select the type of reference action stored in the reference action DB 170. The operation unit 140 may be a keyboard, a mouse, or the like, may receive operations through voice recognition, or may be a touch panel display device equipped with a display such as an LCD (liquid crystal display) that displays images required for operations.

[0022] The ROM 150 is made up of a non-volatile memory such as a flash memory, and stores programs for the control unit 110 to realize various functions as described above. The RAM 160 is made up of a volatile memory, and is used as a work area for the control unit 110 to execute programs for performing various processes. The RAM 160 also stores information such as image data of the user U.

[0023] The reference motion DB 170 stores data indicating the reference motions that the user U wants to learn. The reference motion DB 170 is required to store at least one reference motion, and may store multiple reference motions, such as a reference motion for dance 1 and a reference motion for dance 2. The reference motion is not particularly limited as long as it is a human body motion, and includes dance, sports, and other motions. As shown in FIG. 4, the data indicating the reference motions includes data indicating the three-dimensional position coordinates of reference points RP1 to RPN indicating joints such as the wrist and elbow, stored every predetermined number of seconds (e.g., 0.1 seconds). Specifically, reference point RP1 indicates the position of the right wrist, reference point RP2 indicates the position of the right elbow, and reference point RP3 indicates the position of the right shoulder. The data indicating the reference motions may be data obtained by motion capture of a professional dancer or the like, or may be data obtained by computer analysis of the three-dimensional position coordinates of reference points RP1 to RPN.

[0024] Next, the motion learning assistance process executed by the motion learning assistance device 100 will be described.

[0025] In response to an instruction from the user U to start the motion learning assistance process, the motion learning assistance device 100 starts the motion learning assistance process shown in FIG.

[0026] First, the correction unit 113 acquires data indicating a reference movement from the reference movement DB 170 (step S101). For example, the correction unit 113 acquires data indicating a reference movement stored in the reference movement DB 170 based on an instruction to select the type of reference movement input by the user to the operation unit 140. In this example, a dance movement is acquired as the reference movement.

[0027] Next, the image acquisition unit 111 acquires image data of the user U shown in FIG. 3 captured by the imaging unit 120 (step S102). For example, the image acquisition unit 111 acquires image data of the entire body of the user U captured by the imaging unit 120. The image data includes two-dimensional or three-dimensional image data. If the image data is two-dimensional image data, distance data indicating the distance to the user U is further acquired. The acquired image data and distance data are stored in the RAM 160.

[0028] Next, the skeleton estimation unit 112 estimates the three-dimensional skeleton of the user U based on the acquired image data (step S103). As shown in Fig. 3, the skeleton estimation unit 112 detects feature points FP1 to FPN, such as the person's face and joints, from the two-dimensional image or three-dimensional image through clothing to detect the person's posture, and estimates the three-dimensional skeleton including the three-dimensional position coordinates of the feature points FP1 to FPN. Specifically, the feature point FP1 indicates the position of the right wrist, the feature point FP2 indicates the position of the right elbow, and the feature point FP3 indicates the position of the right shoulder.

[0029] Next, the skeleton estimation unit 112 estimates the length of the skeleton of the user U (step S104). Specifically, the skeleton estimation unit 112 estimates the length between adjacent feature points FPn. For example, the skeleton estimation unit 112 estimates the length from feature point FP1 to feature point FP2. When the imaging unit 120 is a 2D camera that captures two-dimensional images, the skeleton estimation unit 112 estimates the length between adjacent feature points FPn based on the distance from the imaging unit 120 to the user U and the distance between adjacent feature points FPn on the image. When the imaging unit 120 is a 3D camera or LiDAR that captures three-dimensional images, the skeleton estimation unit 112 estimates the length between adjacent feature points FPn based on three-dimensional position coordinates included in the image data.

[0030] Next, the correction unit 113 adjusts the length between the reference points RPn in the reference motion to match the skeletal length in the reference motion to that of the user U (step S105). This makes it possible to obtain data representing the adjusted reference motion. For example, the length from the reference point RP1 to the reference point RP2 shown in FIG. 4 is adjusted to be the same as the length from the feature point FP1 to the feature point FP2. This makes it possible to appropriately compare the reference motion with the motion of the user U even if the skeletal length in the reference motion differs from that of the user U.

[0031] Next, the comparison unit 114 superimposes the feature points FPn in the user U's movements on the reference points RPn in the adjusted reference movements (step S106). Specifically, the feature points FP1, FP2, and FP3 representing the user U's right wrist, right elbow, and right shoulder are superimposed on the reference points RP1, RP1, and RP3 representing the right wrist, right elbow, and right shoulder, respectively, representing the reference movements. The other feature points FPn are superimposed on the corresponding reference points RPn. At this time, the sum of the differences between the feature points FPn and the reference points RPn for n=1 to N is calculated, and the feature points FPn and the reference points RPn are superimposed on the reference points RPn so that this difference is minimized. In this way, it is possible to highlight the feature points FPn that deviate from the reference points RPn. Note that the feature points FPn representing specific body parts of the user U may be aligned with the reference points RPn corresponding to the specific body parts in the reference movements, and the feature points FPn and the reference points RPn may be superimposed on the reference points RPn. In this case, for example, a feature point FPn indicating a specific part of the user U, such as the foot or waist, is matched with a reference point RPn corresponding to a specific part of the user U, such as the foot or waist, in the reference motion, and the feature point FPn and the reference point RPn are superimposed. By superimposing the feature point FPn and the reference point RPn on the specific part, the foot or waist, which is considered to be in a fixed position, it is possible to make the feature point FPn that is deviated from the reference point RPn stand out.

[0032] The comparison unit 114 calculates the amount of deviation between the feature point FPn in the motion of the user U and the reference point RPn in the adjusted reference motion based on the three-dimensional position coordinates of the feature point FPn and the three-dimensional position coordinates of the reference point RPn in the adjusted reference motion (step S107). For example, the amount of deviation between the feature point FP1 indicating the right wrist and the reference point RP1 is calculated. If the amount of deviation is less than a first reference value, it is referred to as a first deviation amount; if it is less than a second reference value that is equal to or greater than the first reference value but greater than the first reference value, it is referred to as a second deviation amount; and if it is equal to or greater than the second reference value, it is referred to as a third deviation amount.

[0033] The comparison unit 114 determines whether or not the amounts of deviation for all feature points FPn in the movements of the user U have been calculated (step S108).

[0034] If there remains a feature point FPn for which the amount of deviation has not been calculated (step S108; No), the process returns to step S107 and steps S107 to S108 are repeated so as to calculate all the amounts of deviation from feature point FP1 to reference point RPN.

[0035] When it is determined that the displacement amounts of all feature points FPn in the user U's movements have been calculated (step S108; Yes), the comparison unit 114 controls the display unit 130 to project and display markers indicating the displacement amounts in real time onto the user U based on the displacement amounts calculated in step S107 (step S109). In this example, the markers are displayed in colors corresponding to the displacement amounts. As shown in FIG. 5, the display unit 130, which is a projector, projects and displays a color corresponding to the displacement amount for each feature point of the user U. Specifically, the display unit 130 projects a first marker 10 indicating a displacement amount of 1 onto the right shoulder, a second marker 20 indicating a displacement amount of 2 onto the right elbow, and a third marker 30 indicating a displacement amount of 3 onto the right wrist. To make it easy to distinguish between the types of markers, the display color of the first marker 10 is blue, the display color of the second marker 20 is yellow, and the display color of the third marker 30 is red. This makes it easy for the user U to grasp the displacement amount from the colors. The size of the marker is set based on the magnitude of the displacement. Specifically, the size of the second marker 20 indicating a second displacement amount greater than the first displacement amount is made larger than the size of the first marker 10, and the size of the third marker 30 indicating a third displacement amount greater than the second displacement amount is made larger than the size of the second marker 20. This allows the marker to be projected onto the user U even when projected onto a portion with a large displacement amount. Note that, if the display unit 130 cannot project the marker indicating the displacement amount onto the part of the user U corresponding to the feature point FPn, the display unit 130 may project the marker indicating the displacement amount onto a portion close to the part of the user U corresponding to the feature point FPn. For example, if the display unit 130 cannot project the marker onto the wrist, it may project it onto the arm.

[0036] Next, the motion learning assistance device 100 determines whether or not an instruction to end the motion learning assistance process has been received (step S110). If the instruction to end the motion learning assistance process has not been received (step S110; No), the process returns to step S102, and steps S102 to S110 are repeated to capture an image of the motion of the user U and project the amount of deviation between the motion of the user U and the reference motion onto the user U in real time.

[0037] When an instruction to end the movement learning support process is received (step S110; Yes), the movement learning support device 100 ends the movement learning support process.

[0038] As described above, according to the movement learning assistance device 100 of the present embodiment, by projecting a marker indicating the amount of deviation onto the user U in real time, the user U can visually recognize the amount of deviation and easily grasp the correct posture. Furthermore, the user U can practice a movement while always knowing the amount of deviation, and can search for the correct position in space by feel. Another advantage is that the user U can directly search for the correct position in three-dimensional space, rather than grasping the amount of deviation by looking at a screen. This allows the user U to easily learn the movement they want to learn. Furthermore, by color-coding the amount of deviation, the user U can easily grasp the amount of deviation. Furthermore, by projecting and displaying a marker indicating the amount of deviation onto the user U, the user U can easily grasp the parts with large amounts of deviation. Furthermore, a display color corresponding to the amount of deviation from the reference movement is projected onto the user U while the user U is learning the movement. By recording the user U with the display color corresponding to the amount of deviation projected onto him / her using another camera and reviewing the recorded images, it becomes possible to check which parts of the movement have the most deviations by the display color.

[0039] (Variation) In the above-described embodiment of the motion learning assistance device 100, the image acquisition unit 111 acquires image data of the entire body of the user U, and the comparison unit 114 calculates the amount of deviation in the user U's entire body movements. Alternatively, the image acquisition unit 111 may acquire image data of a portion of the user U, and the comparison unit 114 may calculate the amount of deviation in the portion of the user U's movements. In this way, when there is not enough space to practice or when large movements such as dancing cannot be performed, the image acquisition unit 111 can acquire image data of the upper body of the user U, and the comparison unit 114 can calculate the amount of deviation in the upper body movements of the user U. This allows the amount of deviation in only the upper body of the user U to be determined, as shown in FIG. 7 . Similarly, it is also possible to determine the amount of deviation in only the lower body movements in a lower body determination mode. The upper body determination mode is used when practicing arm and hand choreography indoors, etc. The lower body determination mode is used when practicing the position and movement of the feet. It is also possible to run the "upper body determination mode" and the "lower body determination mode" simultaneously in parallel. In this case, if the movement of the lower body is out of sync, the upper body will also be out of sync relatively, but even if there is a difference in the amount of movement of the lower body, it is possible to ignore this and determine only the movement of the upper body.

[0040] In the above-described embodiment of the motion learning support device 100, the comparison unit 114 projects the deviation between the user U's motion and the reference motion to the user U in real time. The comparison unit 114 may pause the motion learning support process shown in FIG. 6 at a checkpoint where a motion, such as a signature pose, is checked during the motion and continue to project the deviation at the checkpoint to the user U. In this case, the data indicating the reference motion includes data indicating one or more checkpoints. The display unit 130 displays the deviation in real time up to the checkpoint, pauses at the checkpoint, and continues to display the deviation at the checkpoint. For example, if the pose shown in FIG. 5 is set as a checkpoint, the display unit 130 continues to display the deviation in this state. In this manner, the motion learning support process is paused during practice, allowing the user U to easily check the motion at a signature pose or the like. The pause is canceled by a user operation input to the operation unit 140. For example, the operation unit 140 may receive a signal indicating resumption via voice input.

[0041] In the above-described embodiment of the motion learning assistance device 100, an example has been described in which the display unit 130 includes a projector that projects and displays an image. The display unit 130 is not particularly limited as long as it can display the amount of misalignment, and may include a display panel such as a liquid crystal display panel or an organic EL (Electro Luminescence) panel. In this case, as shown in FIG. 8 , the display unit 130 displays a marker indicating the amount of misalignment superimposed on the image of the user U captured by the imaging unit 120. Even in this case, the user U can visually recognize the amount of misalignment. In this example, the marker corresponding to the amount of misalignment is displayed as a score. This allows the user U to recognize which position has the largest misalignment. The display unit 130 may also display the direction of misalignment at each position. For example, the marker may include "up," "down," "right," "left," "front," or "back." This allows the user U to understand which position has the largest misalignment and in which direction. The display unit 130 may also display a value indicating the amount of misalignment. This allows the user U to understand the direction and extent of the misalignment. For example, the right wrist is shifted upward by 3 cm.

[0042] Furthermore, the core part of the motion learning assistance process executed by motion learning assistance device 100, which is composed of a CPU, RAM, ROM, etc., can be executed using an ordinary mobile information terminal (smartphone, tablet PC), personal computer, etc., without relying on a dedicated system. For example, a computer program for executing the above-described operations may be stored and distributed on a computer-readable recording medium (such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), etc.), and this computer program may be installed on a mobile information terminal, etc., to configure an information terminal that executes the above-described process. Furthermore, this computer program may be stored in a storage device of a server device on a communication network such as the Internet, and an ordinary information processing terminal, etc., may download the computer program to configure an information processing device.

[0043] In addition, when the functions of the motion learning assistance device 100 are realized by sharing the functions between an OS (Operating System) and an application program, or by collaboration between the OS and an application program, only the application program portion may be stored on a recording medium or storage device.

[0044] It is also possible to superimpose a computer program on a carrier wave and distribute it over a communications network. For example, the computer program may be posted on a bulletin board system (BBS) on the communications network and distributed over the network. The computer program may then be started and executed under the control of an OS in the same way as other application programs, thereby enabling the above-described processing to be performed.

[0045] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to such specific embodiments, and the present invention includes the inventions described in the claims and their equivalents. [Explanation of symbols]

[0046] 10...first sign, 20...second sign, 30...third sign, 100...motion learning support device, 110...control unit, 111...image acquisition unit, 112...skeleton estimation unit, 113...correction unit, 114...comparison unit, 120...imaging unit, 130...display unit, 140...operation unit, 150...ROM, 160...RAM, 170...reference movement DB, U...user, RP1, RP2, RP3, RPn, RPN...reference points, FP1, FP2, FP3, FPn, FPN...feature points

Claims

1. an imaging unit that images a user; a skeleton estimation unit that estimates a plurality of feature points that indicate a skeleton of the user based on an image of the user captured by the imaging unit; a comparison unit that calculates the amount of deviation between a plurality of reference points that indicate a skeleton of a pre-stored reference movement and the plurality of feature points that correspond to the plurality of reference points, respectively; a display unit that displays a plurality of markers corresponding to the plurality of deviation amounts calculated by the comparison unit; A movement learning assistance device comprising:

2. the display unit displays the plurality of markers in correspondence with the plurality of feature points, respectively.

2. The movement learning support device according to claim 1.

3. the display unit includes a projector, and projects and displays the plurality of markers to the user in correspondence with the plurality of feature points, respectively.

3. The movement learning assistance device according to claim 2.

4. The indicator is displayed in a display color according to the amount of deviation.

2. The movement learning support device according to claim 1.

5. The size of the mark is set based on the amount of deviation.

2. The movement learning support device according to claim 1.

6. The display unit displays the sign by superimposing it on an image of the user captured by the imaging unit.

2. The movement learning support device according to claim 1.

7. a correction unit that corrects a length between the reference points in the reference motion so that the length between the reference points in the reference motion and the length between the feature points of the user become the same; 2. The movement learning support device according to claim 1.

8. an imaging step of imaging a user; a skeleton estimation step of estimating a plurality of feature points indicating a skeleton of the user based on the image of the user captured in the imaging step; a comparison step of calculating deviation amounts between a plurality of reference points indicating a skeleton of a pre-stored reference movement and the plurality of feature points corresponding to the plurality of reference points, respectively; a display step of displaying a plurality of markers corresponding to the plurality of deviation amounts calculated in the comparison step; A movement learning assistance method comprising:

9. a computer that controls an imaging unit that images a user and a display unit; a skeleton estimation unit that estimates a plurality of feature points that indicate a skeleton of the user based on an image of the user captured by the imaging unit; a comparison unit that calculates amounts of deviation between a plurality of reference points indicating a skeleton of a reference movement stored in advance and the plurality of feature points corresponding to the plurality of reference points, respectively, and causes the display unit to display signs corresponding to the calculated amounts of deviation; A program that functions as a

Images