Movement evaluation system and movement evaluation method

Abstract

A movement evaluation system and a movement evaluation method are provided that are capable of evaluating with high accuracy the similarity of the movement between two subjects or measurement targets that are displaced in accordance with sound information. [Solution] The apparatus includes a sound information output unit 16 that outputs sound information, an imaging unit 13 that captures an image of a subject 1 that is displaced based on the sound information, and a control unit 10, and the control unit 10 acquires motion information of the subject 1 from the output of the imaging unit 13 based on a predetermined timing extracted from the sound information. The control unit 10 then derives a similarity between the motions of the first subject 1a and the second subject 1b based on the motion information of the first subject 1a and the second subject 1b detected based on the predetermined timing extracted from the sound information.

Description

[Technical field]

[0001] The present invention relates to a motion evaluation system and a motion evaluation method for evaluating the similarity of a subject's motion, in particular, the motion between two subjects. [Background technology]

[0002] It is known that dancing, which involves full-body exercise, contributes to improving cardiopulmonary function and muscle strength and balance. By incorporating dance into daily life, it is expected that the risk of falls and injuries, for example, will be reduced in the elderly. It is also expected that learning new dance steps and moving the body to the rhythm of music will improve spatial awareness, memory, attention, etc., in children with developmental disabilities and the elderly.

[0003] In dance lessons, students gradually learn the movements by imitating the steps of the instructor. In this process, by evaluating the similarity (match) of the movements between the instructor and the student and reflecting this in the instruction, it is possible to improve the student's dance performance and further improve motor function. In addition, by incorporating dance into rehabilitation, evaluating the similarity of movements between the doctor or physical therapist and the care recipient, and reflecting this in the care plan, such as how to apply stress, it is thought that rehabilitation can be advanced more efficiently.

[0004] A known technique for evaluating the similarity of subjects appearing in an image includes, for example, a method including the steps of acquiring a digital image capturing an environment including at least a first subject, segmenting a first portion of the digital image depicting the first subject into a plurality of superpixels, for each of the plurality of superpixels, assigning a semantic label to the superpixel, extracting features of the superpixel, and determining a similarity metric between the features extracted from the superpixels and features extracted from a reference superpixel identified in a reference digital image, where the reference superpixel is given a reference semantic label that matches the semantic label assigned to the superpixel, and determining that the first subject appears in the reference image based on a plurality of similarity metric associated with the plurality of superpixels (Patent Document 1).

[0005] According to the patent, it may be determined that a first object is present in a reference image based on a number of similarity measures associated with a number of superpixels. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2021-531539 DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0007] However, while the technology disclosed in Patent Document 1 targets identifying people in digital images and makes it possible to identify people in digital images by combining features such as clothing, accessories, hair, and face, it does not suggest evaluating the similarity in the movements of two subjects that displace in accordance with sound information.

[0008] The present invention has been devised to solve such problems in the prior art, and its purpose is to provide a movement evaluation system and a movement evaluation method that are capable of accurately evaluating the similarity in movement between two subjects or measurement targets that displace in accordance with sound information. [Means for solving the problem]

[0009] The present invention, which has been made to solve the above problems, is a motion evaluation system that includes a sound information output unit that outputs sound information, an imaging unit that images a subject that displaces based on the sound information, and a control unit, where the control unit acquires motion information of the subject from the output of the imaging unit based on a predetermined timing extracted from the sound information. This makes it possible to accurately acquire motion information of the subject that displaces based on the sound information.

[0010] In the present invention, the subject includes a first subject and a second subject, and the control unit derives a similarity between the movements of the first subject and the second subject based on the movement information of the first subject and the second subject detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity between the movements of the first subject 1a and the second subject 1b that are displaced based on the sound information.

[0011] In addition, in the present invention, the control unit calculates at least one representative position information representative of each of the position information of the first subject and the second subject, and derives the similarity based on a time-series change in the movement information extracted from the representative position information. This makes it possible to calculate the similarity with high accuracy and high speed without processing a large amount of movement information.

[0012] In addition, in the present invention, the control unit generates a first motion information image and a second motion information image representing a time-series change in the motion information for each of the first subject and the second subject, and derives the similarity based on the first motion information image and the second motion information image. This makes it possible to represent the motion (trajectory) of a position representative of the subject as a two-dimensional image and derive the similarity based on the difference between the images.

[0013] In addition, the present invention renders the motion information in the first motion information image and the second motion information image as an object of a predetermined size exceeding one pixel size. This makes it possible to appropriately obtain the similarity even when the number of pieces of representative position information is small.

[0014] In addition, in the present invention, the control unit detects a beat or rhythm contained in the sound information, and extracts peak values ​​of the representative position information before and after the timing based on the detected beat or rhythm as the movement information. This makes it possible to accurately obtain movement information even for a subject whose movement is intentionally out of sync with the beat.

[0015] In the present invention, the sound information is music, and the control unit determines the predetermined timing based on a change in sound pressure of the sound information, thereby making it possible to unify the timing of acquiring movement information for a first subject and a second subject performing a dance to music.

[0016] In addition, in the present invention, the control unit determines that the predetermined timing is when the sound pressure of the sound information exceeds a predetermined value, or when a change in the sound pressure of the sound information exceeds a predetermined value, thereby making it possible to easily obtain the timing for acquiring the movement information.

[0017] The present invention is also a motion evaluation system that includes a sound information output unit that outputs sound information, an imaging unit that captures a first subject and a second subject, and a control unit, the first subject and the second subject are displaced based on the sound information output by the sound information output unit, the control unit calculates at least one representative position information representative of each of the position information of the first subject and the second subject, and further generates a first motion information image and a second motion information image that represent time-series changes in each of the representative position information, and derives a similarity based on the first motion information image and the second motion information image. This makes it possible to express the motion (trajectory) of a position representative of the subject as a two-dimensional image and derive a similarity based on the difference between the images.

[0018] In addition, in the present invention, the control unit calculates a structural similarity index (SSIM) as the similarity based on the first motion information image and the second motion information image, which makes it possible to derive the similarity taking into account the characteristics of the human visual system.

[0019] The present invention also provides a motion evaluation system that includes a sound information output unit that outputs sound information, a motion detection unit that detects the motion of a measurement object displaced based on the sound information, and a control unit, the control unit acquiring the motion information of the measurement object based on a predetermined timing extracted from the sound information, thereby making it possible to accurately acquire the motion information of the measurement object displaced based on the sound information.

[0020] In addition, in the present invention, the measurement target includes a first measurement target and a second measurement target, and the control unit derives a similarity between the movements of the first measurement target and the second measurement target based on the movement information of the first measurement target and the second measurement target detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity between the movements of the first measurement target and the second measurement target that are displaced based on the sound information.

[0021] The present invention also provides a motion evaluation method that outputs sound information, captures an image of a subject that moves based on the sound information, and acquires motion information of the subject based on a predetermined timing extracted from the sound information, thereby making it possible to accurately acquire motion information of a subject that moves based on the sound information.

[0022] In addition, in the present invention, the subject includes a first subject and a second subject, and the similarity between the movements of the first subject and the second subject is derived based on the movement information between the first subject and the second subject. This makes it possible to accurately evaluate the similarity between the movements of the first subject and the second subject that are displaced based on sound information.

[0023] The present invention also provides a movement evaluation method for detecting the movement of a measurement object displacing based on sound information and acquiring information on the movement of the measurement object based on a predetermined timing extracted from the sound information, thereby making it possible to accurately acquire information on the movement of the measurement object displacing based on sound information.

[0024] In addition, the present invention is such that the measurement target includes a first measurement target and a second measurement target, and the similarity between the movements of the first measurement target and the second measurement target is derived based on the movement information of the first measurement target and the second measurement target detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity between the movements of the first measurement target and the second measurement target that are displaced based on the sound information. Effect of the Invention

[0025] In this way, according to the present invention, it is possible to accurately evaluate the degree of similarity in movement between two moving subjects or between two measurement targets. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a block diagram showing a configuration of a movement estimation system S1 according to a first embodiment of the present invention. [Diagram 2]FIG. 1A is an explanatory diagram showing a usage mode of the motion evaluation system S1, and FIG. 1B to FIG. 1D are explanatory diagrams explaining pre-processing in the motion evaluation system S1. [Diagram 3] Illustration of pose recognition model 40 [Figure 4] FIG. 1A is an explanatory diagram showing an example of an image of a subject 1, and FIG. 1B is an explanatory diagram showing a key point 41 and a center of gravity CGa of a whole body 1AL in the image of the subject 1. [Diagram 5] Graph (A) shows the movement of the first subject 1a in the x direction, graph (B) shows the movement of the second subject 1b in the x direction, and graph (C) shows the movement of the first subject 1a in the x direction normalized within the range of ±1. [Figure 6] 1A and 1B are diagrams illustrating the timing of acquiring motion information of a subject 1. [Figure 7] 13A and 13B are diagrams illustrating a process of deriving a similarity in a second embodiment of the present invention. [Figure 8] FIG. 11 is an explanatory diagram for explaining a method for visualizing the movement of a subject 1 in a third embodiment of the present invention. [Figure 9] FIG. 11 is a block diagram showing the configuration of a movement estimation system S1 according to a fourth embodiment of the present invention. [Figure 10] Block diagram showing the configuration of the motion detection unit 3 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] (First embodiment) A first embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a movement evaluation system S1 according to the first embodiment of the present invention. The movement evaluation system S1 is composed of a control unit 10, a display unit 15, an imaging unit 13, and a sound information output unit 16. A sound information acquisition unit 17 is provided as necessary, as described later.

[0028] The control unit 10 is composed of a calculation unit 10a, a storage unit 10b, and a communication unit 10c. The calculation unit 10a is composed of a CPU (Central Processing Unit) and the like. The storage unit 10b is composed of a ROM (Read Only Memory), a RAM (Random access memory), and the like, and the calculation unit 10a operates according to a control program stored in the storage unit 10b. The storage unit 10b includes a non-volatile memory (such as an EEPROM (Electrically Erasable Programmable Read-Only Memory)). The non-volatile memory stores music files used to generate sound information. The storage unit 10b may include a so-called storage (large-capacity storage device) such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and the control program and music files may be stored in the storage. The calculation unit 10a and the other components are connected by a bus 20 or the like, and the calculation unit 10a controls the other components via the bus 20 or the like.

[0029] The control unit 10 may be configured, for example, as a PC (Personal Computer) or a server. The display unit 15 may be separate from the control unit 10, or may be configured integrally with the control unit 10, such as a tablet terminal or a notebook PC. The communication unit 10c includes a communication module (not shown) conforming to a wireless communication standard, such as LTE, LTE-M, 4G, or 5G. Furthermore, the communication unit 10c may include a communication module (not shown) conforming to a short-range wireless communication standard, such as BLE (Bluetooth (registered trademark) Low Energy). The communication unit 10c establishes communication with the imaging unit 13, the sound information output unit 16, and the sound information acquisition unit 17, and is capable of transmitting and receiving information between the control unit 10 and these units. Of course, these units may be connected by wire.

[0030] The imaging unit 13 includes an image sensor including a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD). For example, a camera included in an information terminal such as a smartphone may be used as the imaging unit 13. In this case, the information terminal may include a communication module (not shown) that complies with a predetermined wireless communication standard, and may transmit image data to the control unit 10 via a network 50. The imaging unit 13 may be integrated with the control unit 10, in which case the imaging unit 13 transmits image data to the calculation unit 10a via a bus 20.

[0031] The sound information output unit 16 includes an amplifier, a speaker, etc., which are not shown. The control unit 10 generates an analog audio signal based on a music file stored in the storage unit 10b, and outputs the signal to the sound information output unit 16. The sound information output unit 16 reproduces sound information (music, songs) based on the analog sound signal. The sound information may be reproduced through a speaker in a room where the subject 1 is dancing, or may be reproduced through a device such as wireless headphones worn by the subject 1. Here, the sound information refers to any one of digital data constituting a music file, analog data obtained by decoding the digital data, and sound output from the sound information output unit 16 based on the analog data.

[0032] The subject 1 may be fitted with a sound information acquisition unit 17. The sound information acquisition unit 17 includes a microphone, an AD converter, and a communication module conforming to a short-distance wireless communication standard (none of which are shown). In this embodiment, the sound information reproduced by the sound information output unit 16 is acquired and digitized by the sound information acquisition unit 17, and transmitted to the control unit 10 by short-distance wireless communication. The control unit 10 may store the received sound information as a music file in the storage unit 10b, and use this when detecting motion or deriving similarity, which will be described later. As a result, even if the subject 1 and the sound information output unit 16 are far apart from each other, by fitting the sound information acquisition unit 17 to the subject 1, it is possible to eliminate the effect of a time delay until the sound information reaches the subject 1.

[0033] Here, subject 1 is, for example, a human being. Subject 1 performs a dance according to a predetermined choreography in time with the sound information (music) output from sound information output unit 16. The content of the sound information may be selected at will, and it is particularly preferable to select music with a clear beat, rhythm, or tempo. The dance choreography may also be selected at will, and it is particularly preferable to select choreography that increases the movement (displacement) of subject 1 at the timing when the beat occurs (on-beat). Note that sound information is not limited to music or musical pieces, and a sound source that emits periodic sounds, such as a metronome, may be used.

[0034] The imaging unit 13 captures the subject 1 performing the dance. The control unit 10 temporarily stores the image data received from the imaging unit 13 in the storage unit 10b. If the imaging unit 13 has a function of storing image data in a portable storage medium, the control unit 10 may read the image data stored in the portable storage medium and store it in the storage unit 10b. The control unit 10 extracts movement information of the subject 1 using the stored image data. Here, the subject 1 includes a first subject 1a and a second subject 1b. The control unit 10 derives the similarity of the movements of the first subject 1a and the second subject 1b based on the movement information.

[0035] 2(A) is an explanatory diagram showing a usage mode of the movement evaluation system S1, and (B) to (D) are explanatory diagrams explaining pre-processing in the movement evaluation system S1. In the following explanation, the side of the subject 1 viewed in FIG. 2(B) may be referred to as the front, the opposite direction as the rear, the direction of the right arm as the right, the direction of the left arm as the left, the direction of the head 1HD as the top, and the opposite direction as the bottom. In FIG. 2(A), the sound information output unit 16 includes a speaker (not shown), and the sound information output unit 16 and the subject 1 are spatially separated by a predetermined distance. Of course, the arrangement position of the sound information output unit 16 may be determined arbitrarily.

[0036] As for the positional relationship between the subject 1 and the imaging unit 13, it is preferable that the angle θ between a line extending forward from the front of the subject 1 and the optical axis AxL of the imaging unit 13 is in the range of 10°≦θ≦45° [deg], for example. This reduces occlusion during shooting, and many of the pose landmarks described below can be acquired with high reliability. The distance L (or the angle of view) between the subject 1 and the imaging unit 13 may be determined arbitrarily, but it is preferable to determine the distance L (or the angle of view) taking into consideration the range in which the subject 1 moves in dancing, and to capture the whole body 1AL (see FIG. 3) of the subject 1 in a state in which the subject 1 has both arms 1A raised (FIG. 2(B)) and in a state in which the arms 1A are spread out to the left and right (FIG. 2(C)).

[0037] Hereinafter, the description will be continued with reference to FIG. 1. In the pre-processing of the movement evaluation system S1, after adjusting the angle of view of the imaging unit 13, the subject 1 is photographed in the postures of FIG. 2(B) to (D) before the subject 1 starts to move in sync with the dance. The control unit 10 measures the distance between the hand and foot (first hand-foot distance H1a) when the subject 1 (here, the first subject 1a) is in a posture with both arms 1A raised (so-called "Voice!" posture) shown in FIG. 2(B) based on the image data received from the imaging unit 13. In addition, the control unit 10 measures the distance between the hands (first hand-foot distance Wa) when the first subject 1a spreads the arms 1A to the left and right as shown in FIG. 2(C). Furthermore, the control unit 10 measures the distance (first head-foot distance H2a) between the top of the head 1HD and the foot when the first subject 1a is in an upright posture (so-called "Attention!" posture) as shown in FIG. 2(D). Note that a pose landmark, which will be described later, can be used to measure these distance information.

[0038] As with first subject 1a, second limb distance H1b (FIG. 2(B)), second hand-to-hand distance Wb (FIG. 2(C)), and second head-to-foot distance H2b (FIG. 2(D)) are measured for second subject 1b. These distance information items may be obtained by so-called 3D distance measurement, or may be measured manually with a tape measure or the like, and the numerical values ​​input to control unit 10 via an input unit (not shown). Based on the obtained distance information, control unit 10 calculates the following: (i) Width direction (x direction) correction coefficient SFx = first hand distance Wa / second hand distance Wb (ii) Height direction (y direction) correction coefficient SFy = 1st limb distance H1a / 2nd limb distance H1b (or SFy2 = 1st head-to-leg distance H2a / 2nd head-to-leg distance H2b) When evaluating the movement, the control unit 10 corrects the position information relating to the first subject 1a or the second subject 1b using these correction coefficients (details will be described later).

[0039] Fig. 3 is an explanatory diagram of the pose recognition model 40. In the first embodiment, MediaPipe Pose is used as the pose recognition model 40. As shown in Fig. 3, for an image (still image or video) of a person, MediaPipe Pose recognizes key points 41 (pose landmarks) from 0.nose to 32.right_foot_index (a total of 33 points), and outputs position information (coordinate values ​​(x, y coordinates)) of the recognized key points 41.

[0040] Using the obtained position information, the control unit 10 calculates the coordinates of the center of gravity representing the movement of the subject 1 for each of the whole body 1AL, the head 1HD, the torso 1BD, and the legs 1L. Specifically, using the position information of each key point 41, the control unit 10 calculates the following centers of gravity. Center of gravity of whole body 1AL CGa: Average value of each coordinate value from 0.nose to 32.right_foot_index Center of gravity of head 1HD CGh: Average value of each coordinate value from 0.nose to 12.right_shoulder Center of gravity CGb of torso 1BD: Average of each coordinate value from 11.left_shoulder to 24.right_hip Center of gravity CGl of leg 1L: Average value of each coordinate value from 23.left_hip to 32.right_foot_index

[0041] In the calculation of the center of gravity CGh of the head 1HD and the center of gravity CGb of the torso 1BD, 11.left_shouldeu and 12.right_shoulde are referred to, and in the calculation of the center of gravity CGb of the torso 1BD and the center of gravity CGl of the legs 1L, 23.right_hip and 24.left_hip are referred to. However, it is preferable to exclude the coordinate values ​​of the key points 41 that are not acquired due to occlusion or have low reliability in the calculation of each center of gravity. In the following description, the position information of the center of gravity may be referred to as "representative position information". In this way, in the first embodiment, the position information of the center of gravity described above is used as the representative position information, but the representative position information may be calculated based on the key points 41 that reflect the movement of the subject 1 more significantly according to the dance choreography. In addition, the first distance between both hands Wa or the distance between both feet (the distance between 31.left foot index and 32.right foot index) described above may be used instead of (or together with) the representative position information.

[0042] FIG. 4(A) is an explanatory diagram showing an example of an image of the subject 1, and FIG. 4(B) is an explanatory diagram showing key points 41 and the center of gravity CGa of the whole body 1AL in the image of the subject 1. The process from photographing the subject 1 to acquiring representative position information will be described below. The first subject 1a and the second subject 1b perform a dance with the same predetermined choreography to the same music output from the sound information output unit 16. First, as shown in FIG. 4(A), the subject 1 (first subject 1a or second subject 1b) performing the dance is photographed by the imaging unit 13.

[0043] Here, the first subject 1a and the second subject 1b may be photographed at different places and different times using different imaging units 13, or may be photographed simultaneously using the same imaging unit 13. The imaging unit 13 photographs the subject 1 in chronological order at a predetermined frame rate (e.g., 60 fps (frames per second)) and transmits the image data to the control unit 10.

[0044] The control unit 10 generates a video file based on the received image data and stores it in the storage unit 10b. Thereafter, the control unit 10 accesses the storage unit 10b to retrieve the image file, acquires key points 41 (and their coordinate values) from each frame image (evaluation image) constituting the image file, and calculates representative position information (here, the coordinate value of the center of gravity CGa of the entire body 1AL). Specifically, the control unit 10 processes the evaluation image using the API (Application Programming Interface) of the above-mentioned MediaPipe Pose. As a result, as shown in FIG. 4(B), a plurality of key points 41 are recognized for the subject 1 (here, the first subject 1a), and the x, y coordinates and representative position information of each key point 41 are calculated. Then, the subject 1, the key points 41, and the representative position information (center of gravity CGa) are displayed superimposed on the display unit 15, and further, the skeleton connecting the main key points 41 and the outer edge including the key points 41 are displayed as line segments. The evaluation image obtained by photographing the second subject 1b is also processed in the same manner.

[0045] FIG. 5(A) is a graph showing the movement of the first subject 1a in the x direction, FIG. 5(B) is a graph showing the movement of the second subject 1b in the x direction, and FIG. 5(C) is a graph showing the movement of the first subject 1a in the x direction normalized within the range of ±1. Here, the vertical axis (x direction) of FIG. 5(A)-(C) corresponds to the left-right direction of the subject 1 shown in FIG. 4(A)-(B), and the horizontal axis is the time axis t. As described above, each evaluation image is obtained discretely (periodically) in time series, so that the representative position information generated based on the evaluation image is also obtained discretely. However, in the graphs of FIG. 5(A)-(C), each representative position information is interpolated and drawn as a curve (similar to FIG. 6 described later). In addition, the representative position information uses the x coordinate value of the center of gravity CGa of the whole body 1AL described above.

[0046] The process of acquiring the movement information of the subject 1 will be described below. First, when photographing the subject 1, the control unit 10 matches the timing at which sound information is output from the sound information output unit 16 with the timing at which photographing is started by the imaging unit 13. A music file used to play the sound information may be prepared in advance. Of course, the sound information output from the sound information output unit 16 may be acquired (recorded) via the sound information acquisition unit 17 to create a music file, and in this case, the control unit 10 controls so that the timing at which photographing is started and the timing at which recording is started are the same.

[0047] In this way, a music file and a video file are obtained in which the start of music playback and the start of shooting are synchronized. If the distance from the sound information output unit 16 to the subject 1 is large and affects the evaluation of the movement (for example, if the time it takes for sound information to reach the subject 1 from the sound information output unit 16 exceeds 1 / 2 the period of the music beat), it is preferable to adjust the origin of the time axis t by adjusting the timestamp of the music file or the image file. In Figs. 5(A) to (C), the point in time when the subject 1 starts moving is set as the origin (0) of the time axis t, and the timestamp of the music file at this point in time is adjusted to 0.

[0048] The positional relationship between the imaging unit 13 and the first subject 1a (second subject 1b) is usually considered to be different every time shooting is performed, and the control unit 10 executes a process of matching the initial positions (here, the x coordinate value of the representative position information) of the first subject 1a and the second subject 1b. Specifically, when the initial position of the representative position information of the first subject 1a is 540 (FIG. 5(A)), an offset is given to the representative position information of the second subject 1b to match its initial value to 540 (FIG. 5(B) shows a graph after matching the initial positions). In this way, the control unit 10 executes a calibration to match the initial values ​​of the representative position information of the first subject 1a and the second subject 1b. This makes it possible to evaluate the movement with high accuracy even if the positions at which the first subject 1a and the second subject 1b start dancing are different, or even if the positions in the image are different. Note that the calibration is also executed for the y coordinate value of the representative position information in the same way.

[0049] Hereinafter, the time series change in the representative position information of the first subject 1a (i.e., FIG. 5(A)) may be referred to as CGax(x,t), and the time series change in the representative position information of the second subject 1b (i.e., FIG. 5(B)) may be referred to as CGax'(x',t'). Here, the x-coordinate value of the center of gravity CGa of the whole body 1AL is exemplified as the representative position information.

[0050] The control unit 10 calculates an average value of the representative position information in a time series for the first subject 1a based on CGax(x,t). Then, as shown in FIG. 5C, the average value is set to 0, and each representative position information is normalized to be within a range of ±1 (hereinafter, the representative position information after normalization may be referred to as "normalized representative position information"). Hereinafter, the normalized CGax(x,t) may be referred to as CGax_fin(x,τ). Furthermore, the control unit 10 similarly normalizes CGax'(x',t') for the second subject 1b. Hereinafter, the normalized CGax'(x',t') may be referred to as CGax_fin'(x',τ') (see FIG. 6B).

[0051] When normalization is performed, the normalized representative position information may be corrected using the above-mentioned correction coefficient (here, SFx). Specifically, if the above-mentioned SFx=first hand-distance Wa / second hand-distance Wb=0.9, for example, the normalized representative position information of the second subject 1b is multiplied by 0.9. Of course, the normalized representative position information of the first subject 1a may be multiplied by 1 / 0.9. This makes it possible to eliminate the influence based on the difference in the body shape, build, etc. of each subject 1. When evaluating the movement of the subject 1 in the up-down direction (i.e., the y direction), the normalized representative position information may be corrected using SFy or SFy2.

[0052] 6(A) and (B) are explanatory diagrams showing the timing of acquiring the motion information of the subject 1. Here, FIG. 6(A) shows CGax_fin(x, τ) (see FIG. 5(C)) to which the timing (τ1 to τ14) extracted from the sound information, normalized representative position information (x1 to x14) adopted as the motion information, and the timing (τ1a, τ2a, etc.) at which the motion information was acquired are added. FIG. 6(B) shows CGax'(x', t') (see FIG. 5(B)) normalized in the vertical axis direction to ±1, and to which the timing (τ'1 to τ'14) extracted from the sound information, normalized representative position information (x'1 to x'14) adopted as the motion information, and the timing (τ'1a to τ'14a) at which the motion information was acquired are added.

[0053] Hereinafter, the explanation will be continued with reference to FIG. 1. The control unit 10 opens a music file in a digital audio format (WAV, MP3, etc.) stored in the storage unit 10b and executes decoding. Through decoding, the music data is converted into time-series sound pressure data in which sound pressure is sampled at regular intervals. The control unit 10 detects regular beats and irregular beats that make up the music from the sound pressure data. Here, regular beats are related to the rhythm of the music, and from this point of view, it can be said that the control unit 10 detects a tempo (BPM (Beats Per Minute)) based on the rhythm of the music. For example, a method such as FFT (Fast Fourier Transform) can be used to detect the BPM.

[0054] For example, the control unit 10 determines that a beat has occurred when a change in sound pressure data exceeds a predetermined threshold. Alternatively, the control unit 10 may determine that a beat has occurred when the sound pressure data exceeds a predetermined value. Alternatively, the control unit 10 may determine that a beat has occurred when the sound pressure data exceeds a predetermined value and the change in the sound pressure data in time series exceeds a predetermined threshold. That is, the control unit 10 extracts as a predetermined timing a case where the sound pressure of the sound information exceeds a predetermined value or a case where the change in the sound pressure of the sound information exceeds a predetermined value. This makes it possible to easily obtain the timing for acquiring movement information.

[0055] In addition, beat detection may be performed by distinguishing between regularly occurring beats and irregular beats. Regular beat detection can be performed using, for example, an algorithm based on Sound Energy Variation (https: / / mziccard.me / 2015 / 05 / 28 / beats-detection-algorithms-1 / ). This algorithm analyzes the energy of each bar of music and extracts regular beat patterns from these energy peaks. On the other hand, irregular beat detection can be performed using, for example, an algorithm based on multipath search and cluster analysis (Hindawi Complexity Volume 2021, "Music Rhythm Detection Algorithm Based on Multipath Search and Cluster Analysis"). This algorithm converts sample data into the frequency domain using a short-time Fourier transform (STFT), extracts amplitude peak and phase information, and extracts PCM (Pulse Code Modulation) feature values ​​from this information.

[0056] Thus, in the movement evaluation system S1 of the first embodiment, the sound information is music, and the control unit 10 determines a predetermined timing (timing for acquiring movement information) based on a change in sound pressure of the sound information. This makes it possible to unify the timing for acquiring movement information between the first subject 1a and the second subject 1b who are dancing to music.

[0057] Control unit 10 acquires movement information based on the timing at which the beat is detected. In Fig. 6(A), τ1 to τ14 correspond to the timing at which the beat is detected, and in Fig. 6(B), τ'1 to τ'14 correspond to the timing at which the beat is detected. Here, in a situation where first subject 1a and second subject 1b are dancing to the same music, τ1 and τ'1, τ2 and τ'2... and τ14 and τ'14 are the same timing.

[0058] As described above, the evaluation images are taken at a predetermined cycle. The control unit 10 extracts a plurality of evaluation images taken in close time around the timing at which the beat is detected, and acquires motion information of each subject 1 based on the evaluation images. At this time, the normalized representative position information described above is referenced. The control unit 10 acquires normalized representative position information for each of the evaluation images taken within a predetermined period (for example, within ±1 / 3 of the beat cycle) around the timing at which the beat is detected (for example, τ1 shown in FIG. 6A), and adopts the normalized representative position information that satisfies a predetermined criterion as motion information of the subject 1. Then, in combination with time information (τ1a, etc.) at the time when the motion information is acquired, the first x-direction data set: (x1, τ1a), (x2, τ2a)...(x14, τ14a) is acquired from CGax_fin(x, τ).

[0059] Examples of criteria for extracting the motion information of the subject 1 from the normalized representative position information include the following. (C1) When a plurality of peaks of normalized representative position information are detected before and after the timing τ at which a beat is detected, the normalized representative position information having the largest absolute value is adopted as the motion information.

[0060] An example of applying the criterion (C1) will be described below. In FIG. 6B, there are multiple peaks around τ'1, τ'6, and τ'8. By processing according to (C1), P1, P2, and P3 shown in the figure are not adopted as motion information, and as a result, for the second subject 1b, a second data set in the x direction: (x'1, τ'1a), (x'2, τ'2a)...(x'14, τ'14a) is obtained from CGax_fin'(x', τ').

[0061] As described above, the movement evaluation system S1 of the first embodiment includes a sound information output unit 16 that outputs sound information, an imaging unit 13 that captures an image of the subject 1 that is displaced based on the sound information, and a control unit 10, and the control unit 10 acquires movement information of the subject 1 from the output of the imaging unit 13 based on a predetermined timing extracted from the sound information. This means that the control unit 10 acquires movement information substantially in synchronization with the beat or rhythm included in the sound information. This makes it possible to accurately acquire movement information of the subject 1 that is displaced based on the sound information.

[0062] In addition, the control unit 10 detects the beat or rhythm contained in the sound information, and extracts, as movement information, peak values ​​of representative position information that are temporally before and after the timing based on the detected beat or rhythm. It is known that dance experts intentionally deviate from the moment the beat is struck when making large body movements to enhance their expressiveness. Conversely, beginners may not be able to follow the rhythm of the music, and their body movements may lag behind the timing of the beat. The present invention makes it possible to accurately acquire movement information even for such movements that are intentionally (or due to lack of skill, etc.) deviated from the timing of the beat (or are off).

[0063] The application of the above-mentioned criterion (C1) is arbitrary. For example, if there are multiple peaks of normalized representative position information within a predetermined period before and after the timing τ at which the beat is detected, all of these may be adopted as movement information. That is, multiple pieces of movement information may be obtained for one beat. Even if the dance is performed to the same music, the number of peaks detected in the first subject 1a and the second subject 1b may differ, and this difference in the number of peaks may be reflected in the derivation of the similarity.

[0064] The control unit 10 calculates the average time between peaks in the positive region Tpp_P_axn and the average time between peaks in the negative region Tpp_N_axn using the first data set in the x direction, which is the movement information of the first subject 1a. These are calculated specifically as follows. In the following formula, kaxp means the number of peaks in the positive region, and kaxn means the number of peaks in the negative direction. ·Tpp_P_axn ={(τ3a-τ1a)+(τ6a-τ3a)+(τ8a-τ6a)+...+(τ14a-τ12a)} / kaxp =(τ12a-τ1a) / kaxp ·Tpp_N_axn ={(τ4a-τ2a)+(τ5a-τ4a)+(τ7a-τ5a)+...+(τ13a-τ11a)} / kaxn =(τ11a-τ2a) / kaxn

[0065] Similarly, the second x-direction data set, which is the movement information of the second subject 1b, is used to calculate the average time between peaks in the positive region (Tpp_P_axn') and the average time between peaks in the negative region (Tpp_N_axn'). These are calculated specifically as follows. In the following formula, jaxp is the number of peaks in the positive region, and jaxn is the number of peaks in the negative region. ·Tpp_P_axn' =(τ'14a-τ'3a) / jaxp ·Tpp_N_axn' =(τ'13a-τ'1a) / jaxn

[0066] The average peak-to-peak value and the number of peaks in the positive and negative regions of the first subject 1a and the second subject 1b are closely related to the beat, and if there is a difference between them, it can be determined that the second subject 1b (student) has made a mistake in the dance movement. The control unit 10 calculates, for example, F1 as an evaluation function. α and β in F1 are weighting coefficients and may be determined appropriately. Formula 1 =α{|(Tpp_P_axn)-(Tpp_P_axn')|+|(Tpp_N_axn)-(Tpp_N_axn')}+β(|kaxp-jaxp|+|kaxn-jaxn|)

[0067] The control unit 10 may also calculate the difference between the sum of the absolute values ​​of all elements of CGax_fin(x, τ) for the first subject 1a and the sum of the absolute values ​​of all elements of CGax_fin'(x', τ') for the second subject 1b (evaluation function F2). Note that δ is a weighting coefficient and may be determined appropriately. F2 =δ(Σ|CGax_fin(x,τ)|-Σ|CGax_fin'(x',τ')|)

[0068] F1 and F2 can be used as similarities. These are indices that approach zero as the difference in the motion information between first subject 1a and second subject 1b decreases, that is, as the motion similarity increases. Of course, the following evaluation function F3 may be determined using F1 and F2. F3=F1+F2 F3 can also be used as a measure of similarity. F3 is also an index that approaches zero as the similarity between the movements of the two increases.

[0069] Thus, in the movement evaluation system S1 of the first embodiment, the subject 1 includes the first subject 1a and the second subject 1b, and the control unit 10 derives the similarity between the movements of the first subject 1a and the second subject 1b based on the movement information of the first subject 1a and the second subject 1b detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity between the movements of the first subject 1a and the second subject 1b that are displaced based on the sound information.

[0070] In the motion evaluation system S1 of the first embodiment, the control unit 10 calculates at least one representative position information representative of the position information of each of the first subject 1a and the second subject 1b, and derives the similarity based on the time-series change of the motion information extracted from the representative position information. This makes it possible to calculate the similarity with high accuracy and high speed without processing a large amount of motion information.

[0071] The above shows an example of deriving the similarity between the first subject 1a and the second subject 1b based on the average peak-to-peak value and the number of peaks in the positive and negative regions of the movement information of the whole body 1AL in the x direction. Of course, the similarity may be derived similarly based on the movement information in the y direction (height direction) of both subjects 1. Furthermore, the similarity may be derived based on the average peak-to-peak value and the number of peaks in the positive and negative regions of the movement information of the head 1HD, the torso 1BD, and the legs 1L in the x and y directions, respectively, or the similarity may be derived by integrating these pieces of movement information. Note that, when the imaging unit 13 is configured with a stereo camera, the displacement of the subject 1 in the front-rear direction (see FIG. 2(A)) can be measured. The similarity may be derived by extracting the movement information from the amount of displacement in the front-rear direction.

[0072] Second embodiment 7(A) and (B) are explanatory diagrams for explaining the process of deriving similarity in the second embodiment of the present invention. Here, FIG. 7(A) shows the distribution of representative position information of the first object 1a in the x and y directions in a time series, and is an image generated based on the movement of the first object 1a in the x direction (CGax(x, t) shown in FIG. 5(A)) and the movement in the y direction (not shown). Here, if the evaluation image is shot for 24 seconds at 60 fps, for example, 60 [fps] × 24 [s] = 1440 pieces of representative position information (x, y coordinate values) are obtained. These pieces of representative position information are plotted on the x and y coordinates. In FIG. 7(A), the plotted area is shown as Tra. Note that the range of the x and y coordinates is normalized to, for example, a range of 0 to 511. The representative position information is plotted as 8-bit monochrome image data, and the pixel value is set to, for example, 255. Hereinafter, the image shown in FIG. 7(A) is referred to as a "first movement information image".

[0073] FIG. 7(B) shows the distribution of representative position information of second subject 1b in the x and y directions in a time series, and is generated in the same manner as FIG. 7(A) based on the movement of second subject 1b in the x direction (CGax'(x',t') shown in FIG. 5(B)) and the movement in the y direction (not shown). In FIG. 7(B), the plotted area is shown as Trb. Hereinafter, the image shown in FIG. 7(B) will be referred to as the "second movement information image."

[0074] The configuration of the motion evaluation system S1 of the second embodiment is the same as that of the first embodiment. Hereinafter, the description will be continued with reference to Fig. 1. The control unit 10 generates a first motion information image and a second motion information image, and substitutes elements constituting each image into the following [Equation 1] to obtain a structural similarity index (SSIM: Structural Similarity Index Measure).

number

[0075] SSIM provides an evaluation index that considers the characteristics of the human visual system based on three elements of image brightness, contrast, and structure. In [Equation 1], x and y are vectors that represent each pixel in a window (here, 512 × 512) in the first motion information image and the second motion information image, respectively. μ is the average pixel value in the window, σ x ,σ y is the standard deviation of the pixel values ​​in the same window, σ xy is the covariance of x and y. C1 and C2 are constants that prevent the evaluation value from becoming unstable when the denominator value becomes very small. Here, C1=(K1L) 2 , C2=(K2L) 2 where L is the dynamic range of pixel values ​​(here, 8 bit: 255). K1 and K2 are constants, e.g., K1=0.01 and K2=0.03.

[0076] In this way, in the second embodiment, the SSIM is calculated for the first subject 1a and the second subject 1b using representative position information (center of gravity CGa of the whole body 1AL) in the x and y directions of the whole body 1AL (see FIG. 3). When the first movement information image and the second movement information image completely match, SSIM(x, y)=1, and the lower the similarity, the closer the SSIM value is to 0. The control unit 10 displays the calculated SSIM on the display unit 15 as the similarity. Of course, the SSIM may be calculated using representative position information of the head 1HD, torso 1BD, and legs 1L, or these individual SSIM values ​​may be appropriately combined to use as an index of similarity.

[0077] In calculating SSIM(x,y), the first motion information image and the second motion information image may be divided into small regions, and the SSIM may be calculated for each small region, and the SSIM may be averaged to calculate MSSIM (Mean SSIM). Note that, in deriving the similarity between images, for example, SNR (Signal to Noise Ratio) may be used instead of SSIM or MSSIM, or in addition to SSIM, etc.

[0078] In this way, the motion evaluation system S1 of the second embodiment includes a sound information output unit 16 that outputs sound information, an imaging unit 13 that captures the first subject 1a and the second subject 1b, and a control unit 10. The first subject 1a and the second subject 1b are displaced based on the sound information output by the sound information output unit 16, and the control unit 10 calculates at least one representative position information representative of the position information of the first subject 1a and the second subject 1b, and further generates a first motion information image and a second motion information image that represent the time-series change of the representative position information, and derives a similarity based on the first motion information image and the second motion information image. This makes it possible to express the motion (trajectory) of the position representative of the subject 1 as a two-dimensional image and derive a similarity based on the difference between the images.

[0079] Furthermore, in the motion evaluation system S1 of the second embodiment, the control unit 10 calculates a structural similarity index (SSIM) based on the first motion information image and the second motion information image, which makes it possible to replace the motion of the subject 1 with an image and derive the similarity taking into account the characteristics of the human visual system.

[0080] A modified example of the second embodiment will be described below. The control unit 10 generates a first movement information image for the first subject 1a based on a first x-direction data set: (x1, τ1a), (x2, τ2a)...(x14, τ14a) acquired based on CGax_fin(x, τ) shown in Fig. 6(A) and a first y-direction data set: (y1, τ1a), (y2, τ2a)...(y14, τ14a) acquired in a similar manner to the first x-direction data set.

[0081] Furthermore, for the second subject 1b, a second motion information image is generated based on the second x-direction data set: (x'1, τ'1a), (x'2, τ'2a)...(x'14, τ'14a) obtained based on CGax_fin'(x', τ') shown in Fig. 6(B) and the second y-direction data set: (y'1, τ'1a), (y'2, τ'2a)...(y'14, τ'14a) obtained in the same manner as the second x-direction data set. That is, the first x-direction data set, the first y-direction data set, the second x-direction data set, and the second y-direction data set in the modified example all use motion information of the subject 1 obtained in synchronization with sound information.

[0082] In the modified example, the control unit 10 also derives the similarity based on the first motion information image and the second motion information image. In this manner, in the modified example of the motion evaluation system S1, the control unit 10 generates a first motion information image and a second motion information image representing the time-series change in motion information for each of the first subject 1a and the second subject 1b, and derives the similarity based on the first motion information image and the second motion information image. In this case, SSIM may be used as in the second embodiment. This makes it possible to express the motion of a position representative of the subject as a two-dimensional image and derive the similarity based on the difference between the images.

[0083] However, in the modified example, the number of points (pixels) constituting the motion information image is very small, 14 in the above example ((x1, τ1a) to (x14, τ14a) shown in FIG. 6(A)). When an image is composed of a small number of pixels (dots), the structures of the first motion information image and the second motion information image are significantly different, and the SSIM is calculated to be very small, and the similarity may not be evaluated appropriately. Therefore, in the modified example, the pixels constituting the first motion information image and the second motion information image are replaced with an object having an area larger than one pixel. Specifically, for example, one pixel is replaced with a circle having a predetermined radius r (for example, r=5 pixels) centered on the x, y coordinates of the pixel. At this time, the inside of the circle may be filled with a predetermined value (for example, 255), or a gradation may be provided in which the pixel value decreases from the center of the circle toward the radial direction. By providing a gradation, the sensitivity to the edge structure can be reduced. In addition, an area where multiple circles overlap may be replaced with an average value of each gradation, which suppresses the edge of the object and intentionally reduces the characteristics related to the structure of the image.

[0084] In this way, the motion evaluation system S1 of the modified example renders the representative position information (motion information) as an object of a predetermined size exceeding one pixel size in the first motion information image and the second motion information image. This makes it possible to properly obtain the SSIM even if the number of pieces of representative position information is small.

[0085] Third embodiment FIG. 8 is an explanatory diagram for explaining a method for visualizing the movement of the subject 1 in the third embodiment of the present invention. In FIG. 8, CGax(x,t) shown in FIG. 5(A) and CGax'(x',t') shown in FIG. 5(B) are drawn on a radar chart. The movement of the first subject 1a is shown by a solid line (hereinafter referred to as the "first graph"), and the movement of the second subject 1b is shown by a dashed line (hereinafter referred to as the "second graph"). The value of "movement" here means representative position information. The radial direction of the radar chart is the movement (displacement) of the subject 1 in the x direction, and the circumferential direction represents the passage of time (here, one revolution is 24 seconds). The subject 1 starts dancing at 0° and ends dancing at 360°. By expressing the movements of the first subject 1a and the second subject 1b as a radar chart in this way, the similarity between the two can be easily evaluated visually.

[0086] The representative position information of the first subject 1a and the second subject 1b is the same at 0° and 360° of the radar chart, and both the first graph and the second graph are drawn as closed curves. In the radar chart, the radial direction corresponds to the magnitude of the movement of the subject 1, so the larger the movement, the more likely it is to be reflected in the increase in the area of ​​the region surrounded by the closed curve. In this way, in the third embodiment, it is possible to evaluate the dynamism of the movement of the subject 1 using the area of ​​the region surrounded by the first graph and the area of ​​the region surrounded by the second graph. Of course, for example, the ratio between the area of ​​the first graph and the area of ​​the second graph may be used as the similarity.

[0087] (Fourth embodiment) FIG. 9 is a block diagram showing the configuration of a movement evaluation system S1 according to a fourth embodiment of the present invention. In the first embodiment, the movement information of the subject 1 is extracted based on an evaluation image captured by the imaging unit 13 (see FIG. 1), but in the fourth embodiment, the movement information of the measurement target 2 is extracted using the movement detection unit 3. Note that the movement evaluation system S1 of the fourth embodiment is obtained by replacing the imaging unit 13 shown in FIG. 1 with the movement detection unit 3 and replacing the subject 1 with the measurement target 2. That is, the measurement target 2 is, for example, a human being, and includes a first measurement target 2a (corresponding to the first subject 1a in the first embodiment) and a second measurement target 2b (corresponding to the second subject 1b in the first embodiment). As in the first embodiment, the first measurement target 2a and the second measurement target 2b are displaced in accordance with the sound information output from the sound information output unit 16.

[0088] The left and right wrists and the left and right ankles of the measurement subject 2 are respectively fitted with, for example, box-shaped motion detection units 3 using wristbands or the like. In order to detect the motion of the arm 1A or leg 1L (see FIG. 2) with high accuracy, it is preferable that the motion detection units 3 are fitted to parts that experience large displacement. From this perspective, it is preferable that the motion detection unit 3 that detects the motion of the arm 1A is fitted to the wrist or held in the palm of the hand. Also, it is preferable that the motion detection unit 3 corresponding to the leg 1L is fitted to the ankle. The motion detection units 3 may be installed in the head 1HD or torso 1BD (see FIG. 2) of the measurement subject 2.

[0089] Here, it is preferable that the correspondence between each motion detection unit 3 and the part on which it is attached is determined in advance. For example, the part on which each motion detection unit 3 should be attached, such as "for arm (right)", is clearly indicated on each motion detection unit 3, and the measurement subject 2 attaches the motion detection unit 3 to the clearly indicated part.

[0090] FIG. 10 is a block diagram showing the configuration of the motion detection unit 3. As shown in FIG. 10, the motion detection unit 3 is composed of a second control unit 3a, a second storage unit 3b, a second communication unit 3c, and an inertial sensor 3d. The second control unit 3a is composed of a CPU or the like, and operates according to a control program stored in a second storage unit 3b composed of a ROM, a RAM, or the like. The second control unit 3a and other components are connected by a bus or the like, and the control unit 10 controls the other components via the bus or the like. The second storage unit 3b further stores an identifier (ID) representing each of the motion detection units 3. The second communication unit 3c includes a communication module (not shown) that complies with a short-range wireless communication standard, such as BLE. The second control unit 3a acquires the ID stored in the second storage unit 3b and the output of the inertial sensor 3d, and transmits this information to the control unit 10 (see FIG. 9) via the second communication unit 3c.

[0091] The inertial sensor 3d is composed of, for example, a three-axis acceleration sensor and / or a gyro sensor. Here, the three-axis acceleration sensor outputs the direction and degree of speed change of each part of the measurement target 2 (acceleration) for the three axes X, Y, and Z. The gyro sensor outputs the direction and speed at which each part of the measurement target 2 rotates (angular velocity) for the three axes X, Y, and Z. In this way, the inertial sensor 3d detects the movement of the arm 1A and leg 1L of the measurement target 2, and outputs three-axis acceleration information and / or three-axis angular velocity information based on the detection (hereinafter, sometimes referred to as "three-axis acceleration information, etc.") to the control unit 10. At this time, the above-mentioned ID is also output.

[0092] The control unit 10, which has received the triaxial acceleration information and the ID, calculates representative position information based on the triaxial acceleration information. By referring to the ID, the control unit 10 determines which motion detection unit 3 has output the output of the inertial sensor 3d. The control unit 10 integrates the output (triaxial acceleration information) of each inertial sensor 3d to obtain speed information, and further integrates this to obtain position information. Then, the control unit 10 averages the position information of each motion detection unit 3 to obtain representative position information of the measurement target 2. Note that the control unit 10 may obtain representative position information based on the output of a specific inertial sensor 3d by referring to the ID. Furthermore, the control unit 10 obtains motion information of the measurement target 2 based on the beat or rhythm of the sound information, as in the first embodiment. Based on this motion information, the control unit 10 derives the similarity of the motion between the first measurement target 2a and the second measurement target 2b.

[0093] As described above, the movement evaluation system S1 of the fourth embodiment includes a sound information output unit 16 that outputs sound information, a movement detection unit 3 that detects the movement of the measurement target 2 that is displaced based on the sound information, and a control unit 10, and the control unit 10 acquires the movement information of the measurement target 2 based on a predetermined timing extracted from the sound information. This makes it possible to accurately acquire the movement information of the measurement target 2 that is displaced based on the sound information.

[0094] Furthermore, in the movement evaluation system S1 of the fourth embodiment, the measurement target 2 includes a first measurement target 2a and a second measurement target 2b, and the control unit 10 derives the similarity between the movements of the first measurement target 2a and the second measurement target 2b based on the movement information of the first measurement target 2a and the second measurement target 2b detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity between the movements of the first measurement target 2a and the second measurement target 2b that are displaced based on the sound information.

[0095] The motion evaluation system S1 and the motion evaluation method according to the present invention have been described in detail above based on specific embodiments, but these embodiments are merely examples, and the present invention is not limited to these embodiments. For example, the first subject 1a and the second subject 1b (or the first measurement target 2a and the second measurement target 2b) may be the same person. By acquiring video files of the same subject 1 performing a dance at different times and deriving the similarity between them, it becomes possible to express the results of training of the same person as a numerical value.

[0096] Moreover, the subject 1 or the measurement target 2 does not have to be a human. Specifically, for example, either the first subject 1a or the second subject 1b may be a robot. In this case, the robot is programmed to displace in accordance with the sound information. Then, for example, the smoothness of the robot's movement, response speed, and amount of displacement can be evaluated based on the similarity. Of course, both the first subject 1a and the second subject 1b may be robots.

[0097] In the first to third embodiments, the representative position information is calculated based on the coordinate values ​​of the key points 41 of the pose recognition model 40 (see FIG. 3), but the so-called motion capture technique may be used to calculate the representative position information. Specifically, a plurality of reflective markers attached to the subject 1 are photographed by the imaging unit 13, and the representative position information is obtained based on the coordinates of the detected reflective markers.

[0098] In the first embodiment, the peak values ​​of the representative position information before and after the timing based on the detected beat or rhythm are extracted as the motion information, but the frame image (evaluation image) closest in time to the timing when the beat is detected may be selected, and the representative position information obtained from this evaluation image may be used as the motion information. Also, the average value of multiple representative position information obtained within a predetermined period before and after the timing when the beat is detected may be used as the motion information.

[0099] In each embodiment, the first subject 1a is mainly described as a dance instructor and the second subject 1b as his / her student, because even if the dance is performed to the same music and with the same choreography, differences in skill and proficiency will result in differences in the movements of the two subjects 1 (measurement target 2). On the other hand, the present invention may be applied to rehabilitation performed between doctors and elderly people, or to guidance and support for children with developmental disorders. [Industrial Applicability]

[0100] The movement evaluation system S1 and movement evaluation method of the present invention can improve dance performance by evaluating the similarity of movements between an instructor and a student and reflecting this in instruction, and can also further improve the motor function and attention of children with developmental disabilities and the elderly.Therefore, the movement evaluation system S1 and movement evaluation method can be widely used in dance classes, support sites for children with developmental disabilities, elderly care facilities, home care sites, etc. [Explanation of symbols]

[0101] 1. Subject 1a First subject 1b 2nd subject 2. Measurement target 3. Motion detection section 10 Control section 13 Imaging unit 16 Sound information output section 40 Pose Recognition Models 41 Key Points 50 Network S1 Movement Evaluation System

Claims

1. a sound information output unit that outputs sound information; an imaging unit that captures an image of a first object and a second object that are displaced based on the sound information; A control unit; Equipped with The control unit is calculating at least one representative position information representative of each of the position information of the first subject and the second subject in a time series from the output of the imaging unit based on a predetermined timing extracted from the sound information; Furthermore, a first motion information image and a second motion information image are generated by plotting the respective representative position information as pixels in a coordinate space that does not include a time axis; A motion evaluation system comprising: a motion evaluation unit that derives a similarity between the motions of the first subject and the second subject based on the first motion information image and the second motion information image.

2. 2. The motion estimation system according to claim 1, wherein the representative position information is rendered as an object having a predetermined size exceeding one pixel size in the first motion information image and the second motion information image.

3. The sound information is music, The control unit is 3. The movement evaluation system according to claim 1, wherein the predetermined timing is determined based on a change in sound pressure of the sound information.

4. The control unit is 4. The movement evaluation system according to claim 3, wherein the predetermined timing is determined to be when the sound pressure of the sound information exceeds a predetermined value or when a change in the sound pressure of the sound information exceeds a predetermined value.

5. The control unit is 3. The motion estimation system according to claim 1, further comprising a structural similarity index (SSIM) calculated as the degree of similarity based on the first motion information image and the second motion information image.

6. a sound information output unit that outputs sound information; a movement detection unit that detects the movement of a first measurement target and a second measurement target that are displaced based on the sound information; A control unit; Equipped with The control unit is acquiring motion information between the first measurement target and the second measurement target from an output of the motion detection unit based on a predetermined timing extracted from the sound information; calculating at least one representative position information representing each of the position information of the first measurement target and the second measurement target in a time series from the movement information; Furthermore, a first motion information image and a second motion information image are generated by plotting the respective representative position information as pixels in a coordinate space that does not include a time axis; A movement evaluation system comprising: a movement evaluation unit that derives a movement similarity between the first measurement target and the second measurement target based on the first movement information image and the second movement information image.

7. Outputs sound information, Shooting a first object and a second object that are displaced based on the sound information; calculating at least one representative position information representing each of the position information of the first object and the second object in a time series based on a predetermined timing extracted from the sound information; Furthermore, a first motion information image and a second motion information image are generated by plotting the respective representative position information as pixels in a coordinate space that does not include a time axis; A motion evaluation method comprising deriving a similarity between the motions of the first subject and the second subject based on the first motion information image and the second motion information image.

8. Detecting the movements of the first measurement object and the second measurement object which are displaced based on the sound information; calculating at least one representative position information representing each of the position information of the first measurement object and the second measurement object in a time series based on a predetermined timing extracted from the sound information; Furthermore, a first motion information image and a second motion information image are generated by plotting the respective representative position information as pixels in a coordinate space that does not include a time axis; A motion evaluation method comprising deriving a similarity between the motions of a first object and a second object based on the first motion information image and the second motion information image.

Images