Motion evaluation system and method for evaluating motion

Abstract

To provide a motion evaluation system and a method for evaluating a motion capable of accurately evaluating similarity of motion between two subjects or between measurement targets that are displaced in accordance with sound information.SOLUTION: The motion evaluation system includes a sound information output unit 16 for outputting sound information, an imaging unit 13 for capturing an image of a subject 1 displaced on the basis of sound information, and a control unit 10. The control unit 10 acquires motion information of the subject 1 from an output of the imaging unit 13 on the basis of predetermined timing extracted from the sound information. The control unit 10 derives similarity of motion between a first subject 1a and a second subject 1b on the basis of motion information of the first subject 1a and the second subject 1b detected on the basis of the predetermined timing extracted from the sound information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a motion evaluation system and a motion evaluation method for evaluating the similarity of a subject's motion, particularly the motion between two subjects. [Background technology]

[0002] Dancing, which involves full-body exercise, is known to improve cardiopulmonary function and enhance muscle strength and balance. Incorporating dance into daily life is expected to reduce the risk of falls and injuries, for example, among the elderly. Learning new dance steps and moving to the rhythm of music is also expected to improve spatial awareness, memory, and attention in children with developmental disabilities and the elderly.

[0003] In dance lessons, students gradually learn movements by imitating the instructor's steps. During this process, evaluating the similarity (match) of movements between the instructor and student and reflecting this in instruction can improve the student's dance performance and further improve motor function. Furthermore, incorporating dance into rehabilitation, evaluating the similarity of movements between doctors, physical therapists, and care recipients, and reflecting this in care plans such as how to apply stress, may lead to more efficient rehabilitation.

[0004] A known technique for evaluating the similarity of subjects appearing in an image includes, for example, the steps of acquiring a digital image capturing an environment including at least a first subject, segmenting a first portion of the digital image capturing the first subject into a plurality of superpixels, assigning a semantic label to each of the plurality of superpixels, extracting features of the superpixel, and determining a similarity metric between the features extracted from the superpixels and features extracted from reference superpixels identified in a reference digital image, wherein the reference superpixels are assigned a reference semantic label that matches the semantic label assigned to the superpixel, and determining that the first subject appears in the reference image based on a plurality of similarity metric associated with the plurality of superpixels (see Patent Document 1).

[0005] According to Patent Document 1, it can be determined that a first object is present in a reference image based on a plurality of similarity measures associated with a plurality of superpixels. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2021-531539 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0007] However, the technology disclosed in Patent Document 1 is aimed at identifying people in digital images, and makes it possible to identify people in digital images by combining features such as clothing, accessories, hair, and face, but does not suggest evaluating the similarity in the movements of two subjects that move in accordance with sound information.

[0008] The present invention has been devised to solve the problems of the conventional technology, and its purpose is to provide a movement evaluation system and a movement evaluation method that can accurately evaluate the similarity in movement between two subjects or measurement targets that displace in accordance with sound information. [Means for solving the problem]

[0009] The present invention, which has been made to solve the above problems, is a movement evaluation system comprising a sound information output unit that outputs sound information, an imaging unit that images a subject that is displacing based on the sound information, and a control unit, wherein the control unit acquires movement information of the subject from the output of the imaging unit based on a predetermined timing extracted from the sound information. This makes it possible to accurately acquire movement information of a subject that is displacing based on sound information.

[0010] In addition, in the present invention, the subjects include a first subject and a second subject, and the control unit derives the similarity of the movements of the first subject and the second subject based on the movement information of the first subject and the second subject detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity of the movements of the first subject 1a and the second subject 1b that are displaced based on the sound information.

[0011] In addition, in the present invention, the control unit calculates at least one representative piece of position information representing each piece of position information for the first subject and the second subject, and derives the similarity based on a time series change in the movement information extracted from the representative piece of position information, thereby enabling the similarity to be calculated with high accuracy and high speed without processing a large amount of movement information.

[0012] Furthermore, in the present invention, the control unit generates a first motion information image and a second motion information image representing time-series changes in the motion information for the first subject and the second subject, respectively, and derives the similarity based on the first motion information image and the second motion information image. This makes it possible to represent the motion (trajectory) of a position representative of the subject as a two-dimensional image and derive the similarity based on the difference between the images.

[0013] Furthermore, in the present invention, the motion information is depicted as an object of a predetermined size exceeding one pixel in the first motion information image and the second motion information image, thereby making it possible to appropriately obtain the similarity even when the number of pieces of representative position information is small.

[0014] In addition, in the present invention, the control unit detects a beat or rhythm contained in the sound information and extracts, as the movement information, peak values ​​of the representative position information that occur before or after the timing based on the detected beat or rhythm, thereby making it possible to accurately obtain movement information even when the subject moves in a way that intentionally throws off the timing of the beat.

[0015] In addition, in the present invention, the sound information is music, and the control unit determines the predetermined timing based on a change in sound pressure of the sound information, thereby making it possible to unify the timing for acquiring movement information for a first subject and a second subject dancing to music.

[0016] In addition, in the present invention, the control unit determines that the predetermined timing is when the sound pressure of the sound information exceeds a predetermined value or when a change in the sound pressure of the sound information exceeds a predetermined value, thereby easily obtaining the timing to acquire movement information.

[0017] The present invention also provides a motion evaluation system comprising a sound information output unit that outputs sound information, an imaging unit that captures images of a first subject and a second subject, and a control unit, wherein the first subject and the second subject are displaced based on the sound information output by the sound information output unit, the control unit calculates at least one representative position information representative of the position information of each of the first subject and the second subject, and further generates a first motion information image and a second motion information image that represent time-series changes in the representative position information, and derives a similarity based on the first motion information image and the second motion information image. This makes it possible to represent the movement (trajectory) of a position representative of the subject as a two-dimensional image and derive a similarity based on the difference between the images.

[0018] In addition, in the present invention, the control unit calculates a structural similarity index (SSIM) as the similarity based on the first motion information image and the second motion information image, thereby making it possible to derive the similarity taking into account the characteristics of the human visual system.

[0019] The present invention also provides a movement evaluation system that includes a sound information output unit that outputs sound information, a movement detection unit that detects the movement of a measurement object that is displaced based on the sound information, and a control unit, wherein the control unit acquires the movement information of the measurement object based on a predetermined timing extracted from the sound information, thereby making it possible to accurately acquire movement information of the measurement object that is displaced based on the sound information.

[0020] In addition, in the present invention, the measurement targets include a first measurement target and a second measurement target, and the control unit derives the similarity in movement between the first measurement target and the second measurement target based on the movement information of the first measurement target and the second measurement target detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity in movement between the first measurement target and the second measurement target that are displaced based on the sound information.

[0021] The present invention also provides a movement evaluation method that outputs sound information, captures an image of a subject that is displacing based on the sound information, and acquires movement information of the subject based on a predetermined timing extracted from the sound information, thereby enabling accurate acquisition of movement information of a subject that is displacing based on the sound information.

[0022] In addition, in the present invention, the subjects include a first subject and a second subject, and the similarity between the movements of the first subject and the second subject is derived based on the movement information of the first subject and the second subject, thereby making it possible to accurately evaluate the similarity between the movements of the first subject and the second subject that are displaced based on sound information.

[0023] The present invention also provides a movement evaluation method that detects the movement of a measurement object that is displaced based on sound information and acquires movement information of the measurement object based on a predetermined timing extracted from the sound information, thereby making it possible to accurately acquire movement information of the measurement object that is displaced based on sound information.

[0024] Furthermore, in the present invention, the measurement targets include a first measurement target and a second measurement target, and the similarity in the movements of the first measurement target and the second measurement target is derived based on the movement information of the first measurement target and the second measurement target detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity in the movements of the first measurement target and the second measurement target that are displaced based on the sound information. [Effects of the Invention]

[0025] As described above, according to the present invention, it is possible to accurately evaluate the degree of similarity in movement between two moving subjects or two measurement targets. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a block diagram showing the configuration of a movement estimation system S1 according to a first embodiment of the present invention. [Figure 2]FIG. 1A is an explanatory diagram showing a usage mode of the motion evaluation system S1, and FIGS. 1B to 1D are explanatory diagrams explaining pre-processing in the motion evaluation system S1. [Figure 3] Illustration of pose recognition model 40 [Figure 4] 1A is an explanatory diagram showing an example of an image of a subject 1, and FIG. 1B is an explanatory diagram showing a key point 41 and the center of gravity CGa of the entire body 1AL in the image of the subject 1. [Figure 5] (A) is a graph showing the movement of the first subject 1a in the x direction, (B) is a graph showing the movement of the second subject 1b in the x direction, and (C) is a graph showing the movement of the first subject 1a in the x direction normalized within a range of ±1. [Figure 6] 1A and 1B are explanatory diagrams showing the timing of acquiring motion information of a subject 1. [Figure 7] 10A and 10B are explanatory diagrams illustrating a process of deriving similarity in a second embodiment of the present invention. [Figure 8] FIG. 10 is an explanatory diagram illustrating a method for visualizing the movement of the subject 1 in the third embodiment of the present invention. [Figure 9] FIG. 10 is a block diagram showing the configuration of a movement estimation system S1 according to a fourth embodiment of the present invention. [Figure 10] Block diagram showing the configuration of the motion detection unit 3 DETAILED DESCRIPTION OF THE INVENTION

[0027] (First embodiment) A first embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a movement evaluation system S1 according to the first embodiment of the present invention. The movement evaluation system S1 is made up of a control unit 10, a display unit 15, an imaging unit 13, and a sound information output unit 16. A sound information acquisition unit 17 is provided as needed, as will be described later.

[0028] The control unit 10 is composed of a calculation unit 10a, a storage unit 10b, and a communication unit 10c. The calculation unit 10a is composed of a CPU (Central Processing Unit) and the like. The storage unit 10b is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and the calculation unit 10a operates according to a control program stored in the storage unit 10b. The storage unit 10b includes a non-volatile memory (such as an EEPROM (Electrically Erasable Programmable Read-Only Memory)). The non-volatile memory stores music files used to generate sound information. The storage unit 10b may include a so-called storage (large-capacity storage device) such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and the control program and music files may be stored in the storage. The calculation unit 10a and the other components are connected via a bus 20 and the like, and the calculation unit 10a controls the other components via the bus 20 and the like.

[0029] The control unit 10 may be configured as, for example, a PC (Personal Computer) or a server. The display unit 15 may be separate from the control unit 10, or may be configured integrally with the control unit 10, such as a tablet terminal or a notebook PC. The communication unit 10c includes a communication module (not shown) conforming to a wireless communication standard such as LTE, LTE-M, 4G, or 5G. The communication unit 10c may further include a communication module (not shown) conforming to a short-range wireless communication standard such as BLE (Bluetooth (registered trademark) Low Energy). The communication unit 10c establishes communication with the imaging unit 13, the sound information output unit 16, and the sound information acquisition unit 17, enabling mutual transmission and reception of information between the control unit 10 and these units. Of course, these may be connected by wire.

[0030] The imaging unit 13 includes an image sensor configured with a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). A camera included in an information terminal such as a smartphone may be used as the imaging unit 13. In this case, the information terminal may include a communication module (not shown) that complies with a predetermined wireless communication standard, and may transmit image data to the control unit 10 via a network 50. The imaging unit 13 may be configured integrally with the control unit 10, in which case the imaging unit 13 transmits image data to the calculation unit 10a via a bus 20.

[0031] The sound information output unit 16 includes an amplifier, a speaker, etc. (not shown). The control unit 10 generates an analog audio signal based on a music file stored in the storage unit 10b and outputs it to the sound information output unit 16. The sound information output unit 16 plays back sound information (music, songs) based on the analog sound signal. The sound information may be played back via a speaker in a room where the subject 1 is dancing, or may be played back via a device such as wireless headphones worn by the subject 1. Here, the sound information refers to any of the digital data constituting the music file, analog data decoded from the digital data, and sound output from the sound information output unit 16 based on the analog data.

[0032] Subject 1 may be equipped with a sound information acquisition unit 17. Sound information acquisition unit 17 includes a microphone, an AD converter, and a communication module conforming to a short-range wireless communication standard (none of which are shown). In this embodiment, sound information reproduced by sound information output unit 16 is acquired and digitized by sound information acquisition unit 17 and transmitted to control unit 10 via short-range wireless communication. Control unit 10 may store the received sound information as a music file in storage unit 10b, which may be used for motion detection and similarity derivation, which will be described later. In this way, even if subject 1 and sound information output unit 16 are far apart, by attaching sound information acquisition unit 17 to subject 1, the influence of the time delay before sound information reaches subject 1 can be eliminated.

[0033] Here, subject 1 is, for example, a human being. Subject 1 dances according to a predetermined choreography in time with the sound information (music) output from sound information output unit 16. The content of the sound information may be selected arbitrarily, and it is preferable to select music with a clear beat, rhythm, or tempo. The dance choreography may also be selected arbitrarily, and it is preferable to select choreography that increases the movement (displacement) of subject 1 at the timing when the beat occurs (on-beat). Note that sound information is not limited to music or songs, and a sound source that emits periodic sounds, such as a metronome, may also be used.

[0034] The imaging unit 13 captures an image of the subject 1 performing a dance. The control unit 10 temporarily stores the image data received from the imaging unit 13 in the storage unit 10b. If the imaging unit 13 has a function for storing image data in a portable storage medium, the control unit 10 may read the image data stored in the portable storage medium and store it in the storage unit 10b. The control unit 10 extracts movement information of the subject 1 using the stored image data. Here, the subject 1 includes a first subject 1a and a second subject 1b. The control unit 10 derives the similarity of the movements of the first subject 1a and the second subject 1b based on the movement information.

[0035] 2(A) is an explanatory diagram showing a usage mode of the movement evaluation system S1, and 2(B) to 2(D) are explanatory diagrams explaining pre-processing in the movement evaluation system S1. In the following explanation, the side of the subject 1 viewed in FIG. 2(B) may be referred to as the front, the opposite direction as the back, the direction of the right arm as the right, the direction of the left arm as the left, the direction of the head 1HD as the top, and the opposite direction as the bottom. In FIG. 2(A), the sound information output unit 16 includes a speaker (not shown), and the sound information output unit 16 and the subject 1 are spatially separated by a predetermined distance. Of course, the arrangement position of the sound information output unit 16 may be determined arbitrarily.

[0036] Regarding the positional relationship between the subject 1 and the imaging unit 13, it is preferable that the angle θ formed by a line extending forward from the front of the subject 1 and the optical axis AxL of the imaging unit 13 is in the range of 10°≦θ≦45° [deg], for example. This reduces occlusion during shooting and enables many of the pose landmarks described below to be acquired with high reliability. The distance L (or angle of view) between the subject 1 and the imaging unit 13 may be determined arbitrarily, but it is preferable to determine it so that the entire body 1AL of the subject 1 (see FIG. 3) is captured when the subject 1 has both arms 1A raised (FIG. 2(B)) and when the subject 1 has arms 1A spread out to the left and right (FIG. 2(C)), and also taking into consideration the range of movement of the subject 1 during dancing.

[0037] The following description will be continued with reference to FIG. 1. In preprocessing of the movement evaluation system S1, after adjusting the angle of view of the imaging unit 13, the subject 1 is photographed in the postures shown in FIGS. 2B to 2D before the subject 1 starts moving in sync with the dance. Based on the image data received from the imaging unit 13, the control unit 10 measures the distance between the hand and foot (first limb distance H1a) when the subject 1 (here, the first subject 1a) has both arms 1A raised (a so-called "banzai" posture) as shown in FIG. 2B. The control unit 10 also measures the distance between the hands (first limb distance Wa) when the first subject 1a has arms 1A spread out to the sides as shown in FIG. 2C. Furthermore, the control unit 10 measures the distance (first head-to-foot distance H2a) between the top of the head 1HD and the foot when the first subject 1a is standing upright (a so-called "attention posture") as shown in FIG. 2D. Note that pose landmarks, which will be described later, can be used to measure these distance information.

[0038] As with first subject 1a, second inter-limb distance H1b (FIG. 2(B)), second inter-hand distance Wb (FIG. 2(C)), and second head-to-foot distance H2b (FIG. 2(D)) are measured for second subject 1b. These distance information may be obtained by so-called 3D distance measurement, or may be measured manually with a tape measure or the like, and the numerical values ​​input to control unit 10 via an input unit (not shown). Based on the obtained distance information, control unit 10 calculates the following: (i) Width direction (x direction) correction coefficient SFx = distance between first two hands Wa / distance between second two hands Wb (ii) Height direction (y direction) correction coefficient SFy = distance between first limbs H1a / distance between second limbs H1b (or SFy2 = 1st head-to-leg distance H2a / 2nd head-to-leg distance H2b) When evaluating the movement, the control unit 10 uses these correction coefficients to correct the position information relating to the first subject 1a or the second subject 1b (details will be described later).

[0039] Fig. 3 is an explanatory diagram of a pose recognition model 40. In the first embodiment, MediaPipe Pose is used as the pose recognition model 40. As shown in Fig. 3, for an image (still image or video) of a person, MediaPipe Pose recognizes key points 41 (pose landmarks) from 0.nose to 32.right_foot_index (a total of 33 locations), and outputs position information (coordinate values ​​(x, y coordinates)) of the recognized key points 41.

[0040] Using the obtained position information, the control unit 10 calculates the coordinates of the center of gravity representing the movement of the subject 1 for each of the whole body 1AL, head 1HD, torso 1BD, and legs 1L. Specifically, the control unit 10 calculates the following center of gravity using the position information of each key point 41. Center of gravity CGa of the whole body 1AL: Average value of each coordinate value from 0.nose to 32.right_foot_index Center of gravity CGh of head 1HD: Average value of each coordinate value from 0.nose to 12.right_shoulder Center of gravity CGb of the torso 1BD: Average value of each coordinate value from 11.left_shoulder to 24.right_hip Center of gravity CGl of leg 1L: Average value of each coordinate value from 23.left_hip to 32.right_foot_index

[0041] In calculating the center of gravity Cgh of the head 1HD and the center of gravity CGb of the torso 1BD, 11.left_shoulder and 12.right_shoulder are referenced, and in calculating the center of gravity CGb of the torso 1BD and the center of gravity CGl of the legs 1L, 23.right_hip and 24.left_hip are referenced. However, it is preferable to exclude the coordinate values ​​of key points 41 that are not acquired due to occlusion or have low reliability in calculating each center of gravity. In the following description, the position information of the center of gravity may be referred to as "representative position information." As described above, in the first embodiment, the position information of the center of gravity described above is used as the representative position information. However, the representative position information may be calculated based on key points 41 that more significantly reflect the movement of the subject 1 according to the dance choreography. Furthermore, the first inter-hand distance Wa or the distance between both feet (the distance between 31.left foot index and 32.right foot index) described above may be used instead of (or in addition to) the representative position information.

[0042] FIG. 4(A) is an explanatory diagram showing an example of an image of subject 1, and FIG. 4(B) is an explanatory diagram showing key points 41 and the center of gravity CGa of the entire body 1AL in the image of subject 1. The process from photographing subject 1 to acquiring representative position information will be described below. First subject 1a and second subject 1b perform a dance with the same predetermined choreography to the same music output from sound information output unit 16. First, as shown in FIG. 4(A), subject 1 (first subject 1a or second subject 1b) performing the dance is photographed by imaging unit 13.

[0043] Here, first subject 1a and second subject 1b may be photographed at different locations and at different times using different imaging units 13, or may be photographed simultaneously using the same imaging unit 13. Imaging unit 13 photographs subject 1 in chronological order at a predetermined frame rate (for example, 60 fps (frames per second)) and transmits the image data to control unit 10.

[0044] The control unit 10 generates a video file based on the received image data and stores it in the storage unit 10b. The control unit 10 then accesses the storage unit 10b to retrieve the image file, acquires key points 41 (and their coordinate values) from each frame image (evaluation image) constituting the image file, and calculates representative position information (here, the coordinate values ​​of the center of gravity CGa of the entire body 1AL). Specifically, the control unit 10 processes the evaluation image using the API (Application Programming Interface) of the above-mentioned MediaPipe Pose. As a result, as shown in FIG. 4B, multiple key points 41 are recognized for the subject 1 (here, the first subject 1a), and the x and y coordinates and representative position information of each key point 41 are calculated. The subject 1, key points 41, and representative position information (center of gravity CGa) are then displayed superimposed on the display unit 15. Furthermore, the skeleton connecting the main key points 41 and the outer edge encompassing the group of key points 41 are displayed as line segments. The same processing is performed on the evaluation image captured of the second subject 1b.

[0045] FIG. 5(A) is a graph showing the x-direction movement of first subject 1a, FIG. 5(B) is a graph showing the x-direction movement of second subject 1b, and FIG. 5(C) is a graph showing the x-direction movement of first subject 1a normalized within a range of ±1. The vertical axis (x-direction) of FIGS. 5(A) to 5(C) corresponds to the left-right direction of subject 1 shown in FIGS. 4(A) and 4(B), and the horizontal axis represents the time axis t. As described above, each evaluation image is obtained discretely (periodically) in a time series, and therefore, the representative position information generated based on the evaluation images is also obtained discretely. However, in the graphs of FIGS. 5(A) to 5(C), the representative position information is interpolated and plotted as a curve (similar to FIG. 6, described later). The representative position information uses the x-coordinate value of the center of gravity CGa of the entire body 1AL described above.

[0046] The process of acquiring movement information of subject 1 will be described below. First, when photographing subject 1, control unit 10 matches the timing at which sound information is output from sound information output unit 16 with the timing at which photographing is started by imaging unit 13. A music file used to play the sound information may be prepared in advance. Of course, the sound information output from sound information output unit 16 may be acquired (recorded) via sound information acquisition unit 17 to create a music file. In this case, control unit 10 controls so that the timing at which photographing starts and the timing at which recording starts are the same.

[0047] In this way, a music file and a video file are obtained in which the start of music playback and the start of shooting are synchronized. Note that if the distance from sound information output unit 16 to subject 1 is large and affects the evaluation of the movement (for example, if the time it takes for sound information to reach subject 1 from sound information output unit 16 exceeds half the period of the music beat), it is preferable to adjust the origin of the time axis t by adjusting the timestamp of the music file or image file. Note that in Figures 5(A) to (C), the point in time when subject 1 starts moving is set as the origin (0) of the time axis t, and the timestamp of the music file at this point in time is adjusted to 0.

[0048] The positional relationship between the imaging unit 13 and the first subject 1a (second subject 1b) is generally considered to change with each image capture, and the control unit 10 executes a process to align the initial positions (here, the x-coordinate value of the representative position information) of the first subject 1a and the second subject 1b. Specifically, when the initial position of the representative position information of the first subject 1a is 540 (FIG. 5(A)), an offset is applied to the representative position information of the second subject 1b to align its initial value with 540 (FIG. 5(B) shows the graph after the initial positions have been aligned). In this way, the control unit 10 executes a calibration to align the initial values ​​of the representative position information of the first subject 1a and the second subject 1b. This enables highly accurate evaluation of the movements of the first subject 1a and the second subject 1b, even if the positions at which the first subject 1a and the second subject 1b start dancing are different, or even if the positions within the image are different. A similar calibration is also executed for the y-coordinate value of the representative position information.

[0049] Hereinafter, the time series change in the representative position information of the first subject 1a (i.e., FIG. 5(A)) may be referred to as CGax(x,t), and the time series change in the representative position information of the second subject 1b (i.e., FIG. 5(B)) may be referred to as CGax'(x',t'). Here, the x coordinate value of the center of gravity CGa of the whole body 1AL is exemplified as the representative position information.

[0050] Based on CGax(x,t), the control unit 10 calculates the average value of the representative position information in a time series for the first subject 1a. Then, as shown in FIG. 5C, the average value is set to 0, and each representative position information is normalized to fall within a range of ±1 (hereinafter, the normalized representative position information may be referred to as "normalized representative position information"). Hereinafter, the normalized CGax(x,t) may be referred to as CGax_fin(x,τ). Furthermore, the control unit 10 similarly normalizes CGax'(x',t') for the second subject 1b. Hereinafter, the normalized CGax'(x',t') may be referred to as CGax_fin'(x',τ') (see FIG. 6B).

[0051] When normalizing, the normalized representative position information may be corrected using the correction coefficient (here, SFx) described above. Specifically, if the above-mentioned SFx = first hand-to-hand distance Wa / second hand-to-hand distance Wb = 0.9, for example, the normalized representative position information of second subject 1b is multiplied by 0.9. Of course, the normalized representative position information of first subject 1a may also be multiplied by 1 / 0.9. This makes it possible to eliminate the influence of differences in the body shape, build, etc. of each subject 1. When evaluating the movement of subject 1 in the up-down direction (i.e., the y direction), the normalized representative position information may be corrected using SFy or SFy2.

[0052] 6(A) and (B) are explanatory diagrams showing the timings at which motion information of subject 1 is acquired. Here, FIG. 6(A) shows CGax_fin(x, τ) (see FIG. 5(C)) to which the timings (τ1 to τ14) extracted from sound information, normalized representative position information (x1 to x14) adopted as motion information, and the timings (τ1a, τ2a, etc.) at which the motion information was acquired have been added. FIG. 6(B) shows CGax'(x', t') (see FIG. 5(B)) normalized to ±1 in the vertical axis direction, and to which the timings (τ'1 to τ'14) extracted from sound information, normalized representative position information (x'1 to x'14) adopted as motion information, and the timings (τ'1a to τ'14a) at which the motion information was acquired have been added.

[0053] The following description will be continued with reference to FIG. 1. The control unit 10 opens a music file in a digital audio format (WAV, MP3, etc.) stored in the storage unit 10b and performs decoding. Through decoding, the music data is converted into time-series sound pressure data in which sound pressure is sampled at regular intervals. The control unit 10 detects regular and irregular beats that make up the music from the sound pressure data. Here, regular beats are related to the rhythm of the music, and from this perspective, it can be said that the control unit 10 detects the tempo (BPM (Beats Per Minute)) based on the rhythm of the music. To detect the BPM, for example, a method such as FFT (Fast Fourier Transform) can be used.

[0054] For example, the control unit 10 determines that a beat has occurred when a change in sound pressure data exceeds a predetermined threshold. Alternatively, it may determine that a beat has occurred when sound pressure data exceeds a predetermined value. Alternatively, it may determine that a beat has occurred when sound pressure data exceeds a predetermined value and when a change in sound pressure data in a time series exceeds a predetermined threshold. That is, the control unit 10 extracts as a predetermined timing a case where the sound pressure of the sound information exceeds a predetermined value or a case where a change in sound pressure of the sound information exceeds a predetermined value. This makes it possible to easily obtain the timing for acquiring movement information.

[0055] Note that beat detection may distinguish between regularly occurring beats and irregular beats. Regular beats can be detected using an algorithm based on Sound Energy Variation (https: / / mziccard.me / 2015 / 05 / 28 / beats-detection-algorithms-1 / ). This algorithm analyzes the energy of each musical bar and extracts regular beat patterns from these energy peaks. Irregular beats can be detected using an algorithm based on multipath search and cluster analysis (Hindawi Complexity Volume 2021, "Music Rhythm Detection Algorithm Based on Multipath Search and Cluster Analysis"). This algorithm converts sample data into the frequency domain using a short-time Fourier transform (STFT), extracts amplitude peaks and phase information, and then extracts PCM (Pulse Code Modulation) feature values ​​from this information.

[0056] As described above, in the movement evaluation system S1 of the first embodiment, the sound information is music, and the control unit 10 determines the predetermined timing (timing for acquiring movement information) based on changes in the sound pressure of the sound information. This makes it possible to unify the timing for acquiring movement information between the first subject 1a and the second subject 1b, who are dancing to music.

[0057] Control unit 10 acquires movement information based on the timing at which beats are detected. In Fig. 6(A), τ1 to τ14 correspond to the timing at which beats are detected, and in Fig. 6(B), τ'1 to τ'14 correspond to the timing at which beats are detected. Here, when first subject 1a and second subject 1b are dancing to the same music, τ1 and τ'1, τ2 and τ'2... τ14 and τ'14 are the same timing.

[0058] As described above, the evaluation images are captured at a predetermined interval. The control unit 10 extracts multiple evaluation images captured temporally close to the time when the beat was detected, and acquires motion information for each of the subject 1 based on the evaluation images. The control unit 10 references the normalized representative position information described above. The control unit 10 acquires normalized representative position information for each of the evaluation images captured within a predetermined period (e.g., within ±1 / 3 of the beat period) centered on the time when the beat was detected (e.g., τ1 shown in FIG. 6A), and adopts the normalized representative position information that meets a predetermined criterion as motion information for the subject 1. Then, by combining the motion information with time information (e.g., τ1a) when the motion information was acquired, a first x-direction data set: (x1, τ1a), (x2, τ2a), ... (x14, τ14a) is acquired from CGax_fin(x, τ).

[0059] Examples of criteria for extracting the movement information of the subject 1 from the normalized representative position information include the following. (C1) When multiple peaks of normalized representative position information are detected before and after the timing τ at which the beat is detected: the normalized representative position information with the largest absolute value is adopted as the movement information.

[0060] An example of applying this criterion (C1) will be described below. In FIG. 6B, there are multiple peaks before and after τ'1, τ'6, and τ'8. By processing according to (C1), P1, P2, and P3 shown in the figure are not adopted as movement information, and as a result, for the second subject 1b, a second x-direction data set: (x'1, τ'1a), (x'2, τ'2a)...(x'14, τ'14a) is obtained from CGax_fin'(x', τ').

[0061] As described above, the movement evaluation system S1 of the first embodiment includes a sound information output unit 16 that outputs sound information, an imaging unit 13 that captures an image of the subject 1 that is displacing based on the sound information, and a control unit 10. The control unit 10 acquires movement information of the subject 1 from the output of the imaging unit 13 based on a predetermined timing extracted from the sound information. This means that the control unit 10 acquires movement information substantially in synchronization with the beat or rhythm included in the sound information. This makes it possible to accurately acquire movement information of the subject 1 that is displacing based on the sound information.

[0062] The control unit 10 also detects the beat or rhythm contained in the sound information and extracts, as movement information, peak values ​​of representative position information that occur before and after the timing based on the detected beat or rhythm. It is known that skilled dancers intentionally deviate from the moment the beat is struck to increase their expressiveness. Conversely, beginners may be unable to keep up with the rhythm of the music, causing their body movements to lag behind the timing of the beat. The present invention makes it possible to accurately acquire movement information even for such movements that are intentionally (or due to lack of skill, etc.) deviated from the timing of the beat (or are out of sync).

[0063] The application of the above-mentioned criterion (C1) is optional. For example, if there are multiple peaks of normalized representative position information within a predetermined period before and after the timing τ at which the beat is detected, all of these may be used as movement information. In other words, multiple pieces of movement information may be acquired for one beat. Even if the dancers perform a dance to the same music, the number of peaks detected in first subject 1a and second subject 1b may differ, and this difference in the number of peaks may be reflected in the derivation of the similarity.

[0064] The control unit 10 calculates the average time between peaks in the positive region Tpp_P_axn and the average time between peaks in the negative region Tpp_N_axn using the first x-direction data set, which is the movement information of the first subject 1a. These are calculated specifically as follows: In the following equations, k axp represents the number of peaks in the positive region, and k axn represents the number of peaks in the negative region. Tpp_P_axn ={(τ3a-τ1a)+(τ6a-τ3a)+(τ8a-τ6a)+...+(τ14a-τ12a)} / kaxp =(τ12a-τ1a) / kaxp Tpp_N_axn ={(τ4a-τ2a)+(τ5a-τ4a)+(τ7a-τ5a)+...+(τ13a-τ11a)} / kaxn =(τ11a-τ2a) / kaxn

[0065] Similarly, the second x-direction data set, which is the movement information of the second subject 1b, is used to calculate the average time between peaks in the positive region (Tpp_P_axn') and the average time between peaks in the negative region (Tpp_N_axn'). These are calculated specifically as follows: In the following equation, jaxp is the number of peaks in the positive region, and jaxn is the number of peaks in the negative region. ·Tpp_P_axn' =(τ'14a-τ'3a) / jaxp ·Tpp_N_axn' =(τ'13a-τ'1a) / jaxn

[0066] The average peak-to-peak value and the number of peaks in the positive and negative regions of first subject 1a and second subject 1b are closely related to the beat, and if there is a difference between these, it can be determined that second subject 1b (student) has made a mistake in their dance moves. Control unit 10 calculates, for example, F1 as an evaluation function. α and β in F1 are weighting coefficients that may be determined appropriately. Formula 1 =α{|(Tpp_P_axn)-(Tpp_P_axn')|+|(Tpp_N_axn)-(Tpp_N_axn')}+β(|kaxp-jaxp|+|kaxn-jaxn|)

[0067] The control unit 10 may also calculate the difference between the sum of the absolute values ​​of all elements of CGax_fin(x,τ) for the first subject 1a and the sum of the absolute values ​​of all elements of CGax_fin'(x',τ') for the second subject 1b (evaluation function F2). Note that δ is a weighting coefficient, which may be determined appropriately. F2 =δ(Σ|CGax_fin(x,τ)|-Σ|CGax_fin'(x',τ')|)

[0068] F1 and F2 can be used as similarity indices that approach zero as the difference in the motion information between first subject 1a and second subject 1b decreases, that is, as the motion similarity increases. Of course, the following evaluation function F3 may be determined using F1 and F2. F3=F1+F2 F3 can also be used as a measure of similarity. F3 is also an index that approaches zero as the similarity between the movements of both parties increases.

[0069] Thus, in the movement evaluation system S1 of the first embodiment, the subject 1 includes a first subject 1a and a second subject 1b, and the control unit 10 derives the similarity of the movements of the first subject 1a and the second subject 1b based on the movement information of the first subject 1a and the second subject 1b detected based on a predetermined timing extracted from the sound information. This makes it possible to accurately evaluate the similarity of the movements of the first subject 1a and the second subject 1b, which are displaced based on the sound information.

[0070] Furthermore, in the motion evaluation system S1 of the first embodiment, the control unit 10 calculates at least one representative piece of position information representing the position information of each of the first subject 1a and the second subject 1b, and derives the similarity based on the time-series change in the motion information extracted from the representative position information. This makes it possible to calculate the similarity with high accuracy and high speed without processing a large amount of motion information.

[0071] The above describes an example in which the similarity between the first subject 1a and the second subject 1b is derived based on the average peak-to-peak value and the number of peaks in the positive and negative regions of the movement information of the entire body 1AL in the x direction. Of course, similarity may also be derived based on the movement information of both subjects 1 in the y direction (height direction). Furthermore, similarity may be derived based on the average peak-to-peak value and the number of peaks in the positive and negative regions of the movement information of the head 1HD, torso 1BD, and legs 1L in the x and y directions, respectively. Similarity may also be derived by integrating these pieces of movement information. If the imaging unit 13 is configured with a stereo camera, it can measure the displacement of the subject 1 in the front-to-back direction (see FIG. 2(A)). Movement information may be extracted from the amount of displacement in the front-to-back direction to derive similarity.

[0072] (Second embodiment) 7A and 7B are explanatory diagrams illustrating a process for deriving similarity in the second embodiment of the present invention. Here, FIG. 7A shows the distribution of representative position information of first subject 1a in the x and y directions over time, and is an image generated based on the movement of first subject 1a in the x direction (CGax(x,t) shown in FIG. 5A) and the movement in the y direction (not shown). Herein, if the evaluation image is captured for 24 seconds at 60 fps, for example, 60 [fps] × 24 [s] = 1,440 pieces of representative position information (x, y coordinate values) are obtained. These pieces of representative position information are plotted on the x and y coordinates. In FIG. 7A, the plotted area is indicated as Tra. The range of the x and y coordinates is normalized to, for example, a range of 0 to 511. The representative position information is plotted as 8-bit monochrome image data, with a pixel value of, for example, 255. Hereinafter, the image shown in FIG. 7A will be referred to as a "first movement information image."

[0073] FIG. 7(B) shows the distribution of representative position information of second subject 1b in the x and y directions over time, and is generated in the same manner as FIG. 7(A) based on the movement of second subject 1b in the x direction (CGax'(x',t') shown in FIG. 5(B)) and the movement in the y direction (not shown). In FIG. 7(B), the plotted area is indicated as Trb. Hereinafter, the image shown in FIG. 7(B) will be referred to as the "second movement information image."

[0074] The configuration of the motion evaluation system S1 of the second embodiment is the same as that of the first embodiment. The following description will be continued with reference to Fig. 1. The control unit 10 generates a first motion information image and a second motion information image, and substitutes the elements constituting each image into the following [Equation 1] to calculate a structural similarity index (SSIM: Structural Similarity Index Measure).

number

[0075] SSIM provides an evaluation index that takes into account the characteristics of the human visual system based on three elements of image brightness, contrast, and structure. In [Equation 1], x and y are vectors that represent each pixel within a window (here, 512 × 512) in the first motion information image and the second motion information image, respectively. μ is the average pixel value within the window, and σ is the average pixel value within the window. x ,σ y is the standard deviation of the pixel values ​​in the same window, σ xy is the covariance of x and y. Also, C1 and C2 are constants that prevent the evaluation value from becoming unstable when the denominator value becomes very small. Here, C1=(K1L) 2 , C2=(K2L) 2 where L is the dynamic range of pixel values ​​(here, 8 bits: 255). K1 and K2 are constants, e.g., K1=0.01, K2=0.03.

[0076] In this way, in the second embodiment, the SSIM is calculated for the first subject 1a and the second subject 1b using representative position information (center of gravity CGa of the entire body 1AL) in the x and y directions of the entire body 1AL (see FIG. 3). When the first movement information image and the second movement information image perfectly match, SSIM(x, y) = 1, and the lower the similarity, the closer the SSIM value is to 0. The control unit 10 displays the calculated SSIM as the similarity on the display unit 15. Of course, the SSIM may also be calculated using representative position information for the head 1HD, torso 1BD, and legs 1L, or these individual SSIM values ​​may be appropriately combined to use as an index of similarity.

[0077] In calculating SSIM(x,y), the first motion information image and the second motion information image may each be divided into small regions, the SSIM may be calculated for each small region, and the SSIM may be averaged to calculate MSSIM (Mean SSIM). Note that, when deriving the similarity between images, SNR (Signal to Noise Ratio), for example, may be used instead of or in addition to SSIM or MSSIM.

[0078] As described above, the movement evaluation system S1 of the second embodiment includes a sound information output unit 16 that outputs sound information, an imaging unit 13 that captures images of the first subject 1a and the second subject 1b, and a control unit 10. The first subject 1a and the second subject 1b are displaced based on the sound information output by the sound information output unit 16. The control unit 10 calculates at least one representative piece of position information for the first subject 1a and the second subject 1b that represents the respective pieces of position information, generates a first movement information image and a second movement information image that represent the time-series changes in the respective representative position information, and derives a similarity based on the first movement information image and the second movement information image. This allows the movement (trajectory) of a position representative of the subject 1 to be expressed as a two-dimensional image, and makes it possible to derive a similarity based on the difference between the images.

[0079] Furthermore, in the motion evaluation system S1 of the second embodiment, the control unit 10 calculates a structural similarity index (SSIM) based on the first motion information image and the second motion information image, which makes it possible to represent the motion of the subject 1 as an image and derive the similarity taking into account the characteristics of the human visual system.

[0080] A modified example of the second embodiment will be described below. The control unit 10 generates a first movement information image for the first subject 1a based on a first x-direction data set: (x1, τ1a), (x2, τ2a)...(x14, τ14a) acquired based on CGax_fin(x, τ) shown in Fig. 6(A) and a first y-direction data set: (y1, τ1a), (y2, τ2a)...(y14, τ14a) acquired in the same manner as the first x-direction data set.

[0081] Furthermore, for the second subject 1b, a second movement information image is generated based on the second x-direction data set: (x'1, τ'1a), (x'2, τ'2a)...(x'14, τ'14a) obtained based on CGax_fin'(x', τ') shown in Figure 6(B), and the second y-direction data set: (y'1, τ'1a), (y'2, τ'2a)...(y'14, τ'14a) obtained in the same manner as the second x-direction data set. That is, the first x-direction data set, the first y-direction data set, the second x-direction data set, and the second y-direction data set in this modification all use movement information of the subject 1 obtained in synchronization with sound information.

[0082] In this modified example, the control unit 10 also derives the similarity based on the first motion information image and the second motion information image. Thus, in the modified example of the motion evaluation system S1, the control unit 10 generates a first motion information image and a second motion information image representing time-series changes in motion information for each of the first subject 1a and the second subject 1b, and derives the similarity based on the first motion information image and the second motion information image. In this case, SSIM may be used, as in the second embodiment. This allows the motion of a representative position of the subject to be expressed as a two-dimensional image, and the similarity to be derived based on the difference between the images.

[0083] However, in this modification, the number of points (pixels) constituting the motion information image is very small, at 14 ((x1, τ1a) to (x14, τ14a) shown in FIG. 6A) in the above example. If an image is composed of a small number of pixels (dots), the structures of the first and second motion information images will differ significantly, resulting in a very small calculated SSIM and an inappropriate evaluation of the similarity. Therefore, in this modification, the pixels constituting the first and second motion information images are replaced with objects having an area larger than one pixel. Specifically, for example, one pixel is replaced with a circle having a predetermined radius r (e.g., r = 5 pixels) centered on the x and y coordinates of the pixel. In this case, the inside of the circle may be filled with a predetermined value (e.g., 255), or a gradation may be provided in which the pixel value decreases from the center of the circle in the radial direction. By providing a gradation, sensitivity to edge structures can be reduced. Furthermore, areas where multiple circles overlap may be replaced with the average value of each gradation, which suppresses the edges of the objects and intentionally reduces the structural features of the image.

[0084] In this way, the motion estimation system S1 of the modified example renders the representative position information (motion information) as an object of a predetermined size exceeding one pixel size in the first motion information image and the second motion information image, which makes it possible to properly acquire the SSIM even when the number of pieces of representative position information is small.

[0085] (Third embodiment) FIG. 8 is an explanatory diagram illustrating a method for visualizing the movement of subject 1 in a third embodiment of the present invention. FIG. 8 shows CGax(x,t) shown in FIG. 5(A) and CGax'(x',t') shown in FIG. 5(B) superimposed on a radar chart. The movement of first subject 1a is shown by a solid line (hereinafter referred to as the "first graph"), and the movement of second subject 1b is shown by a dashed line (hereinafter referred to as the "second graph"). The "movement" value here refers to representative position information. The radial direction of the radar chart represents the movement (displacement) of subject 1 in the x direction, and the circumferential direction represents the passage of time (here, one rotation is 24 seconds). Subject 1 starts dancing at 0° and finishes dancing at 360°. By representing the movements of first subject 1a and second subject 1b as a radar chart in this way, the similarity between the two can be easily evaluated visually.

[0086] The representative position information of first subject 1a and second subject 1b is the same at 0° and 360° of the radar chart, and both the first graph and the second graph are drawn as closed curves. In a radar chart, the radial direction corresponds to the magnitude of the movement of subject 1, so the larger the movement, the more likely it is to be reflected in the increase in the area of ​​the region surrounded by the closed curve. In this way, in the third embodiment, it is possible to evaluate the dynamism of the movement of subject 1 using the area of ​​the region surrounded by the first graph and the area of ​​the region surrounded by the second graph. Of course, for example, the ratio between the area of ​​the first graph and the area of ​​the second graph may be used as the similarity.

[0087] (Fourth embodiment) FIG. 9 is a block diagram showing the configuration of a movement evaluation system S1 according to a fourth embodiment of the present invention. In the first embodiment, movement information of the subject 1 is extracted based on an evaluation image captured by the imaging unit 13 (see FIG. 1). In the fourth embodiment, movement information of the measurement target 2 is extracted using a movement detection unit 3. Note that the movement evaluation system S1 of the fourth embodiment is obtained by replacing the imaging unit 13 shown in FIG. 1 with the movement detection unit 3 and replacing the subject 1 with a measurement target 2. That is, the measurement target 2 is, for example, a human being, and includes a first measurement target 2a (corresponding to the first subject 1a in the first embodiment) and a second measurement target 2b (corresponding to the second subject 1b in the first embodiment). As in the first embodiment, the first measurement target 2a and the second measurement target 2b are displaced in accordance with the sound information output from the sound information output unit 16.

[0088] A box-shaped motion detection unit 3 is attached to each of the left and right wrists and left and right ankles of the measurement object 2 using a wristband or the like. In order to detect the movement of the arm 1A or leg 1L (see FIG. 2) with high accuracy, the motion detection unit 3 is preferably attached to a part of the body that experiences large displacement. From this perspective, the motion detection unit 3 that detects the movement of the arm 1A is preferably attached to the wrist or held in the palm of the hand. Furthermore, the motion detection unit 3 corresponding to the leg 1L is preferably attached to the ankle. The motion detection unit 3 may also be installed in the head 1HD or torso 1BD (see FIG. 2) of the measurement object 2.

[0089] Here, it is preferable that the correspondence between each motion detection unit 3 and the part of the body where it is to be worn is determined in advance. For example, each motion detection unit 3 is clearly marked with the part of the body where it should be worn, such as "for arm (right)," and the measurement subject 2 wears the motion detection unit 3 on the clearly marked part of the body.

[0090] FIG. 10 is a block diagram showing the configuration of the motion detection unit 3. As shown in FIG. 10, the motion detection unit 3 includes a second control unit 3a, a second storage unit 3b, a second communication unit 3c, and an inertial sensor 3d. The second control unit 3a is configured with a CPU or the like and operates according to a control program stored in a second storage unit 3b, which is configured with a ROM, RAM, or the like. The second control unit 3a is connected to the other components via a bus or the like, and the control unit 10 controls the other components via the bus or the like. The second storage unit 3b further stores an identifier (ID) representing each motion detection unit 3. The second communication unit 3c includes a communication module (not shown) that complies with a short-range wireless communication standard, such as BLE. The second control unit 3a acquires the ID stored in the second storage unit 3b and the output of the inertial sensor 3d and transmits this information to the control unit 10 (see FIG. 9) via the second communication unit 3c.

[0091] The inertial sensor 3d is composed of, for example, a triaxial acceleration sensor and / or a gyro sensor. Here, the triaxial acceleration sensor outputs the direction and degree of speed change of each part of the measurement target 2 (acceleration) for the three axes X, Y, and Z. The gyro sensor outputs the direction and speed of rotation of each part of the measurement target 2 (angular velocity) for the three axes X, Y, and Z. In this way, the inertial sensor 3d detects the movement of the arm 1A and leg 1L of the measurement target 2, and outputs triaxial acceleration information and / or triaxial angular velocity information (hereinafter sometimes referred to as "triaxial acceleration information, etc.") based on this to the control unit 10. At this time, the above-mentioned ID is also output.

[0092] The control unit 10, having received the triaxial acceleration information and the ID, calculates representative position information based on the triaxial acceleration information. By referencing the ID, the control unit 10 determines which motion detection unit 3 the output of the inertial sensor 3d came from. The control unit 10 integrates the output (triaxial acceleration information) of each inertial sensor 3d to obtain velocity information, which it then integrates to obtain position information. The control unit 10 then averages the position information of each motion detection unit 3 to obtain representative position information of the measurement target 2. Note that the ID may also be referenced to obtain representative position information based on the output of a specific inertial sensor 3d. Furthermore, as in the first embodiment, the control unit 10 acquires motion information of the measurement target 2 based on the beat and rhythm of the sound information. Based on this motion information, the control unit 10 derives the degree of similarity between the motions of the first measurement target 2a and the second measurement target 2b.

[0093] As described above, the movement evaluation system S1 of the fourth embodiment includes a sound information output unit 16 that outputs sound information, a movement detection unit 3 that detects the movement of the measurement object 2 that displaces based on the sound information, and a control unit 10, and the control unit 10 acquires the movement information of the measurement object 2 based on a predetermined timing extracted from the sound information. This makes it possible to accurately acquire the movement information of the measurement object 2 that displaces based on the sound information.

[0094] Furthermore, in the movement evaluation system S1 of the fourth embodiment, the measurement target 2 includes a first measurement target 2a and a second measurement target 2b, and the control unit 10 derives the similarity of the movements of the first measurement target 2a and the second measurement target 2b based on movement information of the first measurement target 2a and the second measurement target 2b detected based on a predetermined timing extracted from sound information. This makes it possible to accurately evaluate the similarity of the movements of the first measurement target 2a and the second measurement target 2b, which are displaced based on sound information.

[0095] The movement evaluation system S1 and the movement evaluation method according to the present invention have been described in detail above based on specific embodiments, but these embodiments are merely examples, and the present invention is not limited to these embodiments. For example, the first subject 1a and the second subject 1b (or the first measurement target 2a and the second measurement target 2b) may be the same person. By acquiring video files of the same subject 1 performing a dance at different times and deriving the similarity between them, it is possible to numerically express the results of training for the same person.

[0096] Furthermore, the subject 1 or the measurement target 2 does not have to be a human. Specifically, for example, either the first subject 1a or the second subject 1b may be a robot. In this case, the robot is programmed to displace in accordance with the sound information. Then, based on the similarity, for example, the smoothness of the robot's movement, response speed, and displacement amount can be evaluated. Of course, both the first subject 1a and the second subject 1b may be robots.

[0097] In the first to third embodiments, the representative position information is calculated based on the coordinate values ​​of the key points 41 of the pose recognition model 40 (see FIG. 3), but the so-called motion capture technology may be used to calculate the representative position information. Specifically, a plurality of reflective markers attached to the subject 1 are photographed by the imaging unit 13, and the representative position information is acquired based on the coordinates of the detected reflective markers.

[0098] In the first embodiment, the peak values ​​of representative position information before and after the timing based on the detected beat or rhythm are extracted as the motion information, but the frame image (evaluation image) closest in time to the timing when the beat is detected may be selected, and the representative position information acquired from this evaluation image may be used as the motion information. Also, the average value of multiple representative position information acquired within a predetermined period around the timing when the beat is detected may be used as the motion information.

[0099] Furthermore, in each embodiment, even if two subjects 1 (measurement targets 2) perform the same dance to the same music with the same choreography, differences in skill and proficiency will result in differences in the movements of the two subjects 1. Therefore, the first subject 1a has been described as a dance instructor and the second subject 1b as his / her student. On the other hand, the present invention may also be applied to rehabilitation conducted between doctors and elderly people, or to the guidance and support of children with developmental disorders. [Industrial Applicability]

[0100] The movement evaluation system S1 and movement evaluation method of the present invention can improve dance performance by evaluating the similarity of movements between instructors and students and reflecting this in instruction, and can also further improve the motor functions and attention spans of children with developmental disabilities and the elderly.Therefore, they can be widely used in dance classes, support settings for children with developmental disabilities, elderly care facilities, home care settings, etc. [Explanation of symbols]

[0101] 1. Subject 1a First subject 1b 2nd subject 2. Measurement target 3. Motion detection section 10 Control Unit 13 Imaging unit 16 Sound information output section 40 Pose Recognition Models 41 Key Points 50 Network S1 Movement Evaluation System

Claims

1. a sound information output unit that outputs sound information; an imaging unit that captures an image of a subject that is displaced based on the sound information; A control unit; Equipped with The control unit A movement evaluation system, characterized in that movement information of the subject is acquired from the output of the imaging unit based on a predetermined timing extracted from the sound information.

2. the objects include a first object and a second object, The control unit The movement evaluation system according to claim 1, further comprising: a movement evaluation unit for deriving a degree of similarity between the movements of the first subject and the second subject based on the movement information of the first subject and the second subject detected based on a predetermined timing extracted from the sound information.

3. The control unit calculating at least one representative piece of position information representing each of the position information of the first object and the second object; 3. The movement evaluation system according to claim 2, wherein the similarity is derived based on a time-series change in the movement information extracted from the representative position information.

4. The control unit generating a first motion information image and a second motion information image representing a time-series change in the motion information for each of the first object and the second object; 4. The motion estimation system according to claim 3, wherein the similarity is derived based on the first motion information image and the second motion information image.

5. 5. The motion estimation system according to claim 4, wherein the motion information is rendered as an object of a predetermined size exceeding one pixel size in the first motion information image and the second motion information image.

6. The control unit Detecting a beat or rhythm contained in the sound information; 4. The movement evaluation system according to claim 3, wherein peak values ​​of the representative position information preceding or following a timing based on the detected beat or rhythm are extracted as the movement information.

7. The sound information is music, The control unit 6. The movement evaluation system according to claim 1, wherein the predetermined timing is determined based on a change in sound pressure of the sound information.

8. The control unit 8. The movement evaluation system according to claim 7, wherein the predetermined timing is determined to be when the sound pressure of the sound information exceeds a predetermined value or when a change in the sound pressure of the sound information exceeds a predetermined value.

9. a sound information output unit that outputs sound information; an imaging unit that captures images of a first subject and a second subject; A control unit; Equipped with the first object and the second object are displaced based on the sound information output by the sound information output unit, The control unit calculating at least one representative piece of position information representing each piece of position information for the first object and the second object, and further generating a first movement information image and a second movement information image representing a time-series change in each piece of the representative position information; A motion estimation system, characterized in that a similarity is derived based on the first motion information image and the second motion information image.

10. The control unit 10. The motion estimation system according to claim 4, wherein a structural similarity index (SSIM) is calculated as the similarity based on the first motion information image and the second motion information image.

11. a sound information output unit that outputs sound information; a movement detection unit that detects the movement of a measurement object that is displaced based on the sound information; A control unit; Equipped with The control unit A movement evaluation system, characterized in that movement information of the measurement object is acquired based on a predetermined timing extracted from the sound information.

12. the measurement targets include a first measurement target and a second measurement target, The control unit The movement evaluation system described in claim 11, characterized in that the similarity of the movements of the first measurement object and the second measurement object is derived based on the movement information of the first measurement object and the second measurement object detected based on a predetermined timing extracted from the sound information.

13. Outputs sound information, capturing an image of a subject that is displaced based on the sound information; A movement evaluation method, characterized in that movement information of the subject is acquired based on a predetermined timing extracted from the sound information.

14. the objects include a first object and a second object, 14. The movement evaluation method according to claim 13, further comprising deriving a similarity between the movements of the first object and the second object based on the movement information between the first object and the second object.

15. Detects the movement of the measurement target, which displaces based on sound information, A movement evaluation method, characterized in that movement information of the measurement object is acquired based on a predetermined timing extracted from the sound information.

16. the measurement targets include a first measurement target and a second measurement target, The movement evaluation method according to claim 15, characterized in that the similarity of the movements of the first measurement object and the second measurement object is derived based on the movement information of the first measurement object and the second measurement object detected based on a predetermined timing extracted from the sound information.

Images