Motion evaluation device, motion evaluation method, and program

Abstract

An action evaluation device (100a) comprises: an action specifying means (108a) that extracts skeletal information for a person in an acquired image and, on the basis of the extracted skeletal information for the person and a registered action pattern that comprises stored skeletal information, specifies an action to be evaluated for the body of the person; and an evaluation means (110a) that evaluates the similarity between the action to be evaluated and a sample action pattern that comprises stored skeletal information on the basis of an integrated evaluation value that incorporates a first evaluation value based on the deviation of the skeletal information in a time axis direction and a second evaluation value based on the deviation of the skeletal information for the person in a space axis direction.

Description

[Technical Field]

[0001] The present disclosure relates to a motion evaluation device, a motion evaluation method, and a non-transitory computer-readable medium. [Background technology]

[0002] Systems are being used that detect people's movements and evaluate their similarity and efficiency.

[0003] For example, in Patent Document 1, a means for acquiring a model video and a means for acquiring an imitation video are provided. The present invention discloses a motion similarity evaluation device that includes a means for extracting skeletal information of each motion from a model video and an imitation video, and a means for evaluating the similarity of each motion based on the similarity of each skeletal information at a synchronized time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-195648 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, the similarity of each movement is evaluated based on the similarity of each piece of skeletal information at the synchronized time, and the movement of a person cannot be evaluated appropriately.

[0006] In view of the above-mentioned problems, an object of the present disclosure is to provide an action evaluation device, an action evaluation method, and a non-transitory computer-readable medium that are capable of appropriately evaluating actions. [Means for solving the problem]

[0007] An action evaluation device according to one aspect of the present disclosure includes: a movement identification means for extracting skeletal information of a person in the acquired image, and identifying an evaluation target movement of the person's body based on the extracted skeletal information of the person and a registered movement pattern made up of the stored skeletal information; The apparatus is provided with an evaluation means for evaluating the similarity between the evaluation target action and a sample action pattern consisting of stored skeletal information based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction.

[0008] A motion evaluation method according to one aspect of the present disclosure includes: extracting skeletal information of a person from the acquired image, and identifying an evaluation target motion of the person's body based on the extracted skeletal information of the person and a registered motion pattern consisting of the stored skeletal information; The similarity between the evaluation target action and a sample action pattern consisting of stored skeletal information is evaluated based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction.

[0009] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable medium, comprising: A process of extracting skeletal information of a person in the acquired image, and specifying an evaluation target motion of the person's body based on the extracted skeletal information of the person and a registered motion pattern consisting of the stored skeletal information; a process of evaluating the similarity between the evaluation target action and a sample action pattern made up of stored skeletal information based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction; A program for causing a computer to execute the above is stored. [Effects of the Invention]

[0010] The present disclosure makes it possible to provide a motion evaluation device, a motion evaluation method, and a non-transitory computer-readable medium that are capable of appropriately evaluating motion. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an action evaluation device according to a first embodiment. [Figure 2] 1 is a flowchart showing a movement evaluation method according to the first embodiment. [Figure 3] FIG. 10 is a diagram showing the overall configuration of an action evaluation system according to a second embodiment. [Figure 4] FIG. 10 is a block diagram showing the configuration of a server according to a second embodiment. [Figure 5] FIG. 10 is a diagram showing skeletal information extracted from a frame image included in video data according to the second embodiment. [Figure 6] FIG. 10 is a diagram showing an example of registered skeleton information according to the second embodiment. [Figure 7] 10A and 10B are diagrams illustrating an example of evaluating the similarity in consideration of the amount of deviation in the time axis direction between a sample action and an evaluation target action. [Figure 8] 10A and 10B are diagrams illustrating an example of evaluating similarity in consideration of the amount of deviation in the spatial axis direction at the synchronization time between skeletal information of a sample action and skeletal information of an evaluation target action. [Figure 9] 10 is a flowchart showing a method for acquiring video data by the action evaluation device according to the second embodiment. [Figure 10] 10 is a flowchart showing a method for registering a registration operation ID and a registration operation sequence by a server according to the second embodiment. [Figure 11] 10 is a flowchart showing a movement evaluation method according to a second embodiment. [Figure 12] FIG. 10 is a diagram showing the overall configuration of an action evaluation system according to a third embodiment. [Figure 13] FIG. 10 is a block diagram showing the configuration of a server according to a third embodiment. [Figure 14] FIG. 2 is a block diagram showing an example of a hardware configuration of an action evaluation device, etc. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present disclosure will be described below through embodiments, but the disclosure according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary.

[0013] <Embodiment 1> FIG. 1 is a block diagram showing the configuration of an action evaluation device 100a according to a first embodiment. The action evaluation device 100a is a computer that captures an image of a user U performing a predetermined action, compares the captured image with a sample action, and evaluates the action of the user U. By evaluating the action of a worker, the action evaluation device 100a can evaluate the quality and efficiency of work and identify the proficiency level (e.g., advanced, intermediate, beginner, etc.) of a person's action (e.g., dance). The action evaluation device 100a includes an action identification unit 108a and an evaluation unit 110a.

[0014] The motion identification unit 108a is also called a motion identification means. The motion identification unit 108a extracts skeletal information of a person in the acquired image, and identifies a motion to be evaluated that is performed on the person's body based on the extracted skeletal information of the person and a registered motion pattern made up of the stored skeletal information.

[0015] A person's body may be at least a part of the body that determines the posture, such as the hands, shoulders, torso, legs, face, or neck, and more specifically, may be the entire skeleton or a skeleton corresponding to a part of the body (for example, only the hands or only the upper body).

[0016] The evaluation unit 110a is also called an evaluation means. The evaluation unit 110a evaluates the similarity between the evaluation target motion and a sample motion pattern made up of stored skeletal information based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction.

[0017] The amount of deviation of the skeletal information in the time axis direction is calculated by aligning the start points of the sample movement pattern and the movement to be evaluated, matching each frame based on the similarity of the skeletal information of each frame, and then calculating the distance between the matched frames.

[0018] The amount of deviation of the skeleton information in the spatial axis direction is calculated based on the amount of deviation of the geometric shape of the skeleton information by associating a similar sample action with the action to be evaluated.

[0019] The enrollment motion pattern may be pre-stored reference data for identifying a person's motion. The sample motion pattern may be pre-stored reference data for evaluating a person's motion. In some embodiments, the enrollment motion pattern may include the sample motion pattern.

[0020] FIG. 2 is a flowchart showing the action evaluation method according to the first embodiment. The motion identification unit 108a extracts skeletal information of the person in the acquired image, and identifies a motion to be evaluated that is performed on the person's body based on the extracted skeletal information of the person and a registered motion pattern made up of the stored skeletal information (step S11).The evaluation unit 110a evaluates the similarity between the motion to be evaluated and a sample motion pattern made up of the stored skeletal information based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction (step S12).

[0021] According to the first embodiment, it is possible to evaluate not only the deviation of a movement in the spatial axis direction but also the deviation of a movement in the time axis direction, and therefore it is possible to provide a movement evaluation device, a movement evaluation method, etc. that can appropriately evaluate a person's movement.

[0022] <Embodiment 2> 3 is a diagram showing the overall configuration of a movement evaluation system 1 according to a second embodiment. The movement evaluation system 1 is a computer system that captures an image of a user U performing a predetermined movement, compares the image with a sample movement, and evaluates the movement of the user U. The following description mainly focuses on the evaluation of dance, but the present disclosure is not limited thereto. The present disclosure can also be applied to, for example, evaluating the quality and proficiency of work performed by a worker.

[0023] The action evaluation system 1 includes an action evaluation device 100 and a camera 300. The action evaluation device 100 is communicatively connected to the camera 300 via a network N. The network N may be wired or wireless. The action evaluation device 100 may be a local computer (e.g., a desktop computer, a laptop computer, a tablet, a smartphone, etc.) or a server computer. Furthermore, the action evaluation device 100 may be configured as a single computer or multiple computers.

[0024] The camera 300 captures an image of the user U performing a predetermined action. The camera 300 is disposed at a position and angle that allows it to capture an image of at least a part of the body of the user U. In the second embodiment, the camera 300 may be a plurality of cameras.

[0025] The action evaluation device 100 is a computer device that compares the actions of the user U with sample action data and evaluates them based on the video data received from the camera 300. The action evaluation device 100 can also be used by the user U or a person evaluating the actions of the user U (hereinafter referred to as an evaluator) to visually check the evaluation results.

[0026] FIG. 4 is a block diagram showing the configuration of the action evaluating device 100 according to the second embodiment.

[0027] (Movement evaluation device 100) The action evaluation device 100 includes a communication unit 201 , a control unit 202 , a display unit 203 , an audio output unit 204 , a microphone 205 , and an operation unit 206 .

[0028] The communication unit 201 is also called a communication means. The communication unit 201 is a communication interface with the network N. The communication unit 201 is also connected to the camera 300, and can acquire video data from the camera 300 at predetermined time intervals.

[0029] The control unit 202 is also referred to as a control means. The control unit 202 controls the hardware of the action evaluation device 100. For example, when the control unit 202 detects a start trigger, the action evaluation device 100 starts acquiring video data from the camera 300. The detection of the start trigger refers to, for example, "detecting the start of dance music with a microphone" or "the evaluator operating the action evaluation device 100 to start evaluating the user's action." Furthermore, for example, when the control unit 202 detects an end trigger, the action evaluation device 100 stops acquiring video data from the camera 300. The detection of the end trigger refers to, for example, "detecting the end of dance music with a microphone" or "the evaluator operating the action evaluation device 100 to end evaluation of the user's action." Note that these start triggers and end triggers are merely examples, and various modifications are possible.

[0030] When the action evaluation device 100 evaluates the user's action as good or bad, the control unit 202 may cause the display unit 203 to display a predetermined display in accordance with the evaluation result. Furthermore, the control unit 202 may cause the audio output unit 204 to output a predetermined sound in accordance with the evaluation result.

[0031] The display unit 203 is a display device. The audio output unit 204 is an audio output device including a speaker. The microphone 205 acquires external audio (e.g., dance music). The operation unit 206 is a mouse, keyboard, touch panel, or the like, and receives input from an operator.

[0032] The action evaluation device 100 also includes a registration information acquisition unit 101, a registration unit 102, an action DB 103, a sample action sequence table 104, a selection unit 105, an image acquisition unit 106, an extraction unit 107, an action identification unit 108, a generation unit 109, an evaluation unit 110, and a processing control unit 111.

[0033] The registration information acquisition unit 101 is also referred to as a registration information acquisition means. The registration information acquisition unit 101 acquires multiple registration video data through operations by an administrator of the action evaluation device 100. In the second embodiment, each registration video data may be video data showing a person's movement. Each registration video data may be video data showing a model movement, or may be video data showing a normal person's movement that is not a model. The model movement may be, for example, video data of various dances (e.g., hip hop, tango) performed by an experienced dancer. The registration video data also includes individual movements (e.g., unit movements included in a dance, such as a box step). Note that, in the second embodiment, the registration video data is a video including multiple frame images, but may also be a still image (a single frame image). The registration movement pattern may be used to identify a person's movement. The registration movement pattern related to the model movement may be used to evaluate the person's movement, which will be described later.

[0034] Furthermore, the registered information acquiring unit 101 acquires, through operation by the administrator of the action evaluating device 100, a plurality of registered action IDs and information on the chronological order in which the actions are performed in a series of actions.

[0035] The registration information acquisition unit 101 supplies the acquired information to the registration unit 102.

[0036] The registration unit 102 is also called a registration means. First, the registration unit 102 executes an action registration process in response to an action registration request. Specifically, the registration unit 102 supplies registration video data to the extraction unit 107 (described later), and acquires skeleton information extracted from the registration video data from the extraction unit 107 as registered skeleton information. The registration unit 102 then registers the acquired registered skeleton information in the action DB 103 in association with a registered action ID.

[0037] Next, the registration unit 102 executes a sequence registration process in response to the sequence registration request. Specifically, the registration unit 102 generates a registration movement sequence by chronologically arranging the registration movement IDs based on the chronological order information. Skeleton information extracted from video data of various dances (e.g., hip hop, tango) performed by experienced dancers may be directly registered as a sample movement sequence. A sample movement sequence is also called a sample movement pattern. In this case, if the sequence registration request is for a first sample movement (e.g., hip hop), the registration unit 102 registers the generated registration movement sequence in the sample movement sequence table 104 as a first sample movement sequence SA1. On the other hand, if the sequence registration request is for a second sample movement (e.g., tango), the registration unit 102 registers the generated registration movement sequence in the sample movement sequence table 104 as a second sample movement sequence SA2. Sample movements can be registered by dance type or difficulty level (e.g., for advanced, intermediate, or beginner). Furthermore, even for the same dance, different sample movement sequences may be registered for each body part of interest (e.g., body parts above the neck, lower body, etc.). In other words, different sample movement sequences may have different evaluation criteria. Each registered movement sequence may be registered together with information about the body part that should be focused on during evaluation and the degree of attention for each body part. In some embodiments, the registered movement sequence and the sample movement sequence may be the same.

[0038] The movement DB 103 is a storage device that stores registered skeleton information corresponding to each unit movement (for example, box step) included in a predetermined movement (for example, dance) in association with a registered movement ID.

[0039] The sample movement sequence table 104 stores a large number of sample movement sequences SA1, SA2, ... SAN. The sample movement sequences are also called sample movement patterns, and can be used to evaluate movements by comparing them with human movements and calculating the similarity.

[0040] The selection unit 105 is also called a selection means. operation The selection unit 105 selects at least one desired sample movement pattern from the plurality of sample movement patterns through the selection unit 206. Alternatively, the selection unit 105 may select one corresponding sample movement pattern depending on the music (e.g., dance music) being played and acquired through the microphone 205. When selecting a plurality of sample movement patterns, the selection unit 105 may set different weightings for each sample movement pattern. That is, the evaluation value may be calculated taking into account the different weightings of the plurality of sample movement patterns. Alternatively, the average, median, maximum, minimum, etc. of the evaluation values based on the plurality of sample movement patterns may be used. This allows the evaluator or the like to select a body part to focus on and appropriately evaluate it.

[0041] The image acquisition unit 106 is also called image acquisition means. The image acquisition unit 106 acquires video data captured by the camera 300 when the action evaluation device 100 is in operation. That is, the image acquisition unit 106 acquires video data in response to detection of a start trigger. The image acquisition unit 106 supplies frame images included in the acquired video data to the extraction unit 107.

[0042] The extraction unit 107 is also called extraction means. The extraction unit 107 detects an image area (body area) of a person's body from a frame image included in the video data and extracts (e.g., cuts out) it as a body image. Then, the extraction unit 107 uses a skeleton estimation technique using machine learning to extract skeleton information of at least a part of the person's body based on features such as the person's joints recognized in the body image. The skeleton information is information composed of "key points" that are characteristic points such as joints, and "bones (or bone links)" that indicate links between the key points. The extraction unit 107 may use a skeleton estimation technique such as OpenPose. The extraction unit 107 supplies the extracted skeleton information to the motion identification unit 108.

[0043] The action identification unit 108 is also called action identification means. The action identification unit 108 converts skeletal information extracted from video data acquired during operation into an action ID using the action DB 103. In this way, the action identification unit 108 identifies an action. Specifically, first, the action identification unit 108 identifies registered skeletal information from the registered skeletal information registered in the action DB 103, the registered skeletal information having a similarity to the skeletal information extracted by the extraction unit 107 equal to or greater than a predetermined threshold. Then, the action identification unit 108 identifies the registered action ID associated with the identified registered skeletal information as the action ID corresponding to the person included in the acquired frame image.

[0044] Here, the movement identification unit 108 may identify one behavior ID based on skeletal information corresponding to one frame image, or may identify one movement ID based on time-series data of skeletal information corresponding to each of a plurality of frame images. The movement identification unit 108 may identify skeletal information in which a weighting for a target part included in the sample movement is higher than a threshold value. This allows the movement identification unit 108 to focus on even parts with small movements.

[0045] In another embodiment, when identifying one action ID using multiple frame images, the action identification unit 108 may extract only skeletal information with large movements and compare the extracted skeletal information with registered skeletal information in the action DB 103. Extracting only skeletal information with large movements may mean extracting skeletal information in which the difference between skeletal information of different frame images included within a predetermined period is equal to or greater than a predetermined amount. Since only a small amount of matching is required in this way, the calculation load can be reduced and the amount of registered skeletal information can also be reduced. Furthermore, since the duration of actions varies from person to person, only skeletal information with large movements is subject to matching, thereby making action detection robust.

[0046] In addition to the above-mentioned method, various other methods are possible for identifying the action ID. For example, there is a method of estimating the action ID from the target video data using an action estimation model trained on video data that has been assigned correct answers with action IDs as training data. However, collecting this training data is difficult and expensive. In contrast, in the second embodiment, skeletal information is used to estimate the action ID, and the action DB 103 is used to compare it with pre-registered skeletal information. Therefore, in the second embodiment, the action evaluation device 100 can identify the action ID more easily.

[0047] The generation unit 109 is also called a generation means. The generation unit 109 generates an action sequence based on the multiple action IDs identified by the action identification unit 108. The action sequence is configured to include the multiple action IDs in chronological order. The generation unit 109 supplies the generated action sequence to the evaluation unit 110.

[0048] The evaluation unit 110 is also called an evaluation means. The evaluation unit 110 determines whether the generated operation sequence matches (corresponds to) a sample operation sequence (e.g., a first sample operation SA1) registered in the operation sequence table 104 and selected by the selection unit 105.

[0049] In some embodiments, the evaluation unit 110 can evaluate the similarity between the sample action and the evaluation target action by taking into account the amount of deviation in the time axis direction on the same time axis between the sample action and the evaluation target action. Furthermore, the evaluation unit 110 can evaluate the similarity between the sample action and the evaluation target action by taking into account the amount of deviation in the geometric shape in the space axis direction of the extracted person's skeletal information, regardless of the time axis. This allows the person's action to be appropriately evaluated.

[0050] The process control unit 111 is also called a process control means. The process control unit 111 outputs information related to the evaluation result of the generated operation sequence. In this case, the process control unit 111 is also called an output means. The process control unit 111 can cause the display unit 203 to display the evaluation result. Alternatively, the process control unit 111 can cause the audio output unit 204 to output the evaluation result as audio.

[0051] For example, the display mode (such as font, color, thickness, or blinking of characters) when displaying information related to the evaluation may be changed depending on the evaluation result, or the volume or the voice itself may be changed when outputting information related to the evaluation as audio. This allows the evaluator or the user performing the action to recognize the content of the evaluation and take prompt and appropriate action to improve their action. Furthermore, the processing control unit 111 may record the time, location, and video of the action that received a predetermined evaluation (bad evaluation) as history information along with the evaluation information. This allows the evaluator or the user performing the action to recognize the content of the evaluation and appropriately improve their action so as to receive a good evaluation.

[0052] Fig. 5 is a diagram showing skeletal information extracted from a frame image IMG40 included in video data according to embodiment 2. The frame image IMG40 includes an image area of the entire body of the user U when the user U is dancing and photographed from the front. The skeletal information shown in Fig. 5 also includes a plurality of key points and a plurality of bones detected from the upper body. As an example, in FIG. 5, the key points are shown as right ear A11, left ear A12, right eye A21, left eye A22, nose A3, neck A4, right shoulder A51, left shoulder A52, right elbow A61, left elbow A62, right wrist A71, left wrist A72, right palm A81, left palm A82, center of chest A8, right hip A91, tanden A9, left hip A92, right knee A101, left knee A102, right ankle A111, left ankle A112, right heel A121, left heel A122, right instep A131, and left instep A132.

[0053] The movement evaluation device 100 compares such skeletal information with registered skeletal information corresponding to the entire body in the sample movement and determines whether they are similar, thereby evaluating each movement. FIG. 6 shows an example of registered skeletal information extracted from a sample image SP40 of the corresponding sample movement (also indicated as SP in the figure). Furthermore, the registered skeletal information may also register areas of interest F01 and F02. The areas of interest may include one or more body regions. The areas of interest may be arbitrarily set by the evaluator via the operation unit 206 of the movement evaluation device 100.

[0054] For example, in the dance movement shown in Fig. 6, the gestures of the right and left hands may be important. Therefore, the movement evaluation device 100 may calculate and evaluate the similarity by weighting the part F01 including the right shoulder A51, right elbow A61, right wrist A71, and palm of the right hand A81, and the part F02 including the left shoulder A52, left elbow A62, left wrist A72, and palm of the left hand A82. For example, the similarity of these parts can be evaluated more accurately by weighting them more heavily than other parts. In some embodiments, only the part to be focused on may be evaluated, and other parts may not be evaluated.

[0055] FIG. 7 is a diagram illustrating an example of evaluating similarity by taking into account the amount of deviation along the time axis between a sample motion and a motion to be evaluated. The horizontal axis represents time (seconds). A plurality of frame images related to the sample motion data arranged in chronological order are shown above the horizontal axis. A plurality of frame images related to the motion data to be evaluated are shown below the horizontal axis. The sample motion data and the motion data to be evaluated can be arranged on the same time axis by, for example, matching the start times of motions or dance music. As shown in FIG. 7, the motion to be evaluated is delayed by t1 (seconds) relative to the sample motion. In this way, according to this embodiment, motions can be evaluated by taking into account not only the amount of deviation (or similarity) in the geometric shapes between the skeletal information related to the sample motion and the skeletal information related to the motion to be evaluated, but also the deviation along the time axis on the same time axis.

[0056] FIG. 8 illustrates an example of evaluating similarity by taking into account the amount of deviation in the spatial axis direction at the synchronization time between the skeletal information of the sample movement and the skeletal information of the movement to be evaluated. The upper part of FIG. 8 shows the skeletal information of the sample movement, and the lower part shows the skeletal information of the movement to be evaluated. The sample movement pattern is also registered and stored for each frame. Skeletal information related to the movement to be evaluated is also acquired for each frame. Frames with similar skeletal shapes are matched (or synchronized) with each other regardless of time, and the deviation in the spatial axis direction is compared to calculate the similarity. The angle of each bone can be calculated during the evaluation. Furthermore, to offset differences in body shape between the dancer performing the sample movement and the dancer performing the movement to be evaluated, the skeletal sizes (i.e., bone lengths) of both dancers may be normalized. This allows, for example, the pseudo-skeleton of the movement to be evaluated to indicate that the left elbow is lowered from the left shoulder compared to the pseudo-skeleton of the sample movement. In this case, by calculating the similarity in the geometric shape of the pseudo-skeleton between the sample movement and the movement to be evaluated without taking into account the difference in the time axis direction, it is possible to show, for example, that the movement to be evaluated is slower than the sample movement, but the shape of the movement is consistent.

[0057] 9 is a flowchart showing a method for acquiring video data by the action evaluating apparatus 100 according to the second embodiment. First, the control unit 202 of the action evaluating apparatus 100 determines whether a start trigger has been detected (S20). If the control unit 202 determines that a start trigger has been detected (Yes in S20), it starts acquiring video data from the camera 300 into the action evaluating apparatus 100 (S21). On the other hand, if the control unit 202 does not determine that a start trigger has been detected (No in S20), it repeats the process shown in S20.

[0058] Next, the control unit 202 of the action evaluation device 100 determines whether or not an end trigger has been detected (S22). If the control unit 202 determines that an end trigger has been detected (Yes in S22), it ends the acquisition of video data from the camera 300 to the action evaluation device 100 (S23). On the other hand, if the control unit 202 does not determine that an end trigger has been detected (No in S22), it repeats the process shown in S22 while transmitting the video data.

[0059] In this way, by limiting the video data acquisition period to the period between a predetermined start trigger and end trigger, the amount of communication data can be minimized. Furthermore, outside this period, the action detection process in the action evaluation device 100 can be omitted, thereby saving computational resources.

[0060] 10 is a flowchart showing a method for registering a registration action ID and a registration action sequence by the action evaluation device 100 according to the second embodiment. First, the registration information acquisition unit 101 of the action evaluation device 100 receives a action registration request including registration video data and a registration action ID from the action evaluation device 100 (S30). Next, the registration unit 102 supplies the registration video data to the extraction unit 107. Having acquired the registration video data, the extraction unit 107 extracts body images from frame images included in the registration video data (S31). Next, the extraction unit 107 extracts skeletal information from the body images (S32). Next, the registration unit 102 acquires the skeletal information from the extraction unit 107 and registers the acquired skeletal information as registration skeletal information in the action DB 103 in association with the registration action ID (S33). Note that the registration unit 102 may register all of the skeletal information extracted from the body images as the registration skeletal information, or may register only a portion of the skeletal information (for example, skeletal information of the shoulders, elbows, and hands) as the registration skeletal information.

[0061] Next, the registration information acquisition unit 101 receives a sequence registration request including a plurality of registration action IDs and information on the chronological order of each action from the action evaluation device 100 (S34). Next, the registration unit 102 registers a registration action sequence (sample action sequence SA) in which the registration action IDs are arranged based on the information on the chronological order in the action sequence table 104 (S35). Then, the action evaluation device 100 ends the process.

[0062] Figure 1 11 is a flowchart showing a movement evaluation method by the movement evaluation device 100 according to the second embodiment. First, when the image acquisition unit 106 of the movement evaluation device 100 starts acquiring video data from the movement evaluation device 100 (Yes in S40), the extraction unit 107 extracts body images from frame images included in the video data (S41). Next, the extraction unit 107 extracts skeletal information from the body images (S42). The movement identification unit 108 calculates the similarity between at least a portion of the extracted skeletal information and each piece of registered skeletal information registered in the movement DB 103, and identifies, as a movement ID, a registered movement ID associated with registered skeletal information whose similarity is equal to or greater than a predetermined threshold (S43). Next, the generation unit 109 adds the movement ID to a movement sequence. Specifically, in the first cycle, the generation unit 109 sets the movement ID identified in S43 as a movement sequence, and in subsequent cycles, adds the movement ID identified in S43 to the already generated movement sequence. The action evaluation device 100 then determines whether a predetermined action (for example, dancing) has ended or whether acquisition of video data has ended (S45). Note that the action evaluation device 100 may determine that dancing has ended if the action identified in S43 of the current cycle is the action of a predetermined registered action ID. If the action evaluation device 100 determines that dancing has ended or acquisition of video data has ended (Yes in S45), the process proceeds to S46; otherwise (No in S45), the process returns to S41 and the action sequence addition process is repeated.

[0063] In S46, the evaluation unit 110 determines whether the action sequence to be evaluated corresponds to the sample action sequence SA selected in the sample action sequence table 104. Specifically, the evaluation unit 110 evaluates the similarity of the action sequence to be evaluated to the sample action sequence SA, taking into account the deviation in the time axis direction (S46). Next, the evaluation unit 110 evaluates the similarity of each unit action in the action sequence to be evaluated to each action of the sample action sequence SA, taking into account the deviation in the space axis direction (S47).

[0064] The process control unit 111 outputs evaluation display information according to the evaluation result (for example, to the display unit 203) (S48), and the action evaluation device 100 then ends the process.

[0065] Thus, according to embodiment 2, the action evaluation device 100 can evaluate the flow of user U's actions and the form of movement by comparing the action sequence showing the flow of user U's actions with the sample action sequence SA.

[0066] <Embodiment 3> In the third embodiment, part of the motion detection and evaluation process, which involves a large processing load, is performed by the server. 12 is a diagram showing the overall configuration of a movement evaluation system 1b according to embodiment 3. The movement evaluation system 1b is a computer system for capturing an image of a user U performing a predetermined movement, comparing the image with a sample movement, and evaluating the movement of the user U.

[0067] The action evaluation system 1b includes an action evaluation device 100b, a terminal device 200b, and a camera 300. The action evaluation device 100b is communicably connected to the camera 300 and the terminal device 200 via a network N. The network N may be wired or wireless. The action evaluation device 100b may be a server computer. The terminal device 200 may be a local computer (e.g., a desktop computer, a laptop computer, a tablet, a smartphone, etc.).

[0068] FIG. 12 is a block diagram showing the configurations of an action evaluation device 100b and a terminal device 200b according to the third embodiment.

[0069] (Terminal device 200b) The terminal device 200b includes a communication unit 201b, a control unit 202, a display unit 203, an audio output unit 204, a microphone 205, and an operation unit 206.

[0070] The basic configuration is the same as in the second embodiment, and therefore a detailed description thereof will be omitted here. The communication unit 201 transmits the video data acquired from the camera 300 to the action evaluation device 100b as appropriate.

[0071] (Motion evaluation device 100b) The action evaluation device 100b includes a registration information acquisition unit 101, a registration unit 102, an action DB 103, a sample action sequence table 104, a selection unit 105, an image acquisition unit 106b, an extraction unit 107, an action identification unit 108, a generation unit 109, an evaluation unit 110, and a processing control unit 111b.

[0072] The basic configuration is the same as in the second embodiment, so a detailed description will be omitted here. The image acquisition unit 106b acquires video data from the camera 300 via the network and the communication unit 201b of the terminal device 200b. The action evaluation device 100b evaluates the action as described above. Thereafter, the process control unit 111b returns the evaluation result to the terminal device 200b.

[0073] <Other embodiments> In other embodiments, camera 300 may be an intelligent camera. In this case, camera 300 includes a processor, memory, various image sensors, etc. Such an intelligent camera may also include all or some of the components of action evaluation device 100 described above.

[0074] FIG. 14 is a block diagram showing an example of the hardware configuration of the action evaluation device 100 and the terminal device 200 (hereinafter referred to as the action evaluation device 100, etc.). Referring to FIG. 14, the action evaluation device 100, etc. includes a network interface 1201, a processor 1202, and a memory 1203. The network interface 1201 is used to communicate with other network node devices constituting a communication system. The network interface 1201 may be used for wireless communication. For example, the network interface 1201 may be used for wireless LAN communication defined in the IEEE 802.11 series or mobile communication defined in the 3GPP (3rd Generation Partnership Project). Alternatively, the network interface 1201 may include, for example, a network interface card (NIC) conforming to the IEEE 802.3 series.

[0075] The processor 1202 reads and executes software (computer programs) from the memory 1203 to perform the processing of the action evaluation device 100 and the like described using flowcharts or sequences in the above-described embodiments. The processor 1202 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 1202 may include multiple processors.

[0076] The memory 1203 is configured by a combination of volatile memory and non-volatile memory. The memory 1203 may include storage located remotely from the processor 1202. In this case, the processor 1202 may access the memory 1203 via an I / O interface (not shown).

[0077] 14, the memory 1203 is used to store a group of software modules. The processor 1202 reads out and executes these software modules from the memory 1203, thereby performing the processing of the action evaluation device 100 and the like described in the above-described embodiment.

[0078] As explained using Figures 2, 11, etc., each of the processors included in the action evaluation device 100, etc., executes one or more programs including a group of instructions for causing a computer to perform the algorithm explained using the drawings.

[0079] Although the above-described embodiments have been described as hardware configurations, the present disclosure is not limited to such configurations. Any processing in the present disclosure can also be realized by causing a processor to execute a computer program.

[0080] Although the embodiments of the present invention have been described, these are merely examples of the present invention, and various other configurations may be adopted. The configurations of the above-described embodiments may be combined with each other, or some of the configurations may be replaced with other configurations. Furthermore, various modifications may be made to the configurations of the above-described embodiments without departing from the spirit of the invention. Furthermore, the configurations and processes disclosed in the above-described embodiments and modified examples may be combined with each other.

[0081] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0082] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0083] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) a movement identification means for extracting skeletal information of a person in the acquired image, and identifying an evaluation target movement of the person's body based on the extracted skeletal information of the person and a registered movement pattern made up of the stored skeletal information; and an evaluation means for evaluating the similarity between the evaluation target action and a sample action pattern consisting of stored skeletal information, based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction. (Appendix 2) the amount of deviation of the skeletal information in the time axis direction is calculated from the distance between the associated frames by aligning the start points of the sample movement pattern and the movement to be evaluated, and associating the frames based on the similarity of the skeletal information of each frame. (Appendix 3) The movement evaluation device described in Appendix 1, wherein the amount of deviation of the skeletal information in the spatial axis direction is obtained by matching a sample movement and a movement to be evaluated that are similar to each other and calculating the amount of deviation of the geometric shape of the skeletal information. (Appendix 4) 2. The action evaluation device according to claim 1, wherein the action identification means identifies the action to be evaluated based on instruction information from a user. (Appendix 5) The movement evaluation device according to any one of appendices 1 to 4, further comprising a selection means for selecting at least one sample movement pattern from a plurality of sample movement patterns consisting of skeletal information for evaluating the movement of the person, the sample movement patterns having different evaluation criteria for a body part of interest. (Appendix 6) The action evaluation device described in Appendix 1, wherein the evaluation means evaluates the amount of deviation of the person's skeletal information in the spatial axis direction after normalizing the skeletal information related to the action to be evaluated and the skeletal information related to the sample action pattern. (Appendix 7) 7. The action evaluation device according to any one of claims 1 to 6, further comprising an output means for outputting an evaluation result regarding the evaluation. (Appendix 8) The action evaluation device according to any one of appendices 1 to 7, wherein the action identification means identifies the action to be evaluated relating to a part of the body by setting feature points and a pseudo skeleton of the person's body in the image data. (Appendix 9) 9. The movement evaluation device according to any one of appendices 1 to 8, wherein the movement identification means identifies a body movement along a time series based on a plurality of consecutive image frames. (Appendix 10) 10. The action evaluation device according to any one of appendices 1 to 9, further comprising a storage means for storing a plurality of sample action patterns and a plurality of registered action patterns. (Appendix 11) 11. The action evaluation device according to any one of appendices 1 to 10, further comprising a storage means for storing evaluation criterion information including a plurality of evaluation criteria corresponding to a plurality of different body parts. (Appendix 12) 12. The motion evaluation device according to claim 11, wherein the evaluation means evaluates the degree of similarity for each predetermined part of the body based on the evaluation criteria. (Appendix 13) 6. The movement evaluation device according to claim 5, wherein the selection means selects one sample movement pattern based on input from an input means or voice data relating to the movement. (Appendix 14) 14. The action evaluation device according to any one of claims 1 to 13, wherein the registered action patterns include the sample action pattern. (Appendix 15) extracting skeletal information of a person in the acquired image, and identifying an evaluation target action based on the extracted skeletal information of the person and a registered action pattern consisting of the stored skeletal information; A movement evaluation method in which the similarity between the movement to be evaluated and a sample movement pattern consisting of stored skeletal information is evaluated based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction. (Appendix 16) A process of extracting skeletal information of a person in the acquired image, and specifying an evaluation target action based on the extracted skeletal information of the person and a registered action pattern consisting of the stored skeletal information; a process of evaluating the similarity between the evaluation target action and a sample action pattern made up of stored skeletal information based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction; A non-transitory computer-readable medium storing a program for causing a computer to execute the above. [Explanation of symbols]

[0084] 1,1b Operation evaluation system 100, 100a, 100b Operation evaluation device 101 Registration Information Acquisition Department 102 Registration Department 103 Operation DB 104 Sample Operation Sequence Table 105 Selection section 106 Image acquisition unit 107 Extraction part 108,108a Operation specific part 109 Generation part 110, 110a Evaluation section 111 Processing control unit 200b Terminal device 201 Communications Department 202 Control section 203 Display section 204 Audio output section 205 Mike 206 Operation section 300 cameras SA sample operation sequence IMG40 Frame image N Network

Claims

1. a movement identification means for extracting skeletal information of a person from the acquired image data, and identifying an evaluation target movement of the person's body based on the extracted skeletal information of the person and a registered movement pattern consisting of the stored skeletal information; and evaluation means for evaluating the similarity between the evaluation target movement and a sample movement pattern consisting of stored skeletal information, based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction.

2. 2. The movement evaluation device according to claim 1, wherein the amount of deviation of the skeletal information in the time axis direction is calculated by aligning movement start points of the sample movement pattern and the movement to be evaluated, matching each frame based on similarity of skeletal information of each frame, and calculating the amount of deviation of the skeletal information in the time axis direction from the distance between the matched frames.

3. 2. The action evaluation device according to claim 1, wherein the amount of deviation of the skeletal information in the spatial axis direction is obtained by matching a sample action and an evaluation target action that are similar to each other and calculating the amount of deviation of the geometric shape of the skeletal information.

4. 2. The action evaluation device according to claim 1, wherein the action identification means extracts skeletal information of a person in the acquired image data, and identifies the action to be evaluated based on the extracted skeletal information of the person, a registered action pattern consisting of the stored skeletal information, and user instruction information.

5. The movement evaluation device according to any one of claims 1 to 4, further comprising a selection means for selecting at least one sample movement pattern from a plurality of sample movement patterns consisting of skeletal information for evaluating the movement of the person, the sample movement patterns having different evaluation criteria for a body part of interest.

6. 2. The movement evaluation device according to claim 1, wherein the evaluation means evaluates the amount of deviation of the person's skeletal information in the spatial axis direction after normalizing the skeletal information related to the movement to be evaluated and the skeletal information related to the sample movement pattern.

7. 7. The action evaluation device according to claim 1, further comprising an output unit that outputs an evaluation result regarding the evaluation.

8. The action evaluation device according to any one of claims 1 to 7, wherein the action identification means extracts skeletal information of the part of the person's body in the image data by setting the part of the person's body in the acquired image data, and identifies the action to be evaluated relating to the part of the body based on the skeletal information of the extracted part of the person's body and a registered action pattern consisting of stored skeletal information.

9. A computer-implemented motion evaluation method, comprising: extracting skeletal information of a person from the acquired image data, and identifying an evaluation target motion based on the extracted skeletal information of the person and a registered motion pattern consisting of the stored skeletal information; A movement evaluation method in which the similarity between the movement to be evaluated and a sample movement pattern consisting of stored skeletal information is evaluated based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction.

10. A process of extracting skeletal information of a person from the acquired image data, and specifying an evaluation target action based on the extracted skeletal information of the person and a registered action pattern consisting of the stored skeletal information; a process of evaluating the similarity between the evaluation target movement and a sample movement pattern formed by stored skeletal information based on an integrated evaluation value including a first evaluation value based on the amount of deviation of the skeletal information in the time axis direction and a second evaluation value based on the amount of deviation of the person's skeletal information in the space axis direction; A program that causes a computer to execute the following.

Images