Computer-readable non-transitory storage medium, information processing device, and information processing method

Abstract

This computer-readable non-transitory storage medium stores a program. In a first period of a first moving image which is displayed on a display device, the program causes a computer to acquire a first posture of a first subject included in the first moving image. In a second period of a second moving image which has been acquired from an imaging device, the program causes the computer to estimate a second posture of a second subject included in the second moving image. The program causes the computer to compare the first posture with the second posture. The program causes the computer to evaluate the second posture from the comparison results. The program causes the computer to present to a user evaluation information relating to the evaluation.

Description

Computer-readable non-transitory storage medium, information processing device, and information processing method

[0001] The present disclosure relates to a computer-readable non-transitory storage medium, an information processing device, and an information processing method.

[0002] Incorporating exercise into daily life is effective for maintaining and improving both mental and physical health. Fitness gyms and personal training facilities have long been available as a way to incorporate exercise into daily life. However, there are psychological and physical hurdles to exercising in the first place, and only a small number of people are able to make use of such facilities.

[0003] Furthermore, to exercise more effectively, it is necessary for the exerciser to perform the exercise with correct movements. However, it can be difficult for the user to know whether they are performing the movements correctly.

[0004] For example, a technology has been developed that calculates the similarity between a user's pose and that of another user (e.g., a model user) and provides feedback to the user. Also, a technology is known that calculates an evaluation score based on the measurement results of multiple items.

[0005] JP 2022-532772 A JP 2021-68257 A

[0006] However, the above-mentioned techniques require prior learning of data, making them difficult to apply to arbitrary videos. In addition, the computational load for similarity is large, making real-time processing difficult.

[0007] In this way, in order to more easily understand whether the user is moving their body correctly, it is desirable to present the user with an evaluation of their movement in real time, for example. Conventional techniques have room for improvement in terms of more easily understanding whether the user is moving their body correctly.

[0008] Therefore, the present disclosure proposes a computer-readable non-transitory storage medium, an information processing device, and an information processing method that can more easily determine whether a user is moving their body correctly.

[0009] It should be noted that the above problem or object is merely one of multiple problems or objects that can be solved or achieved by multiple embodiments disclosed in this specification.

[0010] A computer-readable non-transitory storage medium according to the present disclosure stores a program. The program causes a computer to acquire a first pose of a first subject included in a first moving image displayed on a display device during a first period of the first moving image. The program causes a computer to estimate a second pose of a second subject included in a second moving image acquired from an imaging device during a second period of the second moving image. The program causes the computer to compare the first pose with the second pose. The program causes the computer to evaluate the second pose based on a comparison result. The program causes the computer to present evaluation information related to the evaluation to a user.

[0011] 1 is a diagram illustrating an example of information processing related to a proposed technique of the present disclosure. FIG. 2 is a diagram illustrating an example of a flow of information processing related to a proposed technique of the present disclosure. FIG. 3 is a block diagram illustrating an example of the overall configuration of an information processing system according to an embodiment of the present disclosure. FIG. 4 is a block diagram illustrating an example of the configuration of a server device according to an embodiment of the present disclosure. FIG. 5 is a block diagram illustrating an example of the configuration of a terminal device according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of a determination timing according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a scoring period set by a setting unit according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating a first comparison method according to an embodiment of the present disclosure. FIG. 9 is a diagram illustrating a second comparison method according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of estimation of skeleton data of a user according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of calculation of similarity by a comparison unit according to an embodiment of the present disclosure. FIG. 12 is a diagram illustrating an example of calculation of similarity according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example of a selection image according to an embodiment of the present disclosure. FIG. 14 is a diagram illustrating an example of an instruction image according to an embodiment of the present disclosure. FIG. 15 is a diagram illustrating an example of a confirmation image according to an embodiment of the present disclosure. FIG. 16 is a diagram illustrating an example of a start image according to an embodiment of the present disclosure. FIG. 17 is a diagram illustrating an example of a display image according to an embodiment of the present disclosure. FIG. 18 is a diagram illustrating another example of a display image according to an embodiment of the present disclosure. FIG. 19 is a diagram illustrating another example of a display image according to an embodiment of the present disclosure. FIG. 19 is a diagram illustrating another example of a display image according to an embodiment of the present disclosure. Fig. 1 is a diagram showing an example of a comprehensive evaluation image according to an embodiment of the present disclosure; Fig. 2 is a diagram showing another example of a comprehensive evaluation image according to an embodiment of the present disclosure; Fig. 3 is a flowchart showing an example of a flow of a reproduction process according to an embodiment of the present disclosure; Fig. 4 is a flowchart showing an example of a flow of an evaluation process according to an embodiment of the present disclosure; Fig. 5 is a diagram showing an example of a hardware configuration of a device, etc.

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] In this specification and drawings, similar components of the embodiments may be distinguished by adding at least one different alphabet and / or number after the same reference numeral. However, if there is no need to particularly distinguish between the similar components, only the same reference numeral will be used.

[0014] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0015] <<1. Introduction>> <1-1. Issues> As mentioned above, incorporating exercise into daily life is effective in maintaining and improving both mental and physical health, and fitness gyms and personal training facilities have long existed. However, there are psychological and physical hurdles to exercising in the first place, and only a small percentage of people are able to make use of such facilities.

[0016] Meanwhile, advances in ICT (Information and Communication Technology) are expanding the number of ways to exercise at home. For example, training content is being distributed using video distribution services. By viewing this training content, users can exercise at any time and place they like.

[0017] However, compared to training with a trainer, exercising via video streaming has the disadvantage that it is difficult for users to determine whether they are performing the exercise correctly. Furthermore, if there is no feedback from the trainer on the user's movements, the user may feel that the exercise is not working, which can make it difficult for the user to maintain motivation to continue exercising.

[0018] For example, by using two-way communication such as video conferencing, users can receive direct instruction from instructors. In this case, users can receive training at any location they like, but the training time may be dependent on the instructor's availability. Furthermore, if users receive one-on-one instruction from an instructor, the fees tend to be high, making it financially difficult for them to continue.

[0019] One way to solve this economic problem is to have multiple users receive instruction from one instructor in lessons using two-way communication. However, if one instructor teaches multiple users, there is a risk that instruction (feedback) for each user will be neglected, and user satisfaction may decrease.

[0020] Another possible way for users to exercise is through games. For example, there are games in which a user holds a controller and performs exercises such as boxing while watching a TV screen. The controller has a built-in acceleration sensor and vibration function. The game system detects the user's movements using the built-in acceleration sensor in the controller and provides feedback via haptics or other means to determine whether the movements are correct.

[0021] Exercising through games has the advantage of being easy to do at any time and place of your choice. It also provides feedback based on the user's movements, making it easier for the user to feel a sense of accomplishment in their exercise and providing an entertaining exercise experience. Therefore, compared to video streaming, it has the advantage of making it easier for users to maintain their motivation to continue exercising.

[0022] On the other hand, a dedicated device may be required to run the game, which may require the time and effort of purchasing and setting up the dedicated device. Furthermore, the exercises that can be evaluated and the evaluation performance may be limited depending on the performance of the device, for example, the sensors installed.

[0023] Furthermore, because game developers design the exercise evaluation logic, game content can be expensive or it can be difficult to increase the variety of exercises, which can result in gaps in game content and limits the motivation of users to continue exercising.

[0024] As such, conventional exercise content has room for improvement in terms of making it easier for users to understand whether they are moving their bodies correctly, and in terms of maintaining users' motivation to continue exercising.

[0025] In this embodiment, exercise is not limited to simple sports, but may include various body movements such as exercises (fitness) such as yoga and Pilates, and performances such as dance.

[0026] <1-2. Overview of Proposed Technology> Fig. 1 is a diagram illustrating an example of information processing related to the proposed technology of the present disclosure. The information processing related to the proposed technology is executed, for example, by a terminal device 100 shown in Fig. 1. The terminal device 100 is an information processing device such as a smartphone or a tablet terminal.

[0027] While executing information processing related to the proposed technology, the terminal device 100 displays, for example, a moving image (an example of a first moving image; hereinafter also referred to as a model video C0) that serves as a model for exercise on the display device 130. The model video C0 includes a person exercising (an example of a first subject; hereinafter also referred to as a trainer U0).

[0028] The terminal device 100 is equipped with an imaging device 110. The imaging device 110 captures an image of an exerciser (an example of a second subject; hereinafter, also simply referred to as a user U1 or a trainee) who is exercising near the terminal device 100 while watching a model video C0.

[0029] In the example of FIG. 1, the terminal device 100 displays the captured video C1 captured by the imaging device 110 on the display device 130 in a manner that superimposes the captured video C1 on the upper right of the model video C0.

[0030] The imaging device 110 is, for example, an in-camera installed on the same surface of the terminal device 100 as the display device 130. The terminal device 100 can display the model video C0 on the display device 130 and use the imaging device 110 to capture an image of the user U1 exercising while watching the model video C0.

[0031] The terminal device 100 compares the model video C0 displayed on the display device 130 with the captured video C1, and evaluates the exercise of the user U1 based on the comparison result. The terminal device 100 displays the evaluation result on the display device 130, thereby presenting an evaluation of the exercise to the user U1 who is exercising.

[0032] 2 is a diagram illustrating an example of the flow of information processing according to the proposed technique of the present disclosure. The information processing illustrated in FIG. 2 is executed by the terminal device 100 while the model video C0 is being played, for example.

[0033] The terminal device 100 acquires the posture of the trainer U0 included in the model video C0 during a first period of the model video C0 displayed on the display device 130. In the example of Fig. 2, the terminal device 100 estimates skeleton data for the first period from the model video C0 as the posture of the trainer U0.

[0034] Here, skeleton data is data that indicates a skeleton structure that indicates the structure of the body. The skeleton data indicates information about body parts. The information about body parts in the skeleton data includes, for example, information about the position and posture of each body part.

[0035] Note that the parts in the skeleton structure correspond to, for example, extremity parts of the body, joint parts, etc. Furthermore, the skeleton data may also include bones, which are line segments connecting parts.

[0036] The skeleton data of the trainer U0 is estimated based on, for example, the time-series data of the model video C0.

[0037] The first period is, for example, a period during which the movement (posture) of the user U1 is evaluated. The first period is set in advance, for example, by the creator of the model video C0. The first period may be a predetermined length of time or a certain timing in the model video C0. The first period is also referred to as a judgment period or judgment timing.

[0038] The terminal device 100 estimates the posture of the user U1 included in the captured video C1 during a second period of the captured video C1 acquired from the imaging device 110. In the example of Fig. 1 , the terminal device 100 estimates skeleton data during the second period from the captured video C1 as the posture of the user U1.

[0039] The skeleton data of the user U1 is estimated based on, for example, time-series data of the captured video C1.

[0040] Here, the second period is, for example, a period during which the exercise (posture) of the user U1 is evaluated. The second period is a period that includes the timing of judging the model video C0 being played on the display device 130. The second period is also referred to as a scoring period. Details of the scoring period will be described later.

[0041] The terminal device 100 compares the posture of the trainer U0 with the posture of the user U1. For example, the terminal device 100 compares the postures of the trainer U0 and the user U1 by calculating the similarity between these postures.

[0042] The terminal device 100 evaluates the posture of the user U1 based on the comparison result. For example, the terminal device 100 evaluates the posture of the user U1 during the scoring period in stages (for example, three stages of “low,” “medium,” and “high”) according to the degree of similarity.

[0043] The terminal device 100 presents evaluation information regarding the evaluation of the posture of the user U1 to the user U1. For example, after the end of the grading period, the terminal device 100 presents the evaluation result to the user U1 by superimposing an image or the like showing the evaluation result on the model video C0 and displaying it on the display device 130.

[0044] Specifically, for example, if the evaluation is "low," the terminal device 100 displays a text image of "Miss!" superimposed on the model video C0 on the display device 130. For example, if the evaluation is "medium," the terminal device 100 displays a text image of "Good!" superimposed on the model video C0 on the display device 130. For example, if the evaluation is "high," the terminal device 100 displays a text image of "Perfect!" superimposed on the model video C0 on the display device 130.

[0045] The evaluation information is not limited to the text images described above, but may include various information (effects) such as images of stars and fireworks, sounds, and lights.

[0046] In this way, the terminal device 100 evaluates the posture of the user U1 at the determination timing according to the model video C0 displayed on the display device 130. This allows the terminal device 100 to evaluate the exercise posture of the user U1 in real time and feed back the evaluation result to the user U1. Therefore, the user U1 can more easily understand whether or not he or she is moving his or her body correctly.

[0047] As described above, the terminal device 100 captures the user U1 exercising while watching the model video C0 with the imaging device 110, and compares the movements of the model (trainer U0) with those of the user U1 in real time. The terminal device 100 provides feedback on the similarity between the movements of the model and those of the user U1 by sound, vision, etc.

[0048] This allows the terminal device 100 to provide the user U1 with a sense of accomplishment in exercise and a game-like quality, allowing the user U1 to maintain and improve their motivation to continue exercising.

[0049] As described above, the terminal device 100 may be realized by a portable device such as a smartphone or a tablet terminal. These portable devices generally include a display device 130 and an imaging device 110 that is an in-camera.

[0050] Therefore, by using these portable devices, user U1 can take a picture of himself / herself with the imaging device 110 while watching the model video C0 without having to prepare any special substrate.

[0051] The terminal device 100 also evaluates the posture of the user U1 using the model video C0 and the captured video C1. For example, compared to posture detection using a gyro sensor or the like, the terminal device 100 can evaluate whether the posture of the trainer U0 and the posture of the user U1 are the same in a manner similar to human evaluation.

[0052] When the user U1 is included in the captured video C1, the terminal device 100 can estimate the posture of the user U1 and evaluate the movement of the user U1 based on the estimated posture.

[0053] Furthermore, if the posture of the trainer U0 in the model video C0 can be estimated, the terminal device 100 can evaluate the posture of the user U1 based on that posture. In this way, the terminal device 100 can evaluate the posture of the user U1 without any evaluation logic for evaluating the posture of the user U1, and can easily and inexpensively evaluate the exercise of the user U1.

[0054] Furthermore, if the model video C0 is available, the terminal device 100 can evaluate the posture of the user U1 based on this model video C0. Therefore, the creator of the model video C0 does not need to design or develop evaluation logic or the like for evaluating the posture of the user U1. In other words, even if the creator is not an expert in software development technology, anyone with exercise know-how can create the model video C0 to be provided to the user U1.

[0055] 3 is a block diagram showing an example of the overall configuration of an information processing system 10 according to an embodiment of the present disclosure. The information processing system 10 includes a terminal device 100 and a server device 200.

[0056] (Terminal Device 100) As described above, the terminal device 100 is an information processing device such as a portable device such as a smartphone or a tablet terminal, or a personal computer. The terminal device 100 plays back the model video C0 on the display device 130 and acquires from the imaging device 110 a captured video C1 of a user U1 exercising while watching the model video C0.

[0057] The terminal device 100 compares the posture of the trainer U0 in the model video C0 at the evaluation timing with the posture of the user U1 in the captured video C1 during the scoring period, and evaluates the posture of the user U1. The terminal device 100 feeds back the evaluation result to the user U1.

[0058] (Server Device 200) The server device 200 may be a cloud server or a database server. The server device 200 is connected to the terminal device 100 via a network such as the Internet.

[0059] The server device 200 is an information processing device that manages the model video C0, etc. The server device 200 stores at least one model video C0, and transmits the model video C0 to the terminal device 100 in response to a request from the terminal device 100.

[0060] 4 is a block diagram showing an example of the configuration of the server device 200 according to an embodiment of the present disclosure. The server device 200 shown in FIG. 4 includes a communication unit 210, a storage unit 220, and a control unit 230.

[0061] (Communication Unit 210) The communication unit 210 is realized by, for example, a network interface card (NIC), etc. The communication unit 210 is connected to the terminal device 100 via a network in a wired or wireless manner, and transmits and receives various types of information to and from the terminal device 100.

[0062] (Storage Unit 220) The storage unit 220 is realized by, for example, a semiconductor memory element such as a RAM, a ROM (Read Only Memory), or a flash memory, or a storage device such as a hard disk or an optical disk.

[0063] The storage unit 220 stores model information related to the model video C0. The model information may include, for example, at least one of the following information: Video information related to the model video C0 Model video C0 Sound information Timing information related to the judgment timing (or judgment period) Posture information of the trainer U0 Judgment information related to the judgment method Feedback information related to the evaluation feedback

[0064] (Video Information) The video information includes information about the model video C0. The video information may include, for example, a thumbnail image of the model video C0, text information explaining the content of the model video C0 (exercise content and difficulty level), etc. Alternatively, the video information may include information about the score obtained after exercising, etc.

[0065] (Model video C0) The model video C0 is a moving image of the trainer U0 exercising. The model video C0 includes the correct posture (correct movements) of the exerciser (user U1) when the imaging device 110 captures the exerciser.

[0066] In addition to the trainer U0 exercising, the model video C0 may also include explanations of the exercise, scenes of resting between movements, etc. Such explanations of the exercise, scenes of resting, etc. are not subject to evaluation of the posture of the user U1.

[0067] (Sound Information) The sound information is information including background music (BGM), voice, sound effects, etc. that are played along with the model video C0.

[0068] (Timing Information) The timing information is information indicating the timing for evaluating (determining) the posture of user U1. The timing information may be, for example, information indicating the period from the start of the model video C0 to the determination timing (e.g., X1 seconds after the start of playback). Alternatively, the timing information may be, for example, information specifying an image included in the model video C0 (e.g., the X1th frame).

[0069] The timing information may also be information indicating a predetermined period (determination period). For example, the timing information may be information indicating that the period is from X1 to X2 seconds after the start of playback. In this case, the determination period is from X1 to X2 seconds after playback starts. Alternatively, the timing information may be information indicating that the period is from the X1th frame to the X2nd frame.

[0070] When the timing information is a judgment period, the terminal device 100 may compare the average posture of the trainer U0 during the judgment period with the posture of the user U1.

[0071] The terminal device 100 may also compare the posture of the trainer U0 with the posture of the user U1 at any timing during the assessment period. For example, the terminal device 100 may compare the posture of the trainer U0 with the posture of the user U1 in a frame image at the center of the assessment period. Alternatively, the terminal device 100 may compare the posture of the trainer U0 with the posture of the user U1 at the start timing (X1 seconds later in the above example) or the end timing (X2 seconds later in the above example) of the assessment period.

[0072] The timing at which the terminal device 100 evaluates the posture of the user U1 during the determination period is not limited to one. The terminal device 100 may evaluate the posture of the user U1 at multiple timings. For example, the terminal device 100 may compare the posture of the trainer U0 with the posture of the user U1 at both the start timing (X1 seconds later in the above example) and the end timing (X2 seconds later in the above example) of the determination period. Alternatively, the terminal device 100 may evaluate the posture of the user U1 a predetermined number of times or at predetermined intervals during the determination period.

[0073] Furthermore, the terminal device 100 may evaluate the posture of the user U1 throughout the entire evaluation period. For example, the terminal device 100 compares the posture of the trainer U0 with the posture of the user U1 for each frame image of the model video C0 included in the evaluation period. This may be performed for each frame image of the captured video C1 included in the scoring period of the user U1. In other words, the posture of the trainer U0 in each frame image during the evaluation period may be compared with the posture of the user U1 in each frame image during the scoring period.

[0074] (Posture Information of Trainer U0) The posture information of trainer U0 may include the skeleton data of trainer U0 described above. The posture information of trainer U0 may be generated for each frame image of the model video C0. Alternatively, the posture information of trainer U0 may be generated according to timing information. In this case, the posture information of trainer U0 may be generated for each timing (frame image) at which the posture of user U1 is evaluated during the evaluation period.

[0075] (Determination Information) The determination information includes information related to a method for evaluating (determining) the posture of the user U1 by the terminal device 100. For example, the determination information may include at least one of information related to the body parts (joints and bones) to be evaluated (compared) and information related to the weighting of each body part.

[0076] The determination information may also include information indicating which evaluation information to present to the user U1 based on the evaluation results. For example, when a graded evaluation (such as the aforementioned "low," "medium," or "high") is performed based on a value (score) indicating the similarity of the postures, a threshold for classifying the evaluations into each grade may be included in the determination information.

[0077] The evaluation information may also include an evaluation score according to the graded evaluation. The evaluation score may be a specified value or may be a value according to the number of times the evaluation is performed (the number of evaluation timings).

[0078] If the evaluation score is a specified value, a score is determined according to the evaluation, such as 0 points if the evaluation is "low," 2 points if the evaluation is "medium," and 3 points if the evaluation is "high."

[0079] Furthermore, for example, in the case of a value according to the number of times the evaluation is performed, it can be set so that the total evaluation score will be 100 points when all evaluations are "high." For example, suppose the number of times the evaluation is performed is five. In this case, the evaluation score for a "high" evaluation will be 20 points.

[0080] (Feedback Information) The feedback information may include image information to be displayed on the display device 130 when presenting the evaluation result of the posture of the user U1 to the user U1. The feedback information may also include sound information (music, sound effects, etc.) to be notified together with the image information. As described above, the image information may include text such as "Perfect!" or illustrations of stars, flowers, fireworks, etc.

[0081] (Control Unit 230) The control unit 230 may be realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing a program stored inside the server device 200 using RAM (Random Access Memory) or the like as a work area. However, the control unit 230 may also be executed by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0082] The control unit 230 notifies the terminal device 100 of the model information in response to a request from the terminal device 100, for example. The control unit 230 may notify the terminal device 100 of all the model information at once, or may notify the model information in multiple batches. For example, the control unit 230 may first notify the terminal device 100 of the video information among the model information, and then notify the terminal device 100 of the remaining model information in response to a request from the terminal device 100.

[0083] Furthermore, the control unit 230 acquires model information from the developer (creator) of the model video C0 and stores it in the storage unit 220. The control unit 230 acquires the model information from, for example, an information processing device (not shown) used by the developer.

[0084] The control unit 230 may acquire all of the model information from the developer, or may acquire only part of the model information. When acquiring only part of the model information from the developer, the control unit 230 may generate the remaining model information.

[0085] For example, when a model video C0 is acquired, the control unit 230 may generate video information such as a thumbnail image based on this model video C0. Alternatively, the control unit 230 may use pre-stored information as model information, such as using pre-stored feedback information as feedback information for the acquired model video C0.

[0086] The control unit 230 stores the acquired part of the model information and the generated remaining model information in the storage unit 220.

[0087] 2-3. Example Configuration of Terminal Device Fig. 5 is a block diagram showing an example configuration of a terminal device 100 according to an embodiment of the present disclosure. The terminal device 100 in Fig. 5 includes an imaging device 110, an information processing device 120, and a display device 130. Note that the components of the terminal device 100 are not limited to the example in Fig. 5. The terminal device 100 may include an output device such as a speaker, or an input device such as a touch panel.

[0088] (Imaging device 110) The imaging device 110 is, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor. The imaging device 110 captures an image of the surroundings of the terminal device 100 in accordance with instructions from the information processing device 120. The imaging device 110 is disposed, for example, around the display device 130, and captures an image of a user U1 exercising in front of the display device 130.

[0089] The imaging device 110 outputs the captured image to the information processing device 120 .

[0090] (Display Device 130) The display device 130 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) panel, etc. The display device 130 displays the model video C0 and the captured video C1 in accordance with instructions from the information processing device 120.

[0091] (Information Processing Device 120) The information processing device 120 controls the imaging device 110 and the display device 130, and also executes processing for evaluating the posture of the user U1.

[0092] The information processing device 120 shown in FIG. 5 includes a communication unit 121, a storage unit 122, and a control unit 123.

[0093] (Communication Unit 121) The communication unit 121 is realized by, for example, a NIC etc. The communication unit 121 is connected to the server device 200 via a network in a wired or wireless manner, and transmits and receives various information to and from the server device 200.

[0094] (Storage Unit 122) The storage unit 122 is realized by, for example, a semiconductor memory element such as a RAM, a ROM, or a flash memory, or a storage device such as a hard disk or an optical disk.

[0095] (Control unit 123) The control unit 123 can be realized, for example, by a CPU or an MPU executing a program stored inside the information processing device 120 using RAM or the like as a work area. Alternatively, the control unit 123 may be executed by an integrated circuit such as an ASIC or FPGA.

[0096] The control unit 123 shown in FIG. 5 includes an acquisition unit 1231 , a setting unit 1232 , a reproduction unit 1233 , an estimation unit 1234 , a comparison unit 1235 , an evaluation unit 1236 , and a UI (User Interface) control unit 1237 .

[0097] Each block constituting the control unit 123 (for example, the acquisition unit 1231 to the UI control unit 1237) is a functional block that indicates the function of the control unit 123. These functional blocks may be software blocks or hardware blocks.

[0098] For example, each of the above-described functional blocks may be a software module implemented by software (including a microprogram), or may be a circuit block on a semiconductor chip (die).

[0099] Of course, each functional block may be a single processor or a single integrated circuit. The control unit 123 may be configured with functional units different from the above-mentioned functional blocks. The method of configuring the functional blocks is arbitrary.

[0100] (Acquisition unit 1231) The acquisition unit 1231 acquires various information from other devices, etc. For example, the acquisition unit 1231 acquires model information from the server device 200 via the communication unit 121. The acquisition unit 1231 acquires captured images from the imaging device 110 in succession, thereby acquiring the captured video C1.

[0101] The acquisition unit 1231 outputs the model video C0 and sound information from the acquired model information to the playback unit 1233. The acquisition unit 1231 also outputs the video information and feedback information from the acquired model information to the UI control unit 1237.

[0102] The acquisition unit 1231 outputs timing information from the acquired model information to the setting unit 1232. The acquisition unit 1231 outputs posture information and judgment information of trainer U0 from the acquired model information to the comparison unit 1235. The acquisition unit 1231 outputs judgment information from the acquired model information to the evaluation unit 1236.

[0103] The acquisition unit 1231 outputs the acquired captured video C1 to the playback unit 1233.

[0104] (Setting unit 1232) The setting unit 1232 sets various parameters for, for example, information processing for evaluating the exercise of the user U1 (hereinafter also referred to as evaluation processing). The setting unit 1232 sets, for example, a period (scoring period) during which the information processing device 120 executes the evaluation processing.

[0105] The setting unit 1232 sets the scoring period in accordance with the judgment timing included in the timing information. An example of the judgment timing will now be described with reference to FIG.

[0106] 6 is a diagram illustrating an example of a determination timing according to an embodiment of the present disclosure. The upper diagram in FIG. 6 illustrates the determination timing, and the lower diagram illustrates the model video C0. In FIG. 6, the model video C0 starts at time t1.

[0107] 6, the evaluation timing for the model video C0 is set to five times, from time t2 to time t6. In this embodiment, the evaluation process is not performed for the entire duration of the model video C0, in other words, for all frames. This is because the model video C0 includes footage that is not subject to evaluation.

[0108] As described above, the model video C0 may include an explanation of the exercise by the trainer U0 or the like, or footage of the trainer U0 resting. These are not included in the evaluation of the exercise of the user U1. Therefore, the timing for judging the model video C0 may be set to a period excluding these (explanation periods and rest periods).

[0109] The determination timing may also be, for example, the timing when the trainer U0 assumes a predetermined posture. For example, in exercises such as yoga, it may be important to maintain a certain posture. In such cases, the determination timing may be set to the timing when the trainer U0 assumes a predetermined posture.

[0110] The determination timing may be set to a predetermined cycle. For example, in dance, boxing, etc., movements may be performed at a predetermined cycle. In such cases, the determination timing may be set to the cycle (or interval) of the movements.

[0111] The determination timing may be set by the creator of the model video C0, such as the trainer U0, or may be automatically set by an information processing device (such as the server device 200) based on background music, sound effects, or the speech of the trainer U0.

[0112] The setting unit 1232 sets a scoring period for evaluating (scoring) the posture of the user U1 according to this determination timing.

[0113] Here, it is desirable that the timing for evaluating the posture of the user U1 is the same as the timing for judgment, because it is expected that the user U1 will adopt the same posture as the trainer U0 while watching the model video C0.

[0114] However, if the timing for evaluating the posture of the user U1 is the determination timing, there is a risk that the posture of the user U1 will not be evaluated correctly.

[0115] This is because it may take some time for the information processing device 120 to acquire the captured image from the imaging device 110. The time it takes for the information processing device 120 to acquire the captured image from the imaging device 110 is determined depending on, for example, the hardware performance of the terminal device 100. In addition, this time varies depending on the type and number of applications running on the terminal device 100, the temperature of the terminal device 100, etc.

[0116] Therefore, even if the information processing device 120 attempts to evaluate the posture of the user U1 at the desired scoring timing, there is a risk that the captured image acquired by the information processing device 120 from the imaging device 110 will not be an image captured at the judgment timing.

[0117] Furthermore, if the user U1 checks the movements in the model video C0 before performing the movement, there is a risk that the timing at which the posture of the user U1 should be evaluated will be later than the timing at which the posture should be judged.

[0118] Therefore, in this embodiment, the information processing device 120 evaluates the posture of the user U1 during a scoring period. This scoring period is a period around the determination timing. For example, the scoring period is a fixed period that includes the determination timing.

[0119] The information processing device 120 compares the posture of the user U1 with the posture of the trainer U0 at the determination timing multiple times during the scoring period. That is, the scoring period includes multiple scoring timings for comparing these postures.

[0120] The information processing device 120 evaluates the posture of the user U1 based on the results of multiple comparisons during the scoring period. For example, the information processing device 120 calculates the degree of similarity between the posture of the user U1 at multiple scoring times and the posture of the trainer U0 at the judgment time.

[0121] The information processing device 120 determines the similarity with the largest value among the similarities at each of the multiple scoring timings as the similarity for that scoring period (i.e., the evaluation result).

[0122] In this manner, in this embodiment, the information processing device 120 evaluates the posture of the user U1 during a scoring period. The setting unit 1232 sets this scoring period in accordance with the determination timing.

[0123] 7 is a diagram illustrating a scoring period set by the setting unit 1232 according to an embodiment of the present disclosure. In FIG. 7, a judgment timing, a model video C0, a scoring period, and a captured video C1 are illustrated.

[0124] In the example of FIG. 7, the judgment timings for the model video C0 are set in advance at times t02 and t05.

[0125] The setting unit 1232 sets the period from time t01 to time t03 as the scoring period corresponding to the judgment timing of time t02. Time t01 is a period T01 earlier than time t02. Time t03 is a period T02 later than time t02. Periods T01 and T02 may be the same length or different lengths.

[0126] The setting unit 1232 also sets the period from time t04 to time t06 as the scoring period corresponding to the determination timing of time t05. Time t04 is earlier than time t05 by a period T01. Time t06 is later than time t05 by a period T02.

[0127] Here, the length of the scoring period corresponding to the determination timing at time t02 is the same as the length of the scoring period corresponding to the determination timing at time t05, but this is not limited to this. The length of the scoring period corresponding to the determination timing at time t02 may be different from the length of the scoring period corresponding to the determination timing at time t05.

[0128] For example, the setting unit 1232 may adjust the length of the scoring period depending on the interval between judgment timings, the tempo of the background music, the speed at which trainer U0's posture changes, etc. For example, the setting unit 1232 shortens the scoring period when the background music tempo is fast compared to when it is slow.

[0129] Alternatively, the setting unit 1232 shortens the scoring period when the speed at which trainer U0's posture changes is fast compared to when the speed is slow. For example, the setting unit 1232 estimates the speed at which trainer U0's posture changes by analyzing time-series changes in the bones in the skeleton data of trainer U0. The setting unit 1232 sets the scoring period according to the estimated speed. For example, the setting unit 1232 sets the scoring period according to the rhythm at which peaks appear in the angular velocity of the angles between joints in the skeleton data of trainer U0.

[0130] Here, an example of a method for comparing the postures of the user U1 and the trainer U0 at the determination timing of time t02 will be described with reference to Fig. 7. In Fig. 7, it is assumed that playback of the model video C0 and shooting of the shot video C1 are completed up to time t7, which is after time t02.

[0131] The information processing device 120 estimates the posture of the user U1 from the captured images C11 and C12 acquired between time t01 and time t7. For example, after time t01, the information processing device 120 estimates the posture of the user U1 from the acquired captured image each time it acquires a captured image from the imaging device 110.

[0132] The information processing device 120 compares the posture of the user U1 estimated from the captured images C11 and C12 with the posture of the trainer U0 in the frame image (hereinafter referred to as the model image C02) of the model video C0 displayed at time t02. For example, the information processing device 120 calculates the similarity between the postures as a comparison of the postures.

[0133] The information processing device 120 estimates the posture of the user U1 and performs comparison (for example, calculation of similarity) for the captured images acquired from the image capturing device 110 until time t03, which is the end of the scoring period.

[0134] The information processing device 120 determines that the comparison result (e.g., similarity) obtained during the scoring period in which the posture of user U1 is closest to the posture of trainer U0 (e.g., the highest similarity) is the comparison result for that scoring period (from time t01 to time t03).

[0135] Here, it is assumed that the information processing device 120 estimates the posture of the user U1 and compares it with the posture of the trainer U0 each time it acquires a photographed image during the scoring period. On the other hand, after the scoring period ends, the information processing device 120 may estimate the posture of the user U1 for multiple photographed images acquired during the scoring period and compare it with the posture of the trainer U0.

[0136] The information processing device 120 may change the comparison method depending on the length of the scoring period, such as whether to perform posture estimation and comparison each time a captured image is acquired or to perform posture estimation and comparison all at once after the capturing period. Alternatively, the comparison method may be changed depending on the processing capacity of the information processing device 120.

[0137] For example, if the scoring period is shorter than a predetermined value, the information processing device 120 may perform posture estimation and comparison all at once after the shooting period. The predetermined value may be determined in advance, or may be determined based on the processing capacity of the information processing device 120, for example.

[0138] Furthermore, for example, if the information processing device 120 has a high processing capability, the information processing device 120 may perform posture estimation and comparison every time a captured image is acquired.

[0139] Although the setting unit 1232 sets the grading period here, the setting of the grading period may be performed by a party other than the setting unit 1232. The creator who creates the model video C0 may set the grading period, or an information processing device (not shown) such as the server device 200 may set the grading period. In this case, information regarding the grading period may be notified from the server device 200 as timing information.

[0140] In the above example, a method for setting a scoring period was described when evaluating the posture of user U1 at a specific timing corresponding to the posture of trainer U0. As a method for evaluating the posture of user U1, there are cases where it is desired to evaluate whether a certain rhythm is being maintained appropriately, in other words, to evaluate the posture of user U1 at a certain rhythm. In this case, the timing of the scoring can be set at intervals corresponding to the rhythm.

[0141] In this way, when evaluating the posture of user U1, including whether the rhythm is appropriate, the setting unit 1232 can compare the postures of user U1 and trainer U0 using the following two comparison methods.

[0142] (First Comparison Method) Fig. 8 is a diagram illustrating a first comparison method according to an embodiment of the present disclosure. In the example of Fig. 8, the determination timing is set to times t12, t15, and t17 with a cycle of T1.

[0143] In the first comparison method, the setting unit 1232 sets a scoring period corresponding to the determination timing of time t12. This scoring period is set in the same manner as described with reference to Fig. 7. In Fig. 8, the period from time t11 to time t13 is set as the scoring period corresponding to the determination timing of time t12.

[0144] The setting unit 1232 sets the grading periods (here, grading timing) of times t15 and t17 according to the evaluation results from the grading period from time t11 to time t13.

[0145] Here, suppose that as a result of comparing the postures of user U1 and trainer U0 during the scoring period from time t11 to time t13, the posture of user U1 at time t14 is given the highest evaluation.

[0146] In this case, the setting unit 1232 sets the scoring timing for each period T1, for example, from time t14. In the example of Fig. 8, the setting unit 1232 sets the scoring timing corresponding to the determination timing of time t15 to time t16. The setting unit 1232 sets the scoring timing corresponding to the determination timing of time t17 to time t18.

[0147] Here, the setting unit 1232 sets times t16 and t18 as the scoring period. In this way, the scoring period according to this embodiment can include a specific time (scoring timing) in addition to a period of a predetermined length.

[0148] Although the setting unit 1232 sets the times t16 and t18 as the scoring period here, the setting unit 1232 may set a period including the times t16 and t18 as the scoring period. In this case, the setting unit 1232 may set a period shorter than the scoring period corresponding to the determination timing of the time t12 as the scoring period corresponding to the determination timing of the times t15 and t17.

[0149] In this way, by the setting unit 1232 setting the period including times t16 and t18 as the scoring period, the information processing device 120 can more appropriately evaluate the posture of user U1 even if the acquisition of the captured image from the imaging device 110 is delayed from times t16 and t18.

[0150] (Second Comparison Method) A second comparison method of the posture of the user U1, including the rhythm, will be described.

[0151] 9 is a diagram illustrating a second comparison method according to an embodiment of the present disclosure. In the example of Fig. 9, the determination timing is set to times t12, t15, and t17 with a cycle of T1.

[0152] In the second comparison method, the setting unit 1232 sets scoring periods corresponding to the determination timings of times t12, t15, and t17. These scoring periods are set in the same manner as described with reference to FIG. 7.

[0153] 9, the period from time t11 to time t13 is set as the scoring period corresponding to the determination timing of time t12, the period from time t21 to time t22 is set as the scoring period corresponding to the determination timing of time t15, and the period from time t24 to time t25 is set as the scoring period corresponding to the determination timing of time t17.

[0154] The information processing device 120 evaluates the posture of the user U1 in each scoring period. The information processing device 120 also evaluates whether the user U1 is keeping an appropriate rhythm based on the interval between the times when the evaluation result (e.g., similarity) of the posture of the user U1 is highest in each scoring period (hereinafter also referred to as the "high evaluation timing").

[0155] Here, suppose that as a result of comparing the postures of user U1 and trainer U0 during the scoring period from time t11 to time t13, the posture of user U1 at time t14 is given the highest evaluation.

[0156] Furthermore, as a result of comparing the postures of user U1 and trainer U0 during the scoring period from time t21 to time t22, it is assumed that the posture of user U1 at time t23 is given the highest evaluation.

[0157] In this case, the information processing device 120 evaluates the rhythm of the user U1 based on the difference between the period T2 between time t14 and time t23 and the cycle T1 (hereinafter also referred to as the first difference).

[0158] Specifically, the information processing device 120 compares the first difference with a threshold Th1, and if the first difference is equal to or smaller than the threshold Th1, determines that the user U1 is moving in sync with the rhythm. On the other hand, if the first difference is greater than the threshold Th1, the information processing device 120 determines that the user U1's movements are not in sync with the rhythm.

[0159] Furthermore, as a result of comparing the postures of user U1 and trainer U0 during the scoring period from time t24 to time t25, it is assumed that the posture of user U1 at time t26 is given the highest evaluation.

[0160] In this case, the information processing device 120 evaluates the rhythm of the user U1 based on the difference between the period T3 between time t23 and time t26 and the cycle T1 (hereinafter also referred to as the second difference).

[0161] Specifically, the information processing device 120 compares the second difference with a threshold Th1, and if the second difference is equal to or smaller than the threshold Th1, determines that the user U1 is moving in sync with the rhythm. On the other hand, if the second difference is greater than the threshold Th1, the information processing device 120 determines that the user U1's movements are not in sync with the rhythm.

[0162] Here, the information processing device 120 evaluates whether the movements of the user U1 are in rhythm in two stages, but may evaluate in three or more stages. In this case, the information processing device 120 may evaluate in three stages, "very in rhythm," "in rhythm," or "not in rhythm," depending on the interval between adjacent highly rated timings (for example, the first difference or the second difference described above).

[0163] The information processing device 120 evaluates the posture of the user U1 based on the comparison result of the posture of the user U1 in each scoring period and the evaluation result of the rhythm. For example, the information processing device 120 may evaluate the posture of the user U1 by weighting the comparison result of the posture of the user U1 (e.g., similarity) based on the evaluation result of the rhythm.

[0164] Although the determination timing is set at a fixed cycle T1 here, the interval at which the determination timing is set does not have to be a fixed cycle. For example, the interval at which the determination timing is set may be a multiple of a certain period. Furthermore, even if the determination timing is set at a certain cycle, this cycle may change during the model video C0.

[0165] The setting unit 1232 notifies each unit of the control unit 123 of the various set parameters. For example, the setting unit 1232 notifies the estimation unit 1234 of the start and end of a scoring period. Alternatively, the setting unit 1232 may notify the estimation unit 1234 of the scoring timing included in the scoring period.

[0166] 5, the playback unit 1233 plays the model video C0 in accordance with an instruction from the user U1. The playback unit 1233 notifies the estimation unit 1234, for example, of the start of the model video C0.

[0167] 5 receives a notification of the start of playback of the model video C0 from the playback unit 1233. After receiving the notification, when the scoring period begins, the estimation unit 1234 estimates the posture of the user U1 from the captured images acquired from the imaging device 110 during the scoring period.

[0168] The estimation unit 1234 detects the position and posture of each body part as skeleton data, for example, based on time-series data of captured images acquired from the imaging device 110. The estimation unit 1234 estimates the positions and postures of bones in addition to each body part as skeleton data of the user U1, for example.

[0169] Alternatively, the estimation unit 1234 may estimate the posture (skeleton data) of the user U1 using machine learning such as a deep neural network (DNN). For example, an estimator may be used for generating the skeleton data by the estimation unit 1234.

[0170] This estimator is generated by machine learning using, for example, a set of image data acquired by photographing a person and skeleton data of this person as training data.

[0171] However, the method of estimating skeleton data by the estimation unit 1234 is not limited to the method using time-series data or the machine learning method described above.

[0172] 10 is a diagram illustrating an example of estimated skeleton data U10 of a user U1 according to an embodiment of the present disclosure. The skeleton data U10 illustrated in FIG. 10 is estimated by the estimation unit 1234 as an example of posture information indicating the posture of the user U1.

[0173] The skeleton data U10 includes information on each part (joints and extremities) of the user U1 and information on bones. The skeleton data U10 includes at least one of position information, posture information, and skeletal feature information for each part and bone.

[0174] 10, the skeleton data U10 of the user U1 may include a bone B11 connecting the left hand joint point K11 and the left elbow joint point K12, and a bone B12 connecting the left elbow joint point K12 and the left shoulder joint point K13. In this way, the skeleton data U10 is configured to include a plurality of body parts and a plurality of bones connecting the plurality of body parts.

[0175] In the following description, parts may be referred to as joint points or joints, but the joint points here may not necessarily correspond to actual joints of a human being. For example, the joint points may include a head joint point K10 that is different from an actual joint.

[0176] Furthermore, joint points may be provided at the eye positions included in the head joint point K10. A plurality of joint points may be provided between the left hand joint point K1 and the left elbow joint point K2. In this way, as long as the skeleton data U10 can retain the shape of the user U1, joint points and bones may be provided at any positions.

[0177] 10 shows skeleton data U10 of the entire body of the user U1, the estimation unit 1234 does not necessarily need to estimate skeleton data U10 of the entire body. The estimation unit 1234 may estimate skeleton data U10 of only a part (for example, only the upper body or only the hands) corresponding to the posture of the trainer U0 in the model video C0.

[0178] The part of the skeleton data U10 to be estimated may be specified by the model information. In this case, the estimation unit 1234 estimates the part included in the model information, for example, and generates the skeleton data U10 of the user U1.

[0179] The estimation unit 1234 may also calculate a reliability score of the estimation result for each part. For example, the estimation unit 1234 calculates a reliability score of the estimation result of the position information for each joint point. The reliability score is an index indicating the reliability of the estimated value of the joint point.

[0180] For example, when the skeleton data U10 is generated using the above-mentioned estimator, the estimator outputs the reliability score of each part in addition to the skeleton data U10.

[0181] Alternatively, the estimation unit 1234 may calculate the reliability score of the current skeleton data U10 based on the estimation result of the previous skeleton data U10. For example, the estimation unit 1234 calculates the reliability score based on the distance between the previously estimated position of the joint point and the currently estimated position of the joint point. Specifically, for example, the farther the currently estimated position of the joint point is from the previously estimated position of the joint point (the farther the distance between the two joint points), the lower the reliability score.

[0182] Alternatively, the reliability score of a joint point that is outside the captured image is calculated to be low. Also, the reliability score of a joint point behind the body of user U1 is calculated to be low. In this way, the reliability score of a part (joint point) that is not captured in the captured image is calculated to be low.

[0183] This reliability score may also be calculated for the skeleton data of trainer U0. In this case, the reliability score at each joint point of the skeleton data of trainer U0 may be included in the model information (e.g., posture information of trainer U0) described above.

[0184] The estimation unit 1234 outputs the estimated posture information of the user U1 (skeleton data U10 and reliability score) to the comparison unit 1235.

[0185] In addition, if the model information does not include posture information (skeleton data) of trainer U0 at the judgment timing, the estimation unit 1234 may estimate posture information of trainer U0 based on the display image at the judgment timing.

[0186] 5 compares the posture of the trainer U0 at the determination timing with the posture of the user U1 during the scoring period. For example, the comparison unit 1235 calculates the similarity between the skeleton data of the trainer U0 and the skeleton data U10 of the user U1.

[0187] 11 is a diagram illustrating an example of calculation of the similarity by the comparison unit 1235 according to an embodiment of the present disclosure. In Fig. 11, the comparison unit 1235 calculates the similarity between skeleton data U00 of trainer U0 and skeleton data U10 of user U1.

[0188] Here, the skeleton data U00 of the trainer U0 is posture information of the trainer U0 estimated from the display image C0x at time tx (determination timing tx) of the model video C0, and the skeleton data U10 of the user U1 is posture information of the user U1 estimated from the captured image C1x.

[0189] As shown in FIG. 11, the comparison unit 1235 first superimposes (overlaps) the skeleton data of the trainer U0 and the skeleton data U10 of the user U1.

[0190] Simply superimposing the display image C0x and the captured image C1x does not result in the skeleton data U00 and U10 overlapping, so the comparison unit 1235 aligns the skeleton data U00 and U10 before superimposing the skeleton data U00 and U10.

[0191] For example, the comparison unit 1235 moves the positions of each joint point of the skeleton data U10 so as to reduce the sum of the distances between the joint points of the skeleton data U00 and U10. At this time, the comparison unit 1235 moves the positions of each joint point of the skeleton data U10 so as not to change the positional relationships of the joint points of the skeleton data U10 (for example, the postures of the bones, the distances between the joint points, etc.).

[0192] In this way, the comparison unit 1235 overlaps the skeleton data U00 and U10, and fits the skeleton data U10 to the skeleton data U10.

[0193] The method of overlaying the skeleton data U00 and U10 is not limited to a method based on the distance between the joint points of the skeleton data U00 and U10. For example, the comparison unit 1235 may align the display image C0x and the captured image C1x based on the position of the trainer U0 in the display image C0x and the position of the user U1 in the captured image C1x. The comparison unit 1235 may overlay the skeleton data U00 and U10 based on the result of aligning the display image C0x and the captured image C1x.

[0194] Next, the comparison unit 1235 calculates the similarity of the skeleton data U10 of the user U1 to the skeleton data U00 of the trainer U0. The comparison unit 1235 calculates the similarity based on, for example, the distance between a joint point of the skeleton data U10 and a corresponding joint point of the skeleton data U00.

[0195] The comparison unit 1235 first calculates the distance between a joint point of the skeleton data U10 and a corresponding joint point of the skeleton data U00. For example, the comparison unit 1235 calculates the distance d between the joint point K11 and the joint point K01 using the following equation (1): １ Calculate.

[0196] Here, the joint point K11 is a part included in the skeleton data U10 of the user U1. The joint point K01 is a part included in the skeleton data U00 of the trainer U0, and is the part corresponding to the joint point K11. For example, if the joint point K11 is the left hand of the user U1, the joint point K01 is the left hand of the trainer U0.

[0197]

[0198] In addition, x １１ is the x-coordinate value of the joint point K11 in the two-dimensional image in which the skeleton data U00 and U10 are superimposed. １１ is the y coordinate value of the joint point K11 in the two-dimensional image in which the skeleton data U00 and U10 are superimposed.

[0199] x ０１ is the x-coordinate value of the joint point K01 in the two-dimensional image in which the skeleton data U00 and U10 are superimposed. ０１ is the y coordinate value of the joint point K01 in the two-dimensional image in which the skeleton data U00 and U10 are superimposed.

[0200] The comparison unit 1235 compares the distance d １ From the similarity S １ For example, the comparison unit 1235 calculates the similarity S, which takes a value between 0 and 1, using a Gaussian distribution. １ Specifically, the comparison unit 1235 calculates the similarity S of the joint point K11 using, for example, the following equation (2): １Calculate.

[0201]

[0202] Note that σ is an arbitrary value. For example, σ may be a value according to a likelihood function L. σ may be defined as a value obtained by dividing L by a certain value m (σ=L / m).

[0203] The comparison unit 1235 calculates the similarity at each joint point included in the skeleton data U10.

[0204] The comparison unit 1235 calculates the average value S of the similarity at each joint point using the following equation (3): where n is the total number of joint points included in the skeleton data U10 and for which the similarity has been calculated.

[0205]

[0206] The comparison unit 1235 determines this average value S as the comparison result for the captured image C1x, that is, the similarity of the posture of the user U1.

[0207] The comparison unit 1235 may calculate the similarity for all joint points included in the skeleton data U10, or may calculate the similarity for some of the joint points.

[0208] For example, when trainer U0 performs upper body exercise, the comparison unit 1235 calculates the similarity of the joint points corresponding to the upper body of user U1. In this way, the comparison unit 1235 calculates the similarity of the joint points according to the exercise performed by trainer U0. The comparison unit 1235 sets the average of the calculated similarities as the similarity of the posture of user U1.

[0209] The joint points at which the similarity is calculated are determined using, for example, determination information included in the model information.

[0210] In this way, the comparison unit 1235 may calculate the similarity for each body part, such as the upper body or the arms.

[0211] The comparison unit 1235 may also correct the similarity using the reliability score described above. For example, the comparison unit 1235 may weight the similarity according to the reliability score at each joint point and calculate the average value for each joint point. For example, the comparison unit 1235 weights the similarity so that the larger the reliability score, i.e., the higher the reliability, the greater the weight.

[0212] The comparison unit 1235 may weight the similarity according to the reliability score calculated at the joint points of user U1 (an example of second joint reliability), or may weight the similarity according to the reliability score calculated at the joint points of trainer U0 (an example of first joint reliability).

[0213] Alternatively, the comparison unit 1235 may weight the similarity using the smaller of the reliability score at the joint point of the user U1 and the reliability score at the joint point of the trainer U0.

[0214] For example, the comparison unit 1235 weights the similarity using the following formula (4): ｉ ０ is the confidence score of trainer U0 at joint point K0i. ｉ １ is the confidence score at the joint point K1i of the user U1.

[0215]

[0216] The comparison unit 1235 may weight the similarity using the average value of the reliability scores at the joint points of the user U1 and the reliability scores at the joint points of the trainer U0.

[0217] Alternatively, the comparison unit 1235 may select the joint points to be used in calculating the similarity depending on the reliability score.

[0218] 12 is a diagram illustrating an example of calculation of similarity according to an embodiment of the present disclosure, in which the comparison unit 1235 calculates the similarity of skeleton data U10 with respect to skeleton data U00.

[0219] At this time, the comparison unit 1235 excludes joint points whose reliability scores are equal to or less than the threshold Th2 from the similarity calculation. In other words, the comparison unit 1235 calculates similarity for joint points whose reliability scores are greater than the threshold Th2.

[0220] 12, the reliability score of the joint point K14 corresponding to the right hand of the user U1 is equal to or less than the threshold value Th2. In this case, the comparison unit 1235 does not calculate the similarity of the joint point K14 with respect to the joint point K04 corresponding to the right hand of the trainer U0.

[0221] 12, the reliability score of the joint point K05 corresponding to the left foot of the trainer U0 is equal to or less than the threshold value Th2. In this case, the comparison unit 1235 does not calculate the similarity of the joint point K15 corresponding to the left foot of the user U1 with respect to the joint point K05.

[0222] In this way, if at least one of the reliability score at a joint point in the skeleton data U10 and the reliability score at a joint point in the skeleton data U00 is equal to or lower than the threshold value Th2, the comparison unit 1235 excludes that joint point from the calculation of the similarity.

[0223] In other words, the comparison unit 1235 calculates the similarity using joint points whose reliability scores in the skeleton data U10 are greater than the threshold Th2 and whose reliability scores in the skeleton data U00 are greater than the threshold Th2.

[0224] In the example of Figure 12, the comparison unit 1235 calculates the similarity of skeleton data U10 using 11 joint points excluding joint points K14 and K15 of skeleton data U10, and 11 joint points excluding joint points K04 and K05 of skeleton data U00.

[0225] Alternatively, the comparison unit 1235 may determine that joint points in the skeleton data U00 of trainer U0 whose reliability scores are equal to or less than the threshold Th2 are not reliable as models and may not be used in calculating the similarity.

[0226] On the other hand, even if there is a joint point in the skeleton data U10 of the user U1 whose reliability score is equal to or less than the threshold Th2, the comparison unit 1235 may calculate the similarity of the joint point.

[0227] This is because the reliability score of the joint point may have decreased because the image capturing device 110 failed to capture the captured image C1x in which the user U1 appears, or because the user U1 failed to perform a movement, causing the joint point to be hidden, or because the joint point is located outside the angle of view of the captured image C1x.

[0228] In this way, the reliability of the skeleton data U10 of the user U1 may be reduced due to the movement of the user U1 significantly deviating from the movement of the trainer U0, who is the model. In this case, the comparison unit 1235 calculates the similarity of the joint points of the skeleton data U10 regardless of the reliability score of the joint points, thereby enabling the information processing device 120 to evaluate the posture of the user U1.

[0229] This allows the comparison unit 1235 to calculate the similarity of the posture of the user U1 to the posture of the trainer U0 using joint points with higher reliability.

[0230] Alternatively, the comparison unit 1235 may change the method of calculating the similarity depending on whether the joint point is hidden or not.

[0231] For example, if a joint point of user U1 corresponding to a joint point estimated to be hidden by the body of trainer U0 is estimated to be hidden by the body of user U1, the comparison unit 1235 estimates that the hidden joint point of user U1 may be in the correct position.

[0232] In this case, the comparison unit 1235 calculates the similarity at the joint point, or estimates the similarity at the joint point to be a predetermined value (for example, 0.5).

[0233] For example, if a joint point of user U1 corresponding to a joint point estimated to be hidden by the body of trainer U0 is estimated to not be hidden by the body of user U1, the comparison unit 1235 estimates that the joint point of user U1 that is not hidden is not in the correct position.

[0234] In this case, the comparison unit 1235 estimates the similarity at the joint point to be a predetermined value (for example, zero).

[0235] For example, if a joint point of user U1 corresponding to a joint point estimated not to be hidden by trainer U0's body is estimated to be hidden by user U1's body, the comparison unit 1235 estimates that the hidden joint point of user U1 is not in the correct position.

[0236] In this case, the comparison unit 1235 estimates the similarity at the joint point to be a predetermined value (for example, zero).

[0237] For example, if a joint point of user U1 corresponding to a joint point estimated not to be hidden by trainer U0's body is estimated not to be hidden by user U1's body, the comparison unit 1235 estimates that the hidden joint point of user U1 may be in the correct position.

[0238] In this case, the comparison unit 1235 calculates the similarity at the joint point.

[0239] In this way, the comparison unit 1235 can calculate the similarity of the joint point depending on whether the joint point is hidden by the body. In this case, the estimation unit 1234 can estimate whether the joint point is hidden in addition to or instead of the reliability score of the joint point of user U1. Furthermore, the server device 200 can store information indicating whether the joint point is hidden in addition to or instead of the reliability score of the joint point of trainer U0.

[0240] For example, the comparison unit 1235 evaluates the posture of the user U1 at one or more timings (scoring timings) during the scoring period in accordance with instructions from the setting unit 1232. That is, the comparison unit 1235 calculates the similarity of the posture of the user U1 to the posture of the trainer U0 at one or more scoring timings during the scoring period.

[0241] The comparison unit 1235 determines the largest similarity among the similarities calculated at one or more scoring timings as the similarity of user U1 for the scoring period including the scoring timing.

[0242] The comparison unit 1235 notifies the evaluation unit 1236 of the determined similarity as the similarity of the posture of the user U1 during the scoring period.

[0243] (Evaluation Unit 1236) Returning to FIG. 5, the evaluation unit 1236 acquires from the comparison unit 1235 the comparison result (similarity) of the posture of the user U1 with the posture of the trainer U0 during the scoring period.

[0244] The evaluation unit 1236 evaluates the posture of the user U1 during the scoring period based on the acquired comparison result.

[0245] For example, the evaluation unit 1236 calculates a score according to the similarity and sets the calculated score as the evaluation result of the user U1. For example, when the similarity is 1, the evaluation unit 1236 calculates the score (evaluation score) so that the score is 100 points.

[0246] Alternatively, the evaluation unit 1236 may evaluate the posture of the user U1 during the scoring period in stages according to the degree of similarity.

[0247] Here, the comparison unit 1235 calculates the similarity between the posture of the user U1 and the posture of the trainer U0. However, it cannot be said that the similarity calculated by the comparison unit 1235 is necessarily close to the value obtained when a human scores the posture.

[0248] This is because the sense of similarity that a human would have when visually comparing postures does not necessarily correspond to the similarity that is determined by the posture comparison performed by the comparison unit 1235. Therefore, if the evaluation unit 1236 calculates a score according to the degree of similarity as the evaluation score, the user U1 may feel uncomfortable with this evaluation score.

[0249] Therefore, the evaluation unit 1236 evaluates the posture of the user U1 during the scoring period in stages according to the degree of similarity. Specifically, the evaluation unit 1236 classifies the posture of the user U1 into three stages, "high," "medium," and "low," according to the degree of similarity, for example.

[0250] Alternatively, the evaluation unit 1236 may assign points to each stage. For example, if the evaluation of the posture of user U1 is "high," the evaluation unit 1236 assigns 3 points to the posture of user U1. For example, if the evaluation of the posture of user U1 is "medium," the evaluation unit 1236 assigns 2 points to the posture of user U1. For example, if the evaluation of the posture of user U1 is "low," the evaluation unit 1236 assigns 0 points to the posture of user U1.

[0251] The evaluation unit 1236 compares the similarity with a threshold and performs evaluation according to the comparison result, or determines a score according to the comparison result.

[0252] When the rating period ends, the evaluation unit 1236 evaluates the user U1 during that rating period and notifies the UI control unit 1237 of the evaluation result. This allows the UI control unit 1237 to present the evaluation result to the user U1 in real time.

[0253] For example, the UI control unit 1237 presents the evaluation result to the user U1 using text information, effects, sound effects, voice, etc. according to the evaluation result. The presentation of the evaluation result by the UI control unit 1237 will be described in detail later.

[0254] Next, when the playback of the model video C0 is finished, the evaluation unit 1236 evaluates the posture of the user U1 throughout the entire exercise (hereinafter also referred to as the overall evaluation). For example, the evaluation unit 1236 sets the average value of the evaluation points for at least one scoring period as the score of the overall evaluation (overall evaluation point).

[0255] Alternatively, the evaluation unit 1236 determines the overall evaluation based on the total value of the evaluation points for at least one grading period. For example, the evaluation unit 1236 adds up the evaluation points (3 points, 2 points, or 0 points) determined in stages, and determines the total score as the overall evaluation point for the model video C0.

[0256] The evaluation unit 1236 may also evaluate the exercise of the user U1 in stages according to the overall evaluation score. For example, the evaluation unit 1236 compares the overall evaluation score with a threshold and classifies the exercise of the user U1 into three stages: "high," "medium," and "low" according to the comparison result.

[0257] The evaluation unit 1236 notifies the UI control unit 1237 of the result of the overall evaluation. This allows the UI control unit 1237 to present the result of the overall evaluation to the user U1 after playing back the model video C0.

[0258] The UI control unit 1237 presents the result of the overall evaluation to the user U1 using, for example, an overall evaluation score as well as effects, sound effects, voices, etc. according to the graded overall evaluation.

[0259] For example, if the scaled overall rating is "high," the UI control unit 1237 presents the scaled overall rating to the user U1 by displaying three stars on the display device 130. For example, if the scaled overall rating is "medium," the UI control unit 1237 presents the scaled overall rating to the user U1 by displaying two stars on the display device 130. For example, if the scaled overall rating is "low," the UI control unit 1237 presents the scaled overall rating to the user U1 by displaying one star on the display device 130.

[0260] In this way, the evaluation unit 1236 evaluates the exercise so that the UI control unit 1237 can present the evaluation result to the user U1. The evaluation unit 1236 evaluates the posture of the user U1 for each scoring period, so that the information processing device 120 can present the evaluation result to the user U1 in real time. Furthermore, the evaluation unit 1236 performs an overall evaluation at the end of playback of the model video C0, so that the information processing device 120 can present the evaluation of the posture throughout the entire exercise to the user U1.

[0261] 5 generates, for example, a display image and displays it on the display device 130. The UI control unit 1237 also outputs sounds (voice, background music, sound effects, etc.) from a speaker or the like.

[0262] An example of a display image generated by the UI control unit 1237 will be described below.

[0263] (Example of a selected image) For example, the information processing device 120 may present a list of one or more candidate model videos C0 to the user U1, and the user U1 may select a model video C0 that includes the exercise that the user U1 wants to perform from the list.

[0264] In this case, the UI control unit 1237 generates a selection image including a list of candidates for the model video C0, and displays it on the display device 130.

[0265] 13 is a diagram illustrating an example of a selection image according to an embodiment of the present disclosure. The selection image illustrated in FIG. 13 includes a thumbnail image of the model video C0, a description of the model video C0, and the like.

[0266] For example, the selected image includes information indicating the type of included model video C0, such as information indicating that the model video C0 is an exercise to be evaluated (graded) ("graded training" in the example of FIG. 13) and information indicating the effect obtained by performing the exercise ("relieving lack of exercise / improving muscle strength" in the example of FIG. 13).

[0267] As mentioned above, the selected image includes information about the individual model video C0 (hereinafter also referred to as video information), such as a thumbnail image and description. The video information may include at least one of the following. Also, the user U1 may be able to select which video information to include in the selected image. - Thumbnail image - Title - Effect - Body part - Length of the model video C0 - Mark indicating that it is a favorite video - Previous evaluation result

[0268] The title is information indicating the content of the model video C0, and in the example of FIG. 13, "Center of gravity control exercises in a standing position (left and right)" is shown as the title. The effect is the effect expected from performing the exercise in the model video C0. In the example of FIG. 13, the effects shown are "ease of stretching legs" and "ease of lifting legs." The body part indicates the body part where the expected effect can be obtained by performing the exercise. In the example of FIG. 13, "legs" is shown as the body part.

[0269] The mark indicating that a video is a favorite is, for example, a heart mark. When a video is favorited, a heart mark with a color such as yellow or red is displayed, and when a video is not favorited, a heart mark with only a frame such as a white frame is displayed.

[0270] The past evaluation result is information indicating the evaluation of the exercise performed by the user U1 while watching the model video C0. The evaluation result may be, for example, a score (evaluation point) or a number of stars. If the user U1 has watched the model video C0 multiple times, the past evaluation result may be the evaluation result of the previous exercise or the evaluation result with the highest evaluation.

[0271] The user U1 selects one of the one or more candidate model videos C0. The UI control unit 1237 instructs the playback unit 1233 to play the selected model video C0.

[0272] Here, the UI control unit 1237 may present the model video C0, which has been downloaded in advance from the server device 200, to the user U1 as a candidate.

[0273] Alternatively, the UI control unit 1237 may acquire video information of the model video C0 that can be downloaded by the information processing device 120 from the server device 200 and present the video information to the user U1. The video information may be acquired from the server device 200 as part of the model information, for example.

[0274] In this case, for example, the UI control unit 1237 notifies the acquisition unit 1231 of information about the model video C0 selected by the user U1. The acquisition unit 1231 acquires (downloads) the selected model video C0 from the server device 200 and notifies the playback unit 1233 of the information.

[0275] (Example of instruction image) Next, the UI control unit 1237 generates an instruction image that specifies the placement of the terminal device 100, and displays it on the display device 130. The instruction image is an image that indicates the positional relationship between the user U1 and the terminal device 100, etc.

[0276] 14 is a diagram illustrating an example of an instruction image according to an embodiment of the present disclosure. As illustrated in FIG. 14, the instruction image includes information specifying the distance between the terminal device 100 (a smartphone in FIG. 14) and the user U1 (1 meter in FIG. 14).

[0277] In the example of Figure 14, the information processing device 120 uses an instruction image to instruct user U1 to lean the terminal device 100 (here, a smartphone) against the screen and move 1 m away from the terminal device 100 (step 1 m back from the smartphone) so that the user U1's entire body is visible.

[0278] The instruction image may also include information instructing the orientation of the terminal device 100, i.e., whether to place the terminal device 100 vertically or horizontally. For example, the UI control unit 1237 may instruct the orientation of the terminal device 100 according to the size of the model video C0. For example, if the model video C0 is a vertically oriented video, the UI control unit 1237 generates an instruction image instructing the user to place the terminal device 100 vertically.

[0279] Furthermore, the UI control unit 1237 may present the user U1 with an explanatory image explaining the exercise or a confirmation image for confirming whether the user U1 has been photographed correctly before playing the model video C0. Here, the user U1 having been photographed correctly means that the information processing device 120 can evaluate the posture of the user U1 using the photographed image of the user U1.

[0280] (Example of Confirmation Image) An example of a confirmation image will be described below.

[0281] 15 is a diagram illustrating an example of a confirmation image according to an embodiment of the present disclosure. As illustrated in FIG. 15, the confirmation image includes an image instructing confirmation and an image after confirmation.

[0282] As shown in the right diagram of FIG. 15 , the image instructing the user to confirm is an image instructing the user to take a photo of the user U1 using the terminal device 100 (here, a smartphone). The UI control unit 1237 generates an image instructing the user U1 to capture the body part to be evaluated by exercise (evaluation target body part). In the example of FIG. 15 , the UI control unit 1237 generates an image instructing the user U1 to prop up the smartphone so that the entire body is captured. That is, in this example, the evaluation target body part is the entire body.

[0283] The estimation unit 1234 estimates the posture of the user U1 from the captured image acquired from the image capturing device 110 and determines whether the entire body is captured. If it is determined that the entire body is captured, the estimation unit 1234 may determine whether the size of the user U1 is sufficient.

[0284] If the estimation unit 1234 determines that the entire body of the user U1 has been photographed at a sufficient size, the UI control unit 1237 presents to the user U1 an image indicating that the image has been photographed correctly, as shown in the left diagram of Fig. 15. In the example of Fig. 15, the UI control unit 1237 presents to the user U1 information indicating that preparations for exercise have been completed ("Ready!").

[0285] If the user U1 is not photographed correctly, the UI control unit 1237 may, for example, display on the display device 130 an image that guides the user U1 to photograph correctly.

[0286] For example, if the entire body of user U1 is photographed but the image is small, the UI control unit 1237 generates an image instructing user U1 to move closer to the terminal device 100 and presents it to user U1.

[0287] Furthermore, if the entire body of the user U1 is not shown, the UI control unit 1237 generates an image instructing the user U1 to move in a direction in which the entire body will be shown, and presents this image to the user U1.

[0288] In this way, the UI control unit 1237 presents the confirmation image to the user U1, which allows the information processing device 120 to guide the user U1 to a position where the posture of the user U1 can be correctly evaluated before the start of exercise. This allows the information processing device 120 to evaluate the posture of the user U1 with higher accuracy.

[0289] (Example of Start Image) For example, an interval may be provided between when the user U1 instructs the terminal device 100 to play the model video C0 and when the model video C0 is actually played.

[0290] For example, suppose that user U1 starts playing the model video C0 by tapping a video playback button displayed on the screen of terminal device 100. Also, suppose that user U1 exercises, for example, about 1 meter away from terminal device 100.

[0291] In this case, user U1 needs to tap the video playback button and then move to the position where he or she will perform the exercise, but if the model video C0 starts playing immediately after tapping the button, user U1 may not be able to move in time.

[0292] Therefore, in this embodiment, when the user U1 instructs the playback unit 1233 to play the video, the playback unit 1233 starts playing the model video C0 after an interval from the instruction. This interval can be determined according to the recommended distance between the terminal device 100 and the user U1, in other words, the part of the user U1 (e.g., the whole body or the upper body) that the imaging device 110 will capture.

[0293] During the period from when the user U1 issues a video playback instruction until the model video C0 is played, the UI control unit 1237 displays, for example, a start image on the display device 130. The start image is, for example, an image that indicates the remaining time until the model video C0 is played. For example, the start image may be an image that counts down the time until the model video C0 is played.

[0294] 16 is a diagram illustrating an example of a start image according to an embodiment of the present disclosure. As shown in FIG. 16, the UI control unit 1237 causes the display device 130 to display, for example, the first frame image of the model video C0 with a numerical value indicating the time until playback starts superimposed thereon.

[0295] In the example of FIG. 16, the UI control unit 1237 superimposes numerical values ​​"3," "2," and "1" indicating the time until playback starts on the frame image.

[0296] (Display example of model video C0) When playback of the model video C0 starts, the UI control unit 1237 displays the model video C0 on the display device 130. At this time, the UI control unit 1237 may also display the model video C0 and the captured video C1 on the display device 130.

[0297] Display Example 1 For example, the UI control unit 1237 causes the display device 130 to display the captured video C1 superimposed as a wipe image on the model video C0.

[0298] 17 is a diagram illustrating an example of a display image according to an embodiment of the present disclosure, in which a captured video C1 is superimposed on the upper left of a model video C0.

[0299] The UI control unit 1237 displays the model video C0 as a large image and the captured video C1 as a small image, allowing the user U1 to check the movements of the trainer U0 on a large screen while checking their own movements.

[0300] Here, the UI control unit 1237 superimposes the captured video C1 on the model video C0, but the model video C0 may be superimposed on the captured video C1. In this case, the user U1 can check his / her own movements on a large screen. The user U1 may be able to select which video to display larger.

[0301] (Display Example 2) For example, the UI control unit 1237 may cause the display device 130 to display the model video C0 and the captured video C1 side by side.

[0302] 18 is a diagram illustrating another example of a display image according to an embodiment of the present disclosure. In FIG. 18, an example is shown in which a captured video C1 is displayed below a model video C0.

[0303] Here, the UI control unit 1237 displays the model video C0 and the captured video C1 side by side, but these videos may also be displayed side by side. The UI control unit 1237 may switch the direction in which the videos are arranged depending on the orientation of the terminal device 100. Alternatively, the UI control unit 1237 may switch the direction or order in which the videos are arranged depending on an instruction from the user U1.

[0304] (Display Example 3) In the above-described display example, the UI control unit 1237 displays the model video C0 and the captured video C1 on the display device 130. The display by the UI control unit 1237 is not limited to these videos. For example, the UI control unit 1237 may display an image of skeleton data U10 of the user U1 (an example of a posture image) superimposed on the model video C0.

[0305] 19 is a diagram illustrating another example of a display image according to an embodiment of the present disclosure. In FIG. 19, an example is illustrated in which an image of skeleton data U10 of user U1 is superimposed on a model video C0.

[0306] In this case, for example, the estimation unit 1234 estimates the posture of the user U1 while the model video C0 is being played back, and generates skeleton data U10. The UI control unit 1237 displays an image of this skeleton data U10 superimposed on the model video C0.

[0307] Alternatively, the estimation unit 1234 may estimate the posture of the user U1 while the trainer U0 is exercising and generate skeleton data U10. The UI control unit 1237 displays an image of this skeleton data U10 superimposed on the model video C0. The period for generating the skeleton data U10 may be specified in advance, for example, by model information or the like.

[0308] The estimation unit 1234 may estimate the posture of the user U1 during the scoring period and generate skeleton data U10. Alternatively, the estimation unit 1234 may estimate the posture of the user U1 during a predetermined period including the scoring period and generate skeleton data U10. The UI control unit 1237 displays an image of the skeleton data U10 by superimposing it on the model video C0.

[0309] The UI control unit 1237 superimposes and displays an image of the skeleton data U10 generated by the estimation unit 1234 on the model video C0, allowing the user U1 to visually understand how much his or her own movements deviate from the movements of the trainer U0.

[0310] Here, it is assumed that the UI control unit 1237 superimposes an image of the skeleton data U10 of the user U1 on the model video C0. The UI control unit 1237 may superimpose an image of the skeleton data U00 of the trainer U0 on the captured video C1 and display it on the display device 130. The UI control unit 1237 may also superimpose an image of the skeleton data U00 of the trainer U0 and an image of the skeleton data U10 of the user U1 and display them on the display device 130.

[0311] 20 is a diagram illustrating another example of a display image according to an embodiment of the present disclosure. In FIG. 20, an example is shown in which an image of skeleton data U10 of user U1 is superimposed on model video C0, and captured video C1 is superimposed as a wipe image in the upper left corner of model video C0.

[0312] As shown in FIG. 20, the UI control unit 1237 may display an actually shot video C1 in addition to the image of the skeleton data U10, superimposed on the model video C0.

[0313] The user U1 may be able to use, for example, a toggle switch to switch whether or not to superimpose the image of the skeleton data on the video (model video C0 and / or captured video C1).

[0314] (Example of Evaluation Image) As described above, the information processing device 120 evaluates the posture of the user U1 during the scoring period. The result of this evaluation is fed back to the user U1 after the scoring period. The UI control unit 1237 generates an image (hereinafter also referred to as an evaluation image) to be fed back to the user U1.

[0315] An example of the evaluation image will be described with reference to Figures 21 to 23. Figures 21 to 23 show an example of the evaluation image when the evaluation unit 1236 evaluates the posture of the user U1 in three levels: "high," "medium," and "low."

[0316] 21 is a diagram illustrating an example of an evaluation image according to an embodiment of the present disclosure. In FIG. 21, an example is shown in which an evaluation image for a case where the evaluation is "high" is superimposed on the model video C0.

[0317] 21, the evaluation image for a "high" evaluation includes text information of "Perfect" and effects such as stars and flowers. The UI control unit 1237 may output sound effects, voice, and the like along with the evaluation image.

[0318] The time displayed in the upper right corner of the model video C0 in Fig. 21 may be the playback time of the model video C0 or the remaining time of the model video C0. Alternatively, this time may be the time for which the posture to be scored is maintained (or is maintained).

[0319] 22 is a diagram illustrating another example of an evaluation image according to an embodiment of the present disclosure. In FIG. 22, an example is illustrated in which an evaluation image for a case where the evaluation is "medium" is superimposed on the model video C0.

[0320] 22, the evaluation image for a "medium" evaluation includes text information of "Good" and effects such as stars. The number of effects and the color of the text information may be changed from those for a "high" evaluation, so that user U1 can more easily recognize the evaluation of his or her posture.

[0321] The UI control unit 1237 may output sound effects, voice, and the like along with the evaluation image.

[0322] 23 is a diagram illustrating another example of an evaluation image according to an embodiment of the present disclosure. In FIG. 23, an example is shown in which an evaluation image for a case where the evaluation is "low" is superimposed on the model video C0.

[0323] 23, the evaluation image for a "low" evaluation includes text information such as "Good." The presence or absence of effects and the color of the text information may be changed depending on whether the evaluation is "high" or "medium," allowing the user U1 to more easily recognize the evaluation of his or her posture.

[0324] The UI control unit 1237 may output sound effects, voice, and the like along with the evaluation image.

[0325] (Example of Comprehensive Evaluation Image) When playback of the model video C0 is completed, the evaluation unit 1236 determines a comprehensive evaluation of the posture of the user U1. The UI control unit 1237 presents the result of this comprehensive evaluation to the user U1 using a comprehensive evaluation image, sound effects, etc.

[0326] An example of the overall evaluation image will be described with reference to Fig. 24 and Fig. 25. Fig. 24 and Fig. 25 show an example of the overall evaluation image when the evaluation unit 1236 evaluates the posture of the user U1 in three levels: "high", "medium", and "low".

[0327] 24 is a diagram illustrating an example of a comprehensive evaluation image according to an embodiment of the present disclosure, in which the comprehensive evaluation is "low."

[0328] 24, the overall evaluation image for a "low" overall evaluation includes one colored star and two colorless stars. In this way, the UI control unit 1237 presents the overall evaluation result to the user U1 using the number of colored stars.

[0329] 24 includes character (numeric) information indicating the overall evaluation score ("35 points" in the example of FIG. 24). In this way, the UI control unit 1237 may present the overall evaluation score calculated by the evaluation unit 1236 to the user U1.

[0330] 25 is a diagram illustrating another example of a comprehensive evaluation image according to an embodiment of the present disclosure, in which the comprehensive evaluation image is “high.”

[0331] As shown in Fig. 25, when the overall rating is "high," the overall rating image includes three colored stars. The overall rating image also includes character (numeric) information indicating the overall rating score ("99 points" in the example of Fig. 25).

[0332] 25, the overall evaluation image may include other information in addition to the above-mentioned stars and overall evaluation points. For example, the overall evaluation image may include information about points that user U1 can obtain, information for selecting the next action of user U1, etc.

[0333] Here, for example, one method for motivating user U1 to continue exercising is to award points based on an exercise evaluation. For example, by awarding points based on an overall evaluation each time user U1 exercises, it is expected that user U1 will continue exercising in order to earn points.

[0334] The points given to user U1 may be points that can be used within an application for exercising, i.e., an application that plays the model video C0, or may be points that can be used in other applications.

[0335] For example, points that can be used in an application for exercising may be points that can be used for selecting effects such as evaluation images, selecting model videos C0, etc. Furthermore, points that can be used in other applications may be points that can be used for shopping on a shopping site or the like.

[0336] At this time, at least one of the method of awarding points and the method of presenting the scoring results may be changed depending on whether the user is using the application for a fee or free of charge.

[0337] For example, in the case of a user who uses an exercise application for free (hereinafter also referred to as a free user), the UI control unit 1237 may present a reward-based advertisement to the user, and then present the scoring results and award points.

[0338] For example, the UI control unit 1237 may present a short advertisement, then present an overall evaluation image that is the scoring result, and then present a long advertisement to the non-paying user. For example, the non-paying user who has checked the overall evaluation image may be able to earn points by watching the long advertisement.

[0339] Alternatively, the UI control unit 1237 may present a short advertisement, then present an overall evaluation image representing the scoring results and award points, and then present a longer advertisement to non-paying users who wish to earn more points.

[0340] In this way, the information processing device 120 can motivate free users who want to check the overall rating or obtain points to view advertisements by encouraging free users to view advertisements before displaying the overall rating image or awarding points.

[0341] On the other hand, the UI control unit 1237 can present a comprehensive evaluation image and award points to paying users without presenting advertisements, which allows the information processing device 120 to encourage non-paying users who want to avoid viewing advertisements to become paying users.

[0342] By tapping the button labeled 300P shown in FIG. 25, the user U1 can acquire 300 points.

[0343] The next action options for user U1 include two options: "End exercise" or "Perform next exercise (training)." In the example of Fig. 25, user U1 taps the "End" button to select the end of exercise, for example, the end of the application for exercising.

[0344] 25, the user U1 can select the next model video C0 by tapping the “Proceed to next training” button. For example, the UI control unit 1237 presents the selected image to the user U1.

[0345] For example, if user U1 taps the "Proceed to next training" button while the check button next to the text information that reads "Do this training again next time" is checked, user U1 may be able to skip the selection of the model video C0.

[0346] In this case, for example, the UI control unit 1237 starts playing the same model video C0 without presenting the selected image to the user U1.

[0347] <<3. Information Processing Example>> An example of information processing executed by the information processing device 120 according to this embodiment will be described below. The information processing device 120 executes, for example, a playback process and an evaluation process.

[0348] 26 is a flowchart showing an example of the flow of a playback process according to an embodiment of the present disclosure. The playback process shown in Fig. 26 is executed, for example, when the user U1 executes an application for exercising, or when the user U1 instructs the information processing device 120 to execute a process for exercising.

[0349] As shown in FIG. 26, the information processing device 120 presents candidates for the model video C0 to the user U1, for example, by presenting a selection image (step S11).

[0350] Next, the information processing device 120 determines whether the model video C0 is to be selected by the user U1 (step S12). If the model video C0 is not selected by the user U1 (step S12; No), the information processing device 120 returns to step S12 and waits for a selection by the user U1.

[0351] On the other hand, if the model video C0 is selected by the user U1 (step S12; Yes), the information processing device 120 instructs the user U1 to align the terminal device 100 (step S13).

[0352] For example, the information processing device 120 may present an instruction image to instruct the user U1 to change the orientation of the terminal device 100 or the distance from the user U1. At this time, the information processing device 120 may present a confirmation image to instruct the user U1 to move the user U1 or the terminal device 100 so that the user U1 is photographed correctly.

[0353] Thereafter, the information processing device 120 plays the model video C0 (step S14). At this time, the information processing device 120 may present a start image before playing the model video C0.

[0354] The information processing device 120 executes the evaluation process while playing the model video C0 (step S15), and then ends the process.

[0355] 27 is a flowchart showing an example of the flow of the evaluation process according to an embodiment of the present disclosure. The process shown in FIG. 27 is executed by the information processing device 120 when, for example, playback of the model video C0 is started.

[0356] 27, the information processing device 120 first determines whether it is the timing to start scoring, i.e., whether the scoring period has started (step S101). If it is not the timing to start scoring, i.e., if the scoring period has not started (step S101; No), the information processing device 120 returns to step S101 and waits for the start of the scoring period.

[0357] On the other hand, when it is time to start scoring, that is, when the scoring period has started (step S101; Yes), the information processing device 120 estimates the posture of the subject (here, user U1) of the captured video C1 (step S102).

[0358] Next, the information processing device 120 compares the estimated posture of the user U1 with the posture of the subject (trainer U0 in this case) in the model video C0 (step S103). The information processing device 120 calculates, for example, the similarity of the posture of the user U1 to the posture of the trainer U0.

[0359] The information processing device 120 determines whether it is time to end the scoring, i.e., whether the scoring period has ended (step S104). If it is not time to end the scoring, i.e., if the scoring period has not ended (step S104; No), the information processing device 120 returns to step S102 and estimates the posture of the user U1.

[0360] On the other hand, when it is time to end the scoring, that is, when the scoring period has ended (step S104; Yes), the information processing device 120 evaluates the posture of the subject (here, user U1) of the captured video C1 (step S105).

[0361] The information processing device 120 calculates an evaluation score based on the highest value of the similarities calculated during the scoring period, for example. The information processing device 120 may also evaluate the posture of the user U1 in multiple stages based on the evaluation score.

[0362] The information processing device 120 presents the evaluation result to the user U1 (step S106). The information processing device 120 superimposes an evaluation image corresponding to the evaluation level (e.g., “high,” “medium,” “low,” etc.) on the model video C0 and displays it on the display device 130.

[0363] The information processing device 120 determines whether the playback of the model video C0 has ended (step S107). If the playback of the model video C0 has not ended (step S107; No), the information processing device 120 returns to step S101 and waits for the start of the next scoring period.

[0364] On the other hand, when the model video C0 has ended (step S107; Yes), the information processing device 120 calculates an overall evaluation (step S108). For example, the information processing device 120 calculates the total evaluation score as the sum of the evaluation scores during the grading period. Furthermore, the information processing device 120 may evaluate the posture of the user U1 on a multiple-level scale depending on the overall evaluation score.

[0365] The information processing device 120 presents the overall evaluation result to the user U1 (step S109), and ends the process. The information processing device 120 causes the display device 130 to display an overall evaluation image corresponding to the overall evaluation level (e.g., “high,” “medium,” “low,” etc.).

[0366] As described above, the information processing device 120 according to this embodiment compares the posture of the subject in the model video C0 with the posture of the subject in the filmed video C1, and evaluates the posture of the subject in the filmed video C1 based on the comparison result. By evaluating the posture of the subject in the filmed video C1 during the scoring period, the information processing device 120 can present the posture evaluation result to the user U1 in real time.

[0367] <<4. Hardware Configuration Example>> Fig. 28 is a diagram showing an example of the hardware configuration of a device, etc. The terminal device 100 or the server device 200 described above is realized by, for example, a computer 1000 shown in Fig. 28.

[0368] The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected to each other via a bus 1050.

[0369] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0370] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0371] The HDD 1400 is a computer-readable storage medium that non-temporarily stores programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a storage medium that stores a program for the information processing method according to the present disclosure, which is an example of program data 1450.

[0372] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0373] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs stored on a predetermined computer-readable storage medium. Examples of media include optical storage media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical storage media such as MOs (Magneto-Optical Discs), tape media, magnetic storage media, and semiconductor memories.

[0374] When the computer 1000 functions as the terminal device 100 or the server device 200 described above, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the control unit 230 or the control unit 123. The program may be stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain the program from another device via the external network 1550.

[0375] Each of the above components may be configured using general-purpose materials or may be configured using hardware specialized for the function of each component. Such configurations may be changed as appropriate depending on the technical level at the time of implementation.

[0376] <<5. Other Embodiments>> The processing according to the above-described embodiment may be implemented in various different forms other than the above embodiment.

[0377] (3D sensor) In the above-described embodiment, the information processing device 120 estimates the posture of the subject (e.g., user U1) from the captured video C1, but the information processing device 120 may also estimate the posture of the subject using data other than the captured video C1.

[0378] For example, the information processing device 120 may estimate the posture of the subject based on the distance from the distance measurement sensor to the subject. In this case, the distance measurement sensor may be mounted on the terminal device 100, or may be disposed around the subject as a device separate from the terminal device 100.

[0379] In this case, the information processing device 120 estimates the posture of the subject using at least one of the captured video C1 and the distance to the subject.

[0380] (Difficulty Level) In the above-described embodiment, one model video C0 is presented for one exercise. For example, multiple model videos C0 may be presented for one exercise.

[0381] For example, the server device 200 stores a plurality of model videos C0 with different levels of difficulty for one exercise. The user U1 selects one of the model videos C0 with different levels of difficulty by, for example, selecting a level of difficulty.

[0382] Alternatively, the difficulty level of the exercise may be changed by changing the evaluation parameters. For example, the information processing device 120 changes the length of the scoring period, the threshold used for determining the graded evaluation of posture, etc., depending on the difficulty level selected by the user U1. For example, the information processing device 120 may lengthen the scoring period or set stricter thresholds used for determination as the difficulty level increases.

[0383] The information processing device 120 may change the difficulty level of the exercise depending on the posture evaluation results during the scoring period, regardless of the difficulty level selection by user U1, or even if user U1 has not selected the difficulty level.

[0384] For example, if the posture evaluation result continues to receive low ratings, the information processing device 120 lowers the difficulty level of the exercise. On the other hand, if the posture evaluation result continues to receive high ratings, the information processing device 120 raises the difficulty level of the exercise. Alternatively, the information processing device 120 may prompt the user U1 to change the difficulty level when selecting the next model video C0.

[0385] Alternatively, the information processing device 120 may compare the posture of the trainer U0 in each of the multiple model videos C0 with the posture of the user U1 in the captured video C1. That is, the information processing device 120 evaluates the posture of the user U1 for each difficulty level.

[0386] The information processing device 120 determines the highest evaluation result from among the multiple model videos C0 with different levels of difficulty as the evaluation result of the posture of the user U1 during the grading period.

[0387] This allows the information processing device 120 to perform an evaluation according to the exercise proficiency of the user U1.

[0388] (Selecting to Execute Evaluation) In the above-described embodiment, when the model video C0 is selected, the posture evaluation of the user U1 is performed. Alternatively, the user U1 may be allowed to select whether or not to have the posture evaluated.

[0389] For example, if the user U1 selects to have his / her posture evaluated, the information processing device 120 evaluates the posture of the user U1. On the other hand, if the user U1 selects not to have his / her posture evaluated, the information processing device 120 does not evaluate the posture of the user U1, and plays the model video C0.

[0390] (Evaluation of continuity) In the embodiment described above, the information processing device 120 performs a comprehensive evaluation based on the evaluation points during the grading period. In addition, the information processing device 120 may perform a comprehensive evaluation taking into account the continuity of the evaluations during the grading period.

[0391] For example, if there are consecutive scoring periods in which the evaluation result of user U1's posture is equal to or greater than a predetermined threshold, the information processing device 120 changes the overall evaluation or the evaluation of user U1's posture during the scoring period.

[0392] For example, if high ratings are consecutively given during a rating period, the information processing device 120 performs an overall rating based on the number of consecutive high ratings in addition to the rating points for the rating period. For example, the more consecutive high ratings there are, the higher the overall rating point becomes.

[0393] In this way, the information processing device 120 can perform a comprehensive evaluation by taking into consideration items other than the evaluation results during the grading period (for example, the continuity of the evaluation).

[0394] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0395] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0396] <<6. Conclusion>> The effects described in this disclosure are merely examples and are not limited to the disclosed content. Other effects may also be obtained.

[0397] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0398] The present technology may also be configured as follows. (1) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: acquiring a first pose of a first subject included in a first moving image during a first period of the first moving image displayed on a display device; estimating a second pose of a second subject included in a second moving image during a second period of the second moving image acquired from an imaging device; comparing the first pose with the second pose; evaluating the second pose based on a comparison result; and presenting evaluation information related to the evaluation to a user. (2) The computer-readable non-transitory storage medium according to (1), wherein the first period is a predetermined timing of the first moving image. (3) The computer-readable non-transitory storage medium according to (1) or (2), wherein a plurality of the first periods are set within the entire period of the first moving image. (4) The computer-readable non-transitory storage medium according to any one of (1) to (3), wherein the first period is set in advance. (5) The computer-readable non-transitory storage medium according to any one of (1) to (4), storing the program further causing a computer to determine the first period based on at least one of a movement of the first subject in the first video and sound information corresponding to the first video. (6) The computer-readable non-transitory storage medium according to any one of (1) to (5), storing the program further causing a computer to estimate the first posture using at least a part of the first subject included in the first video, and estimate the second posture using at least a part of the second subject included in the second video. (7) The computer-readable non-transitory storage medium according to any one of (1) to (6), storing the program further causing a computer to compare at least a part of the first posture and at least a part of the second posture.(8) The computer-readable non-transitory storage medium according to any one of (1) to (7), wherein the evaluation information includes at least one of an evaluation value calculated according to the evaluation and an evaluation index set according to the evaluation value. (9) The computer-readable non-transitory storage medium according to (8), which stores the program that further causes a computer to perform the following: presenting the evaluation information to the user after the second period ends. (10) The computer-readable non-transitory storage medium according to any one of (1) to (9), which stores the program that further causes a computer to perform the following: when there are multiple second periods, presenting to the user comprehensive evaluation information including at least one of a sum of evaluation values ​​calculated according to the evaluations for each second period and an evaluation index set according to the sum. (11) The computer-readable non-transitory storage medium according to (10), wherein the comprehensive evaluation information is presented to the user after playback of the first moving image ends. (12) The computer-readable non-transitory storage medium according to any one of (1) to (11), storing the program that causes a computer to further execute the following: when there are a plurality of second periods, if the second periods during which the evaluation of the second posture is equal to or greater than a predetermined threshold are consecutive, changing the evaluation. (13) The computer-readable non-transitory storage medium according to (12), wherein the change in the evaluation is made according to the number of consecutive second periods during which the evaluation of the second posture is equal to or greater than the predetermined threshold. (14) The computer-readable non-transitory storage medium according to any one of (1) to (13), storing the program that further causes a computer to execute the following: comparing the first posture of the first subject in at least one of a plurality of first moving images, each of which has a different degree of difficulty of movement of the first subject, with the second posture. (15) The computer-readable non-transitory storage medium according to any one of (1) to (14), which stores the program that causes the computer to further execute the following: displaying both the first moving image and the second moving image on the display device.(16) The computer-readable non-transitory storage medium according to (15), which stores the program that causes a computer to further execute the following: display the second moving image on the display device superimposed on or parallel to the first moving image. (17) The computer-readable non-transitory storage medium according to (15) or (16), which stores the program that causes a computer to further execute the following: switch between displaying and hiding the second moving image in accordance with an instruction from the user. (18) The computer-readable non-transitory storage medium according to any one of (15) to (17), which stores the program that causes a computer to further execute the following: switch at least one of the display size, display position, and display method of the second moving image on the display device in accordance with an instruction from the user. (19) The computer-readable non-transitory storage medium according to any one of (1) to (18), which stores the program that causes a computer to further execute the following: display a posture image according to the posture of the second subject and the first moving image on the display device. (20) The computer-readable non-transitory storage medium according to (19), which stores the program that causes a computer to further execute the following: causing the display device to superimpose the posture image on the second moving image. (21) The computer-readable non-transitory storage medium according to (19), which stores the program that causes a computer to further execute the following: causing the display device to display the posture image superimposed on the first moving image. (22) The computer-readable non-transitory storage medium according to any one of (1) to (21), which stores the program that causes a computer to further execute the following: calculating a similarity between the first posture and the second posture; and evaluating the second posture according to the similarity. (23) The computer-readable non-transitory storage medium according to (22), which stores the program that causes a computer to further execute the following: calculating the similarity based on first joint information included in the first posture and second joint information included in the second posture and corresponding to the first joint information.(24) The computer-readable non-transitory storage medium according to (22) or (23), which stores the program causing a computer to further execute the following: calculating the similarity according to a distance between a first joint included in the first posture and a second joint included in the second posture that corresponds to the first joint when the first posture and the second posture are superimposed. (25) The computer-readable non-transitory storage medium according to any one of (22) to (24), which calculates the similarity for each of the second joints. (26) The computer-readable non-transitory storage medium according to any one of (22) to (25), which stores the program causing a computer to further execute the following: calculating the similarity according to at least one of a first joint reliability for an estimation of a first joint included in the first posture and a second joint reliability for an estimation of a second joint that corresponds to the first joint. (27) The computer-readable non-transitory storage medium according to (26), which stores the program causing a computer to further execute: calculating a similarity between the first joint and the second joint when the first joint reliability is greater than a first threshold and the second joint reliability is greater than a second threshold. (28) The computer-readable non-transitory storage medium according to (27), which stores the program causing a computer to further execute: weighting the similarity according to at least one of the first joint reliability and the second joint reliability. (29) The computer-readable non-transitory storage medium according to any one of (1) to (28), which stores the program causing a computer to further execute: evaluating the second posture according to a first interval in which a plurality of the first periods occur and a second interval in which a plurality of the second periods occur. (30) The computer-readable non-transitory storage medium according to any one of (1) to (29), in which the second period includes the first period.(31) The computer-readable non-transitory storage medium according to any one of (1) to (30), which stores the program causing a computer to further perform the following: comparing the first posture with the second posture a plurality of times during the second period; and determining the highest comparison result among the results of the plurality of comparisons as the comparison result for the second period. (32) The computer-readable non-transitory storage medium according to any one of (1) to (31), in which the second period is set according to at least one of an interval at which the first period appears, a tempo of sound information corresponding to the first moving image, and a speed at which the first posture changes. (33) The computer-readable non-transitory storage medium according to (32), in which the second period is set longer as the tempo becomes slower. (34) The computer-readable non-transitory storage medium according to any one of (1) to (33), storing the program that causes a computer to further perform the following: presenting instruction information for specifying a shooting orientation of the imaging device to the user in accordance with an image size of the first moving image. (35) The computer-readable non-transitory storage medium according to any one of (1) to (34), wherein the second posture is estimated in accordance with a three-dimensional position of the second subject. (36) An information processing device comprising: a control unit that acquires a first posture of a first subject included in a first moving image during a first period of the first moving image displayed on a display device; estimates a second posture of a second subject included in the second moving image during a second period of a second moving image acquired from an imaging device; compares the first posture with the second posture; evaluates the second posture based on a comparison result; and presents evaluation information related to the evaluation to the user.(37) An information processing method including: acquiring a first posture of a first subject included in a first moving image during a first period of the first moving image displayed on a display device; estimating a second posture of a second subject included in a second moving image during a second period of the second moving image acquired from an imaging device; comparing the first posture with the second posture; evaluating the second posture based on a comparison result; and presenting evaluation information related to the evaluation to a user.

[0399] REFERENCE SIGNS LIST 10 Information processing system 100 Terminal device 110 Imaging device 120 Information processing device 121, 210 Communication unit 122, 220 Storage unit 123, 230 Control unit 130 Display device 200 Server device 1231 Acquisition unit 1232 Setting unit 1233 Playback unit 1234 Estimation unit 1235 Comparison unit 1236 Evaluation unit 1237 UI control unit

Claims

1. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: acquiring a first pose of a first subject included in a first video displayed on a display device during a first period of the first video; estimating a second pose of a second subject included in a second video acquired from an imaging device during a second period of the second video; comparing the first pose with the second pose; evaluating the second pose based on the comparison result; and presenting evaluation information related to the evaluation to a user.

2. The computer-readable non-transitory storage medium of claim 1, wherein the first period is a predetermined timing of the first moving image.

3. The computer-readable non-transitory storage medium of claim 1, wherein the evaluation information is presented to the user after the second period of time has expired.

4. A computer-readable non-transitory storage medium as described in claim 1, which stores the program that further causes the computer to present to the user, when there are multiple second periods, comprehensive evaluation information including at least one of the sum of evaluation values ​​calculated according to the evaluation for each second period and an evaluation index set according to said sum.

5. The computer-readable non-transitory storage medium according to claim 4, wherein the comprehensive evaluation information is presented to the user after playback of the first moving image has ended.

6. A computer-readable non-transitory storage medium as described in claim 1, which stores the program that further causes a computer to perform the following: if there are multiple second periods, and if the second periods in which the evaluation of the second posture is equal to or greater than a predetermined threshold are consecutive, change the evaluation.

7. A computer-readable non-transitory storage medium as described in claim 1, which stores the program that further causes a computer to compare the first posture of the first subject with the second posture in at least one of a plurality of first moving images, each of which has a different degree of difficulty of movement of the first subject.

8. A computer-readable non-transitory storage medium as described in claim 1, which stores the program that further causes the computer to display on the display device a posture image corresponding to the second posture of the second subject and the first moving image.

9. A computer-readable non-transitory storage medium according to claim 8, which stores the program that causes the computer to further perform the following: superimposing the posture image on the first moving image and displaying it on the display device.

10. A computer-readable non-transitory storage medium as described in claim 1, which stores the program that causes a computer to further perform the following: calculate a similarity between the first posture and the second posture; and evaluate the second posture according to the similarity.

11. A computer-readable non-transitory storage medium as described in claim 10, which stores the program that causes a computer to further perform the following: when the first posture and the second posture are superimposed, calculate the similarity according to the distance between a first joint included in the first posture and a second joint included in the second posture that corresponds to the first joint.

12. The computer-readable non-transitory storage medium of claim 11, wherein the similarity is calculated for each of the second joints.

13. A computer-readable non-transitory storage medium as described in claim 11, which stores the program that further causes a computer to calculate the similarity depending on at least one of a first joint confidence for the estimation of the first joint and a second joint confidence for the estimation of the second joint.

14. The computer-readable non-transitory storage medium of claim 1, further storing the program that causes a computer to evaluate the second posture according to a first interval in which a plurality of the first periods occur and a second interval in which a plurality of the second periods occur.

15. The computer-readable non-transitory storage medium of claim 1, wherein the second period of time includes the first period of time.

16. The computer-readable non-transitory storage medium of claim 1, which stores the program that causes a computer to further perform the following: comparing the first posture with the second posture multiple times during the second period; and determining the highest comparison result among the results of the multiple comparisons as the comparison result for the second period.

17. A computer-readable non-transitory storage medium as described in claim 1, wherein the second period is set according to at least one of the interval at which the first period appears, the tempo of the sound information corresponding to the first moving image, and the speed at which the first posture changes.

18. The computer-readable non-transitory storage medium of claim 1, wherein the second pose is estimated according to a three-dimensional position of the second object.

19. An information processing device comprising: a control unit that acquires a first posture of a first subject included in a first moving image during a first period of the first moving image displayed on a display device; estimates a second posture of a second subject included in a second moving image during a second period of the second moving image acquired from an imaging device; compares the first posture with the second posture; evaluates the second posture based on the comparison result; and presents evaluation information regarding the evaluation to a user.

20. An information processing method comprising: acquiring a first posture of a first subject included in a first moving image during a first period of the first moving image displayed on a display device; estimating a second posture of a second subject included in a second moving image during a second period of the second moving image acquired from an imaging device; comparing the first posture with the second posture; evaluating the second posture based on the comparison result; and presenting evaluation information related to the evaluation to a user.

Images