Information processing device, information processing program, and information processing method

Abstract

To provide an information processing device, a method, and a program capable of updating a training menu according to a change in athletic ability of a user.SOLUTION: In a menu management system, an administrative server includes: a control unit including an acquisition section 11a for acquiring a user training image, an analysis section 11b for analyzing user motion from the training image; an evaluation section 11c for evaluating, on the basis of the analysis result, proficiency degree, and an update section 11d for updating, on the basis of the evaluation result and a request, the user training menu; a transmission unit that transmits the training image to a trainer terminal; a first reception unit that receives a training scoring result and feedback to the user from the trainer terminal; and a second reception unit that receives a request for the feedback from a user terminal.SELECTED DRAWING: Figure 21

Description

[Technical Field]

[0001] The present invention relates to an information processing device or the like that updates a training menu. [Background technology]

[0002] At exercise facilities such as fitness gyms, training menus suited to each user are created. For example, Patent Document 1 proposes an exercise support system that presents a training menu (exercise menu) that reflects the user's health condition. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-299492 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the exercise support system described above does not anticipate updating the training menu in response to changes in the user's athletic ability. The present invention has been made in light of this situation. Its purpose is to provide an information processing device or the like that is capable of updating the training menu in response to changes in the user's athletic ability. [Means for solving the problem]

[0005] An information processing device according to one aspect of the present application is characterized by comprising an acquisition unit that acquires training images of a user, an analysis unit that analyzes the user's movements from the training images, an evaluation unit that evaluates the level of improvement based on the analysis results, and an update unit that updates the user's training menu based on the evaluation results. [Effects of the Invention]

[0006] In one aspect of the present application, it is possible to update the training menu in accordance with changes in the user's athletic ability. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a menu management system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a management server. [Figure 3] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a user terminal. [Figure 4] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a trainer terminal. [Figure 5] FIG. 10 is an explanatory diagram illustrating an example of a user DB. [Figure 6] FIG. 10 is an explanatory diagram illustrating an example of a trainer DB. [Figure 7] FIG. 2 is an explanatory diagram illustrating an example of a menu DB. [Figure 8] FIG. 10 is an explanatory diagram showing an example of a training history DB. [Figure 9] FIG. 10 is an explanatory diagram showing an example of a video DB. [Figure 10] FIG. 2 is an explanatory diagram illustrating an example of a product DB. [Figure 11] FIG. 10 is an explanatory diagram showing an example of a recommendation history DB. [Figure 12] FIG. 10 is an explanatory diagram showing an example of an analysis result DB. [Figure 13] 10 is a flowchart illustrating an example of a procedure for update processing. [Figure 14] 10 is a flowchart illustrating an example of a procedure for training processing. [Figure 15] 10 is a flowchart illustrating an example of a procedure for a diagnostic process. [Figure 16] 10 is a flowchart illustrating an example of a procedure for a menu update process. [Figure 17] 10 is a flowchart illustrating an example of a procedure for recommendation processing. [Figure 18] 10 is a flowchart illustrating an example of a procedure for evaluation processing. [Figure 19]10 is a flowchart illustrating an example of a procedure for a reference process. [Figure 20] FIG. 10 is an explanatory diagram showing an example of an output screen. [Figure 21] FIG. 2 is a block diagram illustrating an example of a functional configuration of a management server. DETAILED DESCRIPTION OF THE INVENTION

[0008] An embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is an explanatory diagram showing an example of the configuration of a menu management system. The menu management system 100 includes a management server 1, a display device 2, a camera 3, a vending machine 4, a user terminal 5, a trainer terminal 6, and a central server 7. The management server 1, the display device 2, the camera 3, the vending machine 4, the user terminal 5, the trainer terminal 6, and the central server 7 are connected to each other via a network N so that they can communicate with each other.

[0009] The management server 1 is configured from a server computer, a workstation, a PC (Personal Computer), etc. The management server 1 is installed in a fitness gym. Multiple management servers 1 may be installed in a fitness gym. A thin client may be installed in the fitness gym instead of the management server 1, and the processing performed by the management server 1 may be performed by a central server 7. The management server 1 may also be configured as a multi-computer consisting of multiple computers, a virtual machine virtually constructed by software, or a quantum computer. Furthermore, the functions of the management server 1 may be realized by a cloud service.

[0010] The display device 2 is configured with a liquid crystal display, an organic EL (Electro Luminescence) display, etc. The display device 2 presents a training menu to the user.

[0011] Camera 3 is equipped with an imaging element such as a CCD element or a CMOS element. Camera 3 captures images of users who are using the training equipment to train. It is desirable to install multiple cameras 3 in each fitness gym. When multiple cameras 3 are installed, each camera is assigned an ID that allows it to be identified. This ID is called a camera ID.

[0012] The vending machine 4 provides users with soft drinks, jelly drinks, supplements, etc. The vending machine 4 is capable of communicating with the management server 1, and can provide products designated by the management server 1 to users.

[0013] The user terminal 5 is composed of a smartphone, tablet computer, notebook PC, etc. The user terminal 5 is used by a user who uses the fitness gym. The user terminal 5 communicates with the management server 1 and displays the user's training history and training images of the user taken by the camera 3.

[0014] The trainer terminal 6 is composed of a smartphone, tablet computer, notebook PC, etc. The trainer terminal 6 is used by a trainer working at a fitness gym. The trainer uses the trainer terminal 6 to create or update a training menu for the user. The trainer also views the user's training images and gives the user training advice.

[0015] The central server 7 is composed of a server computer, a workstation, a PC, etc. The central server 7 generates and updates learning models. The central server 7 may be composed of a multi-computer consisting of multiple computers, a virtual machine virtually constructed by software, or a quantum computer. The functions performed by the central server 7 may be provided by a cloud service.

[0016] Instead of providing the display device 2 and the camera 3 separately, a device that integrates these may be used. For example, a device that includes a mirror that reflects the user during training, a display (display device 2), a camera 3, etc. and has the functions of the user terminal 5, such as a smart mirror, may be used. In addition to smart mirrors, a mirror with tablet functionality or digital signage with mirror functionality may also be used.

[0017] 2 is a block diagram showing an example of the hardware configuration of the management server 1. The management server 1 includes a control unit 11, a main memory unit 12, an auxiliary memory unit 13, a communication unit 15, and a reading unit 16. The control unit 11, the main memory unit 12, the auxiliary memory unit 13, the communication unit 15, and the reading unit 16 are connected by a bus B.

[0018] The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 11 reads and executes a control program 1P (information processing program, program product) stored in the auxiliary storage unit 13, thereby performing various information processing, control processing, etc. related to the management server 1 and realizing various functional units.

[0019] The main memory unit 12 is a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, etc. The main memory unit 12 mainly temporarily stores data required for the control unit 11 to execute arithmetic processing.

[0020] The auxiliary storage unit 13 is a hard disk or a solid state drive (SSD), etc., and stores the control program 1P and various databases (DBs) required for the control unit 11 to execute processing. The auxiliary storage unit 13 stores a user DB 131, a trainer DB 132, a menu DB 133, a training history DB 134, a video DB 135, a product DB 136, a recommendation history DB 137, and an analysis result DB 138. The auxiliary storage unit 13 also stores a motion analysis model 141, a motion diagnosis model 142, and an update model 143. The auxiliary storage unit 13 may be a storage device separate from and externally connected to the management server 1. The various DBs, etc. stored in the auxiliary storage unit 13 may be mirrored to the central server 7. The various DBs, etc. stored in the auxiliary storage unit 13 may also be stored in a database server or cloud storage different from the management server 1.

[0021] The communication unit 15 communicates with the user terminal 5, the trainer terminal 6, etc. via the network N. In addition, the control unit 11 may use the communication unit 15 to download the control program 1P from another computer via the network N, etc., and store it in the auxiliary storage unit 13.

[0022] The reading unit 16 reads the portable storage medium 1a including a CD (Compact Disc)-ROM and a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the auxiliary storage unit 13. The control unit 11 may also read the control program 1P from the semiconductor memory 1b.

[0023] 3 is a block diagram showing an example of the hardware configuration of a user terminal. The user terminal 5 is configured as a smartphone, tablet computer, notebook computer, etc. The user terminal 5 includes a control unit 51, a main memory unit 52, an auxiliary memory unit 53, a communication unit 54, an input unit 55, and a display unit 56. Each component is connected by a bus B.

[0024] The control unit 51 has one or more arithmetic processing units such as a CPU, an MPU, a GPU, etc. The control unit 51 provides various functions by reading and executing a control program 5P (program product) stored in the auxiliary storage unit 53.

[0025] The main memory unit 52 is an SRAM, a DRAM, a flash memory, etc. The main memory unit 52 mainly temporarily stores data required for the control unit 51 to execute arithmetic processing.

[0026] The auxiliary storage unit 53 is a hard disk or an SSD, etc., and stores various data necessary for the control unit 51 to execute processing. The auxiliary storage unit 53 may be an external storage device that is separate from the user terminal 5 and externally connected. The various DBs, etc. stored in the auxiliary storage unit 53 may be stored in a database server or cloud storage.

[0027] The communication unit 54 communicates with the management server 1 via the network N. In addition, the control unit 51 may use the communication unit 54 to download the control program 5P from another computer via the network N or the like and store it in the auxiliary storage unit 53.

[0028] The input unit 55 is a keyboard and a mouse. The display unit 56 includes a liquid crystal display panel or the like. The display unit 56 displays the training menu and the like output by the management server 1. The input unit 55 and the display unit 56 may be integrated to form a touch panel display. The user terminal 5 may display on an external display device.

[0029] Figure 4 is a block diagram showing an example of the hardware configuration of a trainer terminal. The trainer terminal 6 is configured as a smartphone, tablet computer, laptop computer, etc. The trainer terminal 6 includes a control unit 61, a main memory unit 62, an auxiliary memory unit 63, a communication unit 64, an input unit 65, and a display unit 66. Each component is connected via a bus B. The control unit 61, the main memory unit 62, the auxiliary memory unit 63, the communication unit 64, the input unit 65, and the display unit 66 have the same configuration as the control unit 51, the main memory unit 52, the auxiliary memory unit 53, the communication unit 54, the input unit 55, and the display unit 56 of the user terminal 5, respectively, and therefore their description will be omitted.

[0030] Next, the database (DB) used by the menu management system 100 will be described. Fig. 5 is an explanatory diagram showing an example of a user DB. The user DB 131 stores information about users who use the fitness gym. The user DB 131 includes a user ID column, a name column, an age column, a gender column, a height column, a weight column, and a body fat percentage column. The user ID column stores a user ID that uniquely identifies a user. The name column stores the user's name. The age column stores the user's age. The gender column stores the user's gender. The height column stores the user's height. The weight column stores the user's weight. The body fat percentage column stores the user's body fat percentage.

[0031] FIG. 6 is an explanatory diagram showing an example of a trainer DB. The trainer DB 132 stores information about trainers who provide training advice, etc. to users of a fitness gym. The trainer DB 132 includes a trainer ID column, a name column, an age column, and a gender column. The trainer ID column stores a trainer ID that uniquely identifies a trainer. The name column stores the name of the trainer. The age column stores the age of the trainer. The gender column stores the gender of the trainer.

[0032] FIG. 7 is an explanatory diagram showing an example of the menu DB. The menu DB 133 stores training menus created for each user. The menu DB 133 includes a menu ID column, a user ID column, a creation date and time column, a number of times performed column, a trainer ID column, a sequence number column, a type column, and a training amount column. The menu ID column stores a menu ID that uniquely identifies the training menu. The user ID column stores the user ID of the user for whom the menu is intended. The creation date and time column stores the date and time the menu was created. The number of times performed column stores the number of times the user has performed training according to the menu. The trainer ID column stores the trainer ID of the trainer involved in the menu. Being involved in the menu means, for example, creating or updating the menu, or providing comments or advice to the user regarding the menu. The sequence number column stores a sequence number indicating the order in which multiple training exercises included in the training menu are performed. The type column stores the type of each exercise. The training amount column stores the amount to be performed for each exercise, such as the duration, number of times performed, and the load applied.

[0033] FIG. 8 is an explanatory diagram showing an example of a training history DB. The training history DB 134 stores the history of training performed by a user. The training history DB 134 includes a history ID column, a training date column, a start time column, an end time column, a menu ID column, a user ID column, a video ID column, and an evaluation column. The history ID column stores a history ID that uniquely identifies the history. The implementation date column stores the date and time when the user performed the training. The start time column stores the time when the user started the training. The end time column stores the time when the user finished the training. The menu ID column stores the menu ID of the training menu performed by the user. The user ID column stores the user ID of the user who performed the training menu. The video ID column stores a video ID that identifies a video obtained by capturing the performance of the training menu with camera 3. The evaluation column stores an evaluation of the results of the training menu execution. The evaluation is, for example, a five-point evaluation. The evaluation value is A, B, C, D, or E, with A being the highest evaluation and E being the lowest evaluation.

[0034] FIG. 9 is an explanatory diagram showing an example of a video DB. The video DB 135 stores videos obtained by capturing footage of the training menu being performed by the camera 3. The video DB 135 includes a video ID string, a sequence number string, a camera ID string, and a video string. The video ID string stores a video ID that identifies a video. Here, one video ID is assigned to each execution of a training menu. The sequence number string stores the sequence number of the video. The sequence number corresponds to the sequence number of each training included in the training menu. The camera ID string stores the camera ID that identifies the camera that captured each training. If the training is captured with three cameras, the camera IDs for all three cameras are stored in the camera ID string. The video string stores the actual video data. Since the video data is binary data, it is shown as Binary in FIG. 9.

[0035] FIG. 10 is an explanatory diagram showing an example of a product DB. The product DB 136 stores information about products recommended for use by users who have completed their workouts. The products are available from the vending machine 4. The product DB 136 includes a product ID column, a product name column, and a fatigue level column. The product ID column stores a product ID that uniquely identifies the product. The product ID may be assigned independently by the menu management system, or a more versatile ID such as a Global Trade Item Number (GTIN) or a Japanese Article Number (JAN code). The product name column stores the name of the product. The fatigue level column stores the fatigue level of the user to whom the product should be recommended. The fatigue level may be, for example, on a 10-point scale, with 10 indicating the most fatigued and 1 indicating the least fatigued. Note that the fatigue level is a single value, but this is not limiting. The user's fatigue level may also be determined using multiple scales. For example, the system may be able to select recommended products based on multiple fatigue levels, such as fatigue levels determined from the user's facial expression, pulse wave, or heart rate.

[0036] FIG. 11 is an explanatory diagram showing an example of a recommendation history DB. The recommendation history DB 137 stores the history of products recommended to a user. The recommendation history DB 137 includes a recommendation ID column, a history ID column, an estimated fatigue level column, a product ID column, and a result column. The recommendation ID column stores a recommendation ID that uniquely identifies the recommendation history. The history ID column stores the history ID of the corresponding training history. The estimated fatigue level column stores the estimated fatigue level of the user when a product was recommended to the user. The product ID column stores the product ID of the recommended product. The result column stores whether or not the user purchased the recommended product.

[0037] FIG. 12 is an explanatory diagram showing an example of an analysis result DB. The analysis result DB 138 stores the results of analyzing product purchasing trends from the product recommendation history. In the example shown in FIG. 12, the analysis result DB 138 stores the user demographic that is likely to purchase each product. The analysis result DB 138 includes a product ID column, an age column, a gender column, and a fatigue level column. The product ID column stores the product ID. The age column stores the age of users who have purchased the most products in question. The gender column stores the gender of users who have purchased the most products in question. The fatigue level column stores the fatigue level after training of users who have purchased the most products in question.

[0038] Next, the learning model will be described. The motion analysis model 141 (first learning model) is a learning model that, when an image of a human motion is input, outputs an analysis result of the motion of the human. For example, the motion analysis model 141 is configured using a Pose Proposal Network. When an image of a human motion is input, the motion analysis model 141 outputs a skeletal model that represents the motion of the human. The Pose Proposal Network is a model that detects people using an object detection algorithm, detects joint points for each person, and estimates posture by connecting the joint points. A video of a user training, captured by camera 3, is input into the motion analysis model 141, and the video of the skeletal model obtained by outputting it can be used to obtain the range of motion and speed of motion of specific parts of the user during training as analysis results.

[0039] When the movement diagnostic model 142 (second learning model) receives the analysis results of human movement and reference data, it outputs the improvement level of the movement based on the reference data. If the reference data indicates an ideal movement, the improvement level of the movement diagnostic model 142 indicates the deviation from the ideal movement. If the reference data is past data of the same person, the improvement level is determined by the amount of change in the range of movement and movement speed compared with the past. This specification assumes the latter, and determines the improvement level based on the body movement when the user trains and the body movement when they trained previously. The movement diagnostic model 142 is configured, for example, with a CNN (Convolutional Neural Network).

[0040] Multiple videos of the same user training are prepared as training data for the motion diagnosis model 142. A trainer or the like compares the videos in chronological order and selects two videos in which the user's improvement can be confirmed, and the degree of improvement is used as a label. The two videos are input into the motion analysis model 141 to obtain two videos of a skeletal model, and the labels indicating the degree of improvement become the training data. When the two videos constituting the training data are input into the motion diagnosis model 142, parameters used for calculation processing in the intermediate layer are optimized so that the degree of improvement output by the motion diagnosis model 142 approaches the correct label value. Examples of such parameters include weights (coupling coefficients) between neurons and coefficients of activation functions used in each neuron. The parameter optimization method is not particularly limited, and various parameters are optimized using, for example, backpropagation. The motion diagnosis model 142 is generated by learning using multiple training data.

[0041] A video of the user training and a video of an expert such as a trainer training may be prepared as training data for the motion diagnosis model 142. In this case, the trainer or the like compares the user's level of improvement with the movements of the expert as a standard and labels the results.

[0042] The update model 143 outputs the updated content of the training menu when a training menu and a level of improvement are input. For example, when "leg press, 40 kg, 10 repetitions, 2 sets" and "level of improvement 5" are input, it outputs "leg 45 kg, 12 repetitions, 2 sets." The menu update model is configured, for example, by CNN.

[0043] As training data for the update model 143, a training menu to be proposed next is prepared from the training menu and the improvement level. When the training menu and the improvement level constituting the training data are input to the update model 143, parameters used in the calculation process in the intermediate layer are optimized so that the training menu output by the update model 143 approaches the correct training menu to be proposed next. Examples of the parameters include weights (coupling coefficients) between neurons and coefficients of activation functions used in each neuron. There are no particular limitations on the method for optimizing the parameters, and for example, the error backpropagation method is used to optimize various parameters. The update model 143 is generated by learning using multiple training data. Note that a table defining the correspondence between the training menu, the improvement level, and the training menu to be proposed next may be created, and the table may be used instead of the update model 143 to update the training menu.

[0044] Next, the information processing performed by the menu management system 100 will be described. FIG. 13 is a flowchart showing an example of the procedure for update processing. The update processing is a process of collecting the user's training status and updating the user's training menu after training. The control unit 11 of the management server 1 acquires the user ID (step S1). The user ID is acquired, for example, by having a card reader (ID acquisition device) read a membership card. Alternatively, the user may enter the user ID via a keyboard or touch panel. The user may also be identified by facial recognition. The control unit 11 uses the user ID as a search key to acquire a training menu for the user from the menu DB 133 and displays it on the display device 2 (step S2). The control unit 11 performs training processing (step S3). The training processing will be described later. The control unit 11 performs diagnostic processing (step S4). The diagnostic processing will be described later. The control unit 11 updates the training menu based on the results of the diagnostic processing (step S5). Control unit 11 inputs the training menu performed by the user and the level of improvement of each training obtained in the diagnostic processing into update model 143, and updates the training menu based on the content obtained as output. Control unit 11 then ends the processing.

[0045] FIG. 14 is a flowchart showing an example of the procedure for the training process. The training process corresponds to step S3 in FIG. 13. The control unit 11 of the management server 1 starts shooting with the camera 3 (step S11). The camera 3 used for shooting is selected appropriately according to the content of the training. For example, if the training is chest press, shooting is performed with the camera 3 installed around the chest press machine. The control unit 11 determines whether the training has ended (step S12). If the control unit 11 determines that the training has not ended (NO in step S12), it repeats step S12. If the control unit 11 determines that the training has ended (YES in step S12), it stops shooting (step S13). The control unit 11 stores the shot video in the video DB 135 (step S14). The control unit 11 determines whether the training menu has ended (step S15). If the control unit 11 determines that the training menu has not ended (NO in step S15), it resumes shooting (step S16). The camera 3 used for shooting is selected to match the content of the next training. The control unit 11 returns the process to step S12. If the control unit 11 determines that the training menu has ended (YES in step S15), it ends the training process and returns the process to the caller.

[0046] FIG. 15 is a flowchart showing an example of the diagnostic process. The diagnostic process corresponds to step S4 in FIG. 13. The control unit 11 of the management server 1 selects one training session included in the training menu and acquires a video of the user performing the selected training session from the video DB 135 (step S21). The control unit 11 performs a motion analysis of the user from the acquired video (step S22). The control unit 11 inputs the video session into the motion analysis model 141 and obtains the range of motion and the speed of motion of a specific part of the user during training as analysis results from the video of the skeletal model output by the motion analysis model 141. The control unit 11 performs a motion diagnosis (step S23). The control unit 11 stores the results of the motion diagnosis in the auxiliary storage unit 13 or the like (step S24). The control unit 11 inputs the analysis results of the user's previous training session and the analysis results of this training session into the motion diagnosis model 142 and acquires the improvement level output by the motion diagnosis model 142. The control unit 11 stores the improvement level in a temporary storage area such as the auxiliary storage unit 13. The improvement level may be, for example, on a five-point scale, with 3 indicating the current level, 4 indicating slight improvement, 5 indicating significant improvement, 2 indicating slight regression, and 1 indicating significant regression. The control unit 11 determines whether or not there is any unprocessed training (step S25). If the control unit 11 determines that there is any unprocessed training (YES in step S25), it returns the process to step S21. If the control unit 11 determines that there is no unprocessed training (NO in step S25), it evaluates the training performed by the user this time based on the improvement level of each training stored in the temporary storage area, and stores the evaluation value in the evaluation column of the training history DB 134 (step S26). The evaluation value may be the average, median, mode, or total value of the improvement level of each training. The evaluation value may also be a matrix listing the improvement level of each training. The control unit 11 notifies the user of the evaluation result (step S27). The control unit 11 displays the evaluation result on the display device 2. Alternatively, the control unit 11 transmits the evaluation result to the user terminal 5. The control unit 11 ends the diagnostic process and returns the process to the caller.

[0047] 20 is an explanatory diagram showing an example of an output screen. After the training is completed, the output screen is sent to the user terminal 5 or displayed on the display device 2. The output screen displays the training evaluation and the next training menu based on the evaluation.

[0048] In this embodiment, the training menu can be updated based on the results of the training menu execution.

[0049] The above are the basic functions of the menu management system 100. Next, additional functions of the menu management system 100 will be described.

[0050] (Trainer update function) If the user does not want the training menu updated by the menu management system 100 or does not agree with the updated content of the training menu by the menu management system 100, the trainer can update the training menu. FIG. 16 is a flowchart showing an example of the menu update process. If the user does not want the training menu updated by the menu management system 100, after step S3 in FIG. 13, steps S4 and S5 are replaced by the menu update process. If the user does not agree with the updated content of the training menu by the menu management system 100, after the update process in FIG. 13, the menu update process is executed in response to a request from the user who has viewed the updated training menu. The control unit 11 of the management server 1 sends a training evaluation request to the trainer terminal 6 (step S31). The control unit 61 of the trainer terminal 6 receives the evaluation request and displays it on the display unit 66 (step S32). The trainer instructs the trainer terminal 6 to play the user's training video to perform the evaluation. Upon receiving the instruction, the control unit 61 sends a video request to the management server 1 (step S33). The control unit 11 of the management server 1 receives the video request (step S34). The control unit 11 transmits the video to the trainer terminal 6 (step S35). The control unit 61 of the trainer terminal 6 receives and plays the video (step S36). Note that the video may be played via streaming playback. After watching the video, the trainer inputs the training score and feedback (comments to the user, etc.) into the trainer terminal 6. The control unit 61 transmits the input score and feedback to the management server 1 (step S37). The control unit 11 of the management server 1 receives the score and feedback and transmits the received score and feedback to the user terminal 5 (step S38). The control unit 51 of the user terminal 5 receives the score and feedback and displays them on the display unit 56 (step S39). The user refers to the score and feedback and then inputs a request to the trainer. The request is a request to the trainer to update the training menu. The request may include questions for the trainer or requests for updates. The control unit 51 transmits the input request to the management server 1 (step S40).The control unit 11 of the management server 1 receives the request and transmits it to the trainer terminal 6 (step S41). The control unit 11 also transmits a training menu along with the request. The control unit 61 of the trainer terminal 6 receives the request and the training menu and displays the request and the training menu to be updated on the display unit 66 (step S42). The trainer refers to the request and updates the training menu. The control unit 61 accepts the updated content (step S43). The control unit 61 transmits the accepted updated content to the management server 1 (step S44). The control unit 11 of the management server 1 accepts the updated content, reflects it in the menu DB 133 (step S45), and ends the process.

[0051] (Product recommendation function) After completing training, users often replenish their fluids and nutrients at a fitness gym. The product recommendation function recommends products that can help users replenish their fluids and nutrients after completing training. FIG. 17 is a flowchart showing an example of the recommendation process. The recommendation process is executed after the user finishes training, for example, after step S3 in FIG. 13 is executed. In this case, the control unit 11 of the management server 1 executes the update process and the recommendation process in parallel. The control unit 11 outputs a message addressed to the user to the display device 2 (step S61). The display device 2 may be provided with an audio output unit including a speaker, and the message may be output as audio. The message encourages the user to take action in order to acquire a facial image of the user, which is necessary for assessing the user's fatigue level. For example, a camera may be installed in a position where it can capture a facial image when the user faces the display device 2. The control unit 11 captures a facial image of the user using the camera (step S62). The control unit 11 analyzes the facial expression of the captured facial image (step S63). The control unit 11 diagnoses the user's level of fatigue based on the results of the facial expression analysis (step S64). Since facial expression analysis and fatigue level diagnosis are well-known techniques, their explanation will be omitted. The control unit 11 searches the product DB 136 based on the fatigue level and selects a product to recommend to the user (step S65). The control unit 11 outputs the product recommended to the user (step S66). The control unit 11 displays the recommended product on the display device 2. The control unit 11 may also display the recommended product on a display provided in the vending machine 4. The control unit 11 communicates with the vending machine 4 and determines whether the user purchased the recommended product (step S67). The control unit 11 stores a history including the determination result in the recommendation history DB 137 (step S68) and ends the process. The recommended product may be provided to the user free of charge as a sample. In this case, the control unit 11 instructs the vending machine 4 to dispense the recommended product to a receiving slot without payment if the user performs a predetermined operation.

[0052] (Analysis function) The product recommendation function analyzes the recommendation history accumulated in the recommendation history DB 137 to analyze what kind of users selected each product. For example, the control unit 11 analyzes the age, gender, and post-training fatigue level of the users who most frequently selected each product. The control unit 11 stores the analysis results in the analysis result DB 138.

[0053] (Group lesson diagnostic function) In the above-described embodiment, it is assumed that training is performed using a machine and that only one user appears in the captured image. The following describes the diagnostic function for group lessons, such as aerobics or jazz dance. Figure 18 is a flowchart showing an example of the evaluation process. This evaluation process is performed after the group lesson ends. The control unit 11 of the management server 1 acquires a video of the group lesson (step S81). The control unit 11 performs a motion analysis of each person appearing in the video (step S82). For example, the Pose Proposal Network described above is used. When a video containing multiple people is input, the Pose Proposal Network detects the joint points of each person, connects the joint points to estimate the posture of each person, and outputs a skeletal model. The control unit 11 identifies the instructor from among the skeletal models of multiple people (step S83). The instructor is identified from among the people appearing in the video based on the position and angle of view of the camera 3 that captured the input video, the instructor's standing position, etc. The control unit 11 selects a user to be evaluated (step S84). The control unit 11 evaluates the user (step S85). The control unit 11 uses the skeletal model to compare the movements of the instructor with those of the target user, and determines the degree to which the target user is able to follow the movements of the instructor. The degree serves as the evaluation value. The degree of following may be determined for each frame, and the degree for all frames may be averaged to determine the evaluation value. The control unit 11 stores the evaluation value (step S86). The control unit 11 determines whether there are any unprocessed users (step S87). If the control unit 11 determines that there are any unprocessed users (YES in step S87), it returns the process to step S84. If the control unit 11 determines that there are no unprocessed users (NO in step S87), it notifies the user that the evaluation is complete (step S88). For example, the display device 2 displays a message indicating that the evaluation is complete. Alternatively, the control unit 11 may notify the user terminal 5. It is assumed that the user has read their membership card using a card reader or the like before the group lesson, and the management server 1 has memorized the users who participated in the group lesson. The control unit 11 then ends the process.

[0054] FIG. 19 is a flowchart showing an example of the procedure for the reference process. The reference process is a process performed when a user references an evaluation after completing the evaluation. The control unit 51 of the user terminal 5 makes a login request to the management server 1 (step S101). The control unit 11 of the management server 1 receives the login request (step S102). The control unit 11 performs login authentication and transmits the authentication result to the user terminal 5 (step S103). The control unit 51 of the user terminal 5 receives the authentication result (step S104). The control unit 51 determines whether the authentication result indicates that login is permitted (step S105). If the control unit 51 determines that the authentication result indicates that login is not permitted (NO in step S105), the process ends. If the control unit 51 determines that the authentication result indicates that login is permitted (YES in step S105), the control unit 51 transmits a request to display the evaluation to the management server 1 (step S106). The control unit 11 of the management server 1 receives the display request (step S107). The control unit 11 acquires the video of the lesson (step S108). The control unit 11 processes the other users appearing in the video into avatar displays (step S109). At this time, a numerical value indicating the degree to which the user is able to follow the instructor may be added to each frame. The control unit 11 transmits the processed video and the user's evaluation value for the lesson to the user terminal 5 (step S110). The control unit 51 of the user terminal 5 receives the video and the evaluation value (step S111). The control unit 51 displays the video and the evaluation value (step S112) and ends the process.

[0055] In the above-described embodiment, a smart mirror may be used as the display device 2. A smart mirror has a half mirror positioned opposite the display surface of the display, which reflects the user's image and allows the user to view the content displayed on the display.

[0056] 21 is a block diagram showing an example of the functional configuration of the management server 1. The management server 1 includes an acquisition unit 11a, an analysis unit 11b, an evaluation unit 11c, and an update unit 11d. The control unit 11 executes a control program 1P, causing the management server 1 to operate as follows.

[0057] The acquisition unit 11a acquires training images of the user. The analysis unit 11b analyzes the user's movements from the training images. The evaluation unit 11c evaluates the user's improvement based on the analysis results. The update unit 11d updates the user's training menu based on the evaluation results.

[0058] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0059] 100 Menu Management System 1 Management Server 11 Control section 11a Acquisition part 11b Analysis section 11c Evaluation Section 11d update part 12 Main memory 13 Auxiliary storage 131 User DB 132 Trainer DB 133 Menu DB 134 Training History DB 135 Video DB 136 Product DB 137 Recommended History DB 138 Analysis result DB 141 Motion Analysis Model 142 Operational Diagnostic Model 143 Updated Model 15 Communications Department 16 Reading unit 1P control program 1a Portable storage media 1b semiconductor memory 2 Display device 3 Camera 4. Vending Machines 5. User terminal 6 Trainer terminal 7 Central Server B Bus N Network

Claims

1. an acquisition unit for acquiring training images of a user; an analysis unit that analyzes a user's movement from the training image; an evaluation unit that evaluates the degree of improvement based on the analysis results; an update unit that updates a training menu for the user based on the evaluation result; a detection unit that detects the end of the user's training from the training image; a voice output unit that outputs a voice message urging the user to take action to acquire a face image of the user when the end of training is detected; Equipped with the acquisition unit acquires a face image of the user after the message is output as voice; a diagnosis unit that diagnoses a fatigue level of the user based on the face image acquired by the acquisition unit; a selection unit that selects recommended products based on the fatigue level; An output section that outputs the selected product; 2. An information processing device further comprising:

2. a transmission unit that transmits the training image to a trainer terminal; a first receiving unit that receives a training score and feedback for the user from the trainer terminal; a second receiving unit that receives a request for the feedback from a user terminal; Equipped with The update unit updates the menu based on the request.

2. The information processing apparatus according to claim 1, wherein:

3. accumulating a history of whether or not the user purchased the product output by the output unit; Analyzing the user's preferences based on the accumulated history 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. Acquire the user ID of the user from an ID acquisition device; The update unit updates the menu stored in association with the user ID.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

5. a display unit that displays the menu; a half mirror facing the display surface of the display unit; 5. The information processing apparatus according to claim 1, further comprising:

6. The analysis unit inputs a training image to a first learning model that outputs an analysis result of a human motion when the training image is input, and acquires the analysis result.

6. The information processing device according to claim 1, wherein the information processing device is a computer.

7. The evaluation unit inputs the analysis result of the user to a second learning model that outputs a degree of improvement in the motion when the analysis result of the motion of a human is input, and performs evaluation based on the degree of improvement output by the second learning model.

7. The information processing device according to claim 1, wherein the information processing device is a computer.

8. When a plurality of users appear in the training image, the degree of improvement of each user is evaluated, and users other than a predetermined user in the training image are converted into avatars, and the converted images and the degree of improvement are displayed so that the users can identify them.

8. The information processing device according to claim 1, wherein the information processing device is a computer.

9. Obtaining training images of the user; Analyzing a user's movement from the training images; Evaluate your progress based on the analysis results, updating the training menu for the user based on the evaluation results; Detecting the end of the user's training from the training image; When the end of the training is detected, a message is outputted by voice to prompt the user to take action to acquire a face image of the user; After the message is outputted, a facial image of the user is acquired. Diagnosing the user's fatigue level based on the acquired face image; Selecting recommended products based on the fatigue level, Output selected products An information processing program that causes a computer to perform processing.

10. The computer Obtaining training images of the user; Analyzing a user's movement from the training images; Evaluate your progress based on the analysis results, updating the training menu for the user based on the evaluation results; Detecting the end of the user's training from the training image; When the end of the training is detected, a message is outputted by voice to prompt the user to take action to acquire a face image of the user; After the message is outputted, a facial image of the user is acquired. Diagnosing the user's fatigue level based on the acquired face image; Selecting recommended products based on the fatigue level, Output selected products 2. An information processing method comprising:

Images