Evaluation device, evaluation program, and evaluation method

Abstract

To provide an evaluation device, an evaluation program, and an evaluation method for predicting a future body shape or the like.SOLUTION: In a support system in which a server device 20 is connected to a user terminal 10 and an imaging device 30 via a communication network 40, the server device, which is a support device that supports a user's fitness, includes a future posture generation unit that is a posture image generation unit that generates a future posture image of the user performing a predetermined movement based on a user image of the user and information about the user's physical constitution at a future time point predicted based on fitness activity schedule information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an evaluation device, an evaluation program, and an evaluation method. [Background technology]

[0002] Conventionally, techniques for predicting changes in body shape at a certain time in the future have been disclosed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-147566 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technique of Patent Document 1 cannot predict what changes will occur as a result of actual training or the like.

[0005] The present invention has been made in view of the above background, and aims to provide a technique for predicting future body shape and the like. [Means for solving the problem]

[0006] According to the present disclosure, there is provided a support device that supports a user's fitness, the support device including a posture image generation unit that generates a posture image of the user performing a predetermined movement based on a user image of the user and information about the user's physical constitution at a future point in time predicted based on information about planned fitness activities. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to predict future body shape, etc.

[0008] Other problems and solutions disclosed in this application will be made clear in the section on preferred embodiments of the invention and the drawings. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an evaluation system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of a computer that realizes the user terminal 10 according to the embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the software configuration of the user terminal 10 according to the embodiment. [Figure 4] FIG. 10 is a diagram showing an example of the configuration of physical information according to the embodiment. [Figure 5] FIG. 10 is a diagram showing an example of the configuration of evaluation request information according to the embodiment. [Figure 6] FIG. 10 is a diagram showing an example of the configuration of evaluation information according to the embodiment. [Figure 7] FIG. 10 is a diagram showing an example of the configuration of fitness activity request information according to the embodiment. [Figure 8] FIG. 10 is a diagram showing an example of the configuration of fitness activity information according to the embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device 20 according to the embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of the software configuration of a server device 20 according to the embodiment. [Figure 11] FIG. 10 is a diagram showing an example of the configuration of image information according to the embodiment. [Figure 12] FIG. 10 is a diagram showing an example of the configuration of reference information according to the embodiment. [Figure 13] FIG. 10 is a diagram showing an example of the configuration of evaluation condition information according to the embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of the configuration of an improvement condition according to the embodiment. [Figure 15] FIG. 10 is a diagram illustrating an example of processing of the evaluation system according to the embodiment. [Figure 16] FIG. 10 is a diagram showing an example of a screen generated by the evaluation system according to the embodiment. [Figure 17]FIG. 10 is another diagram showing an example of a screen generated by the evaluation system according to the embodiment. [Figure 18] FIG. 10 is another diagram showing an example of a screen generated by the evaluation system according to the embodiment. [Figure 19] FIG. 10 is another diagram showing an example of a screen generated by the evaluation system according to the embodiment. [Figure 20] FIG. 10 is another diagram showing an example of a screen generated by the evaluation system according to the embodiment. [Figure 21] FIG. 10 is another diagram showing an example of a screen generated by the evaluation system according to the embodiment. [Figure 22] 10A and 10B are diagrams illustrating an example of the positions of the imaging device 30 and mirrors according to the embodiment. [Figure 23] 10 is another diagram showing an example of the positions of the imaging device 30 and the mirror according to the embodiment. FIG. [Figure 24] 10 is another diagram showing an example of the positions of the imaging device 30 and the mirror according to the embodiment. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] The contents of the embodiment of the present invention will be listed and described below. One embodiment of the present invention has the following configuration. [Item 1] An assist device for assisting a user's fitness, a posture image generating unit that generates a posture image of the user performing a predetermined movement based on a user image of the user and physical constitution information of the user at a future time point predicted based on fitness activity implementation schedule information; An assistance device comprising: [Item 2] The posture image generation unit inputs at least the user image and text information generated based on the fitness activity implementation schedule information into a generation AI, and generates the posture image based on output information. Item 1. The assistive device according to item 1. [Item 3] the posture image generation unit corrects the posture image based on fitness activity implementation information. 3. The assistive device according to item 1 or 2. [Item 4] the posture image generation unit inputs text information generated based on at least the user image, the fitness activity implementation schedule information, and the training implementation information into a generation AI, and generates the posture image based on output information; Item 2. The assistive device according to item 2. [Item 5] the fitness activity implementation information includes information about the content of the fitness activity acquired by analyzing the user image; 5. The assistive device according to item 3 or 4. [Item 6] the posture image generation unit corrects the posture image based on the user images taken at a plurality of points in time and training effect information determined based on fitness activity implementation information between the points in time. 5. The assistive device according to item 3 or 4. [Item 7] An assistance program for assisting a user's fitness, The processor a posture image generating step of generating a posture image of the user performing a predetermined movement based on the user image of the user and physical constitution information of the user at a future time point predicted based on fitness activity implementation schedule information; A support program to help you achieve this. [Item 8] A method for supporting a user's fitness, comprising: The processor: a posture image generating step of generating a posture image of the user performing a predetermined movement based on the user image of the user and physical constitution information of the user at a future time point predicted based on fitness activity implementation schedule information; How to support this.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The support system analyzes the user's movements and predicts, visualizes, and presents the user's future physical condition to improve skills, and to increase motivation to undergo training, rehabilitation, and treatments that will produce the correct results. At this time, the support system can use actual images of the user and reflect the user's actual fitness activities in the future physical condition.

[0013] == Overview == FIG. 1 is a diagram showing the overall configuration of the support system. As shown in FIG. 1, the support system includes a server device 20, a user terminal 10, and an imaging terminal 30. The server device 20 is connected to the user terminal 10 and the imaging terminal 30 via a network 40. Although only one user terminal 10 is shown, it goes without saying that there may be more than one user terminal 10. Furthermore, the specific devices of the user terminal 10 and the imaging terminal 30 are not limited to mobile terminals and personal computers, but may also be, for example, smartphones, tablet computers, wearable terminals, or other electronic devices.

[0014] ==Server Device 20== The server device 20 is a computer that evaluates physical exercise. The server device 20 is, for example, a workstation, a personal computer, or a virtual computer logically realized by cloud computing. The server device 20 receives video images captured by the user terminal 10, analyzes the received video images, and evaluates the physical exercise. The server device 20 also makes suggestions related to fitness activities for physical exercise. Details of the evaluation of physical exercise and the suggestions of fitness activities will be described later.

[0015] ==User terminal 10== The user terminal 10 is a computer operated by a user (user) performing physical exercise or their supporter. The user terminal 10 is, for example, a smartphone, a tablet computer, or a personal computer. The user terminal 10 is equipped with an imaging device such as a camera, which can capture images of the user's body during exercise. In this embodiment, video images of the user's body during exercise are transmitted from the user terminal 10 to the server device 20. The user can access the server device 10, for example, using an application or web browser running on the user terminal 10.

[0016] ==Imaging device 30== The imaging device 30 is a computer or a camera with communication capabilities that is installed or attached to the location where the user performs physical exercise and captures images of the user. The camera may be optical, such as a video camera, stereo camera, or depth camera, or a camera with depth sensing or motion tracking capabilities using an infrared light source and an infrared camera. For normal images, which will be described in detail below, a LiDAR (Light Detection and Ranging) may be used as the imaging device 30. The imaging device 30 may be, for example, a smartphone, tablet computer, or personal computer. The imaging device 30 has an imaging function such as a camera, which can capture images of the user's body during exercise. The imaging device 30 does not have to be installed to capture the user's physical exercise; for example, it may be installed for another purpose, such as a security camera installed at the location where the user performs physical exercise. In this embodiment, the images capturing the user's physical exercise may be captured by either the imaging device 30 or the user terminal 10.

[0017] The configuration of the user terminal 10 will be described below.

[0018] 2 is a diagram showing an example of the hardware configuration of the user terminal 10. The user terminal 10 includes a CPU 101, a memory 102, a storage device 103, a communication interface 104, an input device 105, and an output device 106. The storage device 103 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 104 is an interface for connecting to the communication network 40, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, or a USB (Universal Serial Bus) connector or an RS232C connector for serial communication. The input device 105 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, or the like for inputting data. The output device 106 is, for example, a display, a printer, a speaker, or the like for outputting data.

[0019] 3 is a diagram showing an example of the software configuration of user terminal 10. User terminal 10 includes functional units, namely, imaging unit 111, rating request sending unit 112, rating information receiving unit 113, rating display unit 114, checkpoint display unit 115, fitness activity request sending unit 116, fitness activity information receiving unit 117, and fitness activity information display unit 118, as well as storage units, namely, user information storage unit 130, image storage unit 131, rating information storage unit 132, and fitness activity storage unit 133.

[0020] Each of the above functional units is realized by the CPU 101 of the user terminal 10 reading out a program stored in the storage device 103 into the memory 102 and executing it, and each of the above storage units is realized as part of the storage area provided by the memory 102 and storage device 103 of the user terminal 10.

[0021] The imaging unit 111 captures images, including video, of the user performing physical exercise. The imaging unit 111 can acquire video of the physical exercise using tools by controlling the camera included in the user device 106. The user or the user's supporter simply installs the user terminal 10 on a flat surface or a wall, points the optical axis of the camera toward the location where the user is performing exercise, and issues an instruction to start video recording. In response, the imaging unit 111 operates the camera to acquire the video. The imaging unit 111 stores the acquired video in the image storage unit 131.

[0022] In this embodiment, the physical exercise performed by the user may also include exercise using tools.

[0023] The imaging unit 111 acquires at least either a normal image obtained by directly capturing the user or a mirror image obtained by capturing the user reflected in a mirror. Images acquired by one imaging device 30 may include at least either a normal image or a mirror image. The imaging unit 111 distinguishes between normal images and mirror images from the images acquired by the imaging device 30 and acquires them.

[0024] An example of the arrangement of the imaging device 30 will be described below. Imaging devices 401 to 406 are examples of imaging device 30. For example, in FIG. 22, imaging device 401 may be arranged in front of a user making a physical movement, in which case imaging unit 111 can acquire a normal image of the user. Alternatively, like imaging device 402, it may be arranged shifted in the x-axis direction from in front of the user, in which case imaging unit 111 can acquire a normal image capturing the user from the front and a mirror image capturing the user's back.

[0025] Furthermore, as shown in the example of imaging device 404 in Figure 23, it may be positioned at a position shifted in the z-axis direction from the position of imaging device 403 positioned in front of the user, in which case imaging unit 111 can acquire a normal image capturing the user from the front and a mirror image capturing the user from behind.

[0026] It goes without saying that since the user is performing an action, the normal image does not always capture the front of the user and the mirror image does not always capture the back of the user. Also, it goes without saying that the image capture device 30 may be positioned at a location that is shifted in the x-axis and z-axis directions from the position of the image capture device 30 positioned in front of the user, as long as the user's action can be captured in the angle of view, and the position in the y-axis direction may be any position.

[0027] 24, for example, when an obstacle (b001) such as a wall or a pillar is present and the imaging device 405 cannot capture a normal image, the imaging unit 111 may acquire a mirror image reflected in a mirror (m003) installed in the room. In this case, the normal image may be captured by the imaging device 406, which is placed in a position where it can capture the normal image.

[0028] There may be multiple mirrors, and the imaging unit 111 may acquire multiple mirror images reflected in each mirror from an image acquired by one imaging device 30. Also, an image of the user reflected in the mirror may be acquired. Furthermore, the mirror may have a curved surface, and the imaging unit 111 may acquire mirror images of the user from multiple points on the curved surface.

[0029] There may be a plurality of imaging devices 30, and the imaging unit 111 may acquire, as mirror images, a plurality of images of one mirror captured from different directions from the respective imaging devices 30. Furthermore, the imaging unit 111 may acquire, as mirror images, images of a plurality of mirrors captured from the respective imaging devices 30.

[0030] There may be a plurality of imaging devices 30, and the imaging unit 111 may acquire, as mirror images, a plurality of images of one mirror captured from different directions from the respective imaging devices 30. Furthermore, the imaging unit 111 may acquire, as mirror images, images of a plurality of mirrors captured from the respective imaging devices 30.

[0031] The imaging device 30 may be a stereo camera, and the images captured by the stereo camera can acquire depth information whether they are normal images or mirror images. Alternatively, the imaging device 30 may be a regular camera. In this case, multiple cameras capture images of the user from different angles, and if the images captured by each camera contain a common part of the user's body, depth information of that part can be acquired. Similarly, even if a single imaging device 30 is used, normal images and mirror images can be acquired, and if the images contain a common part of the user's body, depth information of that part can be acquired.

[0032] The imaging unit 111 may correct the mirror image captured by the imaging device 30. In this case, the mirror may be equipped with a mechanism for determining the positional relationship between the camera and the mirror. For example, a mark may be provided at a predetermined position on the mirror, and the imaging unit 111 may acquire a mirror image including the mark. Based on information about the position of the mark, the imaging unit 111 may process the mirror image captured by the imaging device 30 from an oblique angle into a front image. The mark may be, for example, a QR code (registered trademark), an AR marker, an ArUco marker, or the like, but is not limited to these. The imaging unit 111 may correct distortions and tilts in the mirror image, for example, to make it appear as if it were an image captured from the front, back, side, or the like of the user.

[0033] When the positional relationship between the imaging device 30 and multiple mirrors results in a two-way mirror state, the imaging unit 111 may edit and acquire the part of the user that appears foremost in the mirror image captured by the imaging device 30, or the image analysis unit 212, which will be described in detail below, may refer to this when identifying body parts, etc. Note that the imaging unit 111 may determine the user that appears foremost in the mirror image captured by the imaging device 30 by recognizing the user that appears largest among the images of multiple users in the image, or may determine the user that appears foremost based on depth information, but is not limited to these methods.

[0034] The image storage unit 131 stores images captured, acquired, corrected, etc. by the imaging unit 111. In this embodiment, the images are moving images, but are not limited to this. The image storage unit 131 can store moving images as, for example, files.

[0035] The user information storage unit 130 stores information (hereinafter referred to as user information) related to the user's body, physical abilities, factors that affect the effectiveness of fitness activities, and the like. FIG. 4 is a diagram showing an example of the configuration of user information stored in the user information storage unit 130. As shown in the figure, the user information may include height, weight, sex, dominant hand, arm length, leg length, hand size, finger length, grip strength, muscle strength, muscle mass, body fat percentage, chest circumference, abdominal circumference, waist circumference, arm and leg thickness, flexibility, shoulder strength, various skill levels, genome, epigenome, genetic polymorphism, intestinal flora, diet, physical condition and disease state, range of joint motion, medical history, level of need for assistance, level of need for care, dementia level, ADL, IADL, purpose of training, goal of training, interests, and beauty-related information (skin condition, oral condition, etc.), which may be referred to as constitution in this embodiment.

[0036] The evaluation request sending unit 112 sends to the server device 20 a request to evaluate the physical exercise based on the image captured by the imaging unit 111 (hereinafter referred to as an evaluation request).

[0037] FIG. 5 is a diagram showing an example of the configuration of a rating request sent by the rating request sending unit 112 to the server device 20. As shown in the figure, the rating request includes a user ID, a mode, physical information, and image data. The user ID is information that identifies the user. The mode is information that indicates the exercise performed by the user. The mode can be, for example, "bench press," "tennis serve," or "walking rehabilitation." It should be noted that the mode is selected from a predetermined selection. The user information is user information stored in the user information storage unit 130. The image data is data of a moving image acquired by the imaging unit 111.

[0038] The evaluation information receiving unit 113 receives information relating to the evaluation of the physical exercise (hereinafter referred to as evaluation information) returned from the server device 20 in response to the evaluation request. The evaluation information receiving unit 113 registers the received evaluation information in the evaluation information storage unit 132.

[0039] 6 is a diagram showing an example of the configuration of evaluation information received by the evaluation information receiving unit 113 from the server device 20. As shown in the figure, the evaluation information includes a mode, a user ID, tool position information, body part position information, posture information, movement information, and checkpoint information.

[0040] The user ID and mode are the user ID and mode included in the evaluation request. The captured image shows the user's body performing the exercise indicated by the mode.

[0041] The tool position information indicates the position of each part of the tool in the image (for example, the entire baseball bat, both ends, the grip, the point where the ball is hit, the center of gravity, or any other part; the entire barbell, the shaft, the plates (weight parts), the gripped part, the center of the shaft, the center of gravity, or any other part; including the entire tool). The tool position information includes each part of the tool and its position, associated with a point on the timeline of the video. Based on the tool position information, the movement of the tool and its relationship with body parts can be displayed. That is, for example, a figure indicating a part (for example, a circle) can be superimposed on the image at the position indicated by the tool position information. Note that the positions of multiple parts may be included for one point in time. Furthermore, tool position information does not need to be included for a part connecting two parts (for example, the midpoint between the parts where a barbell is gripped in the right and left hands). In this case, the part connecting these two parts can be represented by connecting a pair of marks (for example, circles) indicating two specified parts. Tool part position information may be included for each frame constituting the video, for each key frame (including frames related to checkpoints, which will be explained later), for every arbitrary number of frames, or for a random point in time. If My Frame does not include position information, the figure can be displayed based on the position information of the most recent past point in time.

[0042] The body position information indicates the position of each body part (e.g., head, shoulders, elbows, hips, knees, ankles, etc.) in the image. The body position information includes the body part and the position of the part, corresponding to a time point on the time axis of the video. The skeletal state (bones) of the body can be displayed based on the body position information. That is, for example, a figure indicating the body part (e.g., a circle) can be superimposed on the image at the position indicated by the body position information. Note that the positions of multiple body parts can be included for one time point. Note that position information does not need to be included for a part connecting two parts (e.g., the forearm connecting the wrist and elbow, or the thigh connecting the hip and knee). In this case, the part connecting these two parts can be expressed by connecting a pair of marks (e.g., a circle) indicating two specific parts with a line. Position information may be included for each frame constituting the video, for each key frame (including frames related to checkpoints, which will be described later), for every arbitrary number of frames, or for random time points. If My Frame does not include position information, bones can be displayed based on position information from the most recent past time point.

[0043] Tool orientation information is information related to the orientation of the tool used by the user and the direction in which parts of the tool are facing. Tool orientation information includes, in association with a point on the video timeline, the part of the tool to be evaluated, a tool movement value, an evaluation rank, and an evaluation comment. The tool orientation value is a value that represents the orientation of the tool. Examples of the tool orientation value include the distance from the ground to a certain part of the tool, the distance between two parts of the tool, the angle of the parts (the angle between a first part of the tool and, for example, the part where the user is holding the tool, and between a second part of the tool and, for example, the part where the user is holding the tool), and the movement of a certain part of the tool. The evaluation rank is a value that expresses the evaluation value as a rank. The evaluation rank is expressed, for example, on a 5-point scale (1 to 5) or A, B, C, or the like. The evaluation comment is a comment related to the evaluation of posture. For example, if the mode is "upright row" and the right and left ends of the barbell are different distances from the ground, an evaluation comment such as "different forces are being applied to the left and right sides" may be included.

[0044] The tool movement information is information related to the movement of the tool used by the user. The tool movement information includes, in association with a period on the time axis of the video, the part of the tool to be evaluated, a list of the tool orientation values, an evaluation rank, and an evaluation comment. The list of tool orientation values ​​is a chronological list of tool orientation values ​​within the period. The evaluation comment is a comment related to the evaluation of the tool movement. For example, if the mode is "upright row" and the up and down movement of the barbell is insufficient, an evaluation comment such as "The barbell is not lifted sufficiently" may be included.

[0045] The posture information is information related to the posture of the user's body. The posture information includes a part to be evaluated, a posture value, an evaluation rank, and an evaluation comment, all associated with a time point on the time axis of the video. The posture value is a value that represents the posture. Examples of the posture value include the distance from the ground to a part, the distance between two parts, and the angle of the joint (the angle formed by the line from the first end part to the joint part and the line from the second end part to the joint part). The evaluation rank is a value that expresses the evaluation value as a rank. The evaluation rank is expressed, for example, on a 5-point scale from 1 to 5, or A, B, C, or the like. The evaluation comment is a comment related to the evaluation of the posture. For example, if the mode is "lifting" and bending is insufficient, an evaluation comment such as "your hips are not lowered" may be included.

[0046] The body movement information is information related to the user's body movements. The body movement information includes, in association with a period on the time axis of the video, the body part to be evaluated, a list of posture values, an evaluation rank, and an evaluation comment. The list of posture values ​​is a time series of posture values ​​within the period. The evaluation comment is a comment related to the evaluation of the movement. For example, if the mode is "lifting" and the knee extension is not smooth, an evaluation comment such as "The knee movement is not smooth" may be included.

[0047] The relationship information indicates one or more pieces of positional relationship information for each of the tool and the body. The relationship information includes information on the relationship between the position, orientation, and movement of the tool parts and information on the body parts, posture, and movement, etc., in association with a point on the timeline of the video. For example, the positional relationship between the two can be displayed based on the tool position information and the body position information. That is, for example, a figure representing a part (e.g., a circle) can be superimposed on the image at the position indicated by the tool position information, and a figure representing a part (e.g., a circle) can be displayed at the position indicated by the body position information. Note that the positions of multiple parts and parts can be included for one point in time. Note that position information does not need to be included for the point connecting the parts (e.g., the tip of the bat and the center point of the part where the user grips the bat). In this case, the point connecting these two parts and parts can be represented by connecting a pair of two specified parts and figures (e.g., circles) representing these parts with a line. The relationship information may be included for each frame that makes up the video, for each key frame, for each checkpoint (details will be described later), for every arbitrary number of frames, or for a random point in time. If My Frame does not include location information, shapes indicating parts or regions and lines connecting them can be displayed based on location information from the most recent past point in time.

[0048] Checkpoint information is information that indicates points (hereinafter referred to as checkpoints) where the orientation of the tool or the posture of the body should be checked in the movement of the tool used by the user or in a series of movements of the user's body. For example, if the mode is "weightlifting," checkpoints include when the barbell reaches its highest position, when it reaches its lowest position, and the moment it is lifted. If the mode is "pitching," checkpoints include when the foot is raised, when the raised foot is lowered and weight is transferred, and when the ball is released. The checkpoint information stores information that indicates the checkpoint (hereinafter referred to as checkpoint ID) in association with a time point on the time axis of the video. In other words, it is possible to identify the frame (still image) in the video in which the checkpoint indicated by the checkpoint ID is displayed.

[0049] The evaluation display unit 114 displays the evaluation information. For example, the evaluation display unit 114 can superimpose on the video a graphic representing a tool part or a body part (for example, a circle representing the edge of the tool or the center of gravity of the body, and a line connecting them) based on the tool position, tool orientation, tool movement information, body position, posture, body movement information, etc. included in the evaluation information, thereby superimposing the tool part or the movement of the body part on the video. Furthermore, the evaluation display unit 114 can, for example, graphically display the position, orientation, and movement of the tool part, as well as the position, posture, and movement of the body part over time.

[0050] Furthermore, the evaluation display unit 114 can display an evaluation rank and an evaluation comment along with the display of the video, based on the tool orientation information and tool movement information included in the evaluation information. For example, the evaluation display unit 114 can display the evaluation rank and evaluation comment included in the tool orientation information when the playback time of the video reaches a point around the time point included in the tool orientation information (this can be any length, for example, around 5 seconds). The evaluation display unit 114 can also display the evaluation rank and evaluation comment included in the tool movement information when the playback time of the video reaches a period included in the tool movement information. The evaluation display unit 114 can also display the orientation values ​​included in the orientation information. The evaluation display unit 114 can also display a graph of the time-series conversion of the tool orientation values ​​based on a list of tool orientation values ​​included in the tool movement information.

[0051] Furthermore, the evaluation display unit 114 can display an evaluation rank and an evaluation comment in conjunction with the display of the video based on the posture information and movement information included in the evaluation information. For example, the evaluation display unit 114 can display the evaluation rank and evaluation comment included in the posture information when the playback time of the video reaches a point around the time point included in the posture information (for example, a desired length, such as around 5 seconds). The evaluation display unit 114 can display the evaluation rank and evaluation comment included in the movement information when the playback time of the video reaches a period included in the movement information. The evaluation display unit 114 can also display the posture values ​​included in the posture information. The evaluation display unit 114 can also display a graph of the time-series conversion of the posture values ​​based on a list of posture values ​​included in the movement information. The evaluation rank and evaluation comment may be displayed together with the evaluation rank and evaluation comment displayed in conjunction with the display of the video based on the tool orientation information and tool movement information described in the previous paragraph.

[0052] The checkpoint display unit 115 can extract and display images of checkpoints from the video. The checkpoint display unit 115 can read frames corresponding to time points included in the checkpoint information from the video image data stored in the image storage unit 131 and display them as still images. Furthermore, the checkpoint display unit 115 may, for example, extract and display only body parts from the read frames.

[0053] The fitness activity request sending unit 116 sends a request for acquiring fitness activities related to physical exercise (hereinafter referred to as a fitness activity request) to the server device 20. FIG. 7 is a diagram showing an example of the configuration of a fitness activity request. As shown in the figure, the fitness activity request includes a user ID, a mode, a purpose, and the like. The purpose is an objective for which the user wishes to improve. Examples of objectives include "increase ball speed," "increase muscle strength," "walk safely," "keep blood pressure within the reference range," and "improve posture." The objective is also selected from a predetermined list of options.

[0054] The fitness activity in this embodiment is a concept that broadly includes activities for a user to achieve a goal, such as exercise, strength and skill training, rehabilitation, medical treatment, receiving treatment such as chiropractic and massage, eating, sleeping, etc., but is not limited to these.

[0055] Fitness activity information receiving unit 117 receives information about fitness activities (hereinafter referred to as fitness activity information) transmitted from server device 20 in response to a fitness activity request. Fitness activity information receiving unit 117 stores the received fitness activity information in fitness activity storage unit 133. FIG. 8 shows an example of the configuration of fitness activity information. As shown in FIG. 8, the fitness activity information includes a goal, advice, and reference information. In this embodiment, the advice is assumed to be a character string describing the fitness activity. However, it may also be content that presents the fitness activity using images, videos, or the like. The reference information includes a desirable body shape, tool orientation and movement (position, orientation, movement, speed, etc. of each part), body posture (position and angle of each part, etc.), and the content, number of times, intensity, etc. of various actions. Note that fitness activity information receiving unit 117 may also receive fitness activity information transmitted from fitness activity information transmitting unit 216 based on the evaluation results and reference values, even without a fitness activity request.

[0056] Fitness activity information display unit 118 displays fitness activities. Fitness activity information display unit 118 displays advice and the like included in the fitness activity information. Furthermore, if the fitness activity information includes suitable positions and angles for body parts, these may be presented as images.

[0057] The fitness activity information display unit 118 may display a future posture image, which will be described later, on the user terminal 10.

[0058] 9 is a diagram illustrating an example of the hardware configuration of the server device 20. The server device 20 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to the communication network 40, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, or a USB (Universal Serial Bus) connector or an RS232C connector for serial communication. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, or the like for inputting data. The output device 206 is, for example, a display, a printer, a speaker, or the like for outputting data.

[0059] 10 is a diagram showing an example of the software configuration of server device 20. As shown in the diagram, server device 20 includes functional units, namely, evaluation request receiving unit 211, image analysis unit 212, evaluation unit 213, evaluation information sending unit 214, fitness activity request receiving unit 215, and fitness activity information sending unit 216, and storage units, namely, image data storage unit 231, reference information storage unit 232, evaluation condition information storage unit 233, and fitness activity information storage unit 234.

[0060] Each of the above functional units is realized by the CPU 201 of the server device 20 reading out a program stored in the storage device 203 into the memory 202 and executing it, and each of the above storage units is realized as part of the storage area provided by the memory 202 and storage device 203 of the server device 20.

[0061] The rating request receiving unit 211 receives a rating request transmitted from the user terminal 10. The rating request receiving unit 211 registers information including image data included in the received rating request (hereinafter referred to as image information) in the image data storage unit 231. FIG. 11 is a diagram showing an example of the configuration of image information stored in the image data storage unit 231. As shown in the figure, the image information includes image data associated with a user ID indicating the user who captured the image. The image data was included in the rating request.

[0062] The reference information storage unit 232 stores information (hereinafter referred to as reference information) related to physical exercise using a tool, including reference values ​​related to the relationship derived from the relationship between the tool and the body, such as the tool position, tool movement (such as the orientation and movement of the tool), and posture (such as the position and angle of a part), as well as the relationship between the tool and the body. FIG. 12 is a diagram showing an example of the configuration of reference information stored in the reference information storage unit 232. As shown in the figure, the reference information includes, but is not limited to, the absolute position of the tool parts when performing physical exercise using a tool, information on how the tool parts moved (such as the speed, distance, and direction of movement), reference information related to the absolute position of the part or its relative position with respect to other parts or other reference objects (hereinafter referred to as position reference information), reference information on the angles formed by the lines connecting each of two parts and the joint parts for three parts including joint parts (hereinafter referred to as angle reference information), and information related to the relationship between the tool parts and the body parts.

[0063] The tool position reference information includes tool parts and the reference positions of those parts, associated with the mode and checkpoint ID. There may be multiple parts. The vertical position of a part may be, for example, the height from the ground or the distance from one of the toes. Furthermore, for example, if the mode is "weightlifting," the distance from a body part or a line connecting two parts, such as the distance between the shaft and the line connecting both shoulders, may be used. The horizontal position of a part may be the distance from a predetermined reference object (for example, a mound plate or a mark on the floor) or the distance from a reference part such as the shoulders, chest, or feet. The position reference information is assumed to be registered in advance.

[0064] The tool movement reference information includes reference values ​​for information such as the movement speed and distance of tool parts, the direction of movement at a certain point in time, and the trajectory of movement over a certain period of time, in association with the mode and checkpoint ID.

[0065] The position reference information includes a body part and a reference position for that body part, in association with the mode and checkpoint ID. There may be multiple body parts. Regarding the position, the vertical position may be, for example, the height from the ground or the distance from one of the toes. The horizontal position may be the distance from a predetermined reference object (for example, a mound plate or a mark on the floor) or the distance from a reference body part such as the shoulders, chest, or feet. The position reference information is assumed to be registered in advance.

[0066] The angle reference information includes, in association with the mode and checkpoint ID, two parts (part 1 and part 2), one joint part, and a reference value for the angle between a line connecting part 1 and the joint part, and a line connecting part 2 and the joint part.

[0067] The related reference information includes information related to the reference relationship between tool parts and body parts, associated with the mode and checkpoint ID. The related reference information includes information obtained from the movement speed, movement distance, angle, etc. for one or more parts and parts, associated with the mode and checkpoint ID. For example, if the mode is batting, the related reference information includes the movement speed of the tip of the bat at the time the ball is met, and the angle formed by the bat and the dominant arm holding the bat, etc.

[0068] The evaluation condition information storage unit 233 stores information for evaluation (hereinafter referred to as evaluation condition information). FIG. 13 is a diagram showing an example of the configuration of evaluation condition information stored in the evaluation condition information storage unit 233. The evaluation condition information includes a category, a condition, an evaluation rank, and a comment. The category is an evaluation category. Examples of categories include "muscle strength," "ball speed," and "control." The conditions are conditions for the position, orientation, or movement (change in position over time) of each part of the tool in the image, as well as conditions for the position or movement (change in position over time) of each part of the body. For example, when analyzing weightlifting movements, conditions for the elbow angle, the speed at which the arms are extended, and the shaft movement and up-and-down speed during the period of lifting and lowering the barbell can be set in the evaluation condition information for the checkpoint at which the ball is released. Furthermore, when analyzing pitching form, conditions for the elbow angle, the arm speed, and the like can be set in the evaluation condition information for the checkpoint at which the ball is released. The evaluation rank is the evaluation value when the above conditions are met. The comment is an explanation of the body posture and movement when the above conditions are met.

[0069] The image analysis unit 212 (part / region identification unit) analyzes the image data. The image analysis unit 212 analyzes the image data to extract the feature amounts of each part of the tool and each part of the body, and identifies the position of each part and each region in the image. The image analysis unit 212 also analyzes the image data to extract the feature amounts of each part of the tool and identifies the direction in which each part is facing. Note that the image analysis method used by the image analysis unit 212 is a general one, and a detailed description thereof will be omitted here. The image analysis unit 212 may analyze the image data for each frame or each key frame, may analyze the image data for each checkpoint, or may analyze the image data at random timing.

[0070] The image analysis unit 212 also compares the position of each part identified from the image data with the position reference information stored in the reference information storage unit 232 for each checkpoint ID, and identifies the closest time as the checkpoint time.

[0071] The image analysis unit 212 may measure the body size of the user.

[0072] The image analysis unit 212 may estimate the distance and measure the body size from the ratio of the distance between an object of known size in the normal image or mirror image (including, but not limited to, the mirror itself, a door, a window, a training machine, etc.) and each part of the body identified by the image analysis unit 212, for example.

[0073] The image analysis unit 212 may predict the user's body size, body type (including the circumference of body parts such as chest circumference and abdominal circumference), weight, and the like.

[0074] The image analysis unit 212 may estimate the user's body size, body shape (including the circumference of body parts such as chest circumference and waist circumference), and weight using, for example, machine learning. For example, the image analysis unit 212 acquires a dataset labeled with height, weight, body shape, and images (which may be a single image or images of the user captured from multiple angles, including multiple normal images, a normal image and a mirror image, or multiple mirror images. When multiple images are used, depth information can also be added), and extracts body features from the collected images using a convolutional neural network (CNN) or other image processing technology. These features include volume, contours, and the size of specific body parts. The image analysis unit 212 constructs a prediction model using the extracted features as input and weight as output. Specifically, the image analysis unit 212 uses a regression model or a deep learning algorithm to learn the relationship between image features and weight. The image analysis unit 212 may evaluate the trained model and verify the error between the predicted weight and the actual weight using another dataset. The image analysis unit 212 improves prediction accuracy by adjusting the architecture and hyperparameters of the model as necessary. The image analysis unit 212 receives feature amounts obtained from a normal image or a mirror image as input and predicts weight using a prediction model.

[0075] The evaluation unit 213 evaluates the movement of the tool used by the user based on the image data. In this embodiment, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information including conditions satisfied by the positions and movements of each part of the tool identified from the image data, and if there is evaluation condition information for which the conditions are satisfied, acquires the evaluation rank and comments included therein. Note that the evaluation unit 213 may evaluate the movement of the tool and count the number of physical exercises.

[0076] The evaluation unit 213 evaluates the user's body movements based on the image data. In this embodiment, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information including conditions satisfied by the positions and movements of each body part identified from the image data, and if there is evaluation condition information for which the conditions are satisfied, acquires the evaluation rank and comments included therein. Note that the evaluation unit 213 may evaluate the body movements and count the number of body exercises.

[0077] The evaluation unit 213 evaluates the tool used by the user and the body movement based on the image data. In this embodiment, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information including the positions of each part of the tool and each part of the body identified from the image data, and conditions satisfied by the movements or relationships of the parts and the parts, and if there is evaluation condition information for which the conditions are satisfied, acquires the evaluation rank and comments included therein. Note that the evaluation unit 213 may evaluate the tool and the body movement and count the number of body movements.

[0078] When the evaluation unit 213 evaluates at least either the tool used by the user or the body movement based on multiple image data, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information (including depth information) including the positions of each part of the tool and each part of the body identified from the image data and the conditions satisfied by the movements or relationships of said parts and said parts, and if there is evaluation condition information whose conditions are satisfied, it obtains the evaluation rank and comments included therein.

[0079] The evaluation information transmission unit 214 transmits evaluation information to the user terminal 10. The evaluation information transmission unit 214 generates tool position information including the time point on the time axis of the video identified by the image analysis unit 212 and the position of each part of the tool. If the position of the tool part satisfies the conditions for the evaluation rank and comment acquired by the evaluation unit 213, the evaluation information transmission unit 214 generates posture information including the time point, part, and tool orientation value, as well as the evaluation rank and comment. If the movement of the part (change in position over time) satisfies the conditions, the evaluation information transmission unit 214 generates tool movement information including a list of the time point, part, and tool orientation value, as well as the evaluation rank and comment. The evaluation information transmission unit 214 also generates checkpoint information including the time point corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint. The evaluation information transmission unit 214 creates evaluation information including the generated tool position information, tool orientation information, tool movement information, and checkpoint information, and transmits it to the user terminal 10. The evaluation unit 213 and the evaluation information transmission unit 214 may correspond to the comment output unit of the present invention.

[0080] The evaluation information transmitting unit 214 transmits the evaluation information to the user terminal 10. The evaluation information transmitting unit 214 generates position information including the time point on the time axis of the video identified by the image analysis unit 212 and the position of each part. Regarding the evaluation rank and comment acquired by the evaluation unit 213, if the position of the part satisfies the condition, the evaluation information transmitting unit 214 generates posture information including the time point, part, and posture value, as well as the evaluation rank and comment. If the movement of the part (change in position over time) satisfies the condition, the evaluation information transmitting unit 214 generates movement information including a list of time points, part, and posture values, as well as the evaluation rank and comment. The evaluation information transmitting unit 214 also generates checkpoint information including the time point corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint. The evaluation information transmitting unit 214 creates evaluation information including the generated position information, posture information, movement information, and checkpoint information, and transmits it to the user terminal 10. The evaluation unit 213 and the evaluation information transmitting unit 214 may correspond to a comment output unit of the present invention.

[0081] ? Fitness activity information storage unit 234 stores information related to fitness activities (hereinafter referred to as fitness activity information). FIG. 14 is a diagram showing an example of the configuration of fitness activity information stored in fitness activity information storage unit 234. As shown in the diagram, fitness activity information includes advice associated with objectives, categories, and conditions. The conditions may be conditions for the tool itself (such as the weight of a barbell), conditions for how to use the tool, conditions for physical conditions (such as flexibility), conditions for the position, orientation, or movement of a part of the tool, or conditions for the position or movement of a part of the body.

[0082] The fitness activity request receiving unit 215 receives a fitness activity request sent from the user terminal 10 .

[0083] Fitness activity information transmitting unit 216 searches for fitness activity information corresponding to the mode and purpose included in the fitness activity request that satisfies the user's physical information included in the evaluation request and the conditions of the position, orientation, movement, etc. of each part or body part identified by image analysis unit 212. Fitness activity information transmitting unit 216 obtains advice from the searched fitness activity information, creates fitness activity information that sets the purpose and advice, and transmits the created fitness activity information to user terminal 10. Fitness activity information transmitting unit 216 also transmits the position, orientation, speed, angle, etc. of each part or body part included in the reference information together with the fitness activity information. Note that fitness activity information transmitting unit 216 may search for fitness activities based on the evaluation information and reference values ​​even without a fitness activity request, and may transmit the fitness activities to user terminal 10.

[0084] As an example, fitness activity implementation information acquisition unit 217 acquires information about fitness activities actually performed by the user. Fitness activity implementation information acquisition unit 217 may acquire the performed fitness activities by, for example, acquiring information input to user terminal 10. In this case, fitness activity implementation information acquisition unit 217 may present to user terminal 10 a form that allows the user to select or input the performed fitness activities (which may be training, rehabilitation, meals, treatments, etc.) and the number of times, intensity, etc. of the performed fitness activities, and acquire information about the input or selection operation of user terminal 10.

[0085] The fitness activity implementation information acquisition section 217 may acquire, for example, the results of the image analysis of the user by the image analysis section 212 or information on the evaluation performed by the evaluation section 213 as the implemented fitness activity.

[0086] The fitness activity implementation information acquisition unit 217 may acquire the implemented fitness activity information from information from a sensor or the like included in the user terminal 10, for example.

[0087] As an example, the fitness activity implementation information acquisition unit 217 may acquire the completed fitness activity information from an image acquired using an imaging function such as a camera included in the user terminal 10. In this case, the fitness activity implementation information acquisition unit 217 may acquire, for example, an image of a meal, and analyze the image using known image analysis technology to acquire information such as the content and amount of the meal, the nutrients contained therein and their amounts, and calories.

[0088] For example, the future posture generation unit 218 predicts the constitution of the user at a future time point and generates an image of the user in the future based on the predicted constitution. For example, the constitution in this embodiment may be the constitution stored in the user information storage unit 130.

[0089] As an example, future posture generating unit 218 predicts future physical constitution based on physical constitution information included in the user information and fitness activity information (fitness activity implementation schedule information) transmitted by fitness activity information transmitting unit 216. Future posture generating unit 218 can predict how physical constitution will change if the fitness activity implementation schedule information is performed.

[0090] Theoretical value For example, the future posture generation unit 218 may store in advance influence information on the influence of each fitness activity presented to the user on the user's physical constitution, and may predict the user's physical constitution based on the influence information. For example, when the presented fitness activity is "one set of 30 squats, three sets per day," the future posture generation unit 218 may read influence information (which may be the influence per day or the influence when the activity is continued for a predetermined period such as 30 days, and may also be set for the number of times the activity is performed in a predetermined period, but is not limited thereto) stored in the server device 20 and predict how the user's leg muscles will be after, for example, one month.

[0091] Use of predictive models The future posture generating unit 218 may, for example, use a prediction model generated in advance to predict the user's constitution when performing each fitness activity presented to the user. In this case, for example, the prediction model may use the fitness activity information (fitness activity implementation schedule information) and information on the constitution of the user who actually performed the fitness activity information (fitness activity implementation schedule information) as training data, and generate a prediction model with the fitness activity information (fitness activity implementation schedule information) as input information and the constitution as output information.

[0092] For example, if the fitness activity information (fitness activity schedule information) includes information on meal content, future posture generation unit 218 may calculate the balance between calorie intake and calorie expenditure based on information such as calorie intake stored in association with the meal content. Note that if the fitness activity information does not include information on meal content, future posture generation unit 218 may predict calorie intake based on the average value for the category to which the user belongs (such as a category divided by age, gender, etc.), or may calculate the value based on information included in user information, performed fitness activity information, etc., such as the user's past meal content.

[0093] The future image generation unit 218 may, for example, acquire images of the user at any two points in time while performing a fitness activity and re-predict the user's future constitution. For example, the future posture generation unit 218 may compare an image of the user at a first point in time with an image of the user at a second point in time and correct the influence information described above.

[0094] The future posture generation unit 218 may, for example, re-predict the future physical constitution based on the information on the actually performed fitness activity acquired by the fitness activity implementation information acquisition unit 217. The future posture generation unit 218 predicts the future physical constitution based on the fitness activity information (fitness activity implementation schedule information). If the actually performed fitness activity implementation information differs from the fitness activity implementation schedule information, the predicted future physical constitution may be corrected. The future posture generation unit 218 may correct the physical constitution at a future time point predicted using the fitness activity information (fitness activity implementation schedule information). For example, the future posture generation unit 218 may predict the physical constitution by replacing the fitness activity implementation schedule information with the amount of actually performed fitness activity instead of the fitness activity implementation schedule information, or may predict the physical constitution at a future time point if the performed fitness activity is continued for a predetermined period. The correction method by the future posture generation unit 218 may be the same as the method for predicting the physical constitution at a future time point using the fitness activity information (fitness activity implementation schedule information).

[0095] As an example, the future posture generation unit 218 generates a future posture image of the user based on at least information about the user's constitution at a future time point predicted by the future posture generation unit 218. For example, the future posture generation unit 218 may input text information generated based on at least the information about the constitution at a future time point predicted by the future posture generation unit 218 and information about the future time point to be predicted (e.g., one month later, three months later, six months later, etc.) into an image generation AI, and generate a posture image based on the output information. At this time, the future posture generation unit 218 may input an image of the user at the current time point (which may be an image stored in the image data storage unit 231) together with the text information into the image generation AI, and generate a posture image based on the output information, thereby generating a posture image taking into account information that can be obtained from the image, such as the user's facial expression and skin texture.

[0096] FIG. 21 is an example of a posture image generated by the future posture generation unit 218. FIG. 22 is a posture image in which muscles are visualized, but of course the posture image may be based on an image of the user himself. Also, FIG. 22 shows some frames extracted from a posture image of a user walking, and the posture image may be a video. Note that the future posture generation unit 218 may generate an explanatory image for fitness activities based on the generated posture image, and may express the image by superimposing a diagram explaining what to be aware of when training, as shown in 462a and 462b, for example.

[0097] The image generated by the future posture generating unit 218 may be a still image or a moving image showing some kind of movement. Hereinafter, the information on the user's constitution at a future time point and the posture image will be collectively referred to as future posture information.

[0098] The future posture information presenting unit 219 presents information about the user's constitution at a future time point, which is generated by the future posture information generating unit 218, to the user terminal 10. The future posture presenting unit 219 may present at least any of the future posture information to the user terminal 10. Furthermore, as one example, the future posture information presenting unit 219 may present the future posture information when it acquires a request for presentation of future posture information from the user terminal 10. Furthermore, as one example, when the fitness activity implementation information acquiring unit 217 acquires fitness activity implementation information that differs from the fitness activity information (fitness activity implementation plan information), the future posture information presenting unit 219 may present the future posture information corrected by the fitness activity implementation information to the user terminal 10, or when a predetermined difference occurs between the future posture information based on the fitness activity implementation information and the initially predicted future posture information, the future posture information may be presented to the user terminal 10.

[0099] FIG. 15 is a diagram showing an example of the flow of processing executed in the physical exercise support server of this embodiment.

[0100] In the user terminal 10, the imaging unit 111 accepts the input of the mode, captures an image of the user's body during exercise, and acquires video data (S321). The evaluation request sending unit 112 sends an evaluation request including the user ID indicating the user, the accepted mode, physical information, and video data to the server device 20 (S322).

[0101] When the evaluation request receiving unit 211 in the server device 20 receives an evaluation request, the image analysis unit 212 analyzes the video data to extract features (S323) and identifies the position of each part and each region (S324). Here, the image analysis unit 212 may identify the position on the image, or may identify the actual position (height from the ground, distance from a reference point such as the center of gravity of the body, etc.) using body information. The evaluation unit 213 obtains an evaluation rank and comments from evaluation condition information that satisfies the conditions of the position of each part and each region and the movement of each part and region (time-series change in position) (S325). The evaluation information sending unit 214 creates evaluation information and sends it to the user terminal 10 (S326).

[0102] In user terminal 10, evaluation display unit 114 displays the position, orientation, movement, etc. of body parts or tools on the video data based on the received evaluation information (S327). Furthermore, in user terminal 10, evaluation display unit 114 displays the position (bones) of each part indicating the body posture, and may also display an evaluation rank and comments (S327). Here, evaluation display unit 114 may graphically display the position, orientation, movement, etc. of parts, as well as time-series changes in the position and movement of parts. Furthermore, checkpoint display unit 115 may extract and display images of checkpoints from the video. Fitness activity request sending unit 116 sends a fitness activity request to server device 20 in response to an instruction from the user (S328).

[0103] In server device 20, when fitness activity request receiving unit 215 receives a fitness activity request sent from user terminal 10, fitness activity information sending unit 216 searches for fitness activity information that satisfies the conditions, obtains the advice included in the searched fitness activity information (S329), creates fitness activity information including the obtained advice, and sends it to user terminal 10 (S330).

[0104] When the fitness activity information receiving unit 117 in the user terminal 10 receives fitness activity information, the fitness activity information display unit 118 displays the advice contained in the received fitness activity information and can also display appropriate tool usage methods superimposed on video data (S331).

[0105] When the fitness activity information receiving unit 117 in the user terminal 10 receives the fitness activity information, the fitness activity information display unit 118 displays the advice contained in the received fitness activity information and can also display an appropriate body posture in the form of bones superimposed on the video data (S331).

[0106] As described above, the physical exercise support server of this embodiment allows users to easily evaluate physical exercise. In particular, for physical exercise related to sports, rehabilitation, dieting, and other physical activities, the server can evaluate the relative positions and movements of each tool and each body part, which can lead to specific efforts to improve, leading to improved performance and more effective implementation. Furthermore, the support server of this embodiment also provides comments and advice, allowing users to easily understand their current status and fitness activities.

[0107] Fig. 16 is a diagram showing an example of a screen displaying a posture evaluation. Fig. 16 illustrates a case where a video is captured in walking mode. As shown in Fig. 16, on screen 41, marks are displayed on the user's body parts with bars connecting the marks superimposed thereon (411). Analysis results of walking speed, walking rate (cadence), stride length, etc. by image analysis unit 212 are displayed (412). Screens displaying analysis results for categories such as mobility, stability, left-right difference, and gait can be selected (413).

[0108] Fig. 17 is a diagram showing an example of a screen displaying a posture evaluation. Fig. 17 illustrates a case where a video is captured in walking mode. As shown in Fig. 17, the screen 42 has an area (421) showing variations in stride length, and the stride length for each step is measured by the image analysis unit 212 and displayed (423), with the average value also displayed (422). The results of the evaluation by the evaluation unit 213 are also displayed (424).

[0109] Fig. 18 is another diagram showing an example of a screen displaying a posture evaluation. Fig. 18 illustrates a case where a video is captured in walking mode. As a result of analysis by the image analysis unit 212 and evaluation by the evaluation unit 213 in walking mode, for example, numerical values ​​(431) such as the maximum flexion / extension angles of the joints and an evaluation (432) are displayed as evaluation items of the gait.

[0110] Fig. 19 is another diagram showing an example of a screen displaying a posture evaluation. Fig. 19 illustrates a case where a video is captured in walking mode. The evaluation results performed by the evaluation unit 213 in walking mode are displayed as an overall evaluation, for example, an evaluation (451) for a user group (such as a group of users of similar age, sex, playing the same sport, or having the same purpose), a radar chart of evaluations for each evaluation category, and advice (453).

[0111] 20 shows an example of fitness activity information that fitness activity information transmitting unit 216 transmits to user terminal 10, that fitness activity information receiving unit 117 receives, and that fitness activity information display unit 118 displays on user terminal 10. The information displayed includes the title of the fitness activity (441), a list of workouts that are part of the fitness activity (442), and a display (443) of an explanatory video and the number of repetitions (number of repetitions and number of sets) for each workout. Other fitness activity information that may be displayed includes dietary details, recommendations for treatment, and the like.

[0112] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention.

[0113] For example, in this embodiment, the image analysis is performed in the server device 20, but this is not limiting. The image analysis may be performed in the user terminal 10 to identify the positional relationship of each part and each region.

[0114] Even if the entire body of the user is not shown in the normal image or mirror image, the server device 20 only needs to analyze or evaluate the parts or portions of the area that is shown.

[0115] Furthermore, when the user's entire body is not shown in the forward image or mirror image, the server device 20 analyzes the parts or portions shown in each of multiple different forward images, or at least one forward image and mirror image, or multiple different mirror images, and searches the evaluation condition information storage unit 233 for evaluation condition information including the conditions that must be satisfied by the position and movement of each part of the tool identified from each image data, and if there is evaluation condition information that satisfies the conditions, it obtains the evaluation rank and comments included therein.

[0116] In addition, in this embodiment, the positions of parts and regions are assumed to be positions on a two-dimensional image, but this is not limited thereto and may be three-dimensional positions. For example, if the user terminal 10 is equipped with a depth camera in addition to a camera, the three-dimensional positions of the parts and regions can be identified based on an image from the camera and a depth map from the depth camera. Also, for example, the three-dimensional position of the parts and regions may be identified by estimating three dimensions from a two-dimensional image. Note that it is also possible to provide a depth camera instead of a camera and identify the three-dimensional position only from the depth map from the depth camera. In this case, the user terminal 10 can transmit a depth map to the server device 20 together with or instead of the image data, and the image analysis unit 212 of the server device 20 can analyze the three-dimensional position.

[0117] Furthermore, in the present embodiment, an image of the user's body while exercising using a tool is transmitted from the imaging device 30 to the server device 20. However, this is not limiting, and the image may be captured using the user terminal 10. In this case, the imaging unit 111 may acquire an image of at least one of a normal image or a mirror image captured by the user terminal 10, which is placed in a location where the user appears in either a normal image or a mirror image. Alternatively, the user terminal 10 may extract features from the acquired image and transmit the features to the server device 20. Alternatively, the user terminal 10 may estimate parts of the tool or body parts based on the features, acquire absolute positions of the parts or parts (which may be positions on the XY coordinate system of the image, or actual distances from a reference position (e.g., the ground, the toes, the head, the center of gravity of the body, etc.), or positions in any other coordinate system) or relative positional relationships between multiple parts, multiple parts, or multiple part parts, and transmit these absolute positions or relative positional relationships to the server device 20.

[0118] In addition, in this embodiment, the fitness activity information is provided as content prepared by the server device 20, but this is not limiting. For example, reference values ​​may be included, and marks or bones that indicate correct movements and postures (such as the position and orientation of each part, and the position and angle of each part) based on the reference values ​​may be superimposed on a video or a still image extracted from the video. This makes it easy to understand what movements and postures should be performed.

[0119] In addition, in this embodiment, the evaluation is based on the part and orientation of the tool, the position or movement of a body part (position over time), etc., but this is not limited to this, and the evaluation may also be based on identifying the position of the tool worn by the user.

[0120] In addition, while the present embodiment provides content such as advice regarding fitness activities, it may also be possible to recommend tools, for example. In this case, the server device 20 stores tools and standard values ​​for the tool's size (e.g., length) in association with the user's physical information (e.g., height, weight), extracts features of the tool used by the user from image data to identify the tool's shape, estimates the tool's size based on the shape and the user's size (e.g., height) included in the physical information, and recommends a tool of the standard size if the difference between the estimated tool size and the standard value is equal to or greater than a predetermined threshold. Furthermore, it may recommend tools according to the purpose based on information such as the tool's conditions (e.g., the weight of a barbell), how the tool is used, physical conditions (e.g., flexibility), and the position, orientation, and movement of the tool's parts.

[0121] Furthermore, in this embodiment, content such as advice is provided for fitness activities. However, for example, the physical exercise being performed may be interrupted. In this case, the server device 20 stores a reference value for when the physical exercise should be interrupted in association with the user's physical information (purpose, height, weight, etc.), and interrupts the physical exercise if the number of times or speed of the physical exercise being performed by the user (for example, if the speed at which the barbell is lifted drops drastically or if too many repetitions are performed at once) deviates from the reference value based on the image data. In this case, the server device 20 may issue a message to the user terminal 10 telling the user to stop, or may notify the user by changing the display, such as by turning off the screen, or by emitting a sound such as an alert, or may notify the user by vibration.

[0122] Furthermore, in this embodiment, the content provided includes advice and other information about fitness activities. However, for example, the server device 20 may also provide information about physical exercises aimed at diagnosing illnesses and improving them. In this case, the server device 20 extracts potential illnesses that the user may be suffering from based on the symptoms and evaluation information entered by the user in the physical information, and provides a screening test to narrow down the illnesses. Once the user has completed the screening test and narrowed down the illness, the server device 20 may recommend seeing a doctor, recommend physical exercises aimed at improving the illness, or recommend items such as tools and meals for the physical exercises.

[0123] Furthermore, by estimating the positions of the tool parts, the server device 20 can estimate the speed, acceleration, movement distance, trajectory, etc. of the tool. Furthermore, by extracting a pattern of change in the tool position over time, the server device 20 can estimate the number of times a pattern occurs as the number of actions using the tool.

[0124] Furthermore, in this embodiment, the exercise is evaluated, but the present invention is not limited to this. When a certain posture or movement is detected, a task for that movement may be proposed. In this case, the server device 20 may store a task in association with one or a series of postures or movements, instead of an evaluation comment, and output the task.

[0125] Furthermore, in this embodiment, exercise is evaluated, but this is not limiting, and when a certain tool movement, tool orientation, posture, or body part movement is detected, content to improve physical exercise according to the purpose, etc. may be presented, such as training to be done, rehabilitation, musical performance, or preparatory steps such as stretching, strength training, posture, etc. In this case, the server device 20 may store the details of the training, etc., in association with one or a series of tool part movements, tool part orientations, body postures, or body part movements, instead of evaluation comments, and output the details.

[0126] Furthermore, in this embodiment, the exercise is evaluated, but the present invention is not limited to this and the movements performed by the user can also be automatically detected. In this case, the server device 20 stores the positions and postures of each part of the tool used to perform a predetermined movement such as shooting or passing (the positions of each body part) as reference information, and can identify the movement performed by the user in the image by comparing the positions of the tool parts and body parts analyzed from the image with the reference information.

[0127] In addition, in this embodiment, the motion is evaluated by analyzing images captured in the past, but this is not limiting. The analysis process may be performed in real time, and when a predetermined motion is detected, the next tactic to be adopted may be recommended. In this case, the tactic may be stored in association with the posture or motion instead of the evaluation comment, and the tactic may be output in real time.

[0128] In addition, in this embodiment, the execution of a predetermined function and the storage of information are performed by the user terminal 10 or the server device 20, but this is not limiting, and the execution of the function and the storage of information may be performed by either device. Alternatively, the function unit and the storage unit may be provided separately in a form different from this embodiment. [Explanation of symbols]

[0129] 10 User terminal 20 Server device 30 Imaging device 40 Communication Network 111 Imaging unit 112 Evaluation request sending unit 113 Evaluation information receiving unit 114 Evaluation display section 115 Checkpoint display 116 Fitness Activity Request Sending Unit 117 Fitness Activity Information Receiving Unit 118 Fitness activity information display section 130 Physical information storage unit 131 Image storage unit 132 Evaluation information storage unit 133 Fitness Activity Memory Section 211 Evaluation Request Receiving Unit 212 Image Analysis Unit 213 Evaluation Department 214 Evaluation information transmission unit 215 Fitness Activity Request Receiving Department 216 Fitness Activity Information Transmission Department 217 Group Analysis Department 218 Group Analysis Presentation Section 231 Image data storage unit 232 Standard information storage unit 233 Evaluation condition information storage unit 234 Fitness activity information storage unit 235 Group analysis information storage unit

Claims

1. An assist device for assisting a user's fitness, a posture image generating unit that generates a posture image of the user performing a predetermined movement based on a user image of the user and physical constitution information of the user at a future time point predicted based on fitness activity implementation schedule information; An assistance device comprising:

2. The posture image generation unit inputs at least the user image and text information generated based on the fitness activity implementation schedule information to a generation AI, and generates the posture image based on output information. The support device according to claim 1 .

3. the posture image generation unit corrects the posture image based on fitness activity implementation information. The support device according to claim 1 or 2.

4. The posture image generation unit inputs text information generated based on at least the user image, the fitness activity implementation schedule information, and the training implementation information into a generation AI, and generates the posture image based on output information. The support device according to claim 2 .

5. the fitness activity implementation information includes information about the content of the fitness activity acquired by analyzing the user image; 5. The support device according to claim 3 or 4.

6. the posture image generation unit corrects the posture image based on the user images taken at a plurality of points in time and training effect information determined based on fitness activity implementation information between the points in time.

5. The support device according to claim 3 or 4.

7. An assistance program for assisting a user's fitness, The processor a posture image generating step of generating a posture image of the user performing a predetermined movement based on the user image of the user and physical constitution information of the user at a future time point predicted based on fitness activity implementation schedule information; A support program to help you achieve this.

8. A method for supporting a user's fitness, comprising: The processor: a posture image generating step of generating a posture image of the user performing a predetermined movement based on the user image of the user and physical constitution information of the user at a future time point predicted based on fitness activity implementation schedule information; How to support this.

Images