Health support system

Abstract

Provided is a health support system that checks the user's posture, movement, and expression without using special measuring equipment, and supports the user to maintain health in a manner suitable for the individual user. 【Solution means】The health support system of the present invention includes an acquisition means 111 for acquiring a video of the user's movement, an analysis means 112 for analyzing the user's posture, movement, and expression based on the acquired video and specifying the difference from the user's form based on comparison with a predetermined ideal form, and an output means 113 for outputting feedback based on the difference. The analysis means specifies the user's movement form by analyzing the user's posture, movement, and expression and estimating the skeleton, measures the movement parameters based on the skeleton points of each part of the body, and then specifies the difference by comparing with the reference value of the ideal form. The output means outputs improvement advice of content according to the difference and an improvement menu according to the difference.

Description

Technical Field

[0001] The present invention is a health support system with a mechanism that checks the user's posture, movement, and expression without using special measuring equipment and supports the user to maintain health in a manner suitable for the individual user. More specifically, it relates to a health support system that checks the user's posture, movement, and expression and supports the user to be healthy.

Background Art

[0002] Conventionally, fitness apps (health management apps) have been known that promote muscle strength training and calorie intake management to improve health. However, conventional fitness apps have problems such as insufficient individual optimization, reduced effectiveness due to form errors, and difficulty in habituation.

[0003] On the other hand, conventionally, a mechanism has been proposed to adjust the optimal rehabilitation intensity or exercise amount corresponding to an exercise prescription suitable for an individual to enhance the effects of rehabilitation or sports fitness (see, for example, Patent Document 1). In addition, a mechanism that can monitor actions such as getting out of bed, falling, and other actions, and based on the human body head position and the bed-type object position, determines the number of people in the target environment, the state on the bed, the state off the bed, and the movement state of the bed position, and can transmit the posture state to an electronic device to give warnings and notifications has also been conventionally proposed (see, for example, Patent Document 2).

[0004] However, conventionally, a mechanism that focuses on the user's posture, movement, and expression and supports the user to maintain health in a manner suitable for the individual user has not been proposed as far as the applicant knows.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] In view of the above circumstances in the prior art, the present invention has been made, and without using special measuring equipment, it checks the user's posture, movement, and expression, and aims to provide a mechanism that supports the maintenance of health in accordance with the individual user.

Means for Solving the Problems

[0007] To achieve the above object, an aspect of the present invention is a health support system comprising: an acquisition means for acquiring an image of a user; an analysis means for analyzing the user's posture, movement, and expression based on the image, and specifying a difference from the user's form based on a comparison with a predetermined ideal form; and an output means for outputting feedback based on the difference.

[0008] In the above health support system, the output means may output improvement advice corresponding to the difference as the feedback. Also, in the above health support system, the output means may output an evaluation corresponding to the difference as the feedback.

[0009] Also, in the above health support system, the output means may further output support information for providing guidance on improvement according to the difference. At this time, the output means may output the support information according to the magnitude of the difference specified within a predetermined time or the tendency of the difference continuously specified. Also, the output means may output the support information by any of an image, text, or voice.

[0010] In the above-mentioned health support system, it is preferable that the output means further outputs an ideal motion video serving as a model of the user's posture and movement. Also, in the above-mentioned health support system, it is preferable that the output means further outputs an image obtained by analyzing the user's posture, movement, and expression.

[0011] Also, in the above-mentioned health support system, the analysis means may identify the user's form by estimating the user's skeleton, measure motion parameters based on the skeleton, and then compare with the reference value of the ideal form to identify the difference. At this time, after identifying the user's form, the analysis means may measure the motion parameters based on the skeleton points of each part of the user's body.

[0012] Also, the analysis means may measure a skeleton angle indicating a change in posture obtained based on the skeleton points as the motion parameter. At this time, the analysis means may measure the maximum value or the minimum value of the skeleton angle as the motion parameter.

[0013] Also, the analysis means may measure a motion speed indicating a change in motion obtained based on the skeleton points as the motion parameter. At this time, the analysis means may measure the maximum value or the minimum value of the motion speed as the motion parameter.

[0014] Also, the analysis means may estimate the user's psychology or emotion based on the change in expression to identify the concentration or fatigue degree, and the output means may output the feedback taking into account the concentration or the fatigue degree.

[0015] Also, in the above-mentioned health support system, when the image is a video, the video may be divided into a plurality of phases, and the difference from the user's form may be identified for each phase. At this time, after identifying the form of the user by estimating the skeleton of the user, the analysis means may divide the video into a plurality of phases based on changes in the posture and movement of the user, and identify the difference for each phase.

[0016] Also, in the above health support system, the analysis means may immediately analyze the image while the acquisition means acquires the image. Furthermore, in the above health support system, the analysis means may identify an improvement degree based on a comparison of the respective differences identified by analyzing two of the images acquired at different times and dates, and the output means may output the feedback according to the improvement degree.

[0017] Also, the above health support system may further include a reception means for receiving task information such as the schedule and work content of the user, and the output means may output recovery information based on the task information. At this time, the reception means may further receive information regarding rest, and the output means may output the above-mentioned recovery information based on the task information and the information regarding rest.

Effects of the Invention

[0018] According to the present invention, by analyzing the posture, movement, and expression of the user based on the captured image and identifying the difference from a predetermined ideal form, it is possible to identify problems and drawbacks related to the health of the user, such as the user's health condition, exercise form, and decline in concentration, and to provide appropriate feedback. Therefore, it is possible to provide a mechanism that supports maintaining health in a manner suitable for the individual user by checking the posture, movement, and expression of the user without using special measuring equipment.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of the health support system according to the present invention will be described with reference to the drawings. The health support system according to the present invention analyzes the user's posture, movement, and expression based on an image of the user, identifies problems and drawbacks related to health, provides appropriate feedback, and proposes support information that can efficiently and appropriately address the identified problems and drawbacks, aiming to support the user to maintain health in a manner suitable for the individual user.

[0021] The embodiments described below are preferred specific examples of the present invention and are technically subject to various limitations. However, the scope of the present invention is not limited to these embodiments unless otherwise specifically limited in the following description.

[0022] As shown in FIG. 1, the health support system in this embodiment is configured such that the health support device 10 and the user terminal 30 are communicably connected via the network N. Also, the network N is not limited to the Internet and may be a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network), for example.

[0023] The health support device 10 shown in FIG. 1 is a device used for health support processing that receives the image information transmitted from the user terminal 30, analyzes the user's posture, movement, and expression based on the image information to find problems and drawbacks related to the user's health, and outputs feedback and support information. This health support device 10 has a computing function and a communication function and functions as a server in the health support system. It is realized by, for example, an electronic device such as a server device or a personal computer. Therefore, in the following description, the health support device 10 may sometimes be referred to as the "server".

[0024] The user terminal 30 is a device with computing, imaging, and communication functions that is operated by a user who receives health support. For example, it is realized by a desktop computer, a laptop computer, a tablet computer, a smartphone, or the like.

[0025] In the example of FIG. 1, only one user terminal 30 is shown, but the number thereof is not particularly limited and may be a plurality.

[0026] <Hardware Configuration> FIG. 2 is a block diagram showing the hardware configuration of the health support device 10 according to the present embodiment. The health support device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, and a communication unit 19.

[0027] The CPU 11 executes various processes according to a program stored in the ROM 12 or a program loaded from the storage unit 18 into the RAM 13. That is, the CPU 11 is a processor that controls the operation of the entire server 1 by executing various processes according to a program. The program is, for example, an operating system (system software) for operating each part of the health support device 10 or an application software for realizing the functional blocks described later.

[0028] In the RAM 13, data and the like necessary for the CPU 11 to execute various processes are also appropriately stored. The CPU 11, ROM 12, and RAM 13 are interconnected via the bus 14. The input / output interface 15 is also connected to this bus 14. Connected to the input / output interface 15 are an output unit 16, an input unit 17, a storage unit 18, and a communication unit 19.

[0029] The output unit 16 is composed of a display, a speaker, etc., and outputs various information as images and sounds. The input unit 17 is composed of a keyboard, a mouse, a microphone, a camera, etc., and inputs various information according to the received instruction operations. Note that, for example, the output unit 16 and the input unit 17 may be integrated by a touch panel.

[0030] The storage unit 18 stores various data necessary for the server 1 to execute information processing, and is composed of, for example, a hard disk, a DRAM (Dynamic Random Access Memory), etc. The communication unit 19 communicates with other devices via a network N including the Internet.

[0031] Note that the supply and demand prediction device 10 may be provided with a drive to which a removable medium made of a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, etc. is appropriately attached. The program read from the removable medium can be installed in the storage unit 18 by this drive. Also, the removable medium can store various data stored in the storage unit 18 in the same manner as the storage unit 18.

[0032] Note that the health support device 10 is not limited to a device that operates alone, and may be a distributed server system that cooperates by communicating via the network N, or a cloud server. Also, the hardware configuration of the user terminal 30 of the present embodiment is the same as that of the health support device 10 described above.

[0033] <Health support device 10> FIG. 3 is a block diagram for explaining a configuration example of the health support device 10 that functions as a server in the present embodiment. The health support device 10 includes an acquisition unit 111, an analysis unit 112, and an output unit 113. The health support device 10 also includes an ideal form information DB (database) 181, a feedback information DB (database) 182, and a support information DB (database) 183.

[0034] The acquisition unit 111 is a means having a function of acquiring an image of the user. That is, the acquisition unit 111 acquires a camera video or an uploaded image of the user.

[0035] The acquisition of the image by the acquisition unit 111 can be performed using, for example, WebRTC (Web Real-Time Communication), a technology that enables data communication such as audio and video in real time on the Web. Since this WebRTC can directly exchange data between browsers and applications without going through a specific server, low-latency and high-quality streaming are possible. Note that the image is not limited to a video obtained by continuously shooting an operation video, and can be a still image of a characteristic posture or a plurality of still images obtained by continuously shooting a characteristic action.

[0036] The analysis unit 112 is a means having a function of analyzing the user's posture, movement, and expression based on the image, and identifying the difference from the user's form based on comparison with a predetermined ideal form. That is, the analysis unit 112 analyzes the user's posture, movement, and expression based on an image such as an operation video (video) acquired by the acquisition unit 111, and, for example, based on comparison with a predetermined ideal form stored in the ideal form information DB 181, finds (identifies) the difference from the actual form indicating issues and drawbacks related to the user's health.

[0037] The analysis unit 112 is also a means having a function of identifying the user's form by estimating the user's skeleton, measuring operation parameters based on the skeleton, and then identifying the difference by comparing with the reference value of the ideal form. That is, as an analysis of the user's posture and movement, the analysis unit 112 identifies the user's form by estimating the user's skeleton, measures movement parameters based on the estimated user's skeleton, and then compares them with the reference values of the pre-set ideal form to identify the difference from the user's form. In this way, by estimating the user's form based on the skeleton that makes it easy to determine the posture and movement, the user can easily confirm the issues and drawbacks related to the user's health indicated by the difference from the ideal form.

[0038] In addition, the analysis unit 112 is also a means having a function of measuring movement parameters based on the skeleton points of each part of the user's body after identifying the user's form. That is, as an analysis of the user's posture, movement, and expression, after identifying the user's form, the analysis unit 112 measures movement parameters based on the positions of each joint such as the shoulders, elbows, knees, and hip joints as the skeleton points of each part of the body. In this way, by measuring movement parameters based on the skeleton points of each part of the body that makes it even easier to determine what posture, movement, and expression the user has, the user can accurately confirm the issues and drawbacks related to the user's health indicated by the difference from the ideal form.

[0039] In addition, the analysis unit 112 is also a means having a function of measuring, as movement parameters, the skeleton angles indicating the changes in the posture obtained based on the skeleton points. That is, the analysis unit 112 measures, as parameters, the skeleton angles that make it even easier to determine what posture the user is taking. In this way, by judging the change in posture based on the skeleton angle, the difference between the user's form and the ideal form can be confirmed. That is, for example, in an exercise where posture is important, such as yoga, it is possible to assist in achieving a desirable posture based on the skeleton angle.

[0040] In addition, the analysis unit 112 is also a means having a function of measuring the maximum or minimum value of the skeletal angle as an operation parameter. That is, when the skeletal angle needs to be opened (extended) widely as a desirable posture, the analysis unit 112 measures the maximum value as the operation parameter; conversely, when the skeletal angle needs to be bent small, the analysis unit 112 measures the minimum value as the operation parameter. In this way, it becomes possible to confirm the maximum difference between one's own movement form and the ideal form based on the skeletal angle.

[0041] In addition, the analysis unit 112 is also a means having a function of measuring the movement speed indicating the change of the movement obtained based on the skeletal points as an operation parameter. That is, the analysis unit 112 measures the movement speed that is easy to determine what movement the user is performing. In this way, by judging the change and rhythm of the movement based on the movement speed, it becomes possible to confirm the quality of the user's movement from the difference between the user's form and the ideal form. That is, for example, in a sport where posture, movement speed, and rhythm are important, such as a pitching motion, it is possible to assist to achieve a desirable movement based on the movement speed (rhythm) together with the skeletal angle.

[0042] In addition, the analysis unit 112 is also a means having a function of measuring the maximum or minimum value of the movement speed as an operation parameter. That is, when fast movement is required as a desirable movement operation, the analysis unit 112 measures the minimum value of the movement speed as the operation parameter; conversely, when slow movement is required, the analysis unit 112 measures the maximum value of the movement speed as the operation parameter. In this way, it becomes possible to confirm the difference between one's own movement form and the ideal form based on the movement speed.

[0043] Here, the estimation of the skeleton can be performed by directly using an open-source library such as MediaPipe. That is, MediaPipe is a framework equipped with machine learning and image processing functions developed by Google. The analysis unit 112 estimates the user's posture using a posture estimation module such as MediaPipe and identifies the user's form. The form to be identified can be, for example, skeletal data representing the user's hands and feet as if they were sticks, so-called stick figures. Also, by estimating the user's skeleton in an image using MediaPipe or the like, it can be used to recognize the user's posture and movement and to measure the skeletal angle and movement speed.

[0044] In addition, when the image is a video, the analysis unit 112 is also a means having a function of dividing the video into a plurality of phases and identifying the difference from the user's form for each phase. That is, after identifying the user's form by estimating the user's skeleton, the analysis unit 112 divides the video into a plurality of phases (as a reference) based on the changes in the user's posture, movement, and expression obtained based on the skeleton points of each part of the user's body, and identifies the difference for each phase. Specifically, for example, the video can be automatically divided into meaningful motion phases such as a preparatory motion, a motion peak, and a follow-through. Also, for example, characteristic moments or peaks such as "maximum flexion angle" and "end point of motion" can be extracted from 10 seconds of continuous motion to divide the video into a plurality of phases. In this way, by measuring motion parameters such as skeletal angle and motion speed for each divided phase, it becomes possible to confirm the difference between one's own motion form and the ideal form.

[0045] In addition, the analysis unit 112 is also a means having a function of estimating the user's psychology or emotion based on the change in expression and identifying the concentration or fatigue level. That is, the analysis unit 112 estimates the user's mental state, the emotion being held, and the mental state using artificial intelligence (AI) from the change (movement) of the facial muscles (skeletal muscles), and takes this into account in the change of posture and movement to identify the user's concentration and fatigue level. In this way, by considering the psychological, emotional, and mental aspects, it becomes possible to identify psychological issues related to health.

[0046] Further, the analysis unit 112 is also a means having a function of immediately analyzing the image while the acquisition unit 111 acquires the image. That is, based on the camera video or the uploaded motion video (video) acquired by the acquisition unit 111 from the user terminal 30, the analysis unit 112 immediately performs skeleton estimation in real time and acquires the position information of the skeleton points (each joint such as the shoulder, elbow, knee, and hip joint) to measure the motion parameters, thereby analyzing the user's motion form. By sequentially analyzing the images obtained in this way, the user can quickly recognize the issues and deficiencies related to the user's health from their own postures, motions, and expressions, and can efficiently and appropriately work on improving the motions.

[0047] Further, the analysis unit 112 is also a means having a function of specifying the improvement degree based on the comparison of the respective differences specified by analyzing two images acquired at different times and dates. That is, for example, the analysis unit 112 compares the difference from the ideal form specified at the initial stage of analysis with the difference from the ideal form specified at the final stage of analysis on the same day, and specifies the improvement degree according to the difference. Further, for example, the analysis unit 112 also compares the difference from the ideal form specified on the first day of analysis with the difference from the ideal form specified on a later day, and specifies the improvement degree according to the difference. In this way, by showing the improvement degree specified based on the analysis results with different analysis times such as time and date, the user can confirm the improvement state of their own motion form.

[0048] The output unit 113 is a means having a function of outputting feedback based on the difference. That is, based on the differences identified by the analysis unit 112, the output unit 113 generates and outputs appropriate feedback to the user using artificial intelligence (AI). This feedback may be output after the acquisition unit 111 finishes acquiring the user's image, or may be output for each analysis performed by the analysis unit 112 in real time.

[0049] The generation of feedback by artificial intelligence (AI) can be achieved, for example, by editing and adjusting a chat-type open-source AI such as Hugging Face for use. The feedback is not a complete automatic generation by artificial intelligence (AI), and it is preferably configured based on, for example, content supervised by experts such as judo rehabilitators and athletic trainers. This feedback is stored as template data in the feedback information DB 182, for example. Therefore, the output unit 113 proposes the corresponding one from the template data stored in the feedback information DB 182 according to the difference calculated by the analysis unit 112.

[0050] Also, the output unit 113 is a means having a function of outputting improvement advice corresponding to the difference as feedback. That is, the output unit 113 takes the difference identified by the analysis unit 112 as an issue, and generates and outputs advice for improving the issue for each user. This improvement advice can be, for example, feedback such as "lower your shoulders a little more". With this improvement advice, the user can recognize the drawbacks of their posture and movement and appropriately work on improving them.

[0051] Also, the output unit 113 is a means having a function of outputting an evaluation corresponding to the difference as feedback. That is, the output unit 113 outputs an evaluation generated according to the difference between the form of the user identified by the analysis unit 112 based on the comparison with a predetermined ideal form. This evaluation can include a score value calculated based on the difference. Specifically, for example, evaluations such as "elbow angle 104 degrees" and "Evaluation Δ: (elbow angle is) slightly wide" can be made. Through such objective evaluations, the user can confirm the quality of their own form. Also, if an objective evaluation that is easy to recognize, such as a score value, is included, the user can immediately recognize the degree of completion of their own form.

[0052] In addition, the output unit 113 is also a means having a function of outputting support information for guiding improvements to issues and drawbacks related to the user's health according to the difference. That is, the output unit 113 generates and outputs, for each user, an improvement menu (support information) for guiding optimal health improvement for individual users, such as form correction instructions, suggestions for auxiliary training, and suggestions for stretching, separately from feedback such as improvement advice generated according to the difference identified by the analysis unit 112.

[0053] The support information is also not completely automatically generated by artificial intelligence (AI), and it is preferably configured based on content supervised by experts such as judo rehabilitation therapists and athletic trainers, for example. This support information is, for example, stored as template data in the support information DB 183. Therefore, the output unit 113 proposes the corresponding one from the template data stored in the support information DB 183 according to the difference calculated by the analysis unit 112. With this support information, it is possible to efficiently and appropriately address the issues and drawbacks of postures and movements pointed out in the improvement advice.

[0054] In addition, the output unit 113 is also a means having a function of outputting support information according to the magnitude of the difference identified within a predetermined time or the trend of continuously identified differences. That is, the output unit 113 generates and outputs appropriate improvement menus (support information) for each user according to the magnitude of the difference identified by the analysis unit 112 within a predetermined time or the trend of the difference identified and stored by the analysis unit 112 within a predetermined time. Thereby, for each posture, movement, and expression at a predetermined time, issues and drawbacks regarding the user's health are recognized, and the user can efficiently and appropriately work on improving their health.

[0055] Also, the output unit 113 is also a means having a function of outputting support information by any one of an image, text, or voice. That is, the output unit 113 outputs an improvement menu (support information) by any one of an image including a video, a still image, a slide, text, or voice. At this time, the support information by voice can be output (played back) by using TTS (Text to Speech), which is a technology for converting existing text into voice. Thereby, the user can output support information by means of the surrounding environment or a preferable means that is easy to understand.

[0056] Also, the output unit 113 is also a means having a function of further outputting an ideal image that serves as a model for the user's posture and movement. That is, the output unit 113 outputs an ideal image separately from feedback (improvement advice and evaluation) and improvement menus (support information). Thereby, the user can appropriately perform a predetermined posture and movement while checking the ideal image. Also, at this time, the output unit 113 can output feedback such as "Look at the (ideal) image more carefully and imitate it" for each analysis performed by the analysis unit 112 in real time.

[0057] Also, the output unit 113 is also a means having a function of further outputting an image obtained by analyzing the user's posture, movement, and expression. That is, the output unit 113 superimposes (overlays) and displays the skeletal data (stick figure information) of the user's form analyzed by the analysis unit 112 on the user's image acquired by the acquisition unit 111, separately from feedback (improvement advice and evaluation), improvement menu (support information), and ideal image. As a result, the user can appropriately perform predetermined postures and movements while objectively confirming their own postures, movements, and expressions.

[0058] Further, the output unit 113 is a means having a function of outputting feedback according to the degree of improvement. That is, the output unit 113 outputs the degree of improvement specified by the analysis unit 112. As a result, the user can continuously use this system and experience the Before·After effect from the degree of improvement indicating the results of efforts to improve the operation.

[0059] Also, the output unit 113 is a means for outputting recovery information based on task information. That is, when the reception unit 114 receives task information, the output unit 113 analyzes the load and density of the task using artificial intelligence (AI) and proposes appropriate recovery information such as action proposals according to time, stretching, and meditation to the user.

[0060] Also, the output unit 113 is a means for outputting recovery information based on task information and information related to breaks. That is, when the reception unit 114 receives information related to breaks together with task information, the output unit 113 proposes appropriate recovery information to the user in consideration of the information related to breaks.

[0061] Also, the output unit 113 is a means for outputting feedback in consideration of the concentration level or fatigue level. That is, the output unit 113 outputs feedback based on the difference from a predetermined ideal form specified by the analysis unit 112 and the concentration level or fatigue level. As a result, the user can efficiently and appropriately work on improving psychological issues related to health.

[0062] The reception unit 114 is a means for receiving task information such as the user's schedule and work content. That is, in order to encourage the user to continue and establish exercise in daily life, after the analysis based on the user's image is completed and feedback (improvement advice) and support information (improvement menu) are output, the reception unit 114 receives task information such as the user's schedule and work content.

[0063] In addition, the reception unit 114 is a means for further receiving information regarding breaks. That is, the reception unit 114 further receives information regarding breaks such as the timing and acquisition time of breaks, in addition to task information.

[0064] The ideal form information DB 181 is a means for storing ideal forms generated based on ideal images that serve as models for the user's movements. For example, it is a database that stores form data of multiple patterns defined in advance through consultation with experts such as athletic trainers.

[0065] The feedback information DB 182 is a means for storing predetermined feedback information (improvement advice and scores) in association with the differences identified by the analysis unit 112. For example, it is a database that stores template data configured based on the content supervised by experts such as judo rehabilitation therapists and athletic trainers.

[0066] The support information DB 183 is a means for storing predetermined support information (improvement menu) in association with the differences identified by the analysis unit 112. Similar to the feedback information DB 182, for example, it is a database that stores template data configured based on the content supervised by experts such as judo rehabilitation therapists and athletic trainers.

[0067] Note that, although not shown, an analysis result information DB (database) may be provided to store, for each user, identification information for identifying feedback such as operation parameters measured by the analysis unit 112, differences identified by the analysis unit 112, improvement advice and evaluation output by the output unit 113, and identification information for identifying support information. Further, the analysis result information DB may store a plurality of analysis results obtained at different times and dates.

[0068] As described above, in the health support system of the present embodiment, without preparing special measuring equipment, for example, the posture and movement of the user are analyzed in real time from a camera image or an uploaded image captured by the user terminal 30 such as a smartphone, and based on the difference identified by comparing the user's form with a predetermined ideal form, feedback (improvement advice) regarding the user's health and support information (improvement menu) are output.

[0069] As described above, the health support device 10 according to the present embodiment functions as a device including an acquisition unit 111, an analysis unit 112, and an output unit 113 when various programs (OS, applications, etc.) stored in the auxiliary storage device are loaded into the main storage device and executed by the CPU 11.

[0070] Next, the processing flow in the health support system according to an embodiment of the present invention will be described. FIG. 4 is a flowchart for explaining the overall processing flow in the health support system according to the present embodiment.

[0071] First, in order to enjoy the use of this system, the user accesses the site of this system using the communication function of the user terminal 30 and installs (downloads) dedicated application software in advance. Then, after obtaining the dedicated application software, the user logs in to this system, enabling the execution of operations using this system.

[0072] In step S301, each user activates the camera of the user terminal 30, captures the posture and movement of the measuring user, and uploads the camera video to the health support device 10. Then, in step S302, the health support device 10 detects the posture and joints of the user, for example, by MediaPipe processing. That is, using a posture estimation module such as MediaPipe, the posture, movement, and expression of the user are analyzed in real time from the acquired uploaded image, and the skeleton points of each joint such as the shoulder, elbow, knee, and hip joint are detected to perform the user posture estimation process.

[0073] Next, in step S303, the health support device 10 calculates (measures) motion parameters such as joint angles and motion speeds based on the position information of the detected skeleton points. Also, in step S304, the health support device 10 selects feedback using a model by form error estimation by artificial intelligence (AI). That is, by comparing with the reference value of the preset ideal form, the difference from the user's form is identified, and based on the identified difference, appropriate feedback to the user is generated and output using artificial intelligence (AI).

[0074] Subsequently, in step S305, the health support device 10 displays an improvement menu by video, text, voice, etc. That is, in order to improve the problems and drawbacks related to the health of the user identified based on the posture, movement, and expression, an improvement menu (support information) for providing appropriate guidance is generated and output for each user according to the identified difference. Furthermore, in step S306, the health support device 10 has the artificial intelligence (AI) guide the movement. That is, the user receives appropriate guidance to improve the problems and drawbacks related to the user's health using artificial intelligence (AI). Then, in step S307, the health support device 10 will end the improvement menu.

[0075] Also, in step S308, the health support device 10 performs a Before·After comparison by implementing the improvement menu. That is, it compares the operation parameters such as the skeletal angle and movement speed, and the images obtained by analyzing the user's posture, movement, and expression before and after the implementation of the improvement menu, and outputs feedback such as impressions and evaluations. Also, in step S308, the health support device 10 performs history scoring and individual profiling based on the ID for identifying the user. That is, it scores the Before·After evaluations obtained by using this system respectively, and records them as history for each user and performs individual profiling.

[0076] Subsequently, in step S310, the health support device 10 optimizes the recovery proposal on a schedule basis. That is, for the health support device 10, it records the schedule and work content of one day as tasks, arbitrarily sets the rest timing, and the artificial intelligence (AI) analyzes the load and density thereof, and proposes recovery information such as refresh methods like rest, meditation, and stretching, and concentration support methods optimal for improving concentration. Then, in step S311, the health support device 10 performs individual motion pattern evaluation by recording analysis and artificial intelligence (AI) proposal control. After that, a series of processes in the health support device 10 are completed (END).

[0077] Next, the detailed process flow in the health support system according to an embodiment of the present invention will be described. FIG. 5 is a flowchart showing an example of the operation of the feedback output process in the health support device.

[0078] In step S401, the acquisition unit 111 performs a process of determining whether it has acquired (received) user image information obtained by photographing the user transmitted from the user terminal 30. When the acquisition unit 111 has acquired the user image information (YES in S401), the process proceeds to step S402. Otherwise (NO in S401), the process in step S401 is repeated.

[0079] In step S402, the analysis unit 112 performs a process of analyzing the user's posture, movement, and expression based on the image. Subsequently, in step S403, the analysis unit 112 performs a process of identifying the difference from the user's form based on a comparison with a predetermined ideal form. Then, in step S404, the output unit 113 performs a process of outputting feedback based on the difference. At this time, as the feedback, improvement advice corresponding to the difference can be output. After that, a series of operations in the health support device 10 ends (END).

[0080] Also, the flow of other detailed processes in the health support system according to an embodiment of the present invention will be described. FIG. 6 is a flowchart showing an example of the operation of the support information output process in the health support device.

[0081] In step S501, the output unit 113 performs a process of outputting an ideal image that serves as a model of the user's movement. Next, in step S502, the acquisition unit 111 performs a process of determining whether it has acquired (received) user image information in the same manner as in step S401 described above. When the acquisition unit 111 has acquired the user image information (YES in S502), the process proceeds to step S503. Otherwise (NO in S502), the process in step S502 is repeated.

[0082] In step S503, the analysis unit 112 performs a process of analyzing the user's posture and movement based on the image in the same manner as in step S402 described above. Subsequently, in step S504, the analysis unit 112 performs a process of identifying the difference between the user's form based on the comparison with a predetermined ideal form, similar to step S403 described above.

[0083] Also, in step S505, the output unit 113 performs a process of outputting feedback such as improvement advice based on the difference, similar to step S404 described above. Then, in step S506, the output unit 113 further performs a process of outputting an improvement menu (support information) for guiding the user's health improvement according to the difference. At this time, the improvement menu (support information) can be any of video, text, and voice. After that, a series of operations in the health support device 10 ends (END).

[0084] Also, another detailed processing flow in the health support system according to an embodiment of the present invention will be described. FIG. 7 is a flowchart showing an example of the operation of the analysis image output process in the health support device.

[0085] In step S601, the output unit 113 performs a process of outputting an ideal image that serves as a model of the user's movement, similar to step S501 described above. Next, in step S602, the acquisition unit 111 performs a process of determining whether the user image information has been acquired (received), similar to step S401 described above. If the acquisition unit 111 has acquired the user image information (YES in S602), the process proceeds to step S603; otherwise (NO in S602), the process in step S602 is repeated.

[0086] In step S603, the analysis unit 112 performs a process of analyzing the user's posture, movement, and expression based on the image, similar to step S402 described above. Subsequently, in step S604, the output unit 113 further performs a process of outputting an image obtained by analyzing the user's posture, movement, and expression. Also, in step S605, the analysis unit 112 performs a process of identifying the difference between the user's form based on comparison with a predetermined ideal form, similar to step S403 described above.

[0087] Furthermore, in step S606, the output unit 113 performs a process of outputting feedback such as improvement advice based on the difference, similar to step S404 described above. Then, in step S607, the output unit 113 further performs a process of outputting an improvement menu (support information) for providing improvement guidance according to the difference. After that, a series of operations in the health support device 10 ends (END).

[0088] Also, other detailed processing flows in the health support system according to an embodiment of the present invention will be described. FIG. 8 is a flowchart showing other operations of the evaluation information output process in the health support device.

[0089] In step S701, the output unit 113 performs a process of outputting an ideal image serving as a model of the user's posture and movement, similar to step S501 described above. Next, in step S702, the acquisition unit 111 performs a process of determining whether user image information has been acquired (received), similar to step S401 described above. If the acquisition unit 111 has acquired user image information (YES in S702), the process proceeds to step S703; otherwise (NO in S702), the process in step S702 is repeated.

[0090] In step S703, the analysis unit 112 performs a process of analyzing the user's posture, movement, and expression based on the image, similar to step S402 described above. Subsequently, in step S704, the output unit 113 performs a process of identifying the difference between the user's form based on comparison with a predetermined ideal form, similar to step S403 described above. Also, in step S705, the analysis unit 112 performs a process of outputting feedback such as improvement advice based on the difference, similar to step S404 described above. Also, in step S706, the output unit 113 performs a process of further outputting an improvement menu (support information) for guiding the user's health improvement according to the difference, similar to step S506 described above.

[0091] Furthermore, in step S707, the analysis unit 112 performs a process of generating an evaluation according to the difference. Then, in step S708, the output unit 113 performs a process of outputting the evaluation generated by the analysis unit 112. After that, a series of operations in the health support device 10 end (END).

[0092] Also, the detailed processing flow in the health support system according to an embodiment of the present invention will be described. FIG. 9 is a flowchart showing the detailed operation of the analysis process in the health support device.

[0093] In step S801, the acquisition unit 111 performs a process of determining whether the user image information has been acquired (received), similar to step S401 described above. If the acquisition unit 111 has acquired the user image information (YES in S801), the process proceeds to step S802; otherwise (NO in S801), the process in step S801 is repeated.

[0094] In step S802, the analysis unit 112 performs a process of specifying the user's form by estimating the user's skeleton as the analysis of the user's posture, movement, and expression based on the image in step S402 described above. Also, in step S803, the analysis unit 112 performs a process of measuring motion parameters based on the skeleton points of each part of the user's body after identifying the user's form as an analysis of the user's posture, motion, and expression based on the image in step S402 described above. At this time, the motion parameters can be, for example, the skeleton angles indicating changes in posture and the motion speeds indicating changes in motion.

[0095] Subsequently, in step S804, the analysis unit 112 performs a process of identifying the difference from the user's form based on a comparison with the reference value of a predetermined ideal form, in the same manner as in step S403 described above. Then, in step S805, the output unit 113 performs a process of outputting feedback such as improvement advice based on the difference, in the same manner as in step S404 described above. After that, a series of operations in the health support device 10 end (END).

[0096] Also, the detailed processing flow in the health support system according to an embodiment of the present invention will be described. FIG. 10 is a flowchart showing the detailed operation of the analysis process in the health support device.

[0097] In step S901, the acquisition unit 111 performs a process of determining whether or not it has acquired (received) user image information, in the same manner as in step S401 described above. If the acquisition unit 111 has acquired user image information (YES in S901), the process proceeds to step S902; otherwise (NO in S901), the process in step S901 is repeated.

[0098] In step S902, the analysis unit 112 performs a process of identifying the user's form by estimating the user's skeleton, in the same manner as in step S802 described above. Also, in step S903, the analysis unit 112 divides the image into a plurality of parts for each specific phase, and performs a process of analyzing the user's posture, movement, and expression for each phase to measure movement parameters. At this time, the video can be divided into phases based on the changes in the posture and movement obtained based on the skeleton points of each part of the user's body.

[0099] Also, in step S904, the analysis unit 112 performs a process of identifying the difference from the user's form based on the comparison with a predetermined ideal form, in the same manner as step S403 described above. Subsequently, in step S905, the output unit 113 performs a process of outputting feedback such as improvement advice according to the magnitude of the identified difference or the tendency of the continuously identified differences, in the same manner as step S404 described above.

[0100] Also, in step S906, the analysis unit 112 performs a process of determining whether the acquisition of the user image information has been completed in the acquisition unit 111. If the acquisition unit 111 has completed the acquisition of the user image information (YES in S906), the process proceeds to step S907. Otherwise (NO in S906), the process from step S903 is repeated. Then, in step S907, the output unit 113 performs a process of further outputting an improvement menu (support information) for providing improvement guidance according to the difference, in the same manner as step S506 described above. After that, a series of operations in the health support device 10 are completed (END).

[0101] In this way, this system, for example, analyzes the user's posture and movement in real time, and identifies (detects) issues and drawbacks related to the user's health, such as health status, exercise form, and decline in concentration, from the differences identified based on the comparison between the predetermined ideal form and the user's form. Then, it provides feedback such as improvement advice and improvement menus (support information), and by guiding the execution of improvements in posture, movement, and expression, it supports the user to become healthy. Moreover, it can analyze not only the differences in the user's form but also the quality of the user's movement (movement speed and rhythm) and the psychological and emotional aspects, enabling personalized feedback and guidance. This can solve the problems of "insufficient individual optimization", "reduction of effects due to form errors", and "difficulty in habituation" that conventional fitness apps have.

[0102] Also, in this system, by recording the schedule and tasks of the day as tasks and arbitrarily setting the break timing, based on the results of analyzing the user's posture, movement, and expression, it can be an extended mechanism that promotes the user's continuous exercise and daily habituation and further supports the maintenance of the user's health. Figure 11 is a flowchart showing the operation of the extended analysis process in the health support device.

[0103] First, the processing from step S1001 to step S1004 is the same as the processing from step S401 to step S404 described above, so the description thereof is omitted. Next, in step S1005, the reception unit 114 performs a process of determining whether it has received (received) task information such as the user's schedule and work content. If the reception unit 114 has received the task information (YES in S1005), it proceeds to step S1006. Otherwise (NO in S1005), the series of operations in the health support device 10 ends (END).

[0104] In step S1006, the reception unit 114 performs a process of determining whether it has further received (received) information regarding breaks. If the reception unit 114 has received the information regarding breaks (YES in S1006), it proceeds to step S1007. Otherwise (NO in S1006), it proceeds to step S1008. In step S1007, the output unit 113 performs a process of outputting recovery information based on the task and rest information. On the other hand, in step S1008, the output unit 113 performs a process of outputting recovery information based on the task. After that, a series of operations in the health support device 10 are completed (END).

[0105] Next, in this embodiment, an example of the screen configuration displayed on the user terminal 30 will be described. The form check screen displayed on the user terminal 30 when the user logs in to this system can be shown, for example, in FIG. 12. FIG. 12 is a schematic diagram showing a configuration example of the measurement image reception screen displayed on the user terminal 30.

[0106] In FIG. 12, on this measurement image reception screen 300, there are provided a demonstration video display unit 301 for displaying an ideal image that serves as a model for the user's actions, a real-time analysis video display unit 302 for displaying a video obtained by analyzing the acquired user image, and a measurement start button 303.

[0107] That is, when the measurement start button 303 is selected on the measurement image reception screen 300, an ideal image is displayed on the demonstration video display unit 301. Therefore, the user imitates the posture and actions with reference to the ideal image. Also, when the measurement start button 303 is selected, the camera (imaging function) of the user terminal 30 operates to capture the user, and the captured video (action video) is transmitted to the health support device (server) 10 using the communication function provided in the user terminal 30. Then, a video obtained by analyzing the image in the server 10 is displayed on the real-time analysis video display unit 302. The video displayed on this real-time analysis video display unit 302 can be, for example, an image (analysis video) that displays the skeletal data (stick figure information) of the form obtained by analyzing the user's posture and actions overlaid on the captured action video of the user.

[0108] In addition, the form confirmation screen displayed on the user terminal 30 can be shown, for example, in FIG. 13. FIG. 13 is a schematic diagram showing a configuration example of the measurement result screen displayed on the user terminal.

[0109] In FIG. 13, on this measurement result screen 310, there is provided a demonstration video display unit 301 for displaying an ideal image serving as a model of the user's operation, a real-time analysis video display unit 302 for displaying a video obtained by analyzing the acquired user image, and a measurement result display unit 311. In addition, on the measurement result display unit 311, there are provided an actual measurement angle display unit 3111 before improvement, an actual measurement angle display unit 3112 after improvement, an improvement degree display unit 3113, an evaluation display unit 3114, an ideal angle display unit 3115, an actual measurement angle display unit 3116, a difference from ideal display unit 3117, and a degree of coincidence display unit 3118.

[0110] As an example, on the measurement result screen 310 shown in FIG. 13, an ideal image with stretching is displayed on the demonstration video display unit 301, and a video obtained by analyzing the user's posture and expression is displayed on the real-time analysis video display unit 302. In addition, on the measurement result screen 310, "147.1°" is displayed on the actual measurement angle display unit 3111 before improvement, "156.8°" is displayed on the actual measurement angle display unit 3112 after improvement, "+9.7°" is displayed on the improvement degree display unit 3113, and "C" is displayed on the evaluation display unit 3114. Furthermore, "179.8°" is displayed on the ideal angle display unit 3115, the same "156.8°" as on the actual measurement angle display unit 3112 after improvement is displayed on the actual measurement angle display unit 3116, "23.0°" is displayed on the difference from ideal display unit 3117, and "77.0%" is displayed on the degree of coincidence display unit 3118.

[0111] In addition, the form improvement screen displayed on the user terminal 30 can be shown, for example, in FIG. 14. FIG. 14 is a schematic diagram showing a configuration example of the improvement guidance screen displayed on the user terminal.

[0112] In FIG. 14, on this improvement guidance screen 320, there is a demonstration video display unit 301 that displays an improvement menu execution image serving as a model for assisting the user's operation, and a real-time analysis video display unit 302 that displays a video obtained by analyzing the acquired user image. In addition, an improvement comment display unit 321 is provided so as to cover the improvement menu execution image displayed on the demonstration video display unit 301. Also, on this improvement guidance screen 320, there are provided a seek bar 322 indicating the progress of the exercise based on the improvement menu execution image displayed on the demonstration video display unit 301, an elapsed time display unit 323, an execution button 324 for executing the display of the improvement menu execution image on the demonstration video display unit 301, and a skip button 325 for fast-forwarding the improvement menu execution image displayed on the demonstration video display unit 301.

[0113] As an example, on the improvement guidance screen 320 shown in FIG. 14, an improvement menu execution image for stretching the neck is displayed on the demonstration video display unit 301, and a video obtained by analyzing the user's posture and expression is displayed on the real-time analysis video display unit 302. Also, on the improvement comment display unit 321, for example, feedback such as "Press your head with your hand" is displayed so as to appropriately support the execution of the improvement menu. Also, the seek bar 322 displays that the exercise has progressed to the point where the exercise is alternately performed left and right in "Progress 3", and the elapsed time display unit 323 displays "00:42" indicating that 42 seconds have elapsed. Also, the execution button 324 is displayed as "Executing" indicating that it has been selected.

[0114] Note that the above-described embodiments are merely examples for facilitating the understanding of the present invention, and are not for limiting the interpretation of the present invention. Therefore, the present invention is not limited to the above-described embodiments, and modifications, improvements, etc. within the scope that can achieve the object of the present invention are included in the present invention.

Industrial Applicability

[0115] Based on the health support system of this invention, by modeling the knowledge of experts in various fields such as trainers, educators, and caregivers into AI models and expanding the types and numbers of the emerging models, it can be expected to be used in, for example, corporate health management measures, sports team training support, elderly care prevention programs, and refresh support for telecommuters.

[0116] In addition, it can also be expected to build a hybrid service where after the virtual experience in this system, face-to-face personal training by the same trainer who proposed improvement advice and improvement menus can be received. Furthermore, it can be expected to build an ecosystem that seamlessly connects the real and digital, such as performing self-care with this system in the office and continuing the online diet session of the same trainer after returning home.

Explanation of Signs

[0117] 10 Health support device (server) 30 User terminal 111 Acquisition unit 112 Analysis unit 113 Output unit 114 Reception unit 181 Ideal form information DB (database) 182 Feedback information DB (database) 183 Support information DB (database)

Claims

1. An acquisition means for acquiring an image of a user; An analysis means for analyzing the posture, movement, and expression of the user based on the image, and identifying a difference from the user's form based on a comparison with a predetermined ideal form; An output means for outputting feedback based on the difference; A health support system characterized by comprising the above.

2. The health support system according to claim 1, wherein the output means outputs improvement advice on the content corresponding to the difference as the feedback.

3. The health support system according to claim 1, wherein the output means outputs an evaluation of the content corresponding to the difference as the feedback.

4. The health support system according to claim 1, wherein the output means further outputs support information for providing guidance on improvement according to the difference.

5. The health support system according to claim 4, wherein the output means outputs the support information according to the magnitude of the difference identified within a predetermined time or the tendency of the continuously identified difference.

6. The health support system according to claim 4, wherein the output means outputs the support information by any one of an image, text, and voice.

7. The health support system according to claim 1, wherein the output means further outputs an ideal image serving as a model for the posture and movement of the user.

8. The health support system according to claim 1, wherein the output means further outputs an image obtained by analyzing the posture, movement, and expression of the user.

9. The analysis means identifies the user's form by estimating the user's skeleton, measures motion parameters based on the skeleton, and then identifies the difference by comparing with a reference value of the ideal form. The health support system according to claim 1, characterized in that.

10. The health support system according to claim 9, wherein the analysis means measures the motion parameters based on the skeleton points of each part of the user's body after identifying the user's form.

11. The health support system according to claim 10, wherein the analysis means measures a skeleton angle indicating a change in posture obtained based on the skeleton points as the motion parameter.

12. The health support system according to claim 11, wherein the analysis means measures the maximum value or the minimum value of the skeleton angle as the operation parameter.

13. The health support system according to claim 10, wherein the analysis means measures the operation speed indicating the change of the operation obtained based on the skeleton points as the operation parameter.

14. The health support system according to claim 13, wherein the analysis means measures the maximum value or the minimum value of the operation speed as the operation parameter.

15. The analysis means estimates the psychology or emotion of the user based on the change of the expression, specifies the concentration or the fatigue degree, and the output means outputs the feedback in consideration of the concentration or the fatigue degree. The health support system according to claim 1, characterized in that.

16. The health support system according to claim 1, wherein when the image is a moving image, the analysis means divides the moving image into a plurality of phases, and specifies the difference from the form of the user for each phase.

17. The health support system according to claim 16, wherein after specifying the form of the user by estimating the skeleton of the user, the analysis means divides the moving image into a plurality of phases based on the change of the posture and movement of the user, and specifies the difference for each phase.

18. The health support system according to claim 1, wherein the analysis means immediately analyzes the image while acquiring the image by the acquisition means.

19. The analysis means specifies the improvement degree based on the comparison of the differences specified by analyzing two images acquired at different times or dates, and the output means outputs the feedback according to the improvement degree. The health support system according to claim 1, characterized in that.

20. The health support system further includes a reception means for receiving task information such as the schedule and work content of the user, and the output means outputs recovery information based on the task information. The health support system according to claim 1, characterized in that.

21. The reception means further receives information regarding rest, and the output means outputs the recovery information based on the task information and the information regarding rest. The health support system according to claim 20, characterized in that.

Images