Health condition evaluation method and device using skeleton model
The method uses a skeleton model to compare user motions with standard motions, addressing the lack of home-based neurological disease diagnosis by accurately evaluating cognitive impairment and physical exhaustion, enhancing telemedicine services.
Patent Information
- Application Number
- JP2024525530
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-23
- Filing Date
- 2022-11-14
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2042-11-14
AI Technical Summary
There is a lack of effective services for diagnosing neurological diseases at home, especially cognitive disorders, as hospital visits decrease with the rise of telemedicine, and existing technologies do not provide accurate methods for evaluating user motions to assess cognitive impairment or physical exhaustion.
A health condition evaluation method using a skeleton model that compares user motions with standard motions, calculating similarity through feature points and applying weights to body parts, allowing for quick clinical judgment and evaluation of cognitive impairment or physical exhaustion.
Enables accurate assessment of cognitive impairment and physical exhaustion by comparing user motions with standard motions, facilitating quick clinical decisions and improving the quality of home training services.
Smart Images

Figure 2024542010000001_ABST
Abstract
Description
[Technical field]
[0001] The embodiments relate to a method and apparatus for evaluating a health condition, and more particularly, to a method and apparatus for evaluating cognitive impairment, degree of physical exhaustion, etc. by comparing a skeleton model of a standard motion model with motion information of a skeleton model of a user. [Background technology]
[0002] As the culture of non-face-to-face communication spreads, the need for remote medical care is increasing. In order to make clinical judgments about neurological disorders such as cognitive impairment, it is necessary to visually check the user's actions. However, as the number of hospital visits decreases, there is a problem that examinations and treatments decrease accordingly.
[0003] Meanwhile, home training, which involves exercising at home without visiting exercise facilities, has become popular in conjunction with the trend towards non-face-to-face contact culture, and related industries are developing. As the quality of home training services improves, people have more opportunities to experience various types of exercise, providing motivation to continue exercising. However, as mentioned above, no services have been presented that can replace visiting a hospital to diagnose neurological diseases.
[0004] To solve these problems, a service is needed that allows people at home to carry out actions to detect nervous system-related diseases and transmit the images to a medical institution to check their health condition. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] Korean Patent Publication No. 10-2020-0073765 (2020.06.24) Summary of the Invention [Problem to be solved by the invention]
[0006] An object of the present specification is to evaluate a user's ability to simulate movements by comparing the user's movements with standard movements.
[0007] Another object of the present invention is to evaluate the degree of cognitive impairment or physical exhaustion of a user by comparing the user's actions with a standard action.
[0008] Another object of this specification is to compare the two actions using skeletal models of user and standard actions.
[0009] Another object of the present invention is to enable rapid clinical judgment by arranging the actions of multiple users and their analysis results on one screen.
[0010] It is yet another object of the present invention to more accurately determine the similarity between two standard movements by applying weights to major body parts for each standard movement.
[0011] The problem to be solved by this specification is not limited to the above, but can be expanded to various matters that can be derived from the embodiments of the invention described below. [Means for solving the problem]
[0012] A health condition evaluation method using a skeleton model according to one embodiment of the present specification is performed by a health management server, and the method includes the steps of: providing a standard action image of a trainer performing a standard action to a user device; receiving from the user device a user action image of a user action in which the user imitates the standard action; comparing the standard action image with the user action image to calculate a similarity between the standard action and the user action; and evaluating the health condition of the user based on the calculated similarity.
[0013] In one embodiment, the step of calculating the similarity includes the steps of: acquiring a trainer skeleton movement model for a trainer performing a standard movement from the standard movement image; acquiring a user skeleton movement model for a user performing a user movement from the user movement image; and calculating the similarity by comparing corresponding feature points of the trainer skeleton movement model and the user skeleton movement model for each body part, wherein the feature points of each body part may be nodes constituting the skeleton movement model.
[0014] In one embodiment, the step of comparing the feature points for each body part and calculating the similarity may include the steps of: obtaining weights for each body part for the standard movement from a database; and applying the weights for each body part to the similarity calculation.
[0015] In one embodiment, the step of evaluating the health condition includes evaluating the user's level of physical exhaustion or cognitive impairment, and the level of cognitive impairment may be evaluated based on behavioral characteristic data that appears during cognitive impairment for each standard movement.
[0016] In one embodiment, the degree of physical exhaustion may be determined based on the sum of movement trajectories at feature points of each body part of a user skeleton motion model; and a weight of a physical exhaustion evaluation according to a previously defined movement amount of each body part.
[0017] In one embodiment, the method may include steps of receiving user action images for each of a plurality of users from a plurality of user devices; displaying the received plurality of user action images on a display device; calculating a similarity between each of a plurality of user actions in the received plurality of user action images and the standard action; and highlighting at least one of the plurality of user action images displayed on the display device based on the calculated similarity.
[0018] In one embodiment, the highlighting step may highlight a predetermined number of user action images, among the plurality of user action images, whose calculated similarity falls within a range lower or higher than a predetermined criterion.
[0019] In one embodiment, the highlighting step may include a step of comparing skeletal movement models of the multiple user actions to calculate a similarity between the users; a step of grouping the multiple user action images based on the calculated similarity between the users; and a step of displaying the grouped user action images in a distinguishable manner.
[0020] In an embodiment, the step of comparing skeletal movement models of the plurality of user movements to calculate the similarity between the users may not apply weights for each body part to the standard movements.
[0021] A program stored on a computer-readable recording medium according to an embodiment of the present specification may be configured with instructions for executing the above-described method. Effect of the Invention
[0022] The present invention can evaluate the user's motion simulation ability by comparing the user's motion with a standard motion.
[0023] The present specification can evaluate the degree of cognitive impairment or physical exhaustion of the user by comparing the user's movements with standard movements.
[0024] This specification uses skeleton models of user actions and standard actions to allow comparison of the two actions.
[0025] This specification makes it possible to make rapid clinical decisions by arranging the actions of multiple users and their analysis results on a single screen.
[0026] In the present specification, weights are applied to the main body parts for each standard action, so that the similarity between two actions can be determined more accurately.
[0027] The effects of this specification are not limited to the above-mentioned matters, but can be extended to various contents that can be derived from the detailed description of the embodiments of the invention below. [Brief description of the drawings]
[0028] [Figure 1] FIG. 1 illustrates an example of an operating environment of a system according to an embodiment of the present specification. [Diagram 2] FIG. 2 is a block diagram for explaining the internal configuration of a computing device 200 according to an embodiment of the present specification. [Diagram 3] 1 illustrates a system environment in which a health condition assessment method utilizing a skeleton model according to an embodiment of the present specification is performed. [Figure 4] 1 illustrates a scene in which a user assumes a posture according to a standard motion, according to an embodiment of the present disclosure. [Diagram 5] According to an embodiment of the present specification, the results of extracting skeleton models for the trainer 500 and the user 400 from standard action images and user action images are shown. [Figure 6a] FIG. 13 is a diagram showing an example of comparing a trainer skeleton motion model with a user skeleton motion model to calculate a similarity in this specification. [Figure 6b] FIG. 13 is a diagram showing an example of comparing a trainer skeleton motion model with a user skeleton motion model to calculate a similarity in this specification. [Figure 6c] FIG. 13 is a diagram showing an example of comparing a trainer skeleton motion model with a user skeleton motion model to calculate a similarity in this specification. [Figure 7] FIG. 11 is a diagram illustrating weights for each body part according to an embodiment of the present specification. [Figure 8] 1 illustrates the angle of a body part (pose) formed by a node according to one embodiment of the present disclosure. [Figure 9] 1 shows a plurality of user action images and their analysis results provided by the health management server 300 according to an embodiment of the present specification. [Figure 10] 13 illustrates an example of highlighting user action images with low similarity to a standard action according to an embodiment of the present specification. [Figure 11] 13 shows an example in which two or more related user action images are grouped and displayed according to an embodiment of the present specification. [Figure 12] 1 is a flowchart of a health condition assessment method utilizing a skeleton model according to an embodiment of the present specification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] In describing the embodiments of the present specification, if it is determined that a specific description of a known configuration or function may obscure the gist of the embodiments of the present specification, the detailed description will be omitted. In addition, in the drawings, parts that are not related to the description of the embodiments of the present specification are omitted, and similar parts are given similar reference numerals.
[0030] In the embodiments of the present specification, when a certain component is said to be "connected", "coupled" or "connected" to another component, this includes not only a direct connection relationship, but also an indirect connection relationship in which another component is interposed between them. Furthermore, when a certain component is said to "include" or "have" another component, this does not exclude the other component and means that the other component may also be included, unless otherwise specified.
[0031] In the embodiments of the present specification, the terms first, second, etc. are used only for the purpose of distinguishing a certain component from other components, and do not limit the order or importance between the components, unless otherwise specified. Therefore, within the scope of the embodiments of the present specification, a first component in an embodiment may be referred to as a second component in another embodiment, and similarly, a second component in an embodiment may be referred to as a first component in another embodiment.
[0032] In the embodiments of the present specification, components that are distinguished from one another are used to clearly describe the characteristics of each component, and do not necessarily mean that the components are separate. That is, multiple components may be integrated into one hardware or software unit, and one component may be distributed into multiple hardware or software units. Therefore, even if not otherwise stated, such integrated or distributed embodiments are also included in the scope of the embodiments of the present specification.
[0033] In this specification, the term "network" may refer to a concept including both wired and wireless networks. In this case, the term "network" may refer to a communication network capable of exchanging data between devices and systems, and between devices, and is not limited to a specific network.
[0034] The embodiments described herein may have all-hardware, part-hardware and part-software, or all-software aspects. A "unit," "device," or "system," etc., herein refers to a computer-related entity, such as hardware, a combination of hardware and software, or software. For example, a unit, module, device, or system, etc., herein may be, but is not limited to, a running process, a processor, an object, an executable, a thread of execution, a program, and / or a computer. For example, both an application running on a computer and a computer may all be referred to as a unit, module, device, or system, etc., herein.
[0035] In this specification, the device may be a mobile device such as a smartphone, a tablet PC, a wearable device, a head mounted display (HMD), etc., or a fixed device such as a PC or a home appliance with a display function. As an example, the device may be an in-car cluster or an Internet of Things (IoT) device. That is, the device in this specification may refer to a device capable of running an application, and is not limited to a specific type. Hereinafter, for convenience of explanation, a device on which an application runs is referred to as a device.
[0036] In this specification, the communication method of the network is not limited, and each component may not be connected by the same network method. The network may include not only a communication method using a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network, a satellite network, etc.), but also a short-range wireless communication between devices. For example, the network may include all communication methods that allow objects to network with each other, and is not limited to wired communication, wireless communication, 3G, 4G, 5G, or other methods.For example, the wired and / or network may be a LAN (Local Area Network), MAN (Metropolitan Area Network), GSM (Global System for Mobile Network), EDGE (Enhanced Data GSM Environment), HSDPA (High Speed Downlink Packet Access), W-CDMA (Wideband Code Division Multiple Access), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), Bluetooth, ZigBee, Wi-Fi, VoIP (Voice over Internet Protocol), LTE Advanced, IEEE 802.16m, WirelessMAN-Advanced, HSPA+, 3GPP (registered trademark) Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and MBWA (IEEE 802.20) systems, HIPERMAN, Beam-Division Multiple Access The term "radio-frequency communication" may refer to a communication network using one or more communication methods selected from the group consisting of BDMA, Wi-MAX (World Interoperability for Microwave Access), and ultrasonic communication, but is not limited to these.
[0037] The components described in the various embodiments are not necessarily essential components, and some may be optional components. Therefore, an embodiment consisting of a subset of the components described in the embodiments is also included in the scope of the embodiments of the present specification. Note that an embodiment including other components in addition to the components described in the various embodiments is also included in the scope of the embodiments of the present specification.
[0038] Hereinafter, the embodiments of the present specification will be described in detail with reference to the drawings.
[0039] Fig. 1 is a diagram showing an example of an operating environment of a system according to an embodiment of the present specification. Referring to Fig. 1, a user device 110 and one or more servers 120, 130, and 140 are connected through a network 1. Fig. 1 is an example for explaining the present specification, and the number of user devices and the number of servers are not limited to those shown in Fig. 1.
[0040] The user device 110 may be a fixed terminal or a mobile terminal realized by a computer system. Examples of the user device 110 include a smart phone, a mobile phone, a navigation system, a computer, a notebook, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, a smart ring, an IoT (internet of things) device, a VR (virtual reality) device, and an AR (augmented reality) device. As an example, in the embodiment, the user device 110 may mean one of various physical computer systems that can communicate with other servers 120 to 140 through the network 1 using a substantially wireless or wired communication method.
[0041] Each server may be realized by a computer device or multiple computer devices that communicate with the user device 110 through the network 1 and provide instructions, codes, files, contents, services, etc. For example, the server may be a system that provides each service to the user device 110 connected through the network 1. As a more specific example, the server may provide the user device 110 with a service (e.g., information provision, etc.) targeted by an application as a computer program installed and run on the user device 110. As another example, the server may distribute files for installing and running the above-mentioned application to the user device 110, receive user input information, and provide a corresponding service.
[0042] 2 is a block diagram illustrating an internal configuration of a computing device 200 according to an embodiment of the present specification. Such a computing device 200 may be applied to one or more of the user devices 110-1, 110-2, or servers 120-140 described above with reference to FIG. 1, and each device and server may have the same or similar internal configuration by adding or excluding some components.
[0043] Referring to FIG. 2, the computing device 200 may include a memory 210, a processor 220, a communication module 230, and a transceiver 240. The memory 210 is a non-transitory computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. Here, the non-disruptive mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the above-mentioned device or server as a separate permanent storage device separate from the memory 210. In addition, the memory 210 may store an operating system and at least one program code (for example, a browser installed and operated on the user device 110, etc., or a code for an application installed on the user device 110, etc. to provide a specific service, etc.). These software components may be loaded from a computer-readable recording medium separate from the memory 210. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc.
[0044] In another embodiment, the software components may be loaded into the memory 210 through the communication module 230 instead of through a computer-readable recording medium. For example, at least one program may be loaded into the memory 210 based on a computer program (e.g., the above-mentioned application) to be installed by a file provided through the network 1 by a developer or a file distribution system (e.g., the above-mentioned server) that distributes the installation files of the application.
[0045] The processor 220 may be configured to process computer program instructions by performing basic arithmetic, logical, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication module 230. For example, the processor 220 may be configured to execute instructions received by program code stored in a storage device, such as the memory 210.
[0046] The communication module 230 can provide functionality for the user device 110 and the servers 120-140 to communicate with each other over the network 1, and can provide functionality for each of the device 110 and / or the servers 120-140 to communicate with other electronic devices.
[0047] The transceiver 240 may be a means for interfacing with an external input / output device (not shown). For example, the external input device may include a keyboard, a mouse, a microphone, a camera, etc., and the external output device may include a display, a speaker, a haptic feedback device, etc.
[0048] Further, in other embodiments, the computing device 200 may include more components than those of FIG. 2 depending on the characteristics of the device to which it is applied. For example, when the computing device 200 is applied to the user device 110, it may be implemented to include at least some of the input / output devices described above, or may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, a database, etc. As a more specific example, when the user device is a smartphone, it may be implemented to further include various components that are generally included in smartphones, such as an acceleration sensor, a gyro sensor, a camera module, various physical buttons, a button using a touch panel, an input / output port, and a vibrator for vibration.
[0049] Fig. 3 shows a system environment in which a health condition assessment method using a skeleton model according to an embodiment of the present specification is performed. Referring to Fig. 3, a health management server 300 can communicate with one or more user devices 310-330, and the health management server 300 can also communicate with a medical institution server 301. The health management server 300, the user devices 310-330, and the medical institution server 301 in Fig. 3 can be realized as the computing devices described in Fig. 2. Furthermore, although three user devices are shown in Fig. 3, there may be fewer or more user devices, and the present invention is not limited to this.
[0050] In one embodiment, the health management server 300 may provide various standard motion images to the user device, generate health information based on the user's motion after receiving and analyzing the user motion images from the user device, and provide the health information to the user device or the medical institution server. Alternatively, the health management server 300 may first process or analyze the user motion images and provide them to the medical institution server 301, and may receive clinical judgment information from the medical institution server to additionally analyze the user motion images.
[0051] In one embodiment, the health management server 300 can provide a standard movement image in which the trainer performs the standard movement to the user device 310. In the following, one user device 310 will be described as an example, but the same procedure can be performed for other user devices.
[0052] 4 illustrates a scene in which a user assumes a posture according to a standard movement according to an embodiment of the present specification. Referring to FIG. 4, a user 400 can perform a standard movement using a user device 401 in response to a standard movement image 510 in which a trainer 500 performs a standard movement. Such an action of the user 400 can be captured by a camera 401c of the user device 401 and recorded as an image.
[0053] In one example, the image captured by the camera 401c may be in various forms, such as a color image, a depth image, a color depth image, etc. Thus, the camera 401c may include a depth camera or may be configured as a stereo camera, but is not limited thereto.
[0054] Such an image is referred to herein as a user action image 410. The user action image 410 may or may not be displayed on the display unit of the user device.
[0055] The playback of the standard action image and the capture of the user's action can be performed on the user device through an application or web service provided by the health management server 300. Therefore, the health management server can store not only the standard action image, but also the user action image and information before and after the user's action, and basic information about the user (such as the user's age, sex, underlying disease, and life pattern).
[0056] 4 shows the standard motion as a posture in which the hands are gathered together, the legs are spread apart, and the knees are bent, but various standard motions can be provided for evaluating the user's health condition. The health management server 300 can receive from the user device 310 a user motion image of the user imitating the standard motion.
[0057] Then, the health management server 300 can compare the standard action image with the user action image to calculate the similarity between the standard action and the user action. Then, the health condition of the user can be evaluated based on the calculated similarity. For example, the health management server 300 can compare the user action with the standard action to evaluate the user's motor ability, cognitive ability, physical strength, etc., but is not limited thereto.
[0058] In one embodiment, the health management server 300 can calculate the similarity by applying a skeleton model of an object in an image. Figure 5 shows a result of extracting a skeleton model for each of the trainer 500 and the user 400 from the standard movement image and the user movement image according to one embodiment of the present specification. Such a skeleton model may or may not be displayed on the display unit of the user device.
[0059] In one example, the health management server 300 can extract skeletal movement models of the trainer and the user from standard movement images or user movement images using a learning model that identifies the positions of human joints or other body parts based on the human body image. Such skeletal movement models can be composed of a number of nodes (feature points for each body part). The types of body parts to which nodes are assigned and the number of nodes can be set in various ways as needed, but preferably 24 nodes can be applied to each joint part. However, a smaller number of nodes is shown in the drawings to simply and clearly illustrate the features of the invention.
[0060] In this specification, the skeleton movement model may be arranged in a two-dimensional space or a three-dimensional space. This can be determined according to the type of image to be acquired. That is, since it is difficult for the viewpoint of the standard movement image and the viewpoint of the user movement image to exactly match, at least one of the standard movement image and the user movement image must be constructed with a three-dimensional skeleton movement model to match the viewpoint of the skeleton movement model of the other image. In this case, the health management server 300 can switch the viewpoint of the skeleton movement model acquired in three dimensions to the viewpoint of another skeleton movement model acquired in two dimensions to compare the two skeleton movement models.
[0061] Specifically, in one embodiment, the health management server 300 may perform a step of acquiring a trainer skeleton movement model for a trainer performing a standard movement from the standard movement image. Also, a step of acquiring a user skeleton movement model for a user performing a user movement from a user movement image may be performed. Thereafter, the health management server 300 may compare corresponding feature points of body parts of the trainer skeleton movement model and the user skeleton movement model to calculate the similarity. Here, the feature points of body parts may be nodes constituting the skeleton movement model.
[0062] In one example, the similarity is a Euclidean distance and can be determined based on a predetermined formula for the Euclidean distance. That is, the similarity can be measured by comparing the positions of the corresponding body part feature points of the user skeleton motion model and the trainer skeleton motion model. In this case, the similarity can be expressed as a percentage.
[0063] As an example, to compare two skeletal models, the distance between corresponding nodes can be calculated by projecting or superimposing the two skeletal models onto a plane.
[0064] 6A to 6B are diagrams showing an example of comparing a trainer skeleton motion model and a user skeleton motion model to calculate a similarity in this specification. Referring to FIG. 6A, the health management server 300 can calculate a similarity between the two models based on the positions of each node of the acquired trainer skeleton motion model 520 and the user skeleton motion model 420. Meanwhile, as described above, the viewpoint of the user motion image may be different from the viewpoint of the standard motion image. In this case, in order to compare the positions of the nodes of the skeleton motion models arranged on a two-dimensional plane, it is necessary to match the two skeleton models. Hereinafter, the transformation of the user skeleton motion model based on the trainer skeleton motion model will be described, but conversely, the trainer skeleton motion model can also be transformed to match the user skeleton motion model.
[0065] 6B, the health management server 300 can rotate the user skeleton motion model 420 and convert it to the same time as the trainer skeleton motion model. To enable such rotation, the user skeleton motion model may need to be configured as a three-dimensional model.
[0066] For example, the physical states of the trainer and the user may be different from each other. In this case, if two skeleton models are compared as they are, there is a risk that the results will be highly similar regardless of whether the postures match. To solve this problem, it is necessary to make the sizes of the two skeleton models the same.
[0067] Referring to Fig. 6C, the health management server 300 can normalize the size of the user skeleton motion model 420 to make it correspond to the trainer skeleton motion model 520. Specifically, since Fig. 6C corresponds to the case where the user's feet are longer, the distance between the user's ankles and knees can be adjusted to be equal to the trainer's physical condition. This allows the distance between the knees and ankles of each leg to be shortened by d1 and d2.
[0068] The normalization process described in Figures 6A to 6C can be applied to either or both of the two skeleton models, and when applied to both, they can be transformed to conform to the ratio or length values of the reference model.
[0069] FIG. 7 is a diagram for explaining weights for each body part according to an embodiment of the present specification. Referring to FIG. 7, in one example, when the health management server 300 compares the feature points for each body part to calculate the similarity, it can obtain the weights for each body part for the standard motion from a database and apply the weights for each body part according to the type of motion to the similarity calculation. That is, depending on the type of standard motion, there may be body parts with high importance and parts with low importance. Referring to FIG. 7, the standard motion may have high importance of the positions and angles of the neck and waist node and both knee nodes (identification number 710), and low importance of the positions and angles of both wrists and wrists (identification number 720). Based on such differences in importance, it can be used to calculate the similarity by reflecting the weights for the body parts for each standard motion.
[0070] Table 1 shows weight values for body parts for each standard motion, where node n indicates each body part. For example, node 3 may be the right elbow and node 5 may be the right shoulder, but this is for illustrative purposes only. Note that the weights 3, 2, and 1 in Table 1 are merely for illustrative purposes.
[0071] [Table 1]
[0072] Such weights for each body part are given for each standard movement, but can also be subdivided for each cognitive impairment. For example, weights can be given to nodes 8 and 9 in order to evaluate cognitive impairment A for standard movement 4, while weights can be given to nodes 10 and 12 in standard movement 4 in order to evaluate cognitive impairment B. The similarity described above in this specification can be determined not only by the distance between corresponding nodes, but also by using angles formed by three or more nodes. Referring to FIG. 8, the values of the angle a1 between two legs, the bending angle a2 of the right leg, and the bending angle a3 of the left leg in the user skeleton movement model can be compared with the angles of the corresponding parts of the trainer skeleton movement model.
[0073] In addition, since the skeleton movement model has a time element, the similarity can be calculated by further considering the time element. For example, when a difference of 2 cm is maintained for 1 second and a difference of 0.5 cm is maintained for 5 seconds, the latter can be calculated to have a lower similarity value. As an example, the similarity can be determined by applying weights to the product of the distance difference and the time or to each of these values. As another example, the similarity can be determined by further reflecting the direction value in addition to the distance difference. The similarity can also be determined by reflecting the angle between the rotation direction of the node in the basic posture and the rotation direction of the actual node.
[0074] In one embodiment, the health management server 300 may evaluate the health condition of the user based on the above-described similarity calculation. Here, the evaluation of the health condition may be an evaluation of the user's level of physical exhaustion or cognitive impairment.
[0075] In one example, the cognitive impairment level may be determined based on the calculated level of similarity. That is, if the similarity to a specific standard posture is lower than a predetermined value, the probability of having a cognitive impairment corresponding to the specific standard posture may be calculated. Alternatively, the position-movement values of the nodes according to the cognitive impairment level for each specific standard posture may be defined, and the probability of having a cognitive impairment and the degree of the impairment may be determined by comparing with these defined values.
[0076] More specifically, this can be evaluated based on behavioral characteristic data that appears when a person has cognitive impairment for each standard action. That is, if a person cannot imitate a standard action to a certain level, a probability of cognitive impairment can be given. For more accurate judgment, behavioral characteristic data of a person with cognitive impairment for each standard action can also be used.
[0077] For example, the behavioral characteristic data may be data representing the behavioral characteristics of a patient as node data for each standard action and type of cognitive impairment, as shown in Table 2. The weights of each body part for the standard action may be compiled into a database and stored in the health management server 300 or other external device.
[0078] [Table 2]
[0079] The percentage of similarity may be determined by an evaluator's input, or may be determined based on a predefined standard, or may be continuously updated through an artificial intelligence learning model. As an example, the health management server 300 may evaluate the degree of physical exhaustion of the user when evaluating the user's health condition. Specifically, the degree of physical exhaustion is used to grasp the increase in the amount of exercise due to unnecessary movements or postures when performing standard movements.
[0080] For example, the degree of physical exhaustion may be determined based on the sum of the movement trajectories at the feature points of each body part of the user skeleton motion model and the weight of the physical exhaustion evaluation according to the movement amount of each body part defined previously. That is, when the user rotates both arms while imitating a standard movement of sitting on the feet, the degree of physical exhaustion may be calculated by considering the movement trajectories of the nodes of both arms. In another example, a weight according to the movement amount of each body part may be given. Specifically, the weight according to the movement amount of each body part is such that, since the movement of raising the legs consumes more energy among raising the arms and raising the legs, a larger weight of physical exhaustion is given to the act of raising the legs. In addition, since the posture of thrusting the thighs forward and the posture of thrusting the thighs backward consumes more energy, the weight of physical exhaustion according to the body part described above may be used to reflect such a difference.
[0081] 3, the health management server 300 can receive user action images from a plurality of user devices. The received user action images may be real-time images or images taken at different times.
[0082] The health management server 300 can display the received user motion images on a display device. Here, the display device can be a user device or a display device of the medical institution server 301. The health management server 300 can provide the images to a user or a medical institution through a web or an app. Hereinafter, a case will be described in which the health management server 300 provides a plurality of user motion images and a standard motion image to the medical institution server 301. More specifically, an example will be described in which the results of processing and analyzing each image are provided to the medical institution server together with the image. Based on the information provided in this way, medical personnel can check and diagnose the motion images of the multiple users.
[0083] Fig. 9 illustrates a plurality of user action images and their analysis results provided by the health management server 300 according to an embodiment of the present specification. Referring to Fig. 9, a standard action image 510 is displayed, and together with this, a plurality of user action images 901 to 905 are displayed on the display device 90D.
[0084] The health management server 300 can calculate the similarity between the plurality of user actions in the plurality of received user action images and the standard action, respectively. This step has been described in detail above. Through the similarity calculation and the evaluation based on the similarity, it is possible to display the similarity (accuracy) 910 between each user action and the standard action, display the presence, type and probability of cognitive impairment 920, and display the degree of physical exhaustion 930. A medical specialist can check and compare the actions of a plurality of patients through these screens.
[0085] FIG. 10 shows an example of highlighting user action images with low similarity to a standard action in one embodiment of the present specification.
[0086] In one example, the health management server 300 may highlight at least one of the plurality of user action images displayed on the display device based on the calculated similarity. Specifically, as shown in FIG. 10, the health management server 300 may highlight a predetermined number of user action images h1 whose calculated similarity is lower or higher than a predetermined standard among the plurality of user action images. This allows a specialist to quickly identify users who are not good at imitating standard actions. Specifically, a specialist can visually check the general physical behavior of the highlighted user to make a quick clinical judgment. Without such a guide, there is the inconvenience of having to check the entire playback time of each and every image.
[0087] FIG. 11 illustrates an example of grouping and displaying two or more related user action images according to an embodiment of the present specification. In one example, the health management server 300 can compare the skeleton action models of the multiple user actions to calculate the similarity between the users, and group the multiple user action images based on the calculated similarity between the users. Thereafter, the grouped user action images can be displayed in a distinguished manner. That is, the user action images that are recognized in the data as having the same cognitive impairment can be grouped and displayed. In FIG. 11, the highlight is displayed with a dotted box, but this is an example, and various visual effects such as displaying a label, changing shading, and making the image size different may be included in the above-mentioned "highlight". In addition, in one example, the health management server 300 does not need to apply weights for each body part to the standard action when comparing skeleton action models of multiple user actions to calculate the similarity between users. When weights for each body part are applied to the standard action, there is an advantage that the cognitive impairment that is matched with the standard action can be quickly and accurately identified. However, in order to discover unconfirmed disorders or user behavior patterns, it may be useful not to apply such weights. For example, by not applying the weights for each body part as described above, a user group with a pattern of slightly bending the neck backward when imitating standard movement A is discovered, and a specialist can make a visual evaluation of these and make a clinical judgment. For example, this allows a conclusion that the user has cognitive impairment C to be made by clinical judgment.
[0088] FIG. 12 is a flowchart of a health condition evaluation method using a skeleton model according to an embodiment of the present specification. Such an evaluation method can be executed by the above-mentioned health management server. Referring to FIG. 12, the health condition evaluation method using the skeleton model can include a step of providing a standard action image in which a trainer performs a standard action to a user device (S100), a step of receiving a user action image for a user action in which a user imitates the standard action from the user device (S200), a step of comparing the standard action image with the user action image to calculate a similarity between the standard action and the user action (S300), and a step of evaluating the health condition of the user based on the calculated similarity (S400). Here, the step of calculating the similarity (S300) includes a step of acquiring a trainer skeleton movement model for a trainer performing a standard movement from the standard movement image, a step of acquiring a user skeleton movement model for a user performing a user movement from the user movement image, and a step of calculating the similarity by comparing corresponding body part feature points of the trainer skeleton movement model and the user skeleton movement model, and the body part feature points may be nodes constituting the skeleton movement model. More specifically, the step of calculating the similarity by comparing the body part feature points may include a step of acquiring a body part weight for the standard movement from a database, and a step of applying the body part weight to the similarity calculation. The step of evaluating the health state evaluates the user's level of physical exhaustion or cognitive impairment, and the cognitive impairment level may be evaluated based on behavioral feature data appearing during cognitive impairment for each standard movement.
[0089] The above-described embodiments may be at least partially realized by a computer program and recorded on a computer-readable recording medium. The computer-readable recording medium on which the program for implementing the embodiments is recorded includes all kinds of recording devices in which computer-readable data is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, optical data storage device, etc. The computer-readable recording medium may also be distributed among computer systems connected via a network, and computer-readable code may be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the present embodiments may be easily understood by those of ordinary skill in the art to which this specification pertains.
[0090] The above-mentioned specification has been described with reference to the embodiment shown in the drawings, but this is merely an example, and a person having ordinary skill in the art can understand that various modifications and equivalent other embodiments are possible. However, such modifications should be considered to be within the technical scope of protection of the specification. Therefore, the true technical scope of protection of the specification is determined by the technical ideas of the appended claims. [Industrial Applicability]
[0091] This specification can evaluate the user's motion simulation ability by comparing the user's motion with a standard motion.This specification can evaluate the user's cognitive impairment or physical exhaustion level by comparing the user's motion with a standard motion.This specification can compare the two motions using skeleton models of the user's motion and the standard motion.
Claims
1. A health condition evaluation method using a skeleton model, the method being carried out by a health management server, comprising: Providing a standard action image of a trainer performing a standard action to the user device; receiving, from a user device, a user action image corresponding to a user action in which the user imitates the standard action; A step of comparing the standard action image with the user action image to calculate a similarity between the standard action and the user action; and and evaluating the health condition of the user based on the calculated similarity. A health condition evaluation method utilizing a skeleton model, comprising:
2. The step of calculating the similarity includes: obtaining a trainer skeleton action model for a trainer performing a standard action from the standard action image; obtaining a user skeleton action model for a user performing a user action from the user action image; and The method includes a step of calculating the degree of similarity by comparing corresponding feature points of body parts of the trainer skeleton movement model and the user skeleton movement model, the feature points of body parts being nodes constituting the skeleton movement model. A health condition evaluation method utilizing the skeleton model according to claim 1.
3. The step of comparing the feature points of each body part and calculating the similarity includes: Obtaining weights for each body part for the standard movement from a database; and applying the weights for each body part to the similarity calculation. A health condition evaluation method utilizing the skeleton model according to claim 2.
4. The step of assessing the health status includes: The method evaluates the user's level of physical exhaustion or cognitive impairment, The cognitive impairment level is evaluated based on behavioral characteristic data that appears during cognitive impairment for each standard movement. A health condition evaluation method utilizing the skeleton model according to claim 2.
5. The degree of physical exhaustion is determined based on the sum of the movement trajectories at the feature points of each body part of the user skeleton motion model; and the weight of the physical exhaustion evaluation according to the movement amount of each body part that has already been defined. A health condition evaluation method utilizing the skeleton model according to claim 4.
6. receiving user action images for each of a plurality of users from a plurality of user devices; displaying the received plurality of user action images on a display device; Calculating a similarity between each of a plurality of user actions in the plurality of received user action images and the standard action; and highlighting at least one of the plurality of user action images displayed on the display device based on the calculated similarity. A health condition evaluation method utilizing the skeleton model according to claim 2 or 3.
7. The highlighting step includes: A predetermined number of user action images, among the plurality of user action images, whose calculated similarity falls within a range lower or higher than a predetermined standard, are highlighted. A health condition evaluation method utilizing the skeleton model according to claim 6.
8. The highlighting step includes: A step of comparing skeleton motion models of the plurality of user motions to calculate a similarity between the users; Grouping a plurality of user action images based on the calculated similarity between users; and A step of displaying the grouped user action images in a distinctive manner is included. A health condition evaluation method utilizing the skeleton model according to claim 6.
9. The step of comparing the skeleton motion models of the plurality of user motions to calculate the similarity between the users does not apply weights for each body part to the standard motion. A health condition evaluation method utilizing the skeleton model according to claim 8.
10. A method for performing the method according to any one of claims 1 to 5, which is stored on a computer-readable recording medium in combination with hardware. A computer program comprising:
Citation Information
Patent Citations
Fitness coaching system using personalized augmented reality technology
KR101970687B1
Method and computer program for consulting fitness, and consultant device
KR1020160116677A
Training device and method for improving cognitive response
KR102123869B1
Balloon flower root functional beverage and process for preparation thereof
KR102227008B1
Apparatus and method for measuring body motion similarity
KR102292415B1
