Method and system for providing exercise therapy using artificial intelligence posture estimation model and motion analysis model

Abstract

The present invention relates to a method and system for analyzing the motion of a patient. The present invention also relates to a method and system for performing motion posture estimation and motion analysis.

Description

[Technical field]

[0001] The present invention relates to a method and system for providing exercise therapy to patients using an artificial intelligence posture estimation model and an artificial intelligence motion analysis model specialized for musculoskeletal disorders. [Background technology]

[0002] Musculoskeletal disorders refer to pain or injury that occurs in the musculoskeletal system, including muscles, nerves, tendons, ligaments, bones, and surrounding tissues. Musculoskeletal disorders can appear in various parts of the body, including the neck, lower back, arms, and legs.

[0003] According to a report by the World Health Organization (WHO), the economic loss due to musculoskeletal disorders is the fourth highest of all diseases, and musculoskeletal disorders are chronic pain conditions that affect not only daily life but also economic activity.

[0004] On the other hand, the treatment of musculoskeletal disorders should, in principle, be started with minimally invasive treatments, and non-pharmacological conservative treatments (e.g., exercise therapy and education, cognitive therapy, or relaxation therapy) should be performed first, followed by drug therapy and surgical treatment.

[0005] Treatment guidelines actively recommend non-pharmacological conservative treatment of musculoskeletal diseases, and research into methods of non-pharmacological conservative treatment of musculoskeletal diseases is being actively conducted, mainly in the United States and Europe.

[0006] On the other hand, with the development of technology, the use of electronic devices (e.g., smartphones, tablet PCs, etc.) has become widespread, which has led to a gradual increase in dependency on the Internet in many aspects of daily life.

[0007] In this way, with the development of various technologies including the Internet, consumption patterns that were previously highly dependent on offline consumption have gradually shifted to online consumption, and online-centered consumption is currently increasing dramatically.

[0008] In line with these changing trends, industries such as the medical industry, which are based on offline services due to the nature of their services, are increasingly moving to provide medical services online.

[0009] Therefore, nowadays, various medical services are provided online, allowing patients, or users, to consult with medical staff about their illnesses with just a few clicks via electronic devices connected to the Internet.

[0010] As an example of such technology, Korean Patent Registration No. 10-2195512 discloses technology relating to a server and system that provides an online medical platform, and provides patients with information regarding locations where medical services are provided online.

[0011] In line with these trends, there is a need for methods to deliver non-pharmacological conservative treatment of musculoskeletal disorders online. Summary of the Invention [Problem to be solved by the invention]

[0012] The present invention relates to a method and system for providing exercise therapy that can provide exercise therapy for musculoskeletal disorders online.

[0013] In particular, the present invention relates to a method and system for providing exercise therapy that can analyze the exercise motion of a patient performing prescribed exercise based on an exercise video.

[0014] In particular, the present invention relates to a method and system for providing exercise therapy that can analyze a patient's exercise movements from exercise video based on an artificial intelligence model specialized for musculoskeletal disorders.

[0015] Furthermore, the present invention relates to a method and system for providing exercise therapy that can provide a user environment that allows patients to easily access treatment for musculoskeletal disorders.

[0016] Meanwhile, the present invention relates to a method and system for estimating exercise posture and analyzing motion, which can provide rehabilitation therapy on an online basis.

[0017] More specifically, the present invention relates to a method and system for estimating exercise posture and analyzing motion, which can analyze postures related to a user's exercise motion for rehabilitation based on an exercise video.

[0018] In particular, the present invention relates to a method and system for motion posture estimation and motion analysis that enables user-customized motion posture estimation.

[0019] Furthermore, the present invention relates to a method and system for motion posture estimation and motion analysis that can construct a database (DB) used for motion posture estimation. [Means for solving the problem]

[0020] In order to solve the above problems, the method of providing exercise therapy according to the present invention may include the steps of receiving prescription information regarding exercise for a patient from a doctor terminal, assigning an exercise plan including at least one prescribed exercise to the patient account based on the prescription information, receiving an exercise video of the exercise corresponding to the prescribed exercise from the patient terminal, extracting key points corresponding to a plurality of pre-set joint points from the exercise video using an artificial intelligence posture estimation model, analyzing the relative positional relationship between the key points through an artificial intelligence motion analysis model, and analyzing the patient's exercise motion for the prescribed exercise based on the analysis of the positional relationship, and transmitting the analysis result of the patient's exercise motion to the patient terminal.

[0021] Furthermore, the exercise treatment provision estimation method according to the present invention may further include a step of outputting the exercise video to the patient terminal in real time in conjunction with the exercise video being captured by the patient terminal, and a step of providing a graphic object corresponding to the extracted key point by superimposing it on an area in the exercise video where a subject corresponding to the patient is located, so that the patient can recognize the joint points at which analysis is performed on the patient's exercise movement.

[0022] In one example, the step of extracting the key points may include identifying visible joint points of the subject that are visible in the motion image from among the plurality of pre-set joint points, and extracting the identified visible joint points as the key points.

[0023] In one example, the motion analysis model may predict invisible joint points of the subject that are not visible in the motion video among the plurality of pre-set joint points based on the learning data, and analyze the motion of the patient based on the visible joint points and the invisible joint points.

[0024] In this case, the key points may include key points corresponding to the visible joint points and the invisible joint points.

[0025] In one example, the learning data may include a first data group in which learning target visible joint points of a subject included in a learning target video and position information of learning target invisible joint points estimated based on the visible joint points are listed in order, and a second data group including data values ​​indicating whether each of the learning target visible joint points and the learning target invisible joint points is visible or not.

[0026] In this case, the enumeration order of the data values ​​included in the second data group may be the same as the enumeration order of the visible joint points to be trained and the invisible joint points to be trained.

[0027] In one example, the step of analyzing the patient's exercise motion may include analyzing the relative positional relationship between the key points based on rule information regarding the prescribed exercise, and determining whether the relative positional relationship between the key points satisfies the rule information, thereby analyzing the patient's exercise motion.

[0028] In one example, visual appearances of graphic objects overlaid on the motion image may differ depending on whether the relative positional relationship between the extracted key points satisfies the rule information.

[0029] In one example, the analysis results for the patient's exercise movements may include a first analysis result that provides the graphic objects corresponding to the key points superimposed on the exercise video in real time with different visual appearances based on the rule information while the exercise video is being shot on the patient terminal, and a second analysis result including the patient's evaluation score for the prescribed exercise based on key points extracted from each of a plurality of frames constituting the exercise video.

[0030] In this case, the first analysis result may be generated by a behavior analysis model of an application installed on the patient terminal, the second analysis result may be generated by a cloud server linked to the application, and both the first analysis result and the second analysis result may be transmitted to the doctor terminal. Furthermore, the exercise treatment providing system according to the present invention includes a communication unit that receives prescription information regarding exercise for a patient from a doctor terminal, and a control unit that assigns an exercise plan including at least one prescribed exercise to the patient account based on the prescription information, wherein the control unit receives an exercise video of the exercise corresponding to the prescribed exercise from the patient terminal, extracts key points corresponding to a plurality of pre-set joint points from the exercise video, analyzes the relative positional relationship between the key points through an artificial intelligence motion analysis model, analyzes the patient's exercise motion for the prescribed exercise based on the analysis of the positional relationship, and transmits the analysis result of the patient's exercise motion to the patient terminal.

[0031] Furthermore, the exercise treatment provision system of the present invention includes a communication unit that receives prescription information regarding exercise for a patient from a doctor terminal, and a control unit that assigns an exercise plan including at least one prescribed exercise to the patient account based on the prescription information, and the control unit receives exercise video of the exercise corresponding to the prescribed exercise from the patient terminal, analyzes the patient's exercise movements for the prescribed exercise from the exercise video using an artificial intelligence motion analysis model, and transmits the analysis results of the patient's exercise movements to the patient terminal.

[0032] Furthermore, a program executed by one or more processes in an electronic device and stored in a computer-readable recording medium may include instructions to perform the following steps: receiving prescription information regarding exercise for a patient from a doctor terminal; allocating an exercise plan including at least one prescribed exercise to the patient account based on the prescription information; receiving an exercise video of the exercise corresponding to the prescribed exercise from the patient terminal; extracting key points corresponding to a plurality of pre-set joint points from the exercise video; analyzing a relative positional relationship between the key points through an artificial intelligence motion analysis model, and analyzing the patient's exercise motion for the prescribed exercise based on the analysis of the positional relationship; and transmitting an analysis result of the patient's exercise motion to the patient terminal.

[0033] In order to solve the above problem, the motion posture estimation method of the present invention includes the steps of receiving a motion video from a user terminal, analyzing a motion movement related to a specific motion movement of a user included in the motion video based on posture estimation information extracted from a posture estimation model trained using a training dataset 400 (data set) including position information of joint points, and providing the user's motion movement analysis result related to the specific motion to the user terminal based on completion of the analysis, wherein the position information of the joint points included in the training dataset is position information of each of a plurality of pre-specified training target joint points among the joint points of a subject included in the training target motion video, and the training dataset may be composed of data extracted from the training target motion video.

[0034] In one example, the learning data set may be composed of a plurality of data groups each corresponding to a different information attribute, a first data group among the plurality of data groups includes position information of each of the plurality of training target joint points, and the position information included in the first data group may include coordinate information of each of the pre-specified plurality of training target joint points in the training target motion video in a form that is paired with each other.

[0035] In an example, the position information included in the first data group may be defined as different types of information depending on whether the plurality of training target joint points are visible in the training image, and the type of the position information may be defined according to a data value of data included in a second data group different from the first data group.

[0036] In one example, the pose estimation model may set different learning weights for each position information of the plurality of training target joint points included in the first data group based on the data values ​​included in the second data group.

[0037] In one example, the first data group may be arranged such that the position information of each of the plurality of training joint points is sequentially arranged in the first data group based on a predefined order among the plurality of training joint points, and the data values ​​included in the second data group may be arranged in the same order as the predefined order in which the position information of each of the plurality of training joint points is arranged, so as to indicate whether each of the plurality of training joint points is visible or not.

[0038] In one example, the learning data set may further include a third data group including data values ​​related to a shooting direction of the object, and the data values ​​included in the third data group may be configured to have different data values ​​depending on a shooting direction of the object relative to a camera that captured the object.

[0039] In this case, the posture estimation model is configured to be learned taking into account the shooting direction of the subject through the learning dataset having different data values ​​depending on the shooting direction of the subject, and the user's exercise motion analysis result may be a result of analyzing a specific exercise motion of the user based on posture estimation information extracted in the posture estimation model taking into account the shooting direction of the user included in the exercise video.

[0040] In one example, the exercise video received from the user terminal and the training target exercise video correspond to the specific exercise movement having the same exercise code, and the learning dataset may include a fourth data group including an exercise code matched to the specific exercise movement performed by the subject in the training target exercise video.

[0041] Furthermore, the training subject motion video may be a video, and the training data set may be composed of training data extracted from each of reference frames selected based on a preset criterion from among a plurality of frames constituting the training subject motion video, with a focus on the subject included in the training subject motion video.

[0042] The motion posture estimation system according to the present invention includes a posture estimation model that extracts posture estimation information using a learning data set including position information of joint points, a motion analysis model that analyzes a motion movement related to a specific motion movement of a user included in a motion video to be analyzed using the posture estimation information, and a service server that provides a motion movement analysis result of the user related to the specific motion movement to the user terminal based on completion of the analysis, wherein the position information of the joint points included in the learning data set is position information of each of a plurality of pre-specified learning target joint points among the joint points of a subject included in the learning target motion video, and the learning data set may be composed of data extracted from the learning target motion video.

[0043] Further, a program executed by one or more processes in an electronic device and stored in a computer-readable recording medium includes instructions for performing the steps of: receiving a motion video from a user terminal; analyzing a motion movement related to a specific motion movement of a user included in the motion video based on posture estimation information extracted from a posture estimation model trained using a learning dataset including position information of joint points; and providing a result of the motion movement analysis of the user related to the specific motion movement to the user terminal based on completion of the analysis. The position information of the joint points included in the learning dataset may be position information of each of a plurality of pre-specified learning target joint points among joint points of a subject included in a training target motion video, and the training dataset may be composed of data extracted from the training target motion video. Effect of the Invention

[0044] As described above, the system and method for providing exercise therapy according to the present invention can receive prescription information on exercises for a patient from a doctor terminal, and assign an exercise plan including at least one prescribed exercise to the patient account based on the prescription information. This allows the doctor to prescribe to the patient even if the doctor and the patient do not meet in person for exercise therapy for musculoskeletal disorders, and the patient receives an exercise plan according to the doctor's prescription, thereby eliminating spatial, time, and economic constraints on exercise therapy and improving accessibility to exercise therapy.

[0045] Furthermore, the system and method for providing exercise therapy according to the present invention can analyze the user's exercise movements by extracting key points corresponding to a plurality of pre-set joint points from the exercise video, and focusing on the joints required for exercise therapy of musculoskeletal diseases.

[0046] Furthermore, the system and method for providing exercise therapy according to the present invention can analyze exercise movements related to a specific exercise movement of a user included in an exercise video based on a posture estimation model trained using a training data set including position information of joint points. As a result, the present invention can accurately analyze the posture of a patient from an exercise video, and in particular, obtain information on the range of motion of a patient's joints, alignment state, and disengagement state, thereby improving the quality of medical services.

[0047] Furthermore, the system and method for providing exercise therapy according to the present invention transmits analysis results of the patient's exercise movements to the patient terminal, so that the patient can receive feedback on the exercise video without having to go directly to a hospital located far away, thereby improving the effectiveness of the exercise therapy and improving the patient's compliance with exercise.

[0048] As described above, the motion posture estimation and motion analysis method and system according to the present invention can analyze a motion related to a specific motion of a user included in a motion video based on a posture estimation model trained using a training data set including position information of joint points. As a result, the present invention can accurately analyze a user's motion from a motion video, and in particular, can improve the quality of medical services by acquiring information on the user's joint range of motion, alignment state, and separation state.

[0049] Furthermore, the exercise posture estimation method of the present invention provides the user's exercise motion analysis results related to a specific exercise motion to the user terminal, allowing patients to receive rehabilitation treatment without having to go directly to a hospital located far away, making it easy to access rehabilitation treatment, and medical staff can easily monitor the patient's rehabilitation exercise through electronic devices and provide feedback based on the monitoring, thereby improving the effect of the patient's exercise treatment. [Brief description of the drawings]

[0050] [Figure 1] 1 is a conceptual diagram for explaining an exercise therapy providing system according to the present invention. [Diagram 2] 1 is a flowchart for explaining a method for providing exercise therapy according to the present invention. [Diagram 3] 1 is a flowchart for explaining a method for providing exercise therapy according to the present invention. [Figure 4a] FIG. 1 is a conceptual diagram for explaining a doctor's prescription. [Figure 4b] FIG. 1 is a conceptual diagram for explaining a doctor's prescription. [Diagram 5] 1 is a conceptual diagram for explaining a method for analyzing a patient's exercise motion from exercise video. [Figure 6] 1 is a conceptual diagram for explaining a method for analyzing a patient's exercise motion from exercise video. [Figure 7]FIG. 1 is a conceptual diagram for explaining an artificial intelligence posture estimation model. [Figure 8a] FIG. 1 is a conceptual diagram for explaining an artificial intelligence posture estimation model. [Figure 8b] FIG. 1 is a conceptual diagram for explaining an artificial intelligence posture estimation model. [Figure 8c] FIG. 1 is a conceptual diagram for explaining an artificial intelligence posture estimation model. [Figure 8d] FIG. 1 is a conceptual diagram for explaining an artificial intelligence posture estimation model. [Figure 8e] FIG. 1 is a conceptual diagram for explaining an artificial intelligence posture estimation model. [Figure 8f] FIG. 1 is a conceptual diagram for explaining an artificial intelligence posture estimation model. [Figure 9a] 1 is a conceptual diagram for explaining a user environment in which the results of a patient's exercise motion analysis are provided. FIG. [Figure 9b] 1 is a conceptual diagram for explaining a user environment in which the results of a patient's exercise motion analysis are provided. FIG. [Figure 9c] 1 is a conceptual diagram for explaining a user environment in which the results of a patient's exercise motion analysis are provided. FIG. [Figure 10] 1 is a conceptual diagram for explaining an exercise posture estimation and motion analysis system according to the present invention; [Figure 11] 2 is a flowchart illustrating a motion posture estimation and motion analysis method according to the present invention. [Figure 12] FIG. 13 is a conceptual diagram for explaining an example of use for providing a result of a user's behavior analysis. [Figure 13] FIG. 13 is a conceptual diagram for explaining an example of use for providing a result of a user's behavior analysis. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings. The same reference numbers are used for the same or similar components regardless of the drawing numbers, and the duplicated description thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed for the sake of ease of specification writing only, and do not have any meaning or role that is different from each other. In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of related known technology may obscure the gist of the embodiments disclosed in this specification, the detailed description will be omitted. In addition, the accompanying drawings are provided for easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the accompanying drawings, and it should be understood that the accompanying drawings include all modifications, equivalents, and alternatives included in the idea and technical scope of the present invention.

[0052] Terms including ordinal numbers such as first, second, etc. may be used to describe various components, but the components are not limited by the terms. The terms are used only to distinguish one component from another.

[0053] When an element is described as being "coupled" or "connected" to another element, it should be understood that it may be directly coupled or connected to the other element, but that there may be other elements between them. On the other hand, when an element is described as being "directly coupled" or "directly connected" to another element, it should be understood that there are no other elements between them.

[0054] The singular expression includes the plural expression unless the context clearly dictates otherwise.

[0055] In this application, terms such as "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof set forth in the specification, but are to be understood as not precluding the possible presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0056] The present invention relates to a method for analyzing a patient's exercise motion contained in an exercise video based on the exercise video received from a patient's terminal, and providing an analysis result. In particular, the present invention relates to a method for analyzing an exercise motion based on a patient's joint points using an artificial intelligence model specialized for musculoskeletal disorders.

[0057] In the present invention, the analysis of the motion of rehabilitation exercise for musculoskeletal disorders will be mainly described, but is not necessarily limited thereto. That is, the motion analysis in the present invention may include not only the analysis of the motion of exercise, but also the analysis of various motions such as motions in daily life and motions during stretching.

[0058] On the other hand, the "exercise action" described in this invention is a gesture (movement) performed in the process of exercising, and may be used interchangeably with terms such as bodily "movement," "action," "movement," and "gesture."

[0059] Moreover, the "exercise video" is a video (image or video) that captures (includes) the process of the patient performing an exercise movement, as shown in FIG. 6, and includes at least a part of the patient U's body.

[0060] In the present invention, a patient object included in an exercise video is named and described as "subject U." In the present invention, "subject U" may refer to a patient exercising in an exercise video or a part of the patient's body. In the present invention, "subject" and "patient" may be used interchangeably, and are described with the same reference numeral "U."

[0061] Hereinafter, the method and system for providing exercise therapy using an artificial intelligence posture estimation model and a motion analysis model according to the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a conceptual diagram for explaining the system for providing exercise therapy according to the present invention. FIG. 2 and FIG. 3 are flow charts for explaining the method for providing exercise therapy according to the present invention, FIG. 4a and FIG. 4b are conceptual diagrams for explaining a doctor's prescription, FIG. 5 and FIG. 6 are conceptual diagrams for explaining a method for analyzing a patient's exercise motion from an exercise video, FIG. 7, FIG. 8a, FIG. 8b, FIG. 8c, FIG. 8d, FIG. 8e, and FIG. 8f are conceptual diagrams for explaining an artificial intelligence posture estimation model, and FIG. 9a, FIG. 9b, and FIG. 9c are conceptual diagrams for explaining a user environment in which a patient's exercise motion analysis result is provided.

[0062] As shown in Fig. 1, the exercise treatment providing system 1000 according to the present invention uses an artificial intelligence posture estimation and motion analysis model to analyze the exercise motion of a patient in an exercise video received from a patient terminal 10, and may be configured to include at least one of an application 100 installed in the patient terminal 10 and an artificial intelligence server (or cloud server) 200. In addition, the exercise treatment providing system 1000 according to the present invention may include a posture estimation model and a motion analysis model trained using training data.

[0063] The application 100 according to the present invention is installed in the patient terminal 10, and can perform a function of analyzing the exercise motion of a patient U having a musculoskeletal disease and providing feedback information based on the analysis result. Therefore, the application 100 according to the present invention may be named a "digital exercise treatment solution", a "digital rehabilitation treatment solution", a "digital exercise evaluation solution", a "non-face-to-face exercise treatment solution", a "non-face-to-face rehabilitation treatment solution", a "non-face-to-face exercise evaluation solution", a "mobile exercise treatment program", a "mobile rehabilitation treatment program", a "mobile exercise evaluation program", and a "Mobile Orthopedic Rehabilitation Assistant (MORA)".

[0064] The application 100 according to the present invention is installed in a patient terminal 10 and can play a role in connecting a patient U with a musculoskeletal disease with an orthopedic surgeon D and supporting the rehabilitation of the patient U. For the sake of convenience, the application 100 installed in the patient terminal 10 will be described below as an "exercise therapy application."

[0065] Meanwhile, the exercise therapy application 100 according to the present invention can be installed in a patient terminal 10. The patient terminal 10 described in the present invention means an electronic device logged in with a user account of a patient U, and for example, the electronic device may include at least one of a smart phone, a mobile phone, a tablet PC, a kiosk, a computer, a laptop, a digital broadcast terminal, a PDA (Personal Digital Assistants), and a PMP (Portable Multimedia Player).

[0066] Here, the user account of the patient U may refer to an account of the patient U that is registered in advance in the exercise therapy providing system 1000 according to the present invention. Such a user account of the patient U may be understood as a "patient account" or a "patient ID (identification, identification number)." In the present invention, the terms "patient," "patient account (or patient user account)," and "patient terminal" may be used interchangeably.

[0067] Meanwhile, a doctor can prescribe exercise to patient U via doctor terminal 20. Doctor terminal 20 in the present invention may refer to an electronic device logged in with doctor D's user account. Doctor D's user account is an account of doctor D registered in advance in exercise treatment providing system 1000 according to the present invention, and can be understood as a "doctor account" or a "doctor ID (identification, identification number)." In the present invention, the terms "doctor," "doctor account (or doctor's user account)," and "doctor terminal" may be used interchangeably.

[0068] Doctor D can prescribe a medicine to patient U by referring to user DB 30 that contains user information of patient U.

[0069] The user DB30 may contain user information (or patient information) of patient U matched to each patient account. The user information of patient U may include various information necessary for providing exercise therapy. For example, the user information of patient U may include at least one of the following: disease information, age information, gender information, surgical history information, exercise plan information, exercise execution information, height information, and weight information of patient U. However, the above-mentioned user information of the patient is merely an example, and it goes without saying that the user information of the patient may include various information necessary for providing exercise therapy to the patient.

[0070] Meanwhile, the exercise therapy application 100 described in the present invention is installed on the patient terminal 10, and can analyze the exercise movements of a patient who has performed exercise according to a doctor D's prescription through an artificial intelligence posture estimation model and an artificial intelligence motion analysis model, and provide the results on the patient terminal 10.

[0071] In addition, the exercise therapy application 100 is configured to communicate with the artificial intelligence server 200, and can provide the patient's motion analysis results analyzed by the artificial intelligence server 200 on the patient terminal 10. The patient's motion analysis results analyzed by the artificial intelligence server 200 can be generated by at least one of the artificial intelligence motion analysis unit 212 and the rule-based motion analysis unit 213 included in the motion analysis unit 210.

[0072] The exercise therapy application 100 is configured to transmit and receive data to and from the artificial intelligence server 200 via wireless communication, and there is no limitation on the wireless communication method. The exercise therapy application 100 according to the present invention can communicate with the artificial intelligence server 200 using a communication module provided in the patient terminal 10. The communication module provided in the patient terminal 10 can be various.

[0073] For example, the communication module provided in the patient terminal 10 may be a WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance: registered trademark), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth TM The device may be configured to communicate with the artificial intelligence server 200 using at least one of the following technologies: Wireless LAN Card (registered trademark), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Wi-Fi Direct, and Wireless Universal Serial Bus (Wireless USB).

[0074] Meanwhile, the artificial intelligence server 200 described in the present invention may be a cloud server that analyzes the exercise movements of the patient U from the exercise video. Such an artificial intelligence server 200 can analyze the exercise movements of the patient U using the exercise video received from the exercise treatment application 100. The "artificial intelligence server" described in the present invention may be named an "artificial intelligence exercise treatment server", an "artificial intelligence rehabilitation treatment server", a "digital treatment server", etc. Hereinafter, for convenience of explanation, it will be named as an "artificial intelligence server" and described.

[0075] Meanwhile, at least one of the exercise therapy application 100 and the artificial intelligence server 200 according to the present invention can analyze the relative positional relationship between key points P1 and P2 corresponding to a plurality of joint points of the patient U extracted from the exercise video 300 through a posture estimation model 52 (corresponding to the artificial intelligence posture estimation unit 121a in FIG. 1) trained using training data related to the joint points, as shown in FIG. 7. The analysis of the relative positions between the key points can be performed by the motion analysis units 120 and 210. In particular, the analysis of the exercise motion can be performed by one of the artificial intelligence motion analysis units 122 and 212 and the rule-based motion analysis units 123 and 213 of the motion analysis unit. One of the artificial intelligence motion analysis units 122 and 212 or the rule-based motion analysis units 123 and 213 can be named an artificial intelligence motion analysis model.

[0076] Here, "joint points" may refer to multiple joints of patient U (or a part of patient U's body that includes joints).

[0077] Moreover, the term "key point" may refer to an area in the motion video 300 that corresponds to each of a plurality of joint points of the subject U.

[0078] Therefore, in the present invention, the terms "joint point" and "key point" may be used interchangeably, and the joint points and key points are described by using the same reference numerals "P1" and "P2", respectively.

[0079] The exercise therapy providing system 1000 can extract key points P1 and P2 corresponding to joint points from the patient's exercise video using the posture estimation model 52, and analyze the exercise motion of the patient U based on an analysis of the positional relationship between the extracted key points P1 and P2. In the present invention, a series of processes for analyzing the patient's exercise motion from the exercise video using the key points extracted through the artificial intelligence posture estimation model 52 can be named a "exercise motion analysis process."

[0080] Such an exercise motion analysis process can be performed by at least one of the exercise therapy application 100 and the artificial intelligence server 200. Specifically, the exercise motion analysis process may include at least one of i) a first data processing method performed by the exercise therapy application 100, ii) a second data processing method performed by the artificial intelligence server 200, and iii) a third data processing method performed by both the exercise therapy application 100 and the artificial intelligence server 200.

[0081] Here, in the third data processing method, the exercise therapy application 100 and the artificial intelligence server 200 may process data sequentially or simultaneously.

[0082] Therefore, in the present invention, the exercise motion analysis process will be described as being performed in the exercise treatment providing system 1000 without distinguishing between a physical space and a subject in which the exercise motion analysis process is performed.

[0083] 7, the motion analysis process may be performed using key points extracted from the AI ​​posture estimation model 52. The AI ​​posture estimation model 52 can identify or estimate the patient's joint points from the motion video through learning data specialized for the joint points, and extract the corresponding key points.

[0084] In the present invention, the learning data used by AI posture estimation model 52 to learn may be stored in database 40, and such database 40 may be named "learning data DB." The learning data will be described in detail later.

[0085] As shown in Fig. 7, posture estimation server 50 may include at least one of learning unit 51 and posture estimation model 52. Posture estimation server 50 may be provided inside exercise treatment providing system 1000 according to the present invention, or may be formed as an external server. That is, posture estimation server 50 according to the present invention performs the function of learning posture estimation, and can be understood as having no physical space constraints. The posture estimation server 50 will be described in detail later together with learning data.

[0086] Meanwhile, as shown in FIG. 1, a therapeutic exercise application 100 according to the present invention may include at least one of a video receiving unit 110, a motion analyzing unit 120, a video processing unit 130, and a control unit 140.

[0087] The video receiving unit 110 of the exercise therapy application 100 may be configured to receive exercise video including the state of the patient's exercise from the patient terminal 10 in which the application 100 is installed. Such exercise video can be captured by a camera provided in the patient terminal 10. In the present invention, "receiving exercise video from the patient terminal 10" may mean that the video receiving unit 110 of the exercise therapy application 100 accesses the exercise video recorded in the memory of the patient terminal 10.

[0088] The motion analysis unit 120 of the exercise therapy application 100 can analyze the patient's exercise motion (or exercise posture) based on the exercise video received from the patient terminal 10. For this purpose, the motion analysis unit 120 of the exercise therapy application 100 may be configured to include at least one of a key point extraction unit 121, an artificial intelligence motion analysis unit 122, and a rule-based motion analysis unit 123. The artificial intelligence motion analysis unit 122 or the rule-based motion analysis unit 123 may be named an "artificial intelligence motion analysis model."

[0089] The key point extraction unit 121 can extract key points P1 and P2, which are paired pieces of coordinate information on the x-axis and y-axis, from the motion video. In this case, the key point extraction unit 121 can extract key points from the video using an artificial intelligence model.

[0090] In the present invention, the extraction of keypoints using an artificial intelligence model will be described as being performed by an artificial intelligence posture estimation unit 121 a included in the keypoint extraction unit 121 .

[0091] The AI ​​posture estimation unit 121a may be called an "AI posture estimation model" and may extract key points corresponding to the patient's joint points from the motion video using an AI model trained for object detection from the video. The AI ​​posture estimation model may be a model based on object detection. For example, the AI ​​posture estimation unit 121a may extract key points from the motion video using an object detection AI model that ensembles multiple bounding boxes. Meanwhile, the AI ​​posture estimation unit 121a may use various object detection AI models, and the above-mentioned object detection AI model is one example.

[0092] Furthermore, in the present invention, the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 can analyze the patient's exercise motion (or exercise posture) using at least one of the exercise video received from the patient terminal and the key points extracted by the key point extraction unit 120.

[0093] More specifically, the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 can i) analyze the patient's motion based on motion video, ii) analyze the patient's motion based on key points, or iii) analyze the patient's motion using both the motion video and key points.

[0094] For the sake of convenience, the following description will be focused on a method for analyzing a patient's motion based on key points. However, it goes without saying that the AI ​​motion analysis unit 122 and the rule-based motion analysis unit 123 can receive motion video as input data instead of key points, and can immediately analyze a patient's motion from the motion video. Meanwhile, the AI ​​motion analysis unit 122 or the rule-based motion analysis unit 123 can also be expressed as the above-mentioned "AI motion analysis model."

[0095] Meanwhile, the artificial intelligence action analysis unit 122 can classify (or identify) the type of exercise performed by the patient and judge the accuracy of the exercise based on an artificial intelligence model (or posture estimation model, see drawing reference number "52" in FIG. 7) learned to analyze the patient's exercise (or exercise posture) from key points.

[0096] In addition, the rule-based motion analysis unit 123 can classify the type of motion (or identify the type of motion) of the motion performed by the patient and determine the accuracy of the motion, based on rule information defined for analyzing the motion of the patient.

[0097] Here, the "rule information" refers to information that includes various rules used in analyzing exercise movements, and may include, for example, reference joint movable range information for each exercise movement (or type of exercise). Such rule information may be used interchangeably with terms such as "reference information" and "standard information."

[0098] Furthermore, the rule information may include various rule information for analyzing at least one of the movable distance of the joint, the movement speed (or acceleration) of the joint, the body balance, body equilibrium, and body alignment state (e.g., leg axis alignment state, spine alignment state, etc.) of the subject (corresponding to the patient) included in the analysis target motion video other than the movable range of the joint. The rule-based motion analysis unit 123 can derive various analysis results from the analysis target motion video of the patient based on such rule information.

[0099] In the present invention, the patient's exercise motion can be analyzed from the exercise video by at least one of the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123.

[0100] Specifically, in the present invention, i) the patient's movement movements are analyzed by the artificial intelligence motion analysis unit 122 ("first analysis execution method"), ii) the patient's movement movements are analyzed by the rule-based motion analysis unit 123 ("second analysis execution method"), or iii) the patient's movement movements may be analyzed by both the artificial intelligence motion analysis unit 122 and the rule-based motion analysis unit 123 ("third analysis execution method").

[0101] Here, in the third analysis execution method, the AI ​​action analysis unit 122 and the rule-based action analysis unit 123 may process data sequentially or simultaneously.

[0102] Meanwhile, the image processor 130 of the therapeutic exercise application 100 may be configured to overlap and render graphic objects corresponding to the extracted key points P1 and P2 on the subject U of the patient included in the exercise image 300. This allows the patient to intuitively recognize the joint points at which analysis is performed on his / her exercise motion.

[0103] The control unit 140 of the exercise therapy application 100 may be configured to perform overall control over the configuration included in the exercise therapy application 100. The control unit 140 of the exercise therapy application 100 can control the configuration of the exercise therapy application 100 using a CPU (Central Processing Unit) of the patient terminal 10, and can further control the configurations provided in the patient terminal 10 (e.g., a communication module, a camera module, a sensing module, an output module (e.g., a display, a speaker), and an input module (e.g., a touch screen, a microphone).

[0104] On the other hand, as shown in FIG. 1, the artificial intelligence server 200 is a cloud server configured to analyze a patient's exercise posture using an artificial intelligence posture estimation model, and may be configured to include at least one of a motion analysis unit 210 and a control unit 220.

[0105] The motion analysis unit 210 of the artificial intelligence server 200 can analyze the motion (or motion posture) of the patient based on the motion video received from the patient terminal 10.

[0106] The motion analysis unit 210 of the artificial intelligence server 200 can receive the patient's exercise video from the exercise therapy application 100, and the reception of the exercise video may be performed by a communication unit (or communication module) of the artificial intelligence server 200.

[0107] The motion analysis unit 210 of the artificial intelligence server 200 may be configured to include at least one of a key point extraction unit 211, an artificial intelligence motion analysis unit 212, and a rule-based motion analysis unit 213. The artificial intelligence motion analysis unit 212 or the rule-based motion analysis unit 213 may be named an "artificial intelligence motion analysis model."

[0108] The key point extraction unit 211, the AI ​​motion analysis unit 212, and the rule-based motion analysis unit 213 included in the AI ​​server 200 may perform the same functions as the key point extraction unit 121, the AI ​​motion analysis unit 122, and the rule-based motion analysis unit 123 of the above-mentioned exercise therapy application 100. Therefore, detailed description will be omitted.

[0109] The control unit 220 of the artificial intelligence server 200 may be configured to exercise overall control over the components included in the artificial intelligence server 200.

[0110] Hereinafter, an exercise and motion analysis process will be described that uses the above-described configuration of the exercise treatment providing system 1000 according to the present invention to analyze the exercise and motion of the patient U from an exercise video and provide an exercise and motion analysis result.

[0111] 2, the doctor terminal 20 may prescribe an exercise prescription for the patient U (S210). The exercise treatment providing system 1000 can receive prescription information for the exercise prescription from the doctor terminal 20 based on the exercise prescription for the patient being prescribed by the doctor terminal 20.

[0112] The exercise treatment provision system 1000 can assign an exercise plan including at least one prescribed exercise according to the prescription information to the patient account based on the prescription information being received from the doctor terminal 20. The assigned exercise plan can be transmitted to the patient terminal 10 (S220).

[0113] The exercise therapy providing system 1000 may include a communication unit that communicates with at least one of the patient terminal 10, the doctor terminal 20, the user DB 30, and the database 40. For example, the communication unit may be a wireless LAN (WLAN), wireless-fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, digital living network alliance (DLNA: registered trademark), wireless broadband (WiBro), world interoperability for microwave access (WiMAX), high speed downlink packet access (HSDPA), high speed uplink packet access (HSUPA), long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation mobile telecommunication (5G), Bluetooth (Bluetooth TMCommunication can be performed using at least one of the following technologies: Wireless LAN (registered trademark), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Wi-Fi Direct, and Wireless Universal Serial Bus (Wireless USB).

[0114] Meanwhile, the patient terminal 10 can capture an exercise video of the patient performing the prescribed exercise included in the exercise plan (S230). The exercise treatment application 100 can activate the camera provided in the patient terminal 10 and control it to capture the exercise video.

[0115] The exercise video captured by the patient terminal 10 can be used by the exercise treatment providing system 1000 as analysis target data (or analysis target exercise video) for the patient's exercise motion analysis.

[0116] As described above, the exercise motion analysis process may be performed by at least a part of the exercise therapy application 100 and the artificial intelligence server 200 installed on the patient terminal 10, and in the present invention, the exercise motion analysis process is described as being performed in the exercise therapy provision system 1000, without separately distinguishing the physical space and subject in which the exercise motion analysis process is performed.

[0117] Meanwhile, the therapeutic exercise providing system 1000 can extract key points P1 and P2 corresponding to a plurality of joint points from the exercise video. The key points P1 and P2 can be extracted by at least a part of the key point extracting unit 121 included in the therapeutic exercise application 100 and the key point extracting unit 221 included in the artificial intelligence server 200.

[0118] The exercise treatment provision system 1000 can analyze the relative positional relationship between the extracted key points P1 and P2 (S250). The exercise treatment provision system 1000 can also analyze the exercise motion of the patient U based on the analysis of the positional relationship between the key points P1 and P2 (S260). Such exercise motion analysis may be performed by at least a part of the motion analysis unit 120 of the application 100 and the motion analysis unit 210 of the artificial intelligence server 200.

[0119] Furthermore, the exercise treatment providing system 1000 can provide the results of the exercise motion analysis of the patient U to the patient terminal 10 as feedback information and to the doctor terminal 20 as monitoring information (S270).

[0120] In this way, the exercise treatment provision system 1000 can perform overall control over the exercise motion analysis process, which in the present invention can be understood to be performed by the control unit of the exercise treatment provision system 1000. That is, the control unit of the exercise treatment provision system 1000 is a concept including the control unit 140 of the exercise treatment application 100 and the control unit 220 of the artificial intelligence server 200, and can perform overall control over the exercise treatment provision system 1000.

[0121] The exercise motion analysis process performed by the exercise treatment delivery system 1000 will now be described in more detail.

[0122] In the present invention, a process of receiving prescription information regarding exercise for a patient from a doctor terminal can be performed (S310, see FIG. 3).

[0123] 4a and 4b, the exercise treatment providing system 1000 can provide an exercise prescription page (or an exercise allocation page) including a prescription function for patient exercise on the doctor terminal 20 logged in with a doctor account. In the present invention, the term "exercise prescription" may be used interchangeably with "exercise allocation."

[0124] The exercise treatment providing system 1000 can provide an exercise prescription page for each patient account on the doctor terminal 20 so that a prescription can be made for a specific patient U account among the patient accounts matched with the doctor account.

[0125] For example, in the present invention, it is assumed that a first patient account (e.g., patient account of "Kim Woo-young") and a second patient account (e.g., patient account of "Kim So-hee") are matched with a specific doctor D account. The exercise treatment providing system 1000 can provide an exercise prescription page corresponding to the first patient account on the doctor terminal 20 based on receiving an exercise prescription request for the first patient account (e.g., patient account of "Kim Woo-young") from the doctor terminal 20.

[0126] The exercise therapy providing system 1000 can receive prescription information for a specific patient from the doctor terminal 20 based on a user's selection (or user's input) made on an exercise prescription page corresponding to the specific patient. The prescription information may include various information for prescribing exercise to the patient. For example, the prescription information may include at least one of the following (see (a) of FIG. 4a and FIG. 4b): i) information on at least one exercise action to be included in the exercise plan (e.g., "Put your hands against the wall to stretch your calves," "Sit down and roll a ball to massage the soles of your feet"), ii) information on the difficulty of the exercise action, iii) information on the duration of the exercise action, iv) information on the number of times the exercise action is performed, v) information on the schedule of the exercise action, vi) physical information matched to the exercise action (e.g., "ankle," "knee"), and vii) information on precautions (e.g., "Please apply ice after exercise").

[0127] The exercise therapy providing system 1000 can receive prescription information for a specific patient U from the doctor terminal 20 based on the input (or selection) of prescription information for the specific patient on an exercise prescription page corresponding to the specific patient. In this case, guidance information informing that a prescription has been made for the specific patient may be output on the doctor terminal 20 (see (b) of FIG. 4).

[0128] Meanwhile, in the present invention, a process of assigning an exercise plan including at least one prescribed exercise to a patient account based on the prescription information can be performed (S320, see FIG. 3).

[0129] As shown in (a) of FIG. 5, the exercise therapy delivery system 1000 can assign an exercise plan E including at least one prescribed exercise to a specific patient account based on prescription information for a specific patient U, and provide the assigned exercise plan (e.g., “digital therapeutic agent for patellofemoral arthritis”) E on a patient terminal 10 logged in with the specific patient account.

[0130] Here, the "prescribed exercise" can be understood as an exercise action that is identified based on prescription information and assigned to a patient account from among multiple exercise actions (or exercise types) included in the exercise treatment provision system 1000. Therefore, in the present invention, the "prescribed exercise" may be used interchangeably with the "exercise action." Also, in the present invention, the "exercise plan" may be used interchangeably with the "digital therapeutic agent."

[0131] The exercise treatment provision system 1000 can provide an exercise page linked to an exercise guide video provision function on the patient terminal 10 based on a request for an exercise plan assigned to a specific patient account from the patient terminal 10 logged in with a specific patient account, so that the patient can perform the prescribed exercises included in the exercise plan.

[0132] As shown in (b) of FIG. 5, the exercise page may include an exercise list L, which may include items V1 to V6 corresponding to exercise guide images for each of a plurality of prescribed exercises (e.g., “straight leg raise,” “standing and bending the knees”) included in an exercise plan assigned to a particular account.

[0133] When the exercise plan includes a specific prescribed exercise (e.g., "straight leg raise") with multiple exercise sets, the exercise therapy provision system 1000 can control the exercise list L to include items V1 to V3 corresponding to the exercise guide image of the specific prescribed exercise for the number of sets (e.g., "3").

[0134] On the other hand, the exercise treatment provision system 1000 can control, upon receiving a request to start exercise from the patient terminal 10, to play multiple exercise guide videos in sequence on the patient terminal 10 based on the order of items V1 to V6 included in the exercise list L.

[0135] Meanwhile, in the present invention, a process of receiving an exercise video of an exercise corresponding to a prescribed exercise from a patient terminal (S330, see FIG. 3) can be performed.

[0136] As shown in FIG. 6, the exercise treatment providing system 1000 can control the camera provided in the patient terminal 10 to capture an exercise video of the patient U based on the exercise guide video being played on the patient terminal 10.

[0137] The exercise therapy application 100 installed on the patient terminal 10 can control the activation state of the camera on the patient terminal 10 from an inactive state to an active state, thereby controlling the camera to capture exercise video of patient U performing exercise movements in accordance with the exercise guide video.

[0138] As shown in (a) of Figure 6, in order to detect a subject U corresponding to the patient from the exercise video captured by the camera, the exercise therapy application 100 may output a guidance message (e.g., "Please stand within the screen") on the patient terminal 10 so that the patient's entire body is entirely contained within a specific area of ​​the exercise video (or the display of the patient terminal).

[0139] The exercise therapy application 100 can detect the object U from the video 300 using an object detection algorithm, based on the fact that the object U corresponding to the patient's entire body is included within a specific region.

[0140] The exercise therapy application 100 can use various object detection algorithms. For example, the exercise therapy application 100 can use an algorithm (Weighted Box Fusion, WBF) that ensembles multiple bounding boxes. However, it goes without saying that the exercise therapy application 100 is not limited to the above-mentioned object detection algorithm, and can use various object detection algorithms that can detect an object corresponding to the subject U from the training target exercise video 300.

[0141] Furthermore, the exercise therapy application 100 can use a camera to capture an exercise video of the patient performing an exercise movement according to a prescribed exercise, based on detection of a subject U corresponding to the patient's entire body within a specific area.

[0142] In this case, the exercise therapy application 100 can capture an image of the patient performing the prescribed exercise while playing back an exercise guide video corresponding to the prescribed exercise assigned to the patient.

[0143] In addition, the exercise therapy application 100 can match the exercise video captured by the camera of the patient terminal 10 to the exercise plan (or each of multiple prescribed exercises included in the exercise plan) and control it to be recorded in the memory of the patient terminal 10.

[0144] Meanwhile, in the present invention, a process of extracting key points corresponding to a plurality of pre-set joint points from a motion image can be performed (S340, see FIG. 3).

[0145] In the present invention, key points P1 and P2 corresponding to preset joint points P1 and P2 can be extracted from an exercise video by at least a part of the exercise therapy application 100 and the artificial intelligence server 200. As described above, the extraction of key points P1 and P2 may be performed i) by the exercise therapy application 100, ii) by the artificial intelligence server 200, or iii) by both the exercise therapy application 100 and the artificial intelligence server 200. In the following description, the extraction of key points P1 and P2 will be described as being performed by the exercise therapy providing system 1000 without distinguishing between different entities.

[0146] The exercise therapy providing system 1000 can extract, from the exercise video 300, areas corresponding to predefined (or pre-set) joint points among a plurality of joint points of the patient, as key points P1 and P2.

[0147] Here, "joint points" may refer to multiple joints of patient U (or a part of patient U's body that includes joints).

[0148] Moreover, the term "key point" may refer to an area in the motion video 300 that corresponds to each of a plurality of joint points of the subject U.

[0149] In the present invention, the terms "joint point" and "key point" may be used interchangeably, and the joint points and key points are described by using the same reference numerals "P1" and "P2", respectively.

[0150] On the other hand, the human body is made up of about 200 bones, and a joint is a part where bones are connected, and the human body is made up of multiple joints.

[0151] In the present invention, among a plurality of joint points constituting a human body, joint points to be key points may be specified in advance and may exist as joint point definition information 500. For example, in the joint point definition information 500, a first joint point P1 corresponding to a head center 510 and a second joint point P2 corresponding to a neck center 520 may be defined in advance and exist (see FIG. 8d).

[0152] The exercise therapy provision system 1000 can extract key points P1 and P2 corresponding to joint points from the exercise video 300 based on a posture estimation model 52 trained using a training data set (Set) including position information of predetermined joint points.

[0153] In this case, the exercise therapy provision system 1000 can identify the positions of key points P1 and P2 in the exercise video 300 based on the fact that the position information of each of the joint points previously set by the posture estimation model is extracted in the form of pairs of coordinate information on the x-axis and y-axis.

[0154] Meanwhile, the exercise therapy provision system 1000 can extract (or identify) key points P1, P2 corresponding to the joint points by either a first key point extraction process or a second key point extraction process based on whether the joint points are visible in the exercise video 300.

[0155] In the present invention, whether or not a joint point is visible can be understood to mean whether or not the joint point is visible in the motion image 300 .

[0156] The exercise therapy providing system 1000 can determine that a joint point is visible in the exercise video 300 if the exercise video 300 includes a part of the subject U's body that corresponds to the joint point.

[0157] When a specific joint point is visible in the exercise video 300, the exercise therapy providing system 1000 can extract a key point corresponding to the specific joint point through a first key point extraction process.

[0158] Specifically, the exercise treatment providing system 1000 can identify visible joint points of the subject U that are visible in the exercise image 300 from among a plurality of preset joint points. For example, when a first joint point and a second joint point are visible in the exercise image from among a plurality of preset joint points, the exercise treatment providing system 1000 can identify the first joint point and the second joint point as the visible joint points.

[0159] Additionally, the exercise therapy delivery system 1000 can extract the identified visible joint points as key points.

[0160] In this case, the exercise therapy providing system 1000 may extract position information of an area (or pixel) corresponding to a visible joint point from within an exercise image, and extract key points corresponding to the visible joint point. For example, the exercise therapy providing system 1000 may extract position information of a visible joint point using an object detection algorithm, and extract key points corresponding to the visible joint point.

[0161] In the present invention, the position information of the visible joint points extracted by the first keypoint extraction process may be named and described as "first type information (first type position information)" or "actual position information".

[0162] On the other hand, if the exercise therapy providing system 1000 does not include a part of the subject U's body that corresponds to the joint point, the exercise video 300 can determine that the joint point is not visible in the exercise video 300.

[0163] When a specific joint point is not visible in the exercise video 300, the exercise therapy providing system 1000 can predict and extract a key point corresponding to the specific joint point using the posture estimation model 52 through a second key point extraction process.

[0164] The exercise therapy providing system 1000 can predict position information of invisible joint points of the subject U that are not visible in the exercise image 300, among a plurality of preset joint points, based on the posture estimation model 52. In this case, the posture estimation model 52 can predict position information of the invisible joint points based on position information of the visible joint points.

[0165] In the present invention, the position information of the articulation points extracted by the second keypoint identification process may be named and described as "second type information (second type position information)" or "predicted position information".

[0166] The exercise therapy delivery system 1000 can extract (or identify) key points corresponding to invisible joint points by matching predicted position information of invisible joint points with key points corresponding to invisible joint points.

[0167] In this way, in the present invention, key points P1 and P2 corresponding to the joint points can be extracted (or identified) through different processes based on a posture estimation model that has been trained on position information of the joint points depending on whether the predefined joint points are visible in the motion video 300. Therefore, in the present invention, analysis of invisible joint points that cannot be seen in the motion video is also possible.

[0168] Meanwhile, the exercise treatment providing system 1000 can extract key points P1 and P2 from the exercise video in real time in conjunction with the exercise video being captured by the patient terminal 10. Also, the exercise treatment providing system 1000 can provide the extracted key points P1 and P2 on the patient terminal 10 in real time so that the patient can intuitively recognize the joint points at which the exercise motion is analyzed.

[0169] Specifically, as shown in (b) and (c) of Fig. 6, the exercise treatment providing system 1000 can output the exercise video 300 on the patient terminal 10 in real time in conjunction with the exercise video 300 being captured by the patient terminal 10. In addition, the exercise treatment providing system 1000 can provide graphic objects corresponding to the extracted key points P1 and P2 by overlapping them in a region of the subject U corresponding to a preset joint point.

[0170] The data processing for providing the key point graphic object by overlaying it on the exercise image 300 may be performed by the image processor 130 of the exercise therapy application 100. The image processor 130 may render each of the graphic objects corresponding to the extracted key points P1 and P2 on the area of ​​the subject U corresponding to the joint points P1 and P2 matched to the key points P1 and P2.

[0171] Furthermore, when the position of a preset joint point is changed as the patient performs an exercise, the image processor 130 may provide a key point graphic object by overlaying it on an area of ​​the subject U corresponding to the changed joint point. That is, the image processor 130 may overlay a key point graphic object on an area corresponding to the joint point in the exercise image so that the position of the joint point, which is changed in real time, is reflected.

[0172] Meanwhile, in the present invention, a process of analyzing a relative positional relationship between key points from key points extracted through a posture estimation model trained using training data related to joint points, and analyzing a patient's exercise motion for a prescribed exercise based on the analysis of the positional relationship can be performed (S350, see FIG. 3). The analysis of the relative positions between key points can be performed by the motion analysis units 120 and 210. In particular, the analysis of the exercise motion can be performed by one of the artificial intelligence motion analysis units 122 and 212 and the rule-based motion analysis units 123 and 213 of the motion analysis unit.

[0173] 6(d), the exercise therapy application 100 can provide guidance information (e.g., "Calculating result value" or "Providing Kim Cheol-soo's exercise motion analysis result") on the patient terminal 10 to guide the patient through the exercise motion analysis. The patient exercise motion analysis method will be described in detail below.

[0174] The exercise therapy provision system 1000 can analyze the relative positional relationship between key points P1 and P2 corresponding to each of a plurality of pre-set joint points using key points extracted from a posture estimation model trained using training data.

[0175] The exercise therapy delivery system 1000 can analyze the relative positions between key points P1 and P2 corresponding to each of a plurality of pre-set joint points using both key points corresponding to visible joint points and key points corresponding to invisible joint points.

[0176] Here, the "relative position between keypoints" can be understood as the position of a particular keypoint (e.g., a first keypoint, "P1") relative to another keypoint (e.g., a second keypoint, "P2") between at least two keypoints P1 and P2.

[0177] In the following, for convenience of explanation, key points corresponding to visible joint points are named "first type key points" and key points corresponding to invisible joint points are named "second type key points".

[0178] The exercise therapy delivery system 1000 can perform analysis on at least one of i) the relative positional relationship between a plurality of first type key points, ii) the relative positional relationship between the first type key points and the second type key points, and iii) the relative positional relationship between a plurality of second type key points.

[0179] In this case, the exercise therapy providing system 1000 can analyze the relative positional relationship between some of the associated key points among the plurality of joint points based on the type of prescribed exercise performed by the patient.

[0180] For example, when a patient performs a prescribed exercise according to a first type of exercise, the exercise therapy providing system 1000 can analyze the relative positional relationship between key points corresponding to a first joint point and a second joint point among a plurality of joint points.

[0181] As another example, the exercise therapy providing system 1000 may analyze the relative positional relationship between key points corresponding to the first joint point and the third joint point among the multiple joint points when the patient performs a prescribed exercise according to a second type of exercise different from the first type of exercise. Such relative positional relationship may be used for motion analysis as a result.

[0182] The results of the motion analysis performed by the exercise treatment providing system 1000 according to the present invention may be very diverse. For example, the exercise treatment providing system 1000 may analyze at least one of the range of motion of a joint, the distance of motion, the speed (or acceleration) of joint movement, the body balance, body equilibrium, and body alignment (e.g., leg axis alignment, spine alignment, etc.) of a subject (corresponding to a patient) included in the exercise image to be analyzed from the extracted key points or image. Meanwhile, the exercise treatment providing system 1000 according to the present invention may analyze the relative positional relationship between key points based on rule information regarding prescribed exercise.

[0183] Here, the rule information can be understood as information in which rules are predefined for analyzing the relative positional relationship between key points.

[0184] The exercise therapy providing system 1000 can analyze the patient's exercise motion by determining whether the relative positional relationship between key points satisfies the rule information. In the following, as an example, a method for analyzing a joint range of motion based on the relative positional relationship of key points and the rule information will be described. However, the content described below is merely one embodiment for analyzing the patient's motion based on the relative positional relationship of key points and the rule information, and in the present invention, various patient motions can be analyzed based on the relative positional relationship of key points and the rule information.

[0185] The joint range of motion analyzed by the exercise treatment provision system 1000 according to the present invention will be considered in more detail. The exercise treatment provision system 1000 can analyze the patient's joint range of motion according to the relative positional relationship between key points based on rule information for the reference joint range of motion related to the prescribed exercise.

[0186] Furthermore, the exercise treatment provision system 1000 can analyze the relative positional relationship between the key points based on rule information regarding the prescribed exercise. Also, the exercise treatment provision system 1000 can analyze the exercise motion of the patient by determining whether the relative positional relationship between the key points satisfies the rule information.

[0187] The exercise therapy delivery system 1000 can extract relative positional relationships between associated key points matched to a specific prescribed motion from multiple consecutive frames related to the specific prescribed motion, and obtain (or calculate) the patient's joint range of motion for the specific prescribed motion using the extracted relative positional relationships.

[0188] Specifically, it is assumed that a motion video comprises a plurality of frames having a first type corresponding to a first prescribed motion, and a plurality of frames having a second type corresponding to a second prescribed motion.

[0189] When a patient's exercise motion analysis is performed for the first prescribed exercise among a first prescribed exercise and a second prescribed exercise, the exercise treatment provision system 1000 can analyze the patient's exercise motion using key points extracted from a plurality of frames having a first type.

[0190] Meanwhile, when a patient's exercise motion analysis is performed for a second prescribed exercise, the exercise therapy delivery system 1000 can analyze the patient's exercise motion using key points extracted from a plurality of frames having a second type.

[0191] That is, the exercise therapy delivery system 1000 can analyze key point positional relationships for a series of movements (or postures) and obtain (or calculate) the patient's range of motion for a specific prescribed exercise.

[0192] In the following, for ease of explanation, consecutive frames (i.e., frames having a first type) corresponding to a particular prescribed motion (e.g., a first prescribed motion) will be named the "first analyzed frame" and the "second analyzed frame" depending on the time before and after the frames are formed.

[0193] Here, the first analyzed frame can be understood as a frame formed before in time, and the second analyzed frame can be understood as a frame formed after in time.

[0194] The exercise treatment delivery system 1000 can extract key points in each of the first and second frames under analysis.

[0195] A first analysis target keypoint group corresponding to each of the plurality of joint points can be extracted from the first analysis target frame, and a second analysis target keypoint group corresponding to each of the plurality of joint points can be extracted from the second analysis target frame.

[0196] The exercise treatment provision system 1000 can analyze a "first positional relationship" between key points included in a first analysis target key point group to perform a first motion analysis of the subject U included in a first analysis target frame. Also, the exercise treatment provision system 1000 can analyze a "second positional relationship" between key points included in a second analysis target key point group to perform a second motion analysis of the subject U included in a second analysis target frame.

[0197] The exercise therapy delivery system 1000 can obtain (extract or calculate) the patient's joint range of motion for a particular prescribed exercise based on the first key point positional relationship and the second key point positional relationship.

[0198] In this case, the exercise therapy provision system 1000 can obtain the patient's joint range of motion by referring to the user DB 30 and taking into consideration at least one of the patient's age information, gender information, height information, weight information, surgical history information, and musculoskeletal disease information.

[0199] Meanwhile, the exercise therapy providing system 1000 can determine whether the acquired range of motion of the patient satisfies a reference joint range of motion corresponding to rule information regarding a specific prescribed exercise. In the present invention, the rule-based analysis of the range of motion of the patient may be performed by the rule-based motion analysis unit 213 of the artificial intelligence server 200 (see FIG. 1), but is not limited to the analysis by the rule-based motion analysis unit 213.

[0200] In the present invention, rule information for the reference joint movable range may exist for each of a plurality of exercise types. Such rule information may include information on the reference joint movable range that differs from one another for each age, sex, height, weight, and musculoskeletal disease.

[0201] The exercise therapy provision system 1000 can compare the patient's joint range of motion for a specific prescribed exercise with a standard joint range of motion for the specific prescribed exercise included in the rule information to determine whether the patient's joint range of motion satisfies the standard joint range of motion.

[0202] Based on the judgment result, the exercise treatment provision system 1000 can provide the analysis result of the patient's exercise motion on the patient terminal 10 as feedback for the prescribed exercise.

[0203] Meanwhile, in the present invention, a process of transmitting the analysis result of the patient's exercise motion to the patient terminal can be performed (S360, see FIG. 3).

[0204] The exercise therapy delivery system 1000 can provide the motion analysis results in various ways so that the patient can intuitively recognize the analysis results of the exercise motion and increase the patient's compliance with the exercise.

[0205] The exercise treatment providing system 1000 can provide graphic objects corresponding to key points P1 and P2 by superimposing them on the exercise video 300 in real time while the exercise video 300 is being captured by the patient terminal 10 (see FIG. 6).

[0206] In this case, the exercise therapy delivery system 1000 can arrange the patient's joint range of motion information around key points P1 and P2 related to the joint range of motion.

[0207] Furthermore, the exercise therapy provision system 1000 can provide key point graphic objects (or graphic objects corresponding to the positional relationship between key points) having different visual appearances superimposed on the exercise image so that the patient can recognize whether the patient's joint range of motion meets the reference joint range of motion.

[0208] Furthermore, visual appearances of graphic objects superimposed on a motion image may be configured to differ from one another depending on whether the relative positional relationship between the extracted key points satisfies the rule information.

[0209] For example, if the patient's joint range of motion satisfies the reference joint range of motion, a graphic object A having a first visual appearance may be superimposed on the exercise video 300. On the other hand, if the patient's joint range of motion does not satisfy the reference joint range of motion, a graphic object B having a second visual appearance different from the first visual appearance may be superimposed on the exercise video 300.

[0210] Furthermore, the exercise treatment provision system 1000 can provide the patient's evaluation score for the prescribed exercise as a motion analysis result (e.g., "Kim Woo-young's squat posture is 70 points") based on key points extracted from each of the multiple frames that make up the exercise video 300.

[0211] Meanwhile, in the present invention, the exercise therapy application 100 and the artificial intelligence server 200 installed in the patient terminal 10 can each analyze the exercise motion and generate the exercise motion analysis result.

[0212] For example, the exercise therapy application 100 may overlay graphic objects corresponding to the key points P1 and P2 on the exercise video in real time to generate the first analysis result.

[0213] As another example, the artificial intelligence server 200 consisting of a cloud server can generate a patient's evaluation score for the prescribed exercise as a second analysis result based on key points extracted from each of a plurality of frames constituting the exercise video.

[0214] The exercise treatment provision system 1000 can provide analysis results of the patient's exercise motion, including a first analysis result generated by the exercise treatment application 100 and a second analysis result generated by the artificial intelligence server 200, on the patient terminal 10.

[0215] Meanwhile, the exercise treatment providing system 1000 can transmit the patient's exercise motion analysis result to the doctor terminal 20. The doctor terminal 20 may be provided with both the first analysis result and the second analysis result.

[0216] In this way, the present invention provides various user environments related to the provision of analysis results so that the patient can intuitively recognize the analysis results for the exercise movement. Other embodiments related to the provision of analysis results will be described later.

[0217] 7, the present invention is directed to analyzing the exercise motion of a patient U included in an exercise video 300 based on the exercise video 300 received from a patient terminal 10, and providing the analysis result. In particular, the present invention relates to a method of processing a learning data set centered on important joint points and learning the data set in order to analyze the exercise motion of a patient based on artificial intelligence.

[0218] The learning data used to learn the posture estimation model of the present invention will be specifically described below.

[0219] 7, the database 40 is a storage in which a learning data set is stored, and may be provided in the exercise therapy providing system 1000 according to the present invention itself, or may be formed as an external storage (or an external DB). The database 40 according to the present invention can be understood to be sufficient as long as it is a space in which a learning data set is stored, and there is no restriction on the physical space.

[0220] The system may be configured to include at least one of a database 40, a posture estimation server 50, and an exercise therapy providing system 1000.

[0221] In database 40, training data for training pose estimation model 52 may be stored and present as a training data set.

[0222] 8b, the learning data set 400 in the present invention may be composed of a plurality of data groups 410-450 corresponding to different information attributes 410a-450a, respectively. The information included in each of the plurality of data groups 410-450 may be extracted and composed from an exercise video 300 including a subject U performing an exercise motion.

[0223] Here, the "exercise video 300" refers to a video (image or video) that captures (includes) the process of a user performing an exercise motion, as shown in FIG. 8a, and may include at least a part of the user U's body.

[0224] In the present invention, a user object included in the exercise video 300 is named and described as "subject U." In the present invention, "subject U" may refer to a user exercising in the exercise video or a part of the user's body. Therefore, in the present invention, "subject" and "user" may be used interchangeably, and are described with the same reference numeral "U."

[0225] Meanwhile, the 'exercise video 300' described in the present invention may include an 'analysis target exercise video' and a 'learning target exercise video'.

[0226] The "analysis target motion video" can be understood as a motion video that is a target of posture estimation analysis of the subject U, and the "learning target motion video" can be understood as a motion video that is a target of machine learning for a posture estimation model. Here, the posture estimation analysis can mean extracting key points from the video.

[0227] The learning unit 51 may be configured to perform learning for a posture estimation model based on the training object exercise video 300. The learning unit 51 can train the posture estimation model using the training data.

[0228] As shown in (a) of FIG. 8b, the learning unit 51 can detect a subject U from the training object motion video 300 and extract various learning data used for motion posture estimation from the detected subject U. Such learning data may be used interchangeably with "information", "data", "data value", or "data value". Meanwhile, extraction of learning data may be performed by other means instead of the learning unit 51.

[0229] The learning unit 51 can use various algorithms for object detection to detect the subject U from the training object motion video 300. For example, the learning unit 51 can use an algorithm (Weighted Box Fusion, WBF) that ensembles a plurality of bounding boxes. However, it goes without saying that the learning unit 51 is not limited to the above-mentioned object detection algorithm, and can use various object detection algorithms that can detect an object corresponding to the subject U from the training object motion video 300.

[0230] The learning unit 51 can classify the extracted learning data into any one of a plurality of data groups 410 to 450 corresponding to each of a plurality of different information attributes 410a to 450a.

[0231] The multiple information attributes 410a-450a different from each other described in the present invention may be predefined and exist as shown in (b) of Fig. 8b. Also, the multiple data groups 410-450 corresponding to the multiple information attributes 410a-450a, respectively, may include learning data corresponding to the predefined information attributes.

[0232] For example, i) the data group 410 corresponding to the first information attribute 410a may include joint point position information of the subject U, ii) the data group 420 corresponding to the second information attribute 420a may include information indicating whether the joint points of the subject U are visible or not, iii) the data group 430 corresponding to the third information attribute 430a may include information regarding the shooting direction of the subject U, iv) the data group 440 corresponding to the fourth information attribute 440a may include exercise code information for classifying the exercise action (or exercise type) performed by the subject U, and v) the data group 450 corresponding to the fifth information attribute 450a may include the size and center position information of a bounding box for the subject U.

[0233] Here, the "joint points P1, P2" may refer to the user's joints or a region corresponding to the subject U's joints in the motion image 300.

[0234] The learning unit 51 can mutually link a plurality of data groups 410-450 extracted from the training object exercise video 300, and generate (construct) a data set for the training object exercise video 300. Furthermore, the learning unit 51 can store the generated training data set 400 in the database 40. The database 40 can be constructed as the database 40 for the posture estimation model 52 based on the storage of the training data set 400 generated by the learning unit 51.

[0235] Furthermore, the learning unit 51 can perform learning for the posture estimation model 52 based on a learning dataset 400 present in the database 40. As described above, the learning dataset 400 may include position information of joint points.

[0236] The posture estimation model 52 is a posture estimation model trained using a training data set including position information of joint points, and is capable of estimating the motion posture of the subject U from the analysis target motion video.

[0237] Meanwhile, the posture estimation model 52 uses the learning dataset 400 generated by the learning unit 51 to extract key points corresponding to the joint points of the subject from the motion video 300, and at least one of the artificial intelligence motion analysis units 122, 212 and the rule-based motion analysis units 123, 213 can analyze the motion of the subject in the motion video 300 using the extracted key points.

[0238] There may be various motion postures of the subject U that can be estimated from the analysis target motion video 300 using the key points estimated from the posture estimation model 52. For example, at least one of the artificial intelligence motion analysis units 122, 212 and the rule-based motion analysis units 123, 213 may estimate and analyze information regarding at least one of i) positions of joint points, ii) joint movable ranges of the joint points, iii) movement paths of the joint points, iv) connection relationships between the joint points, and v) symmetrical relationships of the joint points for the subject U.

[0239] In addition, the artificial intelligence motion analysis unit 122, 212 can analyze at least one of the following from the key points extracted from the movement video 300 to be analyzed or from the video 300: the moving distance of the joint, the movement speed (or acceleration) of the joint, the body balance, body equilibrium, and body alignment (e.g., leg alignment, spinal alignment, etc.) of the subject (corresponding to the patient) contained in the movement video to be analyzed.

[0240] In the present invention, posture estimation model 52 may be configured to include learning unit 51. Conversely, learning unit 51 may include posture estimation model 52. In this case, the posture estimation function can be achieved by having posture estimation model 52 learn by learning unit 51. Therefore, in the present invention, the functions performed by posture estimation model 52 can be explained by mixing them with those performed by learning unit 51.

[0241] On the other hand, the user terminals 10, 20 may be configured to provide a posture analysis result service that provides the user terminals 10, 20 with a user's movement motion analysis result (or a movement motion analysis report) that is analyzed based on key points extracted and estimated from the posture estimation model 52 (see FIG. 1).

[0242] Here, the user terminals 10, 20 may be at least one of a patient terminal 10, a doctor terminal 20, and a third party terminal.

[0243] Such an exercise treatment provision system 1000 may be configured to communicate with the user terminals 10 and 20. In the present invention, the communication of the exercise treatment provision system 1000 can also be understood as being performed by a communication unit of the exercise treatment provision system 1000.

[0244] For example, the communication unit of the exercise therapy providing system 1000 may be a WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance: registered trademark), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth TM The device may be configured to communicate with the user terminals 10, 20 using at least one of the following technologies: Wireless LAN Card (registered trademark), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Wi-Fi Direct, and Wireless Universal Serial Bus (Wireless USB).

[0245] Meanwhile, the user terminals 10 and 20 described in the present invention refer to electronic devices and may include at least one of a smart phone, a mobile phone, a tablet PC, a kiosk, a computer, a laptop, a digital broadcasting terminal, a PDA (Personal Digital Assistants), and a PMP (Portable Multimedia Player). Furthermore, the user terminals 10 and 20 may be electronic devices to which a user account is logged in, connected, or registered.

[0246] Here, the user account may refer to an account registered in advance in the exercise treatment providing system 1000 according to the present invention. Such a user account may be understood as a user ID (identification, identification number).

[0247] Meanwhile, in the present invention, a process of receiving an exercise video from the user terminal 10, 20 can be performed. The exercise treatment providing system 1000 can receive an exercise video 300 in which a user performs an exercise motion by communicating with the user terminal 10, 20.

[0248] In this case, the exercise video 300 that the exercise treatment providing system 1000 receives from the user terminals 10 and 20 can be understood as an analysis target exercise video that is to be the subject of analysis of the exercise motion of the user.

[0249] The exercise treatment providing system 1000 can receive exercise images to be analyzed from the user terminals 10 and 20 according to various times and routes.

[0250] 1, the exercise treatment providing system 1000 can control the camera state to an activated state so that the camera 201 provided in the user terminal 10, 20 captures an exercise video to be analyzed based on the selection of a graphic object corresponding to "start exercise" from the user terminal 10, 20. In addition, the exercise treatment providing system 1000 can receive the exercise video to be analyzed captured by the camera 201 from the user terminal 10, 20 in real time or based on the completion of the user's exercise.

[0251] Next, in the present invention, a process of analyzing a movement motion related to a specific movement motion of a user included in a movement video can be performed based on key points extracted from a posture estimation model trained using a training data set including position information of joint points.

[0252] When the learning unit 51 receives an analysis target exercise video from the user terminal 10, 20, it can extract key points corresponding to joint points of the user U included in the analysis target exercise video based on the posture estimation model 52 learned using the learning target exercise video 300. In addition, at least one of the artificial intelligence action analysis units 122, 212 and the rule-based action analysis units 123, 213 can analyze the exercise action of the subject in the exercise video 300 using the extracted key points.

[0253] The posture estimation information of the user U estimated by the learning unit 51 may include various information. For example, the learning unit 51 can estimate ii) position information of the joint points P1 and P2 of the subject U, and ii) joint movable range information (angle information) of the subject U.

[0254] Next, in the present invention, based on the completion of the analysis, a process of providing the user U's exercise motion analysis result related to a specific exercise motion to the user terminal 10, 20 can be performed.

[0255] The exercise treatment providing system 1000 can process the analysis result of the user's exercise motion to generate an exercise motion analysis report. Also, the exercise treatment providing system 1000 can provide the exercise motion analysis report on the user terminal 10, 20.

[0256] 1, the exercise treatment providing system 1000 can provide a user's exercise video by rendering joint point graphic objects corresponding to the joint points P1 and P2 of the user U at positions corresponding to the joint points P1 and P2 of the user U. In addition, the exercise treatment providing system 1000 can display joint movable range information 221 of a specific joint point P1 around the specific joint point P1.

[0257] In this way, the exercise therapy providing system 1000 according to the present invention can perform learning for the posture estimation model 52 using the database 40 constructed based on the training object exercise video 300. In addition, the posture estimation model 52 can be used to estimate the user's exercise posture and provide an exercise motion analysis result providing service based on the estimated posture.

[0258] The information included in the analysis result may be various. For example, the analysis result may include analysis information on at least one of the range of motion of a joint, the distance of motion, the speed (or acceleration) of joint movement, the body balance, the body equilibrium, and the body alignment state (e.g., leg axis alignment state, spine alignment state, etc.) of a subject (corresponding to a patient) included in the motion image to be analyzed, analyzed from the extracted key points or image.

[0259] Furthermore, such analytical information may further include a score, which may be an analysis score of the user's athletic movement (or posture). Such an analysis score may be calculated based on various methods (e.g., rule-based analysis based on pre-set criteria or analysis by an artificial intelligence algorithm).

[0260] In the database 40, a learning data set 400 extracted and generated from the training object exercise video 300 by the learning unit 51 may be stored.

[0261] The learning data set 400 used for estimating a user's exercise posture will be described in more detail below.

[0262] As shown in FIG. 8 a, the training data set 400 may be configured to include data related to a subject U extracted from the training object exercise video 300.

[0263] The learning unit 51 can extract data for the subject U from the training object exercise video 300 to configure a training data set 400.

[0264] Such a learning data set 400 may be made up of a plurality of sub-data sets 401 to 403. In the present invention, the learning data set 400 can be understood as a higher-level concept, and the sub-data sets 401 to 403 can be understood as data sets corresponding to lower-level concepts.

[0265] The learning unit 51 may extract data for the subject U from each of the reference frames 301-306 selected based on a preset criterion from among the multiple frames constituting the training object exercise video 300, and construct sub-data sets 401-403.

[0266] The learning unit 51 can select the reference frames 301 to 306 based on various criteria. The training exercise video 300 may be a moving image or a plurality of still images.

[0267] When the training object exercise video 300 is a video, the learning unit 51 may select reference frames 301-306 from among a plurality of frames constituting the training object exercise video 300 based on a certain time interval T. As another example, when the amount of change in the motion of the subject included in the previous and next frames corresponds to a certain amount of change or more, the learning unit 51 may select the previous and next frames as the reference frames 301-306.

[0268] The learning data included in the learning dataset according to the present invention may be composed of learning data extracted from each of reference frames selected based on a preset criterion from among a plurality of frames constituting a training target motion video, with a focus on the subject included in the training target motion video.

[0269] For convenience of explanation, the following explanation will be based on the "learning dataset 400" without distinguishing between the "learning dataset 400" and the "sub-datasets 401-403". The information included in the learning dataset 400 described below may be the information included in the sub-datasets 401-403. In this case, the learning dataset 400 according to the present invention can be understood as including a plurality of sub-datasets 401-403 including the information described below.

[0270] On the other hand, as shown in FIG. 8b(a), the learning data set 400 may be made up of a plurality of data groups 410 to 460 each corresponding to a plurality of different information attributes.

[0271] The learning unit 51 can extract data corresponding to each of multiple information attributes from the training object exercise video, and classify (or match) the extracted data having the same information attributes into the same data group to generate a learning dataset 400.

[0272] Here, the information attributes 410a to 450a can be understood as a criterion for classifying the types of information required for estimating the motion posture of the subject U from the learning motion video 300. As shown in (b) of Fig. 8b, in the present invention, a plurality of different information attributes (first to fifth information attributes 410a to 450a) may be predefined and exist.

[0273] The learning unit 51 can extract learning data corresponding to each of the multiple information attributes 410a to 450a from the training object exercise video, and generate a learning data set 400 by classifying the learning data corresponding to the same information attribute into the same data group.

[0274] Furthermore, the learning unit 51 can identify the association between the plurality of data groups 410 to 460 on a group-by-group basis based on the association between the plurality of information attributes 410a to 450a, and perform learning on the learning data set 400 and the association between each group.

[0275] The multiple data groups and the relationships between the groups will be specifically described below.

[0276] As shown in FIG. 8c, a first data group 410 among the plurality of data groups 410 to 450 may include position information 411 and 412 for joint points P1 and P2 of a subject U included in a motion image 300.

[0277] As shown in (a) of Fig. 8c, the joint points P1 and P2 in the present invention may refer to a region of the subject U corresponding to the user's joints in the training motion video 300. Also, as shown in (b) of Fig. 8c, the position information 411 and 412 of the joint points may be understood as the positions of the regions in the training motion video 300 where the joint points P1 and P2 are located.

[0278] On the other hand, the human body is made up of about 200 bones, and a joint is a part where bones are connected, and the human body is made up of multiple joints.

[0279] The learning unit 51 may have predefined joint points to be learned among a plurality of joint points of the subject U. That is, the "learning target joint points" described in the present invention may be understood as joint points predefined for learning in the present invention among a plurality of joint points of the user.

[0280] 8d, in the database 40, among the multiple joint points, training target joint points to be used as training targets for a posture estimation model may be specified in advance and may exist as reference information 500. Also, in the reference information 500, the order of the multiple training target joint points may be defined in advance.

[0281] The first training joint point can be defined as the center of head. More specifically, the first training joint point can be understood as a point that is predicted or corresponds to the first cervical vertebra level.

[0282] The second learning joint point can be defined as the center of the neck. More specifically, the second learning joint point can be understood as the C3-C4 level, which is the center of the neck lordotic curve, and as the midpoint between the first and third midpoints when viewed from the front.

[0283] The third training joint point can be defined as the lower end of the neck. More specifically, the third training joint point can be understood as the C7-T1 level, which is the center point of the line connecting both clavicle levels.

[0284] The fourth learning joint point may be defined as the shoulder center. More specifically, the fourth learning joint point may be the humerus head center, which may be understood as a position corresponding to the rotation center of a continuous motion in a rotation motion in which the arm is abducted to a position that is the central axis of the shoulder joint rotation motion. In an image that is not a continuous motion of a rotation motion, a point corresponding to predicted position information of the shoulder center may be the fourth learning joint point. In addition, the fourth learning joint point may exist at each of the left shoulder center and the right shoulder center.

[0285] The fifth learning joint point may be defined as the elbow center. More specifically, the fifth learning joint point may be a part corresponding to the humerus medial-lateral epicondyle center, and may be understood as a central point at the elbow level. The fifth learning joint point may exist at each of the left elbow center and the right elbow center.

[0286] The sixth training joint point can be defined as the wrist center. More specifically, the sixth training joint point is the radius-ulnar styloid process center, which can be understood as a central point at the wrist level. The sixth training joint point can be present at each of the left wrist center and the right wrist center.

[0287] The seventh training joint point may be defined as the hand center. More specifically, the seventh training joint point may be understood to correspond to the third metacarpal head, and may exist at the left hand center and the right hand center.

[0288] The eighth learning joint point may be defined as the hip joint center (femoral head center). More specifically, the eighth learning joint point may be understood as a position that is the central axis of the hip joint rotational motion, and as a position that corresponds to the rotation center of a continuous motion in a rotational motion of abducting the leg. In an image that is not a continuous motion of a rotational motion, a point that corresponds to predicted position information of the hip joint center may be understood as the eighth learning joint point. The eighth learning joint point may exist at each of the left hip joint center and the right hip joint center.

[0289] The ninth training joint point can be understood as the knee center. More specifically, the ninth training joint point is the femoral medial-lateral epicondyle center, which can be understood as a central point at the knee level. The ninth training joint point can exist at each of the left knee center and the right knee center.

[0290] The tenth training joint point can be defined as the ankle center. More specifically, the tenth training joint point can be understood as a midpoint at the ankle level between the medial-lateral malleolus centers. The tenth training joint point can be located at the left ankle center and the right ankle center, respectively.

[0291] The eleventh training joint point may be defined as the foot center. More specifically, the eleventh training joint point may correspond to the second metatarsal head and may be present at the left foot center and the right foot center.

[0292] The 12th training joint point can be defined as the heel. More specifically, the 12th training joint point can be present on each of the left and right heels at the level where the heels contact the floor. The 12th training joint point may not be visible when the subject U is standing completely forward in the image, but may be visible if the foot is twisted even slightly.

[0293] The 13th learning target joint point can be defined as the lumbar curve start point (SUP. END OF LORDOSIS). More specifically, the 13th learning target joint point can be understood as the midpoint between the average level of the 4th on both sides and the average level of the 8th on both sides, which is the level of about 8-10T Spine at the level of the xiphoid process of the sternum.

[0294] The 14th learning joint point can be defined as the center of lordosis. More specifically, the 14th learning joint point is at a level of about L2-4 Spine, and can be understood as the midpoint between the 13th level and the average level of the 8th level on both sides.

[0295] The 15th learning joint point can be defined as the lumbar curve end point (INF. END OF LORDOSIS). More specifically, the 15th learning joint point is at a level of about S1-2 spine and can be understood as the midpoint between the 14th level and the midlevel of the average of the 8th levels on both sides.

[0296] Meanwhile, the first learning target joint point P1 may be defined as the head center 510, and the second learning target joint point P2 may be defined as the neck center 520. A first order having the highest priority may be defined for the first learning target joint point P1, and a second order having a lower priority than the second order may be defined for the second learning target joint point P2.

[0297] In this case, the order of the training joint points present corresponding to the left and right sides of the subject U may be such that the training joint point corresponding to the first side (e.g., left side) of the body has a higher priority than the training joint point corresponding to the second side (e.g., right side) of the body. For example, a training joint point P3 corresponding to the left wrist center 530 may be matched with a higher priority than a training joint point P4 corresponding to the right shoulder center 540.

[0298] The learning unit 51 can extract coordinate information as position information 411, 412 of each of a plurality of training target joint points P1, P2 designated in advance from the training target exercise video 300.

[0299] The coordinate information may include at least one of two-dimensional or three-dimensional coordinates. When two-dimensional coordinate information is extracted, the learning unit 120 can extract coordinate information of the x-axis and y-axis of each of the multiple learning target joint points P1 and P2 from the learning target motion video 300. In contrast, when two-dimensional coordinate information is extracted, the learning unit 120 can extract coordinate information of the x-axis, y-axis, and z-axis of each of the multiple learning target joint points P1 and P2 from the learning target motion video 300.

[0300] The coordinate information can be extracted by various methods. In particular, the z-axis coordinate information can be extracted by a camera (e.g., an RGB camera, etc.) or various types of sensors (e.g., a distance measurement sensor, etc.). Furthermore, the z-axis coordinate information can be extracted from the learning target image 300 through various types of artificial intelligence algorithms. When the z-axis coordinate information is extracted through an artificial intelligence algorithm, it can be expressed as being "estimated" or "predicted".

[0301] On the other hand, the learning unit 51 can classify the position information 411, 412 of each of the multiple learning target joint points P1, P2 into a first data group 410 based on the correspondence between the position information 411, 412 and the first information attribute 410a, and generate the first data group 410, and generate a learning dataset 400 including the first data group 410.

[0302] Considering an example in which two-dimensional coordinate information (x, y coordinate information) is extracted, the learning unit 121 can extract position information 411, 412 of a plurality of learning target joint points P1, P2 in a form in which the coordinate information of the x axis and the y axis are paired. The learning unit 51 can extract position information "[599, 463]" of the first learning target joint point P1, and extract position information "[586, 545]" of the second learning target joint point P2. In addition, the learning unit 51 can generate a learning data set 400 constituted of a first data group 410 including the above-mentioned "[599, 463]" and "[586, 545]".

[0303] The learning unit 51 can perform learning to estimate the positions of the joint points P1, P2 of the subject U included in the training object exercise video 300 based on the position information 411, 412 of the training object joint points P1, P2 constituting the first data group 410.

[0304] On the other hand, as shown in (b) of FIG. 8c, the learning unit 51 can configure (generate) a learning dataset 400 by sequentially arranging the position information 411, 412 of each of the multiple learning target joint points P1, P2 in a first data group 410 based on a predefined order between the multiple learning target joint points P1, P2.

[0305] As described above, the database 40 may contain a predefined order of a plurality of training target joint points P1 and P2.

[0306] The learning unit 51 can refer to the database 40 and sequentially arrange the position information 411, 412 of the multiple training target joint points P1, P2 in the first data group 410 in accordance with the order corresponding to the training target joint points P1, P2, to generate a training data set 400. In addition, such a training data set 400 can be stored in the database 40 to construct the database 40 for posture estimation.

[0307] Specifically, as shown in (b) of FIG. 8c, the learning unit 51 can preferentially arrange, within the first data group 410, the first position information 411 of the first learning target joint point P1 corresponding to the first order, and arrange the second position information 412 of the second learning target joint point P2 corresponding to the second order following the first position information 411.

[0308] On the other hand, the learning unit 51 can extract (or identify) the position information 411, 412 of the learning target joint points P1, P2 by either the first process or the second process based on whether the learning target joint points P1, P2 are visible in the exercise video 300.

[0309] In the present invention, whether the training target joint points are visible or not can be understood to mean whether the training target joint points P1 and P2 are visible in the training target motion image 300 or not.

[0310] In the present invention, a visible joint point in a training motion image can be named a "training target visible joint point", and an invisible joint point in a training motion image can be named a "training target invisible joint point".

[0311] If the learning object motion video 300 includes parts of the subject U's body corresponding to the learning object joint points P1 and P2, the learning unit 51 can determine that the learning object joint points are visible in the learning object motion video 300.

[0312] Based on the fact that the learning target joint points P1 and P2 are visible in the learning target movement video 300, the learning unit 51 can extract position information of the actual positions where the learning target joint points P1 and P2 are located from the learning target movement video 300 through the first process.

[0313] In the present invention, the position information of the learning target joint points P1, P2 extracted by the first process can be named and explained as "first type information (first type position information)" or "actual position information".

[0314] On the other hand, if the exercise video 300 does not include a part of the subject U's body corresponding to the learning joint points P1 and P2, the learning unit 51 can determine that the learning joint points P1 and P2 are not visible in the learning exercise video 300.

[0315] Based on the fact that the learning target joint points P1 and P2 are not visible in the movement video 300, the learning unit 51 can predict the expected positions of the learning target joint points P1 and P2 through the second process and extract (or identify) the predicted position information.

[0316] In the present invention, the position information of the learning target joint points P1, P2 extracted by the second process can be named and explained as "second type information (second type position information)" or "predicted position information".

[0317] In this manner, in the present invention, the plurality of position information 411, 412 included in the first data group 410 can be defined to have different extraction processes and type information depending on whether the plurality of training target joint points P1, P2 are visible in the training target exercise image 300.

[0318] Meanwhile, the second process may include various data processing operations for extracting (identifying) predicted position information of the learning target joint points P1 and P2 that are not visible in the motion image 300.

[0319] For example, the learning unit 51, which extracts predicted position information through the second process, can predict predicted position information of the learning target joint points P1 and P2 that are not visible in the exercise video 300 based on the actual position information of the learning target joint points that are visible in the exercise video 300.

[0320] In this case, the learning unit 51 can identify predicted position information by assigning weights to multiple learning target joint points P1, P2 that are visible in the exercise video 300 based on their relevance to learning target joint points P1, P2 that are not visible in the exercise video 300.

[0321] For example, the closer the orders of the learning joint points P1 and P2 are, the higher the relevance between the learning joint points may be set. The relevance between the learning joint points corresponding to the third order may be set such that the learning joint point corresponding to the second order is higher than the learning joint point corresponding to the first order.

[0322] As another example, the relevance between the learning target joint points P1 and P2 may be set highest between the learning target joint points P1 and P2 that exist corresponding to the left and right sides of the subject U. For example, the relevance between the learning target joint point P3 corresponding to the center of the left wrist and the learning target joint point P3 corresponding to the center of the right wrist may be set highest (see (a) of FIG. 8c).

[0323] Furthermore, the learning unit 51 can extract predicted position information of the learning target joint points P1 and P2 that are not visible in the movement video 300 based on motion information of the movement action performed by the subject U in the movement video 300.

[0324] The database 40 may store motion information including a moving path (e.g., a moving position, a moving direction) of a body (or a joint point) during a motion.

[0325] The learning unit 51 can identify predicted position information of the training target joints that are not visible in the exercise video 300 by referring to the position information of the training target joint points P1 and P2 that are visible in the exercise video 300 and the motion information in the database 40.

[0326] Meanwhile, as shown in FIG. 8c, a second data group 420 among the multiple data groups 410 to 450 may include data values ​​421 and 422 indicating whether or not the learning target joint points P1 and P2 of a subject U included in the motion image 300 are visible.

[0327] As shown in (b) of FIG. 8c, the data values ​​of the data 421, 422 included in the second data group 420 may be either a first data value (e.g., “1”) or a second data value (e.g., “2”), depending on whether the learning target joint points P1, P2 are visible or not.

[0328] Data having a first data value (e.g., “1”) is data indicating that the learning target joint points P1 and P2 are visible in the exercise image 300, and can be understood as information indicating that the position information included in the first data group 410 is of the first type (actual position information).

[0329] The learning unit 51 may extract a first type of position information (actual position information) of the learning target joint points when the learning target joint points P1 and P2 are visible in the exercise video 300. Based on the extraction of the first type of position information (actual position information) from the exercise video 300, the learning unit 51 may generate (configure) the learning data set 400 by including data having a first data value (e.g., “1”) in the second data group 420.

[0330] On the other hand, the second data value (e.g., “2”) is data indicating that the learning target joint points P1 and P2 are not visible in the exercise video 300, and can be understood as information indicating that the position information included in the first data group 410 is of the second type (predicted position information).

[0331] The learning unit 51 can extract (or identify) the second type of position information (actual position information) of the learning target joint point when the learning target joint point is not visible in the exercise video 300. Based on the extraction (or identification) of the second type of position information (predicted position information) from the exercise video 300, the learning unit 51 can generate (configure) the learning dataset 400 by including data having a second data value (e.g., “2”) in the second data group 420.

[0332] Meanwhile, as shown in (b) of FIG. 8c, the learning unit 51 can generate (configure) the learning dataset 400 by arranging the data (or data values) 421, 422 included in the second data group 420 in the same order as the predefined order in which the position information 411, 412 of each of the multiple training target joint points is arranged, so as to indicate whether each of the multiple training target joint points P1, P2 is visible or not.

[0333] The learning unit 51 can sequentially arrange the data 421, 422 indicating whether each of the multiple learning target joint points is visible or not in the second data group 420 based on a predefined order between the multiple learning target joint points P1, P2.

[0334] For example, as shown in (b) of FIG. 8c, the learning unit 51 can arrange data 421 having a first data value (e.g., “1”) in a first order corresponding to the first learning target joint point P1 within the second data group 420 based on the first learning target joint point P1 being visible in the motion image 300.

[0335] 8c(a), the second learning target joint point P2 is shown as being visible in the motion video 300, but it is assumed that the second learning target joint point P2 is not visible in the motion video 300. The learning unit 51 can arrange data 422 having a second data value (e.g., “2”) in a second order corresponding to the second learning target joint point P2 based on the fact that the second learning target joint point P2 is visible in the motion video 300.

[0336] Meanwhile, in the present invention, it can be understood that the definition of the type of the location information included in the first data group 410 is performed according to the data value of the data included in the second data group 420.

[0337] Assume that in the second data group 420, data 421 arranged in a first order has a first data value (e.g., “1”), and data 422 arranged in a second order has a second data value (e.g., “2”), as shown in (b) of FIG. 8c.

[0338] In the present invention, based on the fact that the data 421 arranged in a first order in the second data group 420 has a first data value (e.g., “1”), the type of the location information 411 arranged in a first order in the first data group 410 can be defined as a first type of location information (actual location information).

[0339] On the other hand, based on the fact that the data 422 arranged in the second order in the second data group 420 has a second data value (e.g., “2”), the type of the location information 411 arranged in the second order in the first data group 410 can be defined as a second type of location information (predicted location information).

[0340] Meanwhile, the posture estimation model 52 according to the present invention can perform learning by setting different learning weights for each of the position information 411, 412 of a plurality of training target joint points included in the first data group 410 based on the data values ​​included in the second data group 420.

[0341] Specifically, when data 421 arranged in a first order in second data group 420 has a first data value (e.g., “1”), posture estimation model 52 can set a first learning weight to position information 411 arranged in a first order in first data group 410 and perform learning.

[0342] On the other hand, if the data 422 arranged in the second order in the second data group 420 has a second data value (e.g., “2”), the posture estimation model 52 can set a second learning weight for the position information 412 arranged in the second order in the first data group 410 and perform learning.

[0343] That is, the posture estimation model 52 can perform learning by setting different learning weights for the first type of position information (actual position information) and the second type of position information (predicted position information) based on the data values ​​of the second data group 420.

[0344] In this case, posture estimation model 52 may set the first learning weight higher than the second learning weight.

[0345] Meanwhile, as shown in Fig. 8b, the plurality of data groups 410 to 450 may further include a third data group 430 including information related to a photographing direction of a subject included in the motion image 300. The third data group 430 may include a data value indicating a photographing direction in which a subject U included in the motion image 300 was photographed.

[0346] As shown in FIG. 8e(a), the subject U may be photographed from different photographing directions (eg, "front" or "side").

[0347] Here, the term "shooting direction" may be understood as the direction of the axis of a camera (see reference numeral "201" in FIG. 10) relative to the subject U. Here, the camera 201 may be understood as the camera 201 that shoots the motion image 300 including the subject U. Such a camera 201 may include the camera 201 provided in the user terminal 10 or 20.

[0348] The data values ​​included in the third data group 430 shown in Fig. 8e(b) may correspond to the shooting direction of the object U and may have different data values ​​(e.g., "0" or "1"). The data values ​​included in the third data group may be configured to have different data values ​​according to the shooting direction of the object U based on the camera that shot the object. In the following, to avoid confusion with the data values ​​included in the second data group 420, the data values ​​corresponding to the shooting direction are named and described as "data object values".

[0349] Data having a first data object value (e.g., “0”) can be understood as data indicating that the shooting direction of the training object video 300 is a first direction (e.g., forward direction) (see FIG. 8e(b)).

[0350] The learning unit 51 can generate a learning dataset 400 by including data having a first data object value (e.g., “0”) in the third data group 430 based on the fact that the shooting direction of the subject U included in the training object exercise video 300 corresponds to a first direction (e.g., forward direction).

[0351] On the other hand, the second data object value (e.g., “1”) can be understood as data indicating that the shooting direction for the motion image 300 is a second direction (e.g., a side direction) different from the first direction (see FIG. 8e(b)).

[0352] The learning unit 51 can generate a learning dataset 400 by including data having a second data object value (e.g., “1”) in the third data group 430 based on the fact that the shooting direction of the subject U included in the exercise video 300 corresponds to the second direction (e.g., side direction).

[0353] Furthermore, although not shown, a third data object value (e.g., “2”) can be understood as data indicating that the shooting direction for the motion image 300 is a third direction (e.g., diagonal direction) different from the first and second directions.

[0354] The learning unit 51 can store in the database 40 the third data group 430 including data having a third data object value (e.g., "2") based on the fact that the shooting direction of the subject U included in the training object video 300 corresponds to the third direction (e.g., diagonal direction).

[0355] On the other hand, the first to third directions described in the present invention can be understood as the cases where the angles between the subject U and the axis of the camera 201 based on a pre-set direction (e.g., clockwise) correspond to the first to third ranges, respectively.

[0356] For example, the first direction can be understood as a range in which the angle between the subject U and the axis of the camera 201 corresponds to a range from a first angle to a second angle that is larger than the first angle. The second direction can be understood as a range in which the angle between the subject U and the axis of the camera 201 corresponds to a range from the second angle to a third angle that is larger than the second angle. Moreover, the third direction can be understood as a range in which the angle between the subject U and the axis of the camera 201 corresponds to a range from the third angle to a fourth angle that is larger than the third angle.

[0357] Meanwhile, the learning unit 51 may configure an angle (or angle value) between the subject U and the axis of the camera 201 based on a preset direction (e.g., clockwise) as the data value of the data included in the third data group 430. For example, when the subject U and the axis of the camera 201 form a right angle based on the clockwise direction, the learning unit 51 may configure a data value of "90" as the data of the third data group 430. Meanwhile, the learning unit 51 may perform learning for estimating the posture of the subject U by linking the learning data sets 400 having different shooting direction information included in the third data group 430.

[0358] For example, it is assumed that the data included in the third data group 430 includes a first learning data set having a first data object value (e.g., “0”) and a second learning data set having a second data object value (e.g., “1”). The learning unit 51 can perform learning for estimating the posture of the subject U by linking the position information of the first data group 410 included in the first data set with the position information of the first data group included in the second data set.

[0359] On the other hand, when the learning unit 51 estimates the exercise posture of the subject U from the exercise video to be analyzed filmed in the first direction, the learning unit 51 can estimate the exercise posture of the subject U based on a posture estimation model learned from the exercise video to be studied 300 filmed in the first direction.

[0360] The exercise therapy provision system 1000 of the present invention can analyze the exercise movements of subject U based on key points extracted from a posture estimation model trained using a learning dataset 400 including shooting direction information (data object values ​​or data values) corresponding to the shooting direction of the exercise video to be analyzed.

[0361] On the other hand, when the motion analysis units 120 and 210 estimate the movement posture of the subject U from the analysis target movement video 300 captured in a first direction, they can analyze the movement motion of the subject U using posture estimation information estimated from a posture estimation model learned from the training target movement video 300 captured in each of the first and second directions.

[0362] That is, when the motion of the subject U is captured from the analysis target motion video 300 captured in a first imaging direction, the motion analysis units 120 and 210 can analyze the motion of the subject U using posture estimation information estimated from a posture estimation model trained on a first learning data set including data object values ​​corresponding to the first imaging direction and a second learning data set including data object values ​​corresponding to a second imaging direction different from the first imaging direction. In this case, the posture estimation model 52 may correspond to a posture estimation model trained by setting a weight on the first learning data set.

[0363] In this way, the posture estimation model 52 may be configured to be trained by taking into consideration the shooting direction of the subject through the training data set having different data values ​​according to the shooting direction of the subject included in the training video. Furthermore, the user's motion analysis result may be a result of analyzing a specific motion of the user based on posture estimation information extracted in the posture estimation model by taking into consideration the shooting direction of the user included in the motion video.

[0364] Meanwhile, as shown in FIG. 8 b , a fourth data group 440 of the plurality of data groups may include an exercise code that matches an exercise motion performed by a subject U included in the training exercise image 300 .

[0365] As shown in FIG. 8f, in the database 40, a plurality of different exercise actions 710, 720, 730 may be matched with different exercise codes (eg, "502", "503", "504"), respectively.

[0366] The "exercise code" described in the present invention is a data value that distinguishes different exercise actions from each other, and may be used interchangeably with "exercise key", "action code" or "action key".

[0367] The learning unit 51 can generate a learning dataset 400 by including in the fourth data group 440 a specific movement code ("502") that matches a specific exercise movement (e.g., "standing and extending one leg forward") 710 performed by the subject U included in the training exercise video 300.

[0368] The learning unit 51 can perform learning for posture estimation by linking a plurality of learning data sets 400 including the same movement code.

[0369] For example, assume that there is a first learning data set based on a first training object exercise video and a second learning data set based on a second training object exercise video 300. The learning unit 51 can link the first learning data set and the second learning data set with each other and perform learning for posture estimation based on the fact that the exercise code (e.g., “502”) included in the first learning data set and the second learning data set is the same.

[0370] Such an exercise code may be included in the fourth data group 440 based on the exercise movement performed by the subject U being identified by at least one of information received from the user terminal 10, 20, a system administrator, or the learning unit 51.

[0371] The learning unit 51 can identify the exercise action performed by the subject U based on the information received from the user terminals 10 and 20. For example, based on the selection of a graphic object corresponding to "start exercising" from the user terminals 10 and 20, the learning unit 51 can control the state of the camera 201 provided in the user terminals 10 and 20 to an activated state so that the camera 201 captures an exercise video.

[0372] In this case, the graphic object may correspond to a particular exercise movement, and the learning unit 51 may determine that the subject U included in the exercise video 300 received from the user terminals 10 and 20 has performed the particular exercise movement.

[0373] Furthermore, the learning unit 51 can identify the exercise and movement performed by the subject U based on the information input by the system administrator.

[0374] Furthermore, the learning unit 51 can identify the exercise / motion performed by the subject U based on the position information of the learning target joint of the subject U included in the exercise video 300. In this case, the exercise / motion performed by the subject U can be identified by referring to the motion information of the exercise / motion stored in the database 40.

[0375] Meanwhile, the learning unit 51 may match a plurality of learning data sets 400 including the same motion code with each other based on the motion code included in the fourth data group 440 and store the matched data in the database 40 .

[0376] In this case, the learning unit 51 may divide and allocate the memory (or memory space) of the database 40 based on the learning code. In the present invention, dividing the memory (or memory space) of the database 40 may be understood as generating folders on the database 40 based on the motion codes. Furthermore, the learning unit 51 may store the learning data set 400 including a specific motion code in a folder corresponding to the specific motion code.

[0377] On the other hand, the motion analysis units 120 and 210 can analyze the exercise movements in the exercise video to be analyzed using posture estimation information estimated from a posture estimation model learned using a learning dataset including movement codes corresponding to specific exercise movements performed by the subject U.

[0378] Furthermore, the motion analysis unit 120, 210 can estimate the movement movement of subject U for a specific movement movement using posture estimation information estimated from a posture estimation model learned based on the learning target movement video 300 related to the same specific movement movement as that of subject U included in the analysis movement target.

[0379] Meanwhile, as shown in FIG. 8c, a fifth data group 450 among the multiple data groups 410 to 450 may include size information 451 of a bounding box 301 for a subject U included in the learning exercise video 300, and position information 452 of the center of the bounding box 301.

[0380] In the present invention, the "size information" of the bounding box 301 is sometimes mixed with the "scale".

[0381] The learning unit 51 extracts size information 451 of the bounding box 301 corresponding to the subject U detected in the training object motion video 300, extracts center position information 452 of the bounding box 301, and includes this in a fifth data group 450 to generate a learning dataset 400.

[0382] As described above, the learning unit 51 can use various algorithms for object detection to detect the subject U from the training target motion video 300. For example, the learning unit 51 can use an algorithm (Weighted Box Fusion, WBF) that ensembles a plurality of bounding boxes. However, it goes without saying that the learning unit 51 is not limited to the above-mentioned object detection algorithm, and can use various object detection algorithms that can detect an object corresponding to the subject U from the training target motion video 300.

[0383] Based on an object detection algorithm, the learning unit 51 can extract size information 451 and center position information 452 of a bounding box corresponding to the subject U from the learning target video 300, and include the size information 451 and center position information 452 in a fifth data group 450 to generate a learning dataset 400.

[0384] In this case, the learning unit 51 can extract the center position information 452 of the bounding box 301 in the form of a pair of coordinate information on the x-axis and y-axis.

[0385] Meanwhile, the learning unit 51 can configure the learning data set 400 including the video identification information of the training object exercise video 300 .

[0386] Here, "video identification information" refers to information for identifying the video 300 from which the information contained in the learning dataset was extracted, and may include, for example, file name information and file format type information (or extension information, e.g., "JPG", "TIF") of the training object exercise video 300.

[0387] In the present invention, the video identification information can be named a sixth data group corresponding to a sixth information attribute.

[0388] The learning unit 51 can generate the learning data set 400, including the sixth data group constituted by the video identification information.

[0389] Meanwhile, in the present invention, the motion analysis result can be provided to the patient terminal 10 based on the completion of the analysis of the patient's motion based on the motion analysis units 120 and 210. As shown in Figures 9a, 9b and 9c, the motion analysis result can be provided through the exercise therapy application 100 installed in the patient terminal 10.

[0390] As shown in (a) of Fig. 9a, the exercise therapy application 100 can provide a service page configured to enable access to each of the multiple services provided by the present invention on the patient terminal 10. For example, the service page may be configured to enable access to at least one of i) an exercise guide page linked to an exercise guide information providing function for an exercise plan assigned to a patient account, ii) an exercise page related to the execution of an exercise plan assigned to a patient account (see (b) of Fig. 9a), iii) a function evaluation page linked to a function evaluation (see (c) of Fig. 9a), and iv) a plan evaluation page linked to an exercise plan evaluation function.

[0391] Furthermore, as shown in Fig. 9b, the exercise therapy application 100 can provide an exercise report page that provides an exercise report based on the exercise motion analysis result and the exercise execution result on the patient terminal 10. For example, the exercise report page may include at least one of exercise execution rate information (see Fig. 9b (a) and (b)) and exercise plan difficulty information (see Fig. 9b (c)).

[0392] Furthermore, as shown in FIG. 9c, the exercise therapy application 100 can provide the patient's exercise motion analysis results on the patient terminal 10.

[0393] As shown in (a) of Fig. 9c, the exercise therapy application 100 can provide motion analysis information for each of a plurality of joint points of a subject U performing a specific prescribed motion (e.g., "stretching arms out to the side") on the patient terminal 10. For example, the exercise therapy application 100 can provide a joint range of motion of a joint point located on a first side (e.g., left side) of the patient and a joint range of motion of a joint point located on a second side (e.g., right side) of the patient on the patient terminal 10 as a graph.

[0394] As shown in (b) of FIG. 9c, the exercise therapy application 100 can provide daily exercise motion analysis results of a patient who has performed prescribed exercises according to an exercise plan for a certain period of time. For example, the exercise therapy application 100 may provide joint range of motion information corresponding to a first exercise day and joint range of motion information corresponding to a second exercise day. The exercise therapy application 100 may also provide average joint range of motion for the first exercise day and the second exercise day.

[0395] As shown in (c) of Fig. 9c, the exercise therapy application 100 may render graphic objects corresponding to key points P1 and P2 in the exercise video 300 and provide them on the patient terminal 10. In this case, the exercise therapy application 100 may also provide the patient's joint range of motion information (angle information) together.

[0396] Meanwhile, the exercise treatment providing system 1000 according to the present invention can provide the exercise motion analysis results provided to the patient terminal 10 to the doctor terminal 20 so that the doctor can monitor the execution of the patient's exercise plan.

[0397] As shown in Fig. 10, the exercise motion analysis system 100 according to the present invention may be configured to include at least one of a database 110, a motion analysis server 120, and a service server 130. The database 110 is a storage in which a learning data set is stored, and may be provided in the exercise motion analysis system 100 according to the present invention itself, or may be formed as an external storage (or an external DB). The database 110 according to the present invention can be understood to be sufficient as long as it is a space in which a learning data set is stored, and there is no restriction on the physical space.

[0398] In database 110, training data for training pose estimation model (or artificial intelligence pose estimation model) 122 may be stored and present as a training data set.

[0399] 8b, the learning data set 400 in the present invention may be composed of a plurality of data groups 410-450 corresponding to different information attributes 410a-450a, respectively. The information included in each of the plurality of data groups 410-450 may be extracted and composed from an exercise video 300 including a subject U performing an exercise motion.

[0400] Here, the "exercise video 300" refers to a video (image or video) that captures (includes) the process of a user performing an exercise motion, as shown in FIG. 8a, and may include at least a part of the user U's body.

[0401] In the present invention, a user object included in the exercise video 300 is named and described as "subject U." In the present invention, "subject U" may refer to a user exercising in the exercise video or a part of the user's body. Therefore, in the present invention, "subject" and "user" may be used interchangeably, and are described with the same reference numeral "U."

[0402] Meanwhile, the 'exercise video 300' described in the present invention may include an 'analysis target exercise video' and a 'learning target exercise video'.

[0403] The "motion video to be analyzed" is the motion video that is the subject of posture estimation analysis of subject U, and the "motion video to be learned" can be understood as the motion video that is the subject of machine learning for the posture estimation model.

[0404] Meanwhile, in the present invention, the motion analysis server 120 may include at least one of the components of the learning unit 121, the posture estimation model 122, and the motion analysis unit 123. The motion analysis server 120 may be provided inside the motion analysis system 100 according to the present invention, or may be formed as an external server. That is, the motion analysis server according to the present invention performs at least one of the functions of learning for posture estimation, posture estimation, and analysis of motion and can be understood as having no physical space constraints.

[0405] The learning unit 121 may be configured to perform learning for a posture estimation model based on the training object motion video 300. The learning unit 121 can train the posture estimation model using the training data. Furthermore, the learning unit 121 can of course train the motion analysis unit 123.

[0406] As shown in (a) of FIG. 8b, the learning unit 121 can detect a subject U from the training object motion video 300 and extract various learning data used for motion posture estimation from the detected subject U. Such learning data may be used interchangeably with "information", "data", "data value", or "data value". Meanwhile, extraction of learning data may be performed by other means instead of the learning unit 121.

[0407] The learning unit 121 can use various algorithms for object detection to detect the subject U from the training object motion video 300. For example, the learning unit 121 can use an algorithm (Weighted Box Fusion, WBF) that ensembles a plurality of bounding boxes. However, it goes without saying that the learning unit 121 is not limited to the above-mentioned object detection algorithm, and can use various object detection algorithms that can detect an object corresponding to the subject U from the training object motion video 300.

[0408] The learning unit 121 can classify the extracted learning data into any one of a plurality of data groups 410 to 450 corresponding to each of a plurality of different information attributes 410a to 450a.

[0409] The multiple information attributes 410a-450a different from each other described in the present invention may be predefined and exist as shown in (b) of Fig. 8b. Also, the multiple data groups 410-450 corresponding to the multiple information attributes 410a-450a, respectively, may include learning data corresponding to the predefined information attributes.

[0410] For example, i) the data group 410 corresponding to the first information attribute 410a may include joint point position information of the subject U, ii) the data group 420 corresponding to the second information attribute 420a may include information indicating whether the joint points of the subject U are visible or not, iii) the data group 430 corresponding to the third information attribute 430a may include information regarding the shooting direction of the subject U, iv) the data group 440 corresponding to the fourth information attribute 440a may include exercise code information for classifying the exercise action (or exercise type) performed by the subject U, and v) the data group 450 corresponding to the fifth information attribute 450a may include the size and center position information of a bounding box for the subject U.

[0411] Here, the "joint points P1, P2" may refer to the user's joints or a region corresponding to the subject U's joints in the motion image 300.

[0412] The learning unit 121 can mutually link a plurality of data groups 410-450 extracted from the learning object exercise video 300, and generate (construct) a data set for the learning object exercise video 300. Furthermore, the learning unit 121 can store the generated learning data set 400 in the database 110. The database 110 can be constructed as the database 110 for the posture estimation model 122 based on the storage of the learning data set 400 generated by the learning unit 121.

[0413] Furthermore, the learning unit 121 can perform learning for the posture estimation model 122 based on a learning dataset 400 present in the database 110. As described above, the learning dataset 400 may include position information of joint points.

[0414] The posture estimation model 122 is a posture estimation model trained using a training data set including position information of joint points. The posture estimation model 122 can extract (or identify) joint points of the subject U from the analysis target motion video, and further extract position information of the joint points. That is, the posture estimation model 122 may be configured to extract or estimate key points corresponding to the joint points of the subject U included in the analysis target motion video. In the present invention, for convenience of explanation, the process of extracting or estimating joint points (or key points) is collectively referred to as "estimation".

[0415] The motion analysis unit 123 can analyze (or estimate) the motion of the subject U from the analysis target motion video by using position information (or key points) of the joint points of the subject U extracted from the posture estimation model 122. The motion analysis unit 123 may be named an "artificial intelligence motion analysis model" or a "motion analysis model".

[0416] The motion analysis unit 123 can perform motion analysis using at least one of the learning data generated by the learning unit 121 and the posture estimation information estimated from the posture estimation model 122 learned by the learning data. It goes without saying that the motion analysis unit 123 may be configured to learn based on the learning data generated by the learning unit 121.

[0417] In the present invention, "pose estimation" refers to extracting position information of joint points or key points corresponding to joint points from an image, and "motion analysis" refers to analyzing a movement using the extracted position information or key points.

[0418] Meanwhile, the motion analysis information of the subject U that the motion analysis unit 123 can analyze from the analysis target motion image 300 may be various. For example, the motion analysis unit 123 may estimate and analyze, from the joint points extracted or estimated by the posture estimation model 122, at least one of information regarding i) a joint movable range of the joint point, ii) a movement path of the joint point, iii) a connection relationship between the joint points, and iv) a symmetrical relationship of the joint points with respect to the subject U.

[0419] In addition, the motion analysis unit 123 can analyze at least one of the following from the key points extracted from the movement video 300 to be analyzed or from the video 300: the moving distance of the joint, the movement speed (or acceleration) of the joint, the body balance, body equilibrium, and body alignment (e.g., leg alignment, spinal alignment, etc.) of the subject (corresponding to the patient) included in the movement video to be analyzed.

[0420] In the present invention, posture estimation model 122 may be configured to include learning unit 121. Conversely, learning unit 121 may include posture estimation model 122, in which case posture estimation function can be achieved by having posture estimation model 122 learn by learning unit 121. Therefore, in the present invention, the functions performed by posture estimation model 122 can be mixed with those performed by learning unit 121.

[0421] On the other hand, the service server 130 may be configured to provide a motion analysis result service that provides the user's motion analysis result (or motion analysis report) 220 to the user terminal 200 through the motion analysis unit 123 that performs analysis using key points estimated by the posture estimation model 122 (see FIG. 10).

[0422] Such a service server 130 may be configured to communicate with the user terminal 200. In the present invention, the communication by the service server 130 may be understood to mean that the communication is performed by a communication unit of the service server 130.

[0423] For example, the communication unit of the service server 130 may support any of the following standards: WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance: registered trademark), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth TM The wireless communication device may be configured to communicate with the user terminal 200 using at least one of the following technologies: Wireless LAN Card (registered trademark), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Wi-Fi Direct, and Wireless Universal Serial Bus (Wireless USB).

[0424] Meanwhile, the user terminal 200 described in the present invention means an electronic device, and may include at least one of a smart phone, a mobile phone, a tablet PC, a kiosk, a computer, a laptop, a digital broadcast terminal, a PDA (Personal Digital Assistant), and a PMP (Portable Multimedia Player). Furthermore, the user terminal 200 may be an electronic device to which a user account is logged in, connected, or registered.

[0425] Here, the user account may refer to an account that is registered in advance in the exercise motion analysis system 100 according to the present invention. Such a user account may be understood as a user ID (identification, identification number).

[0426] Meanwhile, in the present invention, a process of receiving an exercise video from a user terminal 200 can be performed (S210, see FIG. 11). The service server 130 can receive an exercise video 300 in which a user performs an exercise motion by communicating with the user terminal 200.

[0427] In this case, the exercise video 300 that the service server 130 receives from the user terminal 200 can be understood as an analysis target exercise video that is a target for analyzing the user's exercise motion. Meanwhile, the learning unit 121 can use the analysis target exercise video to not only estimate a posture for extracting key points for the user's joint points, but also use the analysis target exercise video 300 as a learning target exercise video to perform learning for a posture estimation model.

[0428] The service server 130 may receive the exercise video to be analyzed from the user terminal 200 according to various times and routes.

[0429] 10, the service server 130 can control the camera state to an active state so that the camera 201 of the user terminal 200 captures the exercise video to be analyzed based on the selection of a graphic object 220 corresponding to "start exercise" from the user terminal 200. In addition, the service server 130 can receive the exercise video to be analyzed captured by the camera 201 from the user terminal 200 in real time or based on the completion of the user's exercise.

[0430] Next, in the present invention, a process of analyzing a movement motion related to a specific movement motion of a user included in a movement video can be performed based on posture estimation information extracted from a posture estimation model trained using a training data set including position information of joint points (S220, see FIG. 11).

[0431] When receiving an analysis target motion video from the user terminal 200, the motion analysis unit 123 can estimate the motion of the user U included in the analysis target motion video by using posture estimation information extracted based on the posture estimation model 122 learned using the training target motion video 300. Here, the posture estimation information may include at least one of information on joint points (or key points) P1 and P2 and position information therefor.

[0432] The motion of the user U analyzed by the motion analysis unit 123 may include various information. For example, the motion analysis unit 123 can estimate joint movable range information (angle information) of the subject U based on position information of the joint points P1 and P2 of the subject U.

[0433] Next, in the present invention, based on the completion of the analysis, a process of providing the user U's exercise motion analysis result related to a specific exercise motion to the user terminal 200 can be performed (S230, see FIG. 11).

[0434] The service server 130 can process the analysis result of the user's exercise motion to generate an exercise motion analysis report. Also, the service server 130 can provide the exercise motion analysis report on the user terminal 200.

[0435] 10, the service server 130 can provide a user's motion video by rendering joint point graphic objects corresponding to the joint points P1 and P2 of the user U at positions corresponding to the joint points P1 and P2 of the user U. In addition, the service server 130 can display joint movable range information 221 of a specific joint point P1 around the specific joint point P1.

[0436] The exercise motion analysis system 100 according to the present invention can perform learning for the posture estimation model 122 using the database 110 constructed based on the training subject exercise video 300. Also, based on the posture estimation information acquired using the posture estimation model 122, the motion analysis unit 123 analyzes the exercise motion of the user, and as a result, can provide an exercise motion analysis result providing service.

[0437] The information included in the analysis result may be various. For example, the analysis result may include analysis information on at least one of the range of motion of a joint, the distance of motion, the speed (or acceleration) of joint movement, the body balance, the body equilibrium, and the body alignment state (e.g., leg axis alignment state, spine alignment state, etc.) of a subject (corresponding to a patient) included in the motion image to be analyzed, analyzed from the extracted key points or image.

[0438] Furthermore, such analytical information may further include a score, which may be an analysis score of the user's athletic movement (or posture). Such an analysis score may be calculated based on various methods (e.g., rule-based analysis based on pre-set criteria or analysis by an artificial intelligence algorithm).

[0439] The database 110 may store and exist a learning data set 400 extracted and generated from the training object exercise video 300 by the learning unit 121 .

[0440] The learning data set 400 used for estimating a user's exercise posture will be described in more detail below.

[0441] Meanwhile, in the present invention, the analysis of the user's exercise movements contained in the exercise video to be analyzed is completed through posture estimation information extracted based on the posture estimation model 122, and the exercise movement analysis results can be provided to the user terminal 200.

[0442] 12(a), the service server 130 can control the camera state to an active state so that the camera 201 of the user terminal 200 captures the exercise video to be analyzed, based on the selection of a graphic object 210 corresponding to "start exercise" from the user terminal 200. In addition, the service server 130 can receive the exercise video to be analyzed captured by the camera 201 from the user terminal 200 in real time or based on the completion of the user's exercise.

[0443] As shown in (b) of FIG. 12, the service server 130 may provide the user terminal 200 with a motion analysis result using the analysis result based on the posture estimation model 122, when the analysis of the motion of the user included in the analysis target video is completed. The motion analysis result provided on the user terminal 200 by the service server 130 may include various information. For example, the service server 130 may render a graphic object corresponding to joint points P1 and P2 for the subject U corresponding to the user, and provide it on the user terminal 200. In this case, the service server 130 may also provide joint movable range information (angle information) 221 of the subject U.

[0444] 12(c), the service server 130 can provide, on the user terminal 200, motion posture information 820 for each of a plurality of joint points of the subject U performing a specific motion posture (e.g., "stretching arms out to the side") 810. For example, the service server 130 can provide, on the user terminal 200, the joint motion ranges of the joint points located on the first side (e.g., left side) of the user and the joint motion ranges of the joint points located on the second side (e.g., right side).

[0445] Meanwhile, the service server 130 can prescribe (provide) an exercise plan (or exercise program) including at least one exercise action to the user based on the action analysis result for the specific exercise action of the user.

[0446] For example, as shown in (a) of FIG. 13, the service server 130 can add exercise actions to an exercise plan (or exercise program) based on the exercise action analysis results, and provide a service page 910 on the user terminal 200 including information on the exercise plan (or exercise program) including the additional exercise actions.

[0447] As another example, as shown in (b) of FIG. 13, the service server 130 can adjust the difficulty of the exercise plan (or exercise program) based on the movement analysis posture results and provide a service page 920 including the adjusted difficulty information on the user terminal 200.

[0448] As another example, as shown in (c) of FIG. 13, the service server 130 can exclude some of the exercise movements from the exercise plan (or exercise program) based on the results of the exercise movement analysis, and provide a service page 930 on the user terminal 200 informing the user that the exercise movements have been excluded.

[0449] As described above, the motion posture estimation and motion analysis method and system according to the present invention can analyze a motion movement related to a specific motion movement of a user included in a motion video based on a posture estimation model and a motion analysis model (or a motion analysis unit) trained using a training data set including position information of joint points. As a result, the present invention can accurately analyze a user's posture from a motion video, and in particular, can improve the quality of medical services by acquiring information on the user's joint range of motion, alignment state, and disengagement state.

[0450] Furthermore, the exercise posture estimation and motion analysis method of the present invention provides the user's exercise motion analysis results related to a specific exercise motion to the user terminal, allowing patients to receive rehabilitation treatment without having to go directly to a hospital located far away, making it easy to access rehabilitation treatment, and medical staff can easily monitor the patient's rehabilitation exercise through electronic devices and provide feedback based on the monitoring, thereby improving the effectiveness of the patient's exercise treatment.

[0451] As described above, the system and method for providing exercise therapy using an AI posture estimation model and a motion analysis model according to the present invention can receive prescription information on exercises for a patient from a doctor terminal, and assign an exercise plan including at least one prescribed exercise to the patient account based on the prescription information. This allows the doctor to prescribe to the patient even if the doctor and the patient do not meet in person for exercise therapy for musculoskeletal disorders, and the patient receives an exercise plan according to the doctor's prescription, thereby eliminating spatial, time, and economic constraints on exercise therapy and improving access to exercise therapy.

[0452] Furthermore, the system and method for providing exercise therapy using the artificial intelligence posture estimation model and motion analysis model of the present invention can analyze the user's exercise movements by extracting key points corresponding to a plurality of pre-set joint points from the exercise video, focusing on the joints required for exercise therapy of musculoskeletal diseases.

[0453] Furthermore, the system and method for providing exercise therapy using an AI posture estimation model and a motion analysis model according to the present invention can analyze a motion related to a specific motion of a user included in an exercise video based on a posture estimation model trained using a training data set including position information of joint points. As a result, the present invention can accurately analyze the posture of a patient from an exercise video, and in particular, obtain information on the range of motion of the patient's joints, alignment state, and disengagement state, thereby improving the quality of medical services.

[0454] Furthermore, the system and method for providing exercise therapy using the artificial intelligence posture estimation model and motion analysis model of the present invention transmits the analysis results of the patient's exercise motion to the patient terminal, so that the patient can receive feedback on the exercise video and improve the effectiveness of the exercise therapy without having to go directly to a hospital located far away.

[0455] Meanwhile, the present invention described above can be embodied as a program that is executed by one or more processes in a computer and that can be stored in a medium (or recording medium) that can be read by such a computer.

[0456] Furthermore, the present invention described above may be embodied as computer readable codes or instructions on a medium having a program recorded thereon, that is, the present invention may be provided in the form of a program.

[0457] On the other hand, the computer readable medium includes any kind of recording device in which data readable by a computer system is stored. Examples of the computer readable medium include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0458] Furthermore, the computer-readable medium may include a storage and may be a server or cloud storage that the electronic device can access through communication. In this case, the computer can download the program according to the present invention from the server or cloud storage through wired or wireless communication.

[0459] Furthermore, in the present invention, the above-mentioned computer is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit), and the type of the computer is not particularly limited.

[0460] However, the above detailed description should not be construed as limiting in all respects, but should be considered as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the scope of the present invention are included in the scope of the present invention.

Claims

1. Receiving prescription information for a patient regarding exercise from a doctor terminal; assigning an exercise plan to the patient account based on the prescription information, the exercise plan including at least one prescribed exercise; receiving, from a patient terminal, an exercise video of the exercise corresponding to the prescribed exercise; Extracting key points corresponding to a plurality of pre-defined joint points from the motion video using an artificial intelligence posture estimation model; Analyzing the relative positional relationship between the key points through an artificial intelligence motion analysis model, and analyzing the patient's exercise motion with respect to the prescribed exercise based on the analysis of the positional relationship; A method for providing exercise therapy using an artificial intelligence motion analysis model, comprising a step of transmitting analysis results of the patient's exercise motion to the patient terminal.

2. outputting the exercise video to the patient terminal in real time in conjunction with the exercise video being captured by the patient terminal; 2. The method for providing exercise therapy using an artificial intelligence motion analysis model of claim 1, further comprising: providing a graphic object corresponding to the extracted key point in an area where a subject corresponding to the patient is located in the exercise video so that the patient can recognize the joint point where analysis is performed on the patient's exercise posture.

3. In the step of extracting key points, Identifying visible joint points of the subject that are visible in the motion image from among the plurality of preset joint points; The method for providing exercise therapy using an artificial intelligence motion analysis model according to claim 2, further comprising extracting the identified visible joint points as the key points.

4. The motion analysis model predicts invisible joint points of the subject that are not visible in the motion image among the plurality of preset joint points based on the learning data, and analyzes the motion of the patient based on the visible joint points and the invisible joint points; The method of claim 3, wherein the key points include key points corresponding to the visible joint points and the invisible joint points.

5. The learning data is a first data group in which position information of a training target visible joint point of a subject included in a training target image and a training target invisible joint point estimated based on the visible joint point are sequentially listed; a second data group including data values ​​indicating whether each of the visible and invisible joint points to be trained is visible; The method for providing exercise therapy using an artificial intelligence motion analysis model according to claim 4, wherein the enumeration order of the data values ​​included in the second data group is the same as the enumeration order of the visible joint points and the invisible joint points.

6. The step of analyzing the patient's athletic motion comprises: Analyze the relative positional relationship between the key points based on the rule information regarding the prescribed exercise; The method for providing exercise therapy using the artificial intelligence motion analysis model of claim 2, further comprising: analyzing the patient's exercise motion by determining whether the relative positional relationship between the key points satisfies the rule information.

7. The method for providing exercise therapy using an artificial intelligence motion analysis model according to claim 6, wherein visual appearances of the graphic objects superimposed on the exercise video are different depending on whether a relative positional relationship between the extracted key points satisfies the rule information.

8. The analysis result of the patient's exercise motion is a first analysis result in which the graphic objects corresponding to the key points are superimposed on the exercise video in real time with different visual appearances based on the rule information while the exercise video is being captured by the patient terminal; A second analysis result including an evaluation score of the patient for the prescribed exercise based on key points extracted from each of the plurality of frames constituting the exercise video; The first analysis result is generated by a behavior analysis model of an application installed on the patient terminal; The second analysis result is generated by a cloud server that is in conjunction with the application; The method for providing exercise therapy using an artificial intelligence motion analysis model according to claim 7, wherein both the first analysis result and the second analysis result are transmitted to the doctor's terminal.

9. A communication unit that receives prescription information regarding exercise for a patient from a doctor terminal; and a control unit that assigns an exercise plan, including at least one prescribed exercise, to the patient account based on the prescription information; The control unit is receiving, from a patient terminal, an exercise video of the exercise corresponding to the prescribed exercise; Analyzing the patient's exercise motion for the prescribed exercise from the exercise video using an artificial intelligence motion analysis model; The system for providing exercise therapy further comprises transmitting an analysis result of the exercise motion of the patient to the patient terminal.

10. A program executed by one or more processes in an electronic device and stored on a computer-readable recording medium, The program is Receiving prescription information for a patient regarding exercise from a doctor terminal; assigning an exercise plan to the patient account based on the prescription information, the exercise plan including at least one prescribed exercise; receiving, from a patient terminal, an exercise video of the exercise corresponding to the prescribed exercise; extracting key points corresponding to a plurality of preset joint points from the motion image; Analyzing the relative positional relationship between the key points through an artificial intelligence motion analysis model, and analyzing the patient's exercise motion with respect to the prescribed exercise based on the analysis of the positional relationship; transmitting an analysis result of the exercise movement of the patient to the patient terminal.

11. receiving an exercise video from a user terminal; Analyzing a movement motion related to a specific movement motion of the user included in the movement video based on posture estimation information extracted from a posture estimation model trained using a training data set including position information of joint points; and providing the user terminal with an athletic motion analysis result of the user related to the specific athletic motion based on completion of the analysis of the athletic motion; The position information of the joint points included in the learning data set is position information of each of a plurality of pre-specified learning target joint points among joint points of a subject included in a learning target motion video, A method for estimating exercise postures using training data, characterized in that the training data set is composed of data extracted from the training target exercise video.

12. The learning data set is composed of a plurality of data groups each corresponding to a different information attribute, A first data group among the plurality of data groups includes position information of each of the plurality of training target joint points, The method for estimating a motion posture using training data according to claim 11, characterized in that the position information included in the first data group includes coordinate information of each of the pre-specified training target joint points in the training target motion image in a form of a pair.

13. The location information included in the first data group is The plurality of training object joint points are defined as different types of information depending on whether they are visible in the training object image, The definition of the type of the position information is as follows: The method of claim 12, wherein the estimation is performed according to a data value of data included in a second data group different from the first data group.

14. A learning unit that learns the posture estimation model using the learning data, 14. The method of claim 13, further comprising: setting different learning weights for each of the position information of the plurality of training target joint points included in the first data group based on the data values ​​included in the second data group.

15. The position information of each of the plurality of training target joint points included in the first data group is sequentially arranged in the first data group based on a predefined order among the plurality of training target joint points; 15. The method of claim 14, wherein the data values ​​included in the second data group are arranged in the second data group in the same order as the predefined order in which the position information of each of the plurality of training target joint points is arranged so as to indicate whether each of the plurality of training target joint points is visible.

16. The training data set further includes a third data group including data values ​​related to a shooting direction of the object; The data values ​​included in the third data group are configured to have different data values ​​according to a photographing direction of the object based on a camera that photographs the object, the pose estimation model is configured to be trained taking into account a shooting direction of the subject through the training data set having different data values ​​according to a shooting direction of the subject; The exercise posture estimation method using learning data according to claim 15, characterized in that the user's exercise motion analysis result is a result of analyzing a specific exercise motion of the user based on posture estimation information extracted in the posture estimation model by taking into account the shooting direction of the user included in the exercise video.

17. The exercise video received from the user terminal and the training target exercise video correspond to the specific exercise motion having the same exercise code, The method of claim 11, wherein the training data set includes a fourth data group including a motion code that is matched to the specific motion movement performed by the subject in the training motion video.

18. The training exercise video is a video; The training data set is The motion posture estimation method according to claim 11, characterized in that the method is composed of learning data extracted from each of the reference frames selected based on a preset criterion from among the multiple frames constituting the training motion video, with a focus on the subject included in the training motion video.

19. A posture estimation model that extracts posture estimation information using a training data set including position information of joint points; a trained motion analysis model that analyzes a motion associated with a specific motion of a user included in an analysis target motion video using the posture estimation information; a service server that provides the user terminal with an exercise motion analysis result of the user related to the specific exercise motion based on completion of the analysis; The position information of the joint points included in the learning data set is position information of each of a plurality of pre-specified learning target joint points among joint points of a subject included in a learning target motion video, The exercise posture estimation system, wherein the training data set is composed of data extracted from the training target exercise video.

20. receiving an exercise video from a user terminal; Analyzing a movement motion related to a specific movement motion of the user included in the movement video based on posture estimation information extracted from a posture estimation model trained using a training data set including position information of joint points; and providing, to the user terminal, an athletic motion analysis result of the user related to the specific athletic motion based on completion of the analysis, The position information of the joint points included in the learning data set is position information of each of a plurality of pre-specified learning target joint points among joint points of a subject included in a learning target motion video, A program executed by one or more processes in an electronic device and stored on a computer-readable recording medium, wherein the learning data set is composed of data extracted from the training subject motion video.

Images