Exercise assistance service providing robot and exercise assistance service providing method

Abstract

A robot for providing an exercise assistance service using posture recognition, according to embodiments of the present invention, comprises: a content reproduction unit that reproduces exercise content for assisting exercise performance of a user; a photographing unit that photographs the user performing exercise according to the exercise content and generates first photographing information; a posture scoring unit that recognizes a posture of the user from the first photographing information generated by the photographing unit, and generates, on the basis of the degree of matching between the recognized posture of the user and an exemplary posture presented in the exercise content, posture score information for a score given to the posture of the user; and an update unit that updates, on the basis of the posture score information, the exercise content provided to the user.

Description

Exercise assistance service providing robot and exercise assistance service providing method

[0001] The present invention relates to a motion assistance service providing robot for providing a service that assists a user's motion and a motion assistance service providing method using the motion assistance service providing robot.

[0002] Recent robot development is being reoriented towards the development of companion robots that are closely related to and can deeply penetrate human lives, rather than simply mechanical computing devices as in the past. These companion robots are called interaction robots because they interact with humans.

[0003] However, most of the interactions performed by currently released interactive robots, such as smart speakers and toy robots, are simple and passive actions that receive input from users and output responses accordingly. Therefore, it is difficult for robots to recognize the user's situation and actively provide the necessary services to the user according to the situation. Since it is more appropriate to view them as simple computing devices, the differentiation from them is becoming more important.

[0004] Meanwhile, with the improvement of living standards and advancement of medical technology, the number of elderly people is increasing, and the number of people with motor disabilities due to various industrial accidents, traffic accidents, stroke, etc. is also increasing.

[0005] For the physical promotion of elderly people and musculoskeletal exercise and rehabilitation training for patients with motor disabilities, regular prescribed exercise or rehabilitation training is required at a set time every day.

[0006] Regularly prescribed exercise or rehabilitation training like this should be able to include at least one of neuromuscular exercise, flexibility exercise, resistance exercise, and aerobic exercise tailored to the user's condition in order to strengthen muscle strength, improve endurance, restore sensorimotor control, and restore cardiovascular function. Therefore, special exercise elements and specialized exercise equipment are required, and special attention is required according to the prescription of a specialist.

[0007] Therefore, in the past, there was a problem in that one had to rely on specialized institutions such as hospitals and use various exercise machines installed in specialized institutions such as hospitals, and proceed by changing various exercise equipment under the guidance of a physical therapist.

[0008] For the elderly or patients with motor impairments, it is difficult to visit a specialized institution every day and perform prescribed exercises at a set time, and even within a specialized institution such as a hospital, it is difficult to move exercise equipment depending on the type of exercise.

[0009] In addition, even in the case of the elderly or patients with motor impairments, not all situations are the same, and although they all differ depending on gender, age, constitution, and health condition, there is a problem in that it is impossible to collect and evaluate continuous data of the user's daily condition or exercise posture, and apply exercise treatment tailored to the user by modifying it in various ways according to the exercise purpose according to the kinematic principle.

[0010] In addition, conventionally, one-way exercise coaching services have been provided using multimedia content, which has the problem of not being able to provide feedback on whether the user is performing the correct movements or performing the appropriate exercises.

[0011] In addition, there were problems such as the content for exercise coaching being one-time and not being able to expect changes in user level or consistent exercise effects, the lack of control over one's own will making it difficult to exercise regularly, and the lack of motivation through competition or empathy with others making it easy to lose interest.

[0012] The present invention provides a robot for providing exercise assistance services and a method for providing exercise assistance services that recognize the user's posture when the user performs exercise using exercise content and provide differences from the exemplary posture of the exercise content, thereby enabling the user to accurately identify and correct any deficiencies.

[0013] In addition, the present invention aims to provide a robot for providing exercise assistance services and a method for providing exercise assistance services that can alleviate the self-occlusion effect, in which the accuracy of posture recognition is reduced by the overlapping body of the user, by photographing the posture of the user using exercise content from various angles.

[0014] In addition, the present invention seeks to provide an exercise assistance service providing robot and an exercise assistance service providing method that display the posture of a user using exercise content and an exemplary posture overlappingly so that the user can more easily follow the exemplary posture and increase the effect of exercise.

[0015] In addition, the present invention aims to provide an exercise assistance service providing robot and an exercise assistance service providing method that accumulate information on the posture of a user using exercise content, checks the health status of the user that changes over time, and provides continuous health management services.

[0016] In addition, the present invention seeks to provide a robot providing exercise assistance service and a method for providing exercise assistance service that can enhance the concentration and interest in exercise content by allowing users using exercise content to share or compete with other users, thereby increasing the effectiveness of exercise.

[0017] However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

[0018] As a means for achieving the above-described technical problem, one embodiment of the present invention includes a robot that provides an exercise assistance service through posture recognition, the robot including: a content playback unit that plays exercise content for assisting a user's exercise performance; a shooting unit that shoots a user performing an exercise according to the exercise content to generate first shooting information; a posture score unit that recognizes the posture of the user from the first shooting information generated through the shooting unit and generates posture score information for a score assigned to the posture of the user according to the degree of matching between the recognized posture of the user and an exemplary posture presented in the exercise content; and an update unit that updates the exercise content provided to the user according to the posture score information.

[0019] The above-described problem-solving methods are merely exemplary and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, additional embodiments may exist, as described in the drawings and detailed description of the invention.

[0020] According to any one of the problem solving means of the present invention described above, when a user performs an exercise using exercise content, the user's posture can be recognized and differences from the exemplary posture of the exercise content can be presented, so that the user can accurately identify and correct any shortcomings.

[0021] In addition, the present invention can alleviate the self-occlusion effect, which reduces the accuracy of posture recognition due to the overlapping body of the user, by capturing the posture of the user using exercise content from various angles.

[0022] In addition, the present invention displays the posture of a user using exercise content and a model posture overlappingly, so that the user can more easily follow the model posture, thereby increasing the effect of exercise.

[0023] In addition, the present invention can accumulate information on the posture of a user using exercise content to check the health status of the user that changes over time and provide continuous health management services.

[0024] In addition, the present invention can enhance the concentration and interest in exercise content by allowing users who use exercise content to share or compete with other users, thereby increasing the effectiveness of exercise.

[0025] Figure 1 is a configuration diagram of a robot providing exercise assistance service according to one embodiment of the present invention.

[0026] FIG. 2 is a drawing for explaining an example of a user using a robot providing exercise assistance service according to one embodiment of the present invention.

[0027] FIG. 3 is a drawing illustrating a model for recognizing a user's posture by analyzing shooting information in a motion assistance service providing robot according to one embodiment of the present invention.

[0028] FIG. 4 is a drawing for explaining an example of a process of superimposing an exemplary posture on a user's photographed information in a motion assistance service providing robot according to one embodiment of the present invention.

[0029] FIG. 5, FIG. 6, FIG. 7a and FIG. 7b are drawings illustrating a process of generating photographing information by photographing a user from different angles in a motion assistance service providing robot according to one embodiment of the present invention.

[0030] FIG. 8 is a drawing illustrating an algorithm used to generate photographing information by photographing a user from different angles in a motion assistance service providing robot according to one embodiment of the present invention.

[0031] FIG. 9 is a drawing illustrating a process for providing exercise assistance services between multiple users in a motion assistance service providing robot according to one embodiment of the present invention.

[0032] Figure 10 is a flowchart illustrating a method for providing exercise assistance services according to one embodiment of the present invention.

[0033] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement them. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity of description, and similar reference numerals have been used throughout the specification to indicate similar elements.

[0034] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between. Furthermore, when a part is said to "include" a component, this should be understood to mean that, unless specifically stated to the contrary, it may include other components rather than excluding them, and does not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0035] In this specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Furthermore, a single unit may be realized using two or more pieces of hardware, and two or more units may be realized by a single piece of hardware.

[0036] Some of the operations or functions described herein as being performed by a terminal or device may instead be performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by a server may also be performed by a terminal or device connected to the server.

[0037] Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0038] Figure 1 is a configuration diagram of a robot (100) providing exercise assistance service according to one embodiment of the present invention.

[0039] Referring to FIG. 1, the exercise assistance service providing robot (100) may include a content playback unit (110), a photographing unit (120), a posture scoring unit (130), an update unit (140), a database (150), a health status determination unit (160), a driving unit (170), a location tracking unit (180), and a communication unit (190). In addition, in another embodiment, the exercise assistance service providing robot (100) may include a robot arm (171) and a moving device (172).

[0040] The exercise assistance service providing robot (100) may be exemplified as a robot that operates based on a built-in power source or an external power source and communicates or interacts with a user. More specifically, the exercise assistance service providing robot (100) may include a body (not shown) and a movement device (172) for implementing a function for interacting with a user, and may also further include additional components necessary for driving other robots.

[0041] The body part is a central configuration in which major components of the exercise assistance service providing robot (100), such as a CPU (not shown), a battery (not shown), a communication unit (190), etc., are mounted inside or other components are connected to the outside, and may further include a shooting unit (120) for shooting a user, a microphone (not shown) for sensing the user's voice or ambient sound to generate related information, and a content playback unit (110) for outputting content composed of at least one of images, videos, texts, and sounds.

[0042] The content playback unit (110) may include at least one of a display and a speaker, wherein the display is a type of liquid crystal display device, and at least one of text, images, and videos of predetermined information may be output. Here, the predetermined information may include status information of the exercise assistance service providing robot (100), such as communication status strength information, remaining battery level information, and wireless Internet ON / OFF information. The display may display exercise content, which will be described later, and may also display information converted into text from voice output through the speaker of the content playback unit (110). For example, when the exercise assistance service providing robot (100) provides a voice corresponding to the exercise content to the user and gives an action instruction, text such as "Raise your right arm higher" may be output on the display.

[0043] The text output on such a display may be one of the previously described pieces of information repeatedly output, or multiple pieces of information may be output alternately, or specific information may be set as default and output. For example, status information of the exercise assistance service providing robot (100), such as communication status strength information, remaining battery level information, and wireless Internet ON / OFF information, may be set as small text at the top or bottom of the display by default and continuously output, and other information may be output alternately.

[0044] Meanwhile, since the display can output one or more of images and videos, in this case, it is preferable to implement the display as a liquid crystal display device with a high resolution and a large size, rather than a display that only outputs text, in order to improve visibility. As described below, the display may be configured on the exterior or interior of the exercise assistance service providing robot (100), or may be configured as a separate device on the exterior of the exercise assistance service providing robot (100).

[0045] The speaker can output various sounds, including voice. Here, voice corresponds to auditory information output by the exercise assistance service providing robot (100) to interact with the user, and the type of voice can be set in various ways through a media-only application installed on the user's terminal (not shown) or direct control of the exercise assistance service providing robot (100).

[0046] For example, you can choose the type of voice output through the speaker from among various voices, such as male voice, female voice, adult voice, and child voice, and you can even choose the type of language, such as Korean, English, Japanese, and French.

[0047] Meanwhile, although the speaker outputs audio, it can also perform the functions of a typical speaker, such as general sound output. For example, if a user wishes to listen to music through the exercise assistance service robot (100), the music can be output through the speaker. Furthermore, if a video is output through the display, sound synchronized with the video can be output through the speaker.

[0048] Each component included in the exercise assistance service providing robot (100) described below and the functions performed by the corresponding components are performed by a control signal output from a control unit (not shown) within the body part. However, for convenience of explanation, the explanation will continue below as being performed by the exercise assistance service providing robot (100).

[0049] The content playback unit (110) can play exercise content to assist the user in performing exercise.

[0050] Here, exercise content is content to assist the user's exercise, and may be input in advance into the exercise assistance service providing robot (100) from an external device, or may be provided in the form of real-time streaming from an external server to the exercise assistance service providing robot (100).

[0051] Exercise content may include content aimed at improving a user's physical abilities. For example, exercise content may include gymnastics, stretching, bodyweight exercises, dance, aerobics, yoga, and health clapping. It may also include content aimed at maintaining and improving physical function, such as simple movements, mental and physical relaxation, stretching, flexibility recovery, and muscle strengthening.

[0052] That is, exercise content may include content that assists the user in performing exercise by providing exercise methods that the user can view, hear, and follow. Furthermore, the content playback unit (110) may play the exercise content through at least one of the displays and speakers provided in the exercise assistance service providing robot (100).

[0053] In embodiments of the present invention, exercise content may be comprised of content that outputs exemplary postures to assist users in performing exercises or stretching. The exemplary postures may be implemented through silhouettes or model poses that users can follow by watching the exercise content being played.

[0054] On the other hand, the content playback unit (110) may play the exercise content through at least one of an external display device (not shown) connected wired or wirelessly to the exercise assistance service providing robot (100) and an external speaker device (not shown) connected wired or wirelessly. The external display device has the advantage of providing the exercise content on a larger screen than the display provided in the exercise assistance service providing robot (100), thereby allowing the user to recognize the exercise content more accurately. In addition, the external speaker device has the advantage of playing the exercise content with higher output than the speaker provided in the exercise assistance service providing robot (100), thereby allowing the user to accurately recognize the sound being played.

[0055] The exercise assistance service providing robot (100) can be connected to an external device or server via a wired or wireless communication unit (190), and can play exercise content on an external display device or external speaker device through data transmission and reception.

[0056] The exercise assistance service providing robot (100) can be connected to an external server or device through a wired or wireless network using a communication unit (190). For example, the exercise assistance service providing robot (100) can be connected to an external server (not shown) or another robot (not shown) through a network simultaneously or at time intervals. In addition, a network refers to a connection structure that enables information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), the Internet (WWW: World Wide Web), a wired and wireless data communication network, a telephone network, a wired and wireless television communication network, etc. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth, infrared, ultrasonic, visible light communication (VLC), and LiFi.

[0057] The photographing unit (120) can capture images of the user or the environment around the user through a visual sensor such as a camera. Examples of the visual sensor include, but are not limited to, a camera that generates photographing information of general RGB (Red Green Blue) pixels, an infrared camera for capturing an object in a dark environment, and a depth camera that can generate depth information regarding the distance to the subject. The photographing information may refer to an image including the user captured by the camera of the photographing unit (120). The photographing information may be visual information including an image or video of the user as a subject, and the photographing information may also include the user's body and objects or backgrounds around the user. The photographing information may then be analyzed by the posture scoring unit (130) and utilized to provide subsequent exercise assistance services.

[0058] The camera (120) can capture a user's image through a camera, and can rotate the camera in the up and down or left and right directions to change the capture angle as needed. A user image captured in a first direction by the camera (120) can be defined as first capture information, and a user image captured in a second direction by the camera (120) can be defined as second capture information.

[0059] Although not illustrated in FIG. 1, the exercise assistance service providing robot (100) may be equipped with a microphone, and the microphone may receive sounds from the user or the environment around the user through a sound sensor. Here, since the user's voice or ambient sounds are important information that are treated as user input in interactions with the user, it is preferable that the microphone be placed in a location where there are no elements that interfere with receiving the user's voice or ambient sounds. For example, if the microphone is placed immediately behind the speaker hole formed on the front of the exercise assistance service providing robot (100), the user's voice or ambient sounds will be able to be directly received.

[0060] Meanwhile, the microphone can be integrated with the speaker, rather than being a separate, independent component. In this case, the microphone placement considerations described above will be eliminated, and the internal space utilization within the body can also be improved.

[0061] In addition, the exercise assistance service providing robot (100) may be equipped with a proximity sensor, a laser scanner (lidar sensor), an RGBD sensor, a geomagnetic sensor, an ultrasonic sensor, an inertial sensor, a UWB sensor, etc. to obtain data on the user or the situation around the exercise assistance service providing robot (100).

[0062] The posture score unit (130) can recognize the user's posture from the shooting information generated through the shooting unit (120), and generate posture score information for the score given to the user's posture based on the degree of matching between the recognized user's posture and the exemplary posture presented in the exercise content. More specifically, the posture score unit (130) can include a skeletal information extraction model (131) that outputs joint coordinate information for the position of each joint of the user being shot, and a posture judgment model (132) that recognizes the user's posture from the joint coordinate information and gives a score for the posture the user is currently taking.

[0063] Here, the skeletal information extraction model (131) can analyze the above-described photographing information based on a deep neural network. More specifically, the skeletal information extraction model (131) analyzes the above-described photographing information based on a convolutional neural network among deep neural networks, and can output two-dimensional or three-dimensional joint coordinate information for the joint positions that constitute the user for each frame of the photographing information.

[0064] And the posture judgment model (132) can derive a joint-specific posture score based on the similarity between the user's joint-specific bending degree and the joint-specific bending degree of the exemplary posture presented in the exercise content based on the joint coordinate information, and derive the user's posture score information by synthesizing the joint-specific posture scores. And the posture judgment model (132) can reflect the joint-specific weights preset in the exercise content to the derived joint-specific posture scores, and derive the user's posture score information by synthesizing the joint-specific posture scores to which the weights are reflected.

[0065] The update unit (140) can update the exercise content provided to the user based on the posture score information. More specifically, when the exercise content is played through a display, the update unit (140) can superimpose an image of the user captured on the exemplary posture of the exercise content so that the user can follow the exemplary posture. Furthermore, when the exercise content is played through a speaker so that the user can follow the exemplary posture, the update unit (140) can provide voice guidance to improve the score given to the user's posture.

[0066] FIG. 2 is a drawing for explaining an example of a user using a robot providing exercise assistance service according to one embodiment of the present invention.

[0067] Referring to FIG. 2, it can be seen that the exercise assistance service providing robot (100) takes a picture of the user (10) through the photographing unit (120) near the user (10) and generates photographing information. In addition, exercise content can be played through the display of the content playback unit (110), and the user's score posture derived by analyzing the user's posture can be displayed. In addition, it can be seen that a voice guidance such as "Your posture is very good!" (201) is provided to the user through the speaker of the content playback unit (110). In addition, the exercise assistance service providing robot (100) is equipped with a moving device (172) at the bottom, and the wheels of the moving device (172) can be rotated by the control of the driving unit to move the exercise assistance service providing robot (100).

[0068] FIG. 3 is a drawing illustrating a model for recognizing a user's posture by analyzing shooting information in a motion assistance service providing robot according to one embodiment of the present invention.

[0069] For skeletal extraction, a skeletal extraction algorithm based on a convolutional neural network (CNN), one of the deep neural networks, can be used on a 2D or 3D image to extract 2D or 3D skeletal information. A convolutional neural network-based feature detection model can be applied as a skeletal information extraction model, and algorithms such as MoveNet, PoseNet, OpenPose, MediaPipe, AlphaPose, Nuitrack, and Kinect SDK can be utilized, for example. Skeletal information and values ​​calculated based on it can be effectively utilized to analyze a user's posture.

[0070] Referring to FIG. 3, the skeletal extraction model (131) can receive a 2D or 3D image as input and extract 2D or 3D coordinates and their reliability for each joint. At this time, the extracted joint data may be in the form of a graph in which the human body, such as the head, shoulders, chest, arms, hips, and legs, is represented as nodes and each node is connected through edges.

[0071] In order to improve the accuracy of the results, the skeleton extraction model (131) can extract joint information by selecting a model that extracts only the skeleton of one person or a model that extracts the skeletons of multiple people, depending on the type of service provided by the robot. For example, in the case of a service provided in a 1:1 format, the robot can use a model that extracts only the skeleton of one person, and when providing a service to multiple people in a household, it can use a model that can extract the skeletons of multiple people. In addition, the skeleton extraction model (131) can select a skeleton extraction model for one person or multiple versions depending on the number of people detected through the object detection model.

[0072] In the present invention, the skeleton extraction model (131) can recognize a human body from an image input through photographing information. As described above, when expressing a human body in the form of a graph, the two-dimensional or three-dimensional coordinates of joints including at least one of the head, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, chest, abdomen, pelvis, left knee, right knee, left foot, and right foot corresponding to each node may correspond to the output values ​​of the skeleton extraction model (131). The coordinate information for each joint generated by the skeleton extraction model (131) can be defined as joint coordinate information.

[0073] The posture determination model (132) can classify the user's posture based on the user's joint coordinate information generated through the skeleton extraction model (131). The posture determination model (132) can extract the position of a specific joint or the degree of bending of a specific joint and use it as a basis for posture determination. More specifically, when the user's body is graphed as described above, when each joint corresponds to a node of the graph and the body part connecting each joint corresponds to an edge, the posture determination model (132) can derive the angle between two edges connected to each node. For example, a node corresponding to the left elbow may be connected by an edge to a node corresponding to the left shoulder and a node corresponding to the left hand. In this case, the posture determination model (132) can derive the interior angle between the two edges and the edge generated from one node based on the coordinate information between the node corresponding to the left elbow and the three nodes connected through the two edges. This interior angle can be derived for each node connected to at least two edges.

[0074] Similarly, the posture judgment model (132) can analyze the exemplary posture of the exercise content and derive the internal angle of each joint of the model or avatar demonstrating the exemplary posture.

[0075] The posture judgment model (132) can compare the derived internal angles of each joint of the user with the internal angles of each joint of the exemplary posture of the currently playing exercise content. Through this, if there is a difference of a predetermined angle or more between the internal angles of each joint of the currently playing user and the exemplary posture, the posture judgment model (132) can determine that the user is not properly following the currently playing exemplary posture. In addition, the posture judgment model (132) can derive a posture score for each joint of the user based on the difference in the bending angle of each joint of the user and the joint of the exemplary posture. At this time, different scores can be derived depending on the degree of difference in the bending angle. For example, if the internal angle of the left elbow in the exemplary posture is 180 degrees, a method can be utilized in which 90 points are awarded if the user's left elbow satisfies the range of 160 to 170 degrees, and 80 points are awarded if the user's left elbow satisfies the range of 150 to 160 degrees.

[0076] Thereafter, the posture judgment model (132) can derive the user's overall posture score by adding up the posture scores for each joint. In other words, the higher the overall posture score, the more accurately the user can be considered to be following the exemplary posture of the currently playing exercise content.

[0077] On the other hand, the posture judgment model (132) can assign different weights to each joint when deriving a joint-to-joint posture score for each exemplary posture. This is because, for each exemplary posture, the user must follow certain joints more precisely than others to enhance the exercise effect. For example, in an arm extension motion, the posture score related to the inner angle of the elbow may have a higher weight than the posture score related to the inner angle of the knee. In this way, the joint-specific weights that vary by posture can be preset for each joint according to the type of exemplary posture and stored in a state matched with the exercise content.

[0078] As described above, when the posture judgment model (132) considers weights in the process of deriving a posture score for each joint, there is an effect that enables the user's overall posture score to be calculated more accurately according to each exemplary posture.

[0079] FIG. 4 is a drawing for explaining an example of a process of superimposing an exemplary posture on a user's photographed information in a motion assistance service providing robot according to one embodiment of the present invention.

[0080] FIG. 4 illustrates an example in which a user (410) being filmed by a filming unit is displayed in real time within a display (400) corresponding to a content playback unit, and a portion of the content of the currently played exercise content is also displayed to the user (410). It can be seen that the user (410) and a model posture (420) that the user (410) should follow are displayed overlapping on the display (400). In FIG. 4, the model posture (420) is displayed in the shape of a silhouette, but the appearance of an actual model demonstrating the model posture may be expressed by overlapping the user, and in this case, the model posture may be displayed with adjusted transparency so that the appearance of the user and the appearance of the model posture can be viewed together. On the other hand, the model posture (420) may be displayed separately in an area separate from the area in which the user being filmed is displayed without applying the above-described algorithm or model.

[0081] In order for the exemplary posture (420) to be displayed overlapping with the user (410), the exercise assistance service providing robot may use an object detection model to identify the location of a person within an image from the shooting information. The object detection model may be an object detection model based on a convolutional neural network (CNN), and examples of object detection models include Single Shot Detector (SSD), Fast R-CNN, and Yolo. The object detection model may output a bounding box corresponding to the area where the user's body is located. On the other hand, the previously used skeleton extraction model may be used to more accurately detect the user's location. The skeleton extraction model may be used to identify the locations of the user's joints within the shooting information, and to superimpose the shape of a person taking the exemplary posture onto the user (410) by corresponding each joint to the joint location of the exemplary posture.

[0082] The posture judgment model of the exercise assistance service providing robot (100) can derive a posture score for each joint of the user (410), and for joints with a posture score lower than a predetermined standard, can display the relevant part (430) superimposed on the shape of the user (410) so that the user can accurately follow the model posture (420).

[0083] In the example of FIG. 4, if the elbow angle (440) of the user (410) is not sufficiently extended compared to the elbow angle of the exemplary posture, the posture judgment model can display a guide superimposed on the elbow side of the user to inform the user (410) how to extend his / her arm to properly follow the exemplary posture.

[0084] Returning to FIG. 1 and continuing with a description of other configurations, the database (150) can store the user's posture score information. Additionally, the database (150) can store information about the user's history, such as the user's age and medical conditions, entered in advance. The database (150) may typically be configured as a storage device located within the exercise assistance service providing robot (100). However, in other embodiments, a portion of the database (150) may be located in an external cloud connected to the exercise assistance service providing robot (100) to secure a wider data storage space.

[0085] The health status determination unit (160) can retrieve at least one piece of past posture score information of the user from the database (150), compare the past posture score information with the current posture score information of the user, and determine the health status of the user based on the comparison result to generate health status change information.

[0086] The health status determination unit (160) can compare posture information derived from exemplary postures on a daily, monthly, and yearly basis when the same exercise content is played, and determine whether the posture score is gradually improving, maintaining a similar level, or decreasing. The health status determination unit (160) can determine the time-series changes in the user's health status information based on the results determined through the above process. The health status information is accumulated and stored in the database (150), and can then be provided to the user's guardian or manager to be used for checking the user's health status.

[0087] When the above exercise content is provided, the driving unit (170) can control the robot arm (171) so that the exercise assistance service providing robot (100) assumes a posture corresponding to the exemplary posture reproduced in the exercise content. The robot arm (171) provided in the exercise assistance service providing robot (100) allows the user to more easily follow the exemplary posture of the exercise content by observing the position, direction, and posture of the robot arm. For example, if the posture presented in the exemplary posture is a movement of extending both arms to the left and right, the driving unit (170) can control the two robot arms to extend in opposite directions.

[0088] In addition, the driving unit (170) can change the position of the exercise assistance service providing robot (100) by controlling the moving device (172) for moving the exercise assistance service providing robot (100). Accordingly, the photographing unit (120) can photograph the user at the changed position of the exercise assistance service providing robot (100) to generate photographing information.

[0089] Here, the moving device (172) can move the exercise assistance service providing robot (100). More specifically, the moving device (172) provides a means for the exercise assistance service providing robot (100) to move within a specific space in relation to driving according to a movement command of the control device. More specifically, the moving device includes a motor and a plurality of wheels, which can be combined to perform the functions of driving, changing direction, and rotating the exercise assistance service providing robot (100).

[0090] Hereinafter, a user image captured in a first direction by the camera unit (120) can be defined as first shooting information, and a user image captured in a second direction by the camera unit (120) can be defined as second shooting information.

[0091] The position tracking unit (180) can generate three-dimensional position coordinate information about the positions of the exercise assistance service providing robot (100) and the user, and update the position coordinate information by tracking the position of the exercise assistance service providing robot (100) that changes according to the movement of the exercise assistance service providing robot (100). In addition, the posture scoring unit (130) can recognize the user's posture from the second shooting information by reflecting the change in the shooting angle according to the changed position of the robot.

[0092] When the exercise assistance service providing robot (100) photographs the user and analyzes the posture, the user's posture continuously changes, and depending on the user's posture or direction, a part of the user's body obscures another part of the user's body, resulting in degraded self-occlusion. In such cases, it becomes difficult for the exercise assistance service providing robot (100) to accurately determine the user's posture, and there is a problem of a reduced posture recognition rate. However, since it may be cumbersome for the user performing the exercise to turn his / her body every time according to the shooting direction of the exercise assistance service providing robot (100), the exercise assistance service providing robot (100) can directly move around the user and photograph the user from different directions. This allows the user's posture to be accurately recognized regardless of the user's posture. For example, if the exemplary posture of the exercise content presents a posture in which the body is stretched forward and backward, if the exercise assistance service providing robot (100) photographs the user from the front, the user's body that is stretched backward may be obscured due to the user's body protruding forward. At this time, if the driving unit (170) moves the exercise assistance service providing robot (100) to the side of the user to photograph the side of the user, the user's posture can be recognized more accurately.

[0093] The posture scoring unit (130) receives the first shooting information and analyzes the user's posture. If the accuracy of the joint coordinates recognized through the skeletal information extraction model (131) is lower than a predetermined value, it is determined that a self-occlusion phenomenon has occurred, and can move along the user's surroundings. At this time, the driving unit (170) controls the position of the exercise assistance service providing robot (100) to change, so that the shooting unit (120) can generate second shooting information. At this time, the second shooting information and the first shooting information are both image information generated by shooting the user, but may be image information shot from different angles. That is, it can be understood that the first shooting information has a low posture recognition rate due to self-occlusion, but the second shooting information can derive all joint positions of the user.

[0094] On the other hand, the driving unit (170) may control the moving device (172) to move to a predetermined shooting position according to the type of exemplary posture being played through the exercise content so that the user can be filmed. That is, even if the posture recognition rate is not lowered below a predetermined value due to self-occlusion, the exercise assistance service providing robot (100) may move to a designated position according to the exercise content being played in response to the shooting position for each exemplary posture specified in the exercise content.

[0095] As in the example above, when a posture of stretching the body forward and backward is presented as a model posture of exercise content, the model posture can be matched with movement information indicating that the exercise assistance service providing robot (100) should move to the user's side and stored. When the model posture is presented through the exercise assistance service providing robot (100), the movement information is loaded, and the driving unit (170) can control the movement device (172) to move the exercise assistance service providing robot (100) to the user's side.

[0096] In order to accurately track and photograph the movement of the exercise assistance service providing robot (100) and the location of the user after the movement, the locations of the exercise assistance service providing robot (100) and the user must be accurately tracked.

[0097] The position tracking unit (180) can confirm its own position (localization) on a three-dimensional space coordinate. Thereafter, the position tracking unit (180) can map the user's two-dimensional or three-dimensional joint coordinate information or user coordinate information measured based on the position on the exercise assistance service providing robot (100) onto a three-dimensional global space coordinate. The exercise assistance service providing robot (100) can continuously measure and update the user's joint coordinate information measured from various directions on a single coordinate system through the mapped coordinates.

[0098] FIGS. 5 to 7 are drawings illustrating a process of generating photographing information by photographing a user from different angles in a motion assistance service providing robot according to one embodiment of the present invention.

[0099] The left drawing of Fig. 5 illustrates that the user's (530) body is not properly included within the range (540) of the shooting information due to limitations in the angle of view or field of view of the shooting unit (520) equipped in the exercise assistance service providing robot (510). If at least one of the user's upper body, lower body, left body, and right body is not sufficiently included in the shooting information, the exercise assistance service providing robot (510) cannot accurately determine the user's skeletal position from the shooting information, which causes a problem in that the posture recognition rate is lowered.

[0100] The right drawing of FIG. 5 illustrates that the shooting direction of the shooting unit (520) of the exercise assistance service providing robot (510) is controlled so that the body of the user (530) is sufficiently included within the range of shooting information (550).

[0101] The exercise assistance service providing robot (510) can control the shooting direction of the shooting unit (520) to alleviate the phenomenon in which the user's body is not properly captured depending on the distance between the robot and the user or the position of the robot, resulting in a decrease in the posture recognition rate. To this end, a shooting angle changing device such as a camera rotation device (not shown) or a camera tilt device (not shown) can be provided so that the shooting angle of the camera corresponding to the shooting unit (520) can be changed up / down or left / right within a predetermined range.

[0102] The motion assistance service providing robot (510) can control a shooting angle change device so that the user's target joint (560) is positioned at the center of the shooting information. Here, the target joint (560) may refer to a joint located at the center of the user's body or a joint that is preferably located at the center of the shooting information to improve posture recognition rate. For example, it is preferable to set the pelvic joint or the lumbar joint as the target joint (560).

[0103] If the robot (510) providing the exercise assistance service determines that the user's posture recognition rate is low or that the user's posture is not properly included in the shooting information, the robot can control the shooting angle change device so that the user's target joint (560) is located at the center of the shooting information using the previously analyzed information.

[0104] The left drawing of Fig. 6 illustrates a case where a robot providing an exercise assistance service (610) photographs a user (620) performing an exercise according to exercise content from the side of the user (620) and analyzes the posture. The left drawing of Fig. 6 illustrates an example of performing a motion of stretching the body left and right. In this case, the posture of the user photographed from the photographing direction of the robot providing an exercise assistance service (610) is such that one arm and shoulder are covered by the other arm and shoulder, and when the photographing information (630) generated in this state is analyzed, there is a problem in that the position of each skeleton is inaccurately recognized, resulting in a decrease in the posture recognition rate.

[0105] On the other hand, the right drawing of FIG. 6 illustrates that the exercise assistance service providing robot (610) is photographing the user (620) from the front while the user (620) is performing exercise according to the exercise content, analyzing the posture of the user (620), and analyzing the joint coordinate information of the user according to the analyzed posture. At this time, since the user (620) performs a motion of stretching the body left and right, when the exercise assistance service providing robot (610) photographs the user (620), some body parts of the user (620) are not covered by other parts, and since the photographing information (640) generated in this state accurately recognizes the skeletal position of the user (620), the posture recognition rate of the exercise assistance service providing robot (610) can be improved.

[0106] FIG. 7A illustrates an example of a motion assistance service providing robot (710) moving around a user (720). In FIG. 7, the motion assistance service providing robot (710) can photograph the user (720) at location A. However, if the posture recognition rate for the user (720) photographed at location A is low or a location corresponding to an exemplary posture reproduced in the motion content is designated, the motion assistance service providing robot (710) can move to location B and continue photographing the user (720).

[0107] In this case, the exercise assistance service providing robot (710) can photograph the user (720) from various angles and alleviate the self-obscuring effect on the exercise posture of the user (720), thereby enabling accurate posture recognition of the user.

[0108] In the process of the exercise assistance service providing robot (710) changing its location, the location tracking unit can know the relative location of the user (720) with respect to the exercise assistance service providing robot (710) through the shooting information generated by the shooting unit and the Lidar. The exercise assistance service providing robot (710) can know the location of the exercise assistance service providing robot (710) on a virtual map where the exercise assistance service providing robot (710) is located using the Lidar. In addition, the relative location of the user (720) with respect to the exercise assistance service providing robot can be determined based on the shooting information, and the location of the user (720) on the virtual map can be calculated using the location of the exercise assistance service providing robot (710) on the virtual map. In this case, the exercise assistance service providing robot (710) can use the Lidar to move to a different location on the virtual map and then control the shooting unit to face the user (720).

[0109] FIG. 7B illustrates an example of a virtual map representing the relative positions between the user (720) and the exercise assistance service providing robot (710) illustrated in FIG. 7A. The exercise assistance service providing robot (710) can calculate coordinate information of the destination B on the virtual map and an angle (θ) to look at the user (720) after moving to the destination. Thereafter, the exercise assistance service providing robot (710) can move to the coordinate information of the destination, and upon arriving at the destination, control the exercise assistance service providing robot (710) to rotate by the angle (θ) to photograph the user (720).

[0110] In the example of Fig. 7b, the exercise assistance service providing robot (710) located at A can have a coordinate value (0,0) on a virtual map corresponding to location A, and can move to location B by receiving the coordinate value (-300,300) of location B. Thereafter, the robot can rotate by the input rotation angle (θ) to photograph the user (720), and at this time, the coordinate value (-300, 0) of the user (720) and the coordinate value of the exercise assistance service providing robot (710) can be continuously updated by the location tracking unit.

[0111] As another embodiment of the present invention, unlike the above-described case, the exercise assistance service providing robot (710) can control the movement of the exercise assistance service providing robot (710) by generating a virtual map only with the photographing information generated through the vision sensor, rather than based on the photographing information generated through the vision sensor and the lidar. For this purpose, Vision SLAM (Visual Simultaneous Localization and Mapping) technology can be used.

[0112] FIG. 8 is a drawing illustrating Vision SLAM used to generate shooting information by shooting a user from different angles in a motion assistance service providing robot according to one embodiment of the present invention.

[0113] Visual SLAM is a technique for visual simultaneous mapping and robot localization, used in the fields of computer vision and robotics, and can be used to enable robots to explore and map their surroundings using cameras or other visual sensors.

[0114] Visual SLAM can be used to detect and extract visual features from the surrounding environment and use them to estimate the location of a robot providing motion assistance services. The robot can then use these features to model the structure of its surroundings and estimate its own position. Through this process, the robot can build its own virtual map, which it can then use to estimate its own location.

[0115] Returning to Figure 1, the communication unit (190) can transmit or receive data via wired or wireless means, and can connect the robot to a server online. Furthermore, the user (10) can connect with other robot users via the communication unit (190) and share exercise content in real time. At this time, filming information can also be shared between users sharing exercise content.

[0116] FIG. 9 is a drawing illustrating a process for providing exercise assistance service between multiple users (920 to 950) by a robot (910) providing exercise assistance service according to one embodiment of the present invention.

[0117] Referring to FIG. 9, an exercise assistance service providing robot (910) can connect multiple users (920, 930, 940, 950) online to share the same exercise content and provide an environment in which the same exemplary posture can be followed in real time. Each of the multiple users (920, 930, 940, 950) can have an exercise assistance service providing robot (910) positioned nearby, and each of the exercise assistance service providing robots can photograph the nearby users. The user images photographed by each of the exercise assistance service providing robots (910) can also be transmitted to each user (920, 930, 940, 950) along with the exercise content. Through this, multiple users (920, 930, 940, 950) sharing the same exercise content can compete with each other or cooperate to perform the content of the exercise content. Additionally, in order to connect multiple users (920, 930, 940, 950) online, the exercise assistance service providing robot (910) can be connected to a server (960) via a communication unit, either wired or wirelessly.

[0118] Therefore, it can stimulate a sense of competition and make users focus more on the exercise content rather than exercising alone, thereby increasing the effectiveness of the exercise.

[0119] The exercise assistance service providing robot (910) located near a plurality of users (920, 930, 940, 950) does not necessarily have to be a robot with the same specifications and size, and may be a robot with the function of playing the exercise content, taking pictures of nearby users, and sharing the content online.

[0120] When sharing exercise content online with other users, users can invite designated users using the online addresses of other users of their choice. Alternatively, the server can match users based on certain conditions to create groups that share the same exercise content. For example, the server can match users of similar ages based on their personal or historical information. As another example, the server can match users with similar medical conditions. As another example, the server can match users using robots with similar hardware or software performance. This is because board performance can vary across robots, leading to differences in posture recognition rates due to differences in recognition speed. Furthermore, sensor installation locations can vary across robots, leading to differences in posture recognition rates. Therefore, board performance can vary even within the same robot. Therefore, before sharing exercise content, the time required for posture recognition can be measured to determine the software or hardware performance of each robot. In this example, if the server cannot match users with robots with similar software or hardware performance into a group that shares a single exercise content due to an insufficient number of connected users, the server may award points related to the performance of the exercise content to each user based on the software or hardware performance of the robot used by each user.

[0121] Additionally, the exercise assistance service robot can be controlled so that, when users share exercise content online, matched users are linked in real time and adopt exemplary postures from the exercise content being played. At this time, the posture scores of each analyzed user (901, 902, 903, 904) can be uploaded to the server in real time, and the user interface can be displayed differently on the screen depending on the user's posture score. For example, the user interface can be changed so that the user with the highest posture score glows yellow, and the user with the lowest posture score glows red.

[0122] When the exercise content playback ends, the robot providing the exercise assistance service can determine the user with the highest accumulated posture score on the server and notify each user (901, 902, 903, 904) of the results. This sharing of exercise content is not a one-time event; it provides a game performance history and allows users to monitor changes before and after sharing their exercise content, as well as receive feedback.

[0123] Figure 10 is a flowchart illustrating a method for providing exercise assistance services according to one embodiment of the present invention.

[0124] Referring to FIG. 10, a method for providing an exercise assistance service may include a content provision step (S100), a shooting step (S200), a posture scoring step (S300), and an update step (S400).

[0125] The content provision step (S100) may be a step of providing exercise content to assist the user in performing exercise.

[0126] The shooting step (S200) may be a step of generating shooting information by shooting a user performing exercise according to exercise content.

[0127] The posture scoring step (S300) may be a step of recognizing the user's posture from the shooting information and generating posture score information for a score given to the user's posture based on the degree of matching between the recognized user's posture and the exemplary posture presented in the exercise content.

[0128] The update step (S400) may be a step of updating exercise content provided to the user based on the detailed score information.

[0129] The above method for providing exercise assistance services can be further divided into additional steps or combined into fewer steps, as described in the embodiments described above with reference to FIGS. 1 through 9. Furthermore, some steps may be omitted as needed, or the order of steps may be changed.

[0130] The above-described method for providing exercise assistance services may also be implemented in the form of a computer program stored on a computer-readable recording medium executed by a computer or a recording medium containing computer-executable commands. Furthermore, the above-described method for providing exercise assistance services may also be implemented in the form of a computer program stored on a computer-readable recording medium executed by a computer.

[0131] A computer-readable recording medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Furthermore, a computer-readable recording medium may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0132] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

[0133] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims

1. In a robot that provides exercise assistance services through posture recognition, A content playback unit that plays exercise content to assist the user in performing exercise; A shooting unit that generates first shooting information by shooting a user performing exercise according to the above exercise content; A posture scoring unit that recognizes the user's posture from the first shooting information generated through the shooting unit and generates posture score information for the score given to the user's posture based on the degree of matching between the recognized user's posture and the exemplary posture presented in the exercise content; and An exercise assistance service providing robot characterized by including an update unit that updates exercise content provided to the user according to the above-mentioned detailed score information.

2. In paragraph 1, The above exercise content includes content to assist the user in performing exercise by providing an exercise method that the user can see, hear, and follow. A robot providing exercise assistance service, characterized in that the content playback unit plays the exercise content through at least one of a display and a speaker provided in the robot.

3. In paragraph 1, The above exercise content includes content to assist the user in performing exercise by providing an exercise method that the user can see, hear, and follow. A robot providing exercise assistance service, characterized in that the content playback unit plays the exercise content through at least one of an external display device connected to the robot by wire or wirelessly and an external speaker device connected to the robot by wire or wirelessly.

4. In paragraph 1, A robot providing exercise assistance service, characterized in that the above-mentioned photographing unit photographs the user through an RGB (Red Green Blue) camera or a depth camera to generate the first photographing information.

5. In paragraph 1, The above-mentioned posture score section is a skeletal information extraction model that receives the first shooting information, analyzes it, and outputs joint coordinate information for the location of each joint of the user being shot. A motion assistance service providing robot characterized by including a judgment model that recognizes the user's posture from the joint coordinate information and assigns a score to the posture currently being taken by the user.

6. In paragraph 5, A robot providing a movement assistance service, characterized in that the above skeletal information extraction model analyzes the first photographed information based on a deep neural network and outputs two-dimensional or three-dimensional joint coordinate information about the user's joint position displayed through the first photographed information.

7. In paragraph 5, The above posture judgment model derives a posture score for each joint based on the difference between the degree of bending of each joint of the user and the degree of bending of each joint of an exemplary posture presented in the exercise content based on the joint coordinate information, and derives the posture score information of the user by synthesizing the posture scores for each joint.

8. In paragraph 7, A robot providing an exercise assistance service, characterized in that the posture judgment model reflects a joint-specific weight preset in the exercise content to the derived joint-specific posture score, and derives the user's posture score information by synthesizing the joint-specific posture scores to which the weights are reflected.

9. In paragraph 1 The above update unit superimposes the image of the user being photographed with the example posture when the exercise content is played through the display so that the user can follow the example posture. A robot providing an exercise assistance service characterized in that, when the exercise content is played through a speaker so that the user can follow the exemplary posture, it performs at least one of the following: providing voice guidance so that the score given to the user's posture can be further improved.

10. In paragraph 1 a database storing the user's detailed score information; and A robot providing an exercise assistance service, further comprising a health status determination unit that retrieves at least one piece of past posture score information of the user from the database, compares the past posture score information with the current posture score information of the user, and determines the health status of the user based on the comparison result to generate health status change information.

11. In paragraph 10, The above health status judgment department A robot providing an exercise assistance service characterized in that it retrieves the user's age information that has been input in advance and reflects the user's health status in the judgment based on average posture information according to the user's age.

12. In paragraph 1 A robot providing exercise assistance service, characterized in that it further includes a driving unit for controlling a robot arm so that, when the exercise content is provided, the robot assumes a posture corresponding to the exemplary posture reproduced in the exercise content.

13. In paragraph 1, A robot providing a motion assistance service characterized in that it changes the shooting direction of the shooting unit so that the target joint of the user is displayed in the center of the first shooting information.

14. In paragraph 1, Further comprising a driving unit for changing the position of the robot by controlling a moving device for moving the robot; A robot providing exercise assistance service, characterized in that the above-mentioned photographing unit photographs the user at a changed location to generate second photographing information.

15. In paragraph 14, A position tracking unit that generates three-dimensional position coordinate information about the positions of the robot and the user, and updates the position coordinate information by tracking the position of the robot that changes according to the movement of the robot; further comprising; A robot providing an exercise assistance service, characterized in that the posture score unit recognizes the user's posture from the second shooting information by reflecting a change in the shooting angle according to the changed position of the robot.

16. In paragraph 15, The above-mentioned posture score unit includes a skeletal information extraction model that receives the first shooting information, analyzes it, and outputs joint coordinate information for the location of each joint of the user being shot. If the accuracy of the joint coordinates derived from the first photographing information by the above skeletal information extraction model is lower than a predetermined value, A robot providing an exercise assistance service, characterized in that the driving unit controls the position of the robot to change, and the photographing unit generates the second photographing information.

17. In paragraph 15, A robot providing an exercise assistance service, characterized in that the driving unit controls the moving device to move to a pre-designated shooting position according to the type of exemplary posture reproduced through the exercise content so that the user can be photographed.

18. In paragraph 1, A communication unit for transmitting or receiving data wired or wirelessly and connecting the robot to a server online; further comprising; Connects to users of other robots through the above communication unit and shares the exercise content in real time. A robot providing exercise assistance service, characterized in that first shooting information is shared between users who share the above exercise content.

19. In paragraph 1, Users who are connected through the above communication unit and share exercise content in real time are based on the history information of users who use the above exercise content. A robot providing exercise assistance service, characterized in that the user's history information includes at least one of the user's age information, disease information, and performance information of the robot used by the user using the exercise content.

20. A method for providing exercise assistance service using a robot that provides exercise assistance service through posture recognition, A content provision step that provides exercise content to assist the user in performing exercise; A shooting step of generating first shooting information by shooting a user performing exercise according to the above exercise content; A posture scoring step for recognizing the user's posture from the first shooting information and generating posture score information for a score given to the user's posture based on the degree of matching between the recognized user's posture and the exemplary posture presented in the exercise content; and A method for providing an exercise assistance service, characterized by comprising: an update step for updating exercise content provided to the user according to the above-mentioned detailed score information;

Images