Method for providing center of gravity information using video image

Abstract

To disclose a method for providing center of gravity information using a video image.SOLUTION: A method for providing center of gravity information according to an embodiment of the present invention comprises: a step for generating a multi-angle two-dimensional pose corresponding to a posture of a user based on a video image of the user captured from multiple angles; a step for generating a three-dimensional pose by combining the multi-angle two-dimensional poses; a step for generating a heat map (HEATMAP) visually indicating the pressure distribution of a sole of a foot based on the three-dimensional pose, and extracting a center of gravity point based on the heat map; and a step for providing the center of gravity information corresponding to the posture of the user using a weight balance calculated based on the center of gravity point.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for providing center of gravity information using image and apparatus therefor, and more particularly to a technology capable of analyzing images of a person captured from various angles and providing information related to the center of gravity of the person in the captured images in the front, back, left and right directions.

[0002] The present invention claims the benefit of the filing date of Korean Patent Application No. 10-2020-0140532, filed on October 27, 2020, the entire contents of which are incorporated herein by reference. [Background technology]

[0003] In many sports, the position and movement of the center of gravity is an important factor in improving one's performance. For example, in golf, the degree and timing of the movement of the body's center of gravity can play an important role in creating a correct swing posture.

[0004] However, in order to measure the center of gravity of a person and the movement of the center of gravity, separate equipment such as a floor mat containing a pressure measurement sensor or special measurement footwear worn by the user is required. Such special equipment requires high initial costs for manufacturing and purchasing, additional costs for charging the battery and repairing damage, and various restrictions are imposed, such as limited space depending on the method of measuring the center of gravity.

[0005] In addition, such dedicated measurement equipment cannot be freely used on mobile platforms, etc., and is restricted to using the platform provided by the equipment manufacturer. Summary of the Invention [Problem to be solved by the invention]

[0006] The object of the present invention is to provide center of gravity information regarding a user's posture and balance using only images captured by a camera in everyday locations where no other measurement equipment is provided, without any restrictions on location or equipment.

[0007] Another object of the present invention is to provide a center of gravity information service in the form of an application that can be freely operated in the environment of a user's personal PC, smartphone, or the like.

[0008] Furthermore, an object of the present invention is to provide a service that assists users who enjoy sports that are affected by exercise posture and center of gravity movement to correct their posture and move more efficiently by providing center of gravity information that corresponds to the user's movements contained in the video.

[0009] The object of the present invention is to provide a service technology that can provide services utilizing exercise indicators such as posture correction, posture accuracy, and weight transfer timing, and can be utilized in a manner of recording and monitoring the degree of correctness of exercise posture as an indicator, and can provide feedback not only in ball games such as golf and baseball, but also in various exercise fields such as gymnastics and bodyweight training. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the center of gravity information providing method of the present invention includes the steps of generating a multi-angle two-dimensional pose corresponding to the user's posture based on images of the user captured from multiple angles, generating a three-dimensional pose by combining the multi-angle two-dimensional poses, generating a heat map (HEATMAP) that visually shows the pressure distribution on the soles of the feet based on the three-dimensional pose, extracting a center of gravity point based on the heatmap, and providing center of gravity information corresponding to the user's posture using a weight balance calculated based on the center of gravity point.

[0011] At this time, the weight balance may include front-rear weight balance and left-right weight balance.

[0012] At this time, the method may further include outputting the center of gravity information corresponding to the image based on a user interface.

[0013] At this time, the outputting step can output at least one of the heat map, the weight balance, the center of gravity, and a movement line of the center of gravity.

[0014] In this case, the step of generating a multi-angle 2D pose may include a step of converting the image into array data consisting of 8-bit integers, and a step of inputting the array data into a deep learning-based joint estimation model to display multiple joints corresponding to the user's posture in a 2D coordinate system.

[0015] In this case, the multi-angle two-dimensional pose can correspond to data expressed by connecting the multiple joints on the two-dimensional coordinate system.

[0016] In this case, the step of extracting the center of gravity can input the 3D pose into a deep learning-based center of gravity estimation model to generate the heat map.

[0017] In this case, the step of extracting the center of gravity may include extracting the center of gravity by normalizing an output value of the heat map.

[0018] In this case, the step of providing the center of gravity information may include the steps of generating a center of gravity correction value, calculating the left and right weight balance for the user's left and right feet based on the center of gravity correction value, and normalizing values ​​corresponding to the left and right feet in the heat map, respectively, to extract the center of gravity of the left foot and the center of gravity of the right foot, and correcting the extracted center of gravity points to calculate the front and rear weight balance corresponding to each of the left and right feet.

[0019] In this case, the images captured from multiple angles may include a front image captured from the front of the user and a side image captured from the side of the user.

[0020] In this case, the step of converting into array data may include extracting valid frames by deleting unnecessary frames from all frames corresponding to the image, and converting the valid frames into the array data.

[0021] At this time, the heat map is displayed so that the greater the pressure applied to the sole of the foot, the more visually emphasized it is.

[0022] In addition, a center of gravity information providing device according to one embodiment of the present invention includes a processor that generates multi-angle 2D poses corresponding to a user's posture based on an image of the user captured from multiple angles, generates a 3D pose by combining the multi-angle 2D poses, generates a heat map (HEATMAP) that visually indicates pressure distribution on the soles of the feet based on the 3D pose, extracts a center of gravity based on the heatmap, and provides center of gravity information corresponding to the user's posture using a weight balance calculated based on the center of gravity, and a memory that stores at least one of the image, the heatmap, and the center of gravity information.

[0023] At this time, the weight balance may include front-rear weight balance and left-right weight balance.

[0024] At this time, the processor may output the center of gravity information to correspond to the image based on a user interface.

[0025] At this time, the processor can output at least one of the heat map, the weight balance, the center of gravity point, and the movement line of the center of gravity point.

[0026] At this time, the processor converts the image into array data consisting of 8-bit integers, and inputs the array data into a deep learning-based joint estimation model to display multiple joints corresponding to the user's posture in a two-dimensional coordinate system.

[0027] In this case, the multi-angle two-dimensional pose can correspond to data expressed by connecting the multiple joints on the two-dimensional coordinate system.

[0028] The processor can then input the 3D pose into a deep learning based centroid estimation model to generate the heatmap.

[0029] At this time, the processor may normalize the output values ​​of the heat map to extract the center of gravity.

[0030] At this time, the processor generates a center of gravity correction value, calculates the left and right weight balance for the user's left and right feet based on the center of gravity correction value, normalizes values ​​corresponding to the left and right feet in the heat map, respectively, to extract the center of gravity of the left foot and the center of gravity of the right foot, and corrects the extracted center of gravity points to calculate the front and rear weight balance corresponding to each of the left and right feet.

[0031] In this case, the images captured from multiple angles may include a front image captured from the front of the user and a side image captured from the side of the user.

[0032] At this time, the processor may delete unnecessary frames from all frames corresponding to the image, extract valid frames, and convert the valid frames into the array data.

[0033] At this time, the heat map is displayed so that the greater the pressure applied to the sole of the foot, the more visually emphasized it is. Effect of the Invention

[0034] According to the present invention, center of gravity information regarding a user's posture and balance can be provided without restrictions on location or equipment, using only images captured by a camera in everyday locations where no other measurement equipment is provided.

[0035] In addition, the present invention can provide a center of gravity information service in the form of an application that can be freely operated in an environment such as a user's personal PC or smartphone.

[0036] Furthermore, the present invention can provide a service that assists users who enjoy sports that are affected by exercise posture and center of gravity movement to correct their posture and move more efficiently by providing center of gravity information that corresponds to the user's movements contained in the video.

[0037] In addition, the present invention can provide services utilizing exercise indicators such as posture correction, posture accuracy, and weight transfer timing, or can be used in a manner of recording and monitoring the correctness of exercise posture as an indicator, and can provide a service technology that can provide feedback not only in ball games such as golf and baseball, but also in various exercise fields such as gymnastics and bodyweight training. [Brief description of the drawings]

[0038] [Figure 1] 1 is an operational flowchart illustrating a method for providing center of gravity information using an image according to an embodiment of the present invention. [Diagram 2] 5A to 5C are diagrams illustrating an example of a process for generating center of gravity information based on a frontal image of a user according to the present invention. [Diagram 3] 5A to 5C are diagrams illustrating an example of a process for generating center of gravity information based on a frontal image of a user according to the present invention. [Figure 4] 5A to 5C are diagrams illustrating an example of a process for generating center of gravity information based on a frontal image of a user according to the present invention. [Diagram 5] 5A to 5C are diagrams illustrating an example of a process for generating center of gravity information based on front and side images of a user according to the present invention. [Figure 6]5A to 5C are diagrams illustrating an example of a process for generating center of gravity information based on front and side images of a user according to the present invention. [Figure 7] 5A to 5C are diagrams illustrating an example of a process for generating center of gravity information based on front and side images of a user according to the present invention. [Figure 8] 5A to 5C are diagrams illustrating an example of a process for generating center of gravity information based on front and side images of a user according to the present invention. [Figure 9] 5A to 5C are diagrams illustrating an example of a process for generating center of gravity information based on front and side images of a user according to the present invention. [Figure 10] FIG. 13 is a diagram showing an example of extracting valid frames according to the present invention. [Figure 11] FIG. 13 illustrates an example of a user interface for providing center of gravity information in accordance with the present invention. [Figure 12] FIG. 13 illustrates an example of a user interface for providing center of gravity information in accordance with the present invention. [Figure 13] 4 is a detailed diagram illustrating a process of providing center of gravity information according to an embodiment of the present invention; [Figure 14] 4 is a detailed diagram illustrating a process of providing center of gravity information according to an embodiment of the present invention; [Figure 15] 1 is a block diagram showing a center of gravity information providing device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, repeated explanations and detailed explanations of known functions and configurations that may unnecessarily obscure the gist of the present invention will be omitted. The embodiments of the present invention are provided to more completely explain the present invention to those having average knowledge in the art. Therefore, the shapes and sizes of elements in the drawings are exaggerated for clearer explanation.

[0040] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0041] FIG. 1 is an operational flowchart showing a method for providing center of gravity information using an image according to an embodiment of the present invention.

[0042] Referring to FIG. 1, in a method for providing center of gravity information using an image according to an embodiment of the present invention, a multi-angle 2D pose corresponding to a user's posture is generated based on an image of a user captured from multiple angles (S110).

[0043] In other words, the present invention can generate a 2D pose used to generate center of gravity information using only images captured by a camera even in everyday locations where separate center of gravity measurement equipment is not installed.

[0044] For example, if a user wants to know whether their posture and balance are correct while doing home training at home, they can take a picture of their home training posture using a camera on their smartphone, and input the home training video via a user interface provided on the smartphone according to one embodiment of the present invention.

[0045] In this case, the images captured from multiple angles may include a front image captured from the front of the user and a side image captured from the side of the user.

[0046] Therefore, the multi-angle 2D poses can include 2D poses generated corresponding to the user's front posture extracted from the front image and 2D poses generated corresponding to the user's side posture extracted from the side image.

[0047] In this case, in the present invention, the left and right weight balance of the user can be estimated using a two-dimensional pose generated corresponding to the front posture of the user.

[0048] For example, as shown in Fig. 2, when a user is photographed using one camera 200 positioned in front of the user, an image 310 including only the frontal pose of the user can be acquired as shown in Fig. 3. In this case, a 2D pose 330 corresponding to the frontal pose of the user can be generated through a process of converting the image 310 into array data 320, which will be described later.

[0049] Using the two-dimensional pose 330 thus generated, the left and right weight balance 400 for the frontal posture of the user can be estimated, as shown in Fig. 4. The process of estimating the left and right weight balance using the two-dimensional pose for the frontal posture of the user will be described in detail later.

[0050] At this time, the left and right weight balance 400 shown in FIG. 4 corresponds to a value estimated based on both feet of the user captured in the image 310 shown in FIG. 3, and the balance of the right foot and the balance of the left foot are estimated to be values ​​inversely proportional to each other.

[0051] For example, if the user's body is analyzed to be leaning to the right based on the 2D pose, the balance value of the right foot increases and the balance value of the left foot decreases. Conversely, if the user's body is analyzed to be leaning to the left based on the 2D pose, the balance value of the left foot increases and the balance value of the right foot decreases.

[0052] In addition, in the present invention, the front-to-back weight balance of the user can also be estimated using a two-dimensional pose generated corresponding to the user's side posture.

[0053] For example, as shown in Fig. 5, when a user is photographed simultaneously using a camera 510 positioned in front of the user and a camera 520 positioned to the side of the user, an image 610 corresponding to the user's side pose at the same time can be obtained as shown in Fig. 6 along with an image 310 corresponding to the user's front pose as shown in Fig. 3. In this case, when the image 610 corresponding to the user's side pose is converted into array data 620, which will be described later, a two-dimensional pose 630 corresponding to the user's side pose can be generated.

[0054] That is, the multi-angle two-dimensional pose in the present invention can be a concept that includes both the two-dimensional pose 330 in the front pose shown in FIG. 3 and the two-dimensional pose 630 in the side pose shown in FIG.

[0055] In order to generate a multi-angle 2D pose, the images captured from multiple angles must be input into a deep learning-based joint estimation model to estimate the position of each of the user's joints. To do this, the images captured from multiple angles can first be converted into array data for input into the deep learning-based joint estimation model.

[0056] At this time, the joint estimation model can correspond to a deep learning-based model.

[0057] For example, a joint estimation model according to an embodiment of the present invention may correspond to a deep learning model designed using the TensorFlow Framework.

[0058] In this case, the array data may correspond to data consisting of 8-bit integers.

[0059] At this time, the image converted into array data is input into a deep learning-based joint estimation model, and multiple joints corresponding to the user's posture in the image can be displayed in a two-dimensional coordinate system.

[0060] In this case, the multiple joints can correspond to nine joints corresponding to the head, right shoulder, left shoulder, right buttock, left buttock, right knee, left knee, right foot, and left foot.

[0061] In other words, according to the present invention, simply by inputting a captured image of a user into a joint estimation model, the positions of nine joints corresponding to the user's posture in the image can be estimated and displayed in a two-dimensional coordinate system.

[0062] In this case, the multi-angle two-dimensional pose can correspond to data expressed by connecting multiple joints on a two-dimensional coordinate system.

[0063] For example, as shown in Figures 3 and 6, two-dimensional poses 330 and 630 can be generated by representing the nine joints estimated by the joint estimation model as two-dimensional points and connecting each point according to the user's body structure.

[0064] In this case, the video input for generating multi-angle 2D poses is not video edited by an expert, but video taken by a user in their daily life, so that unnecessary frames 1021 and 1022 are included at the beginning and end of the video, as shown in FIG. 10.

[0065] For example, the video contains unnecessary parts, such as a part that is shot when the user presses the camera's start shooting button to adjust the camera angle, and a part that moves to press the camera's end shooting button.

[0066] In the present invention, in order to prevent calculations from being performed on such unnecessary frames, which would cause unnecessary load on the system, unnecessary frames 1021 and 1022 are deleted from the entire frames corresponding to the image, and an effective frame 1010 is extracted, and only the effective frame 1010 is converted into array data.

[0067] For example, the present invention may provide a video editing function that allows a user to directly select only valid frames via a user interface provided to the user.

[0068] As another example, any portion of the video in which the user's entire body is not captured may be determined as an unnecessary frame and deleted.

[0069] At this time, the method of deleting unnecessary frames from the image and extracting only valid frames may use various methods applicable to the present invention, and is not limited to a specific method.

[0070] In addition, the method for providing center of gravity information using an image according to an embodiment of the present invention generates a 3D pose by combining multiple 2D poses based on the orientation information of the camera that captured the image (S120).

[0071] For example, the 2D coordinates (X f ,Y f ) and obtain 2D coordinates (X s ,Y s ) can be obtained. The 2D coordinates obtained in this way are combined for each joint position according to the camera orientation to obtain 3D coordinates (X, Y, Z = X f ,Y f ,X s ) can be obtained. At this time, the Z value in the 3D coordinates corresponds to the X value of the 2D coordinates (X s ) can be used.

[0072] At this time, the acquired 3D coordinates are scaled based on the user's head and toes, and are displayed corresponding to vector values ​​on the 3D coordinates, as shown in Figure 7, so that a 3D pose corresponding to the user's posture in the video can be generated.

[0073] For example, the scaling of the three-dimensional coordinates is performed according to equation (1).

[0074]

number

[0075] In addition, the method for providing center of gravity information using an image according to one embodiment of the present invention generates a heat map (HEATMAP) that visually shows the pressure distribution of the soles of the feet based on the 3D pose, and extracts the center of gravity based on the heat map (S130).

[0076] At this time, the heat map visually indicates the pressure distribution on the soles of the feet according to the user's posture, and is generally provided in a manner that measures the center of gravity based on a computing device connected to the pressure sensor when the user stands directly on a step equipped with a pressure sensor.

[0077] However, in the present invention, even without a pressure sensor, a heat map can be generated by inputting the 3D pose generated based on the video into a deep learning-based center of gravity estimation model.

[0078] In this case, the center of gravity estimation model can correspond to a deep learning-based artificial neural network.

[0079] For example, the center of gravity estimation model according to an embodiment of the present invention may correspond to a deep learning model-based artificial neural network designed using the TensorFlow Framework.

[0080] At this time, the generated 3D pose makes it possible to know all the front and side poses of the user in the video, so that the 3D pose can be input to the center of gravity estimation model to estimate the front-rear and left-right weight balance for the user's posture in the video. The estimated front-rear and left-right weight balance can be expressed as the pressure received by the soles of the user's feet, and a heat map such as that shown in FIG. 8 can be provided.

[0081] At this time, the heat map is displayed so that the greater the pressure applied to the sole of the foot, the more visually emphasized it is.

[0082] For example, if you check the weight balance between the left and right sides shown numerically on the heat map in Figure 8, you can see that the weight balance is biased to the left, with LEFT at 89 and RIGHT at 11. Therefore, the left foot can be emphasized more than the right foot on the heat map to clearly see at a glance that higher pressure is applied to the left foot than the right foot.

[0083] FIG. 9 shows the process described above at a glance. Based on images captured from multiple angles, 2D poses are generated for each of the front and side images of the user in the images, and a heat map can be generated using the 3D poses generated by combining the 2D poses.

[0084] In addition, the method for providing center of gravity information using an image according to an embodiment of the present invention provides center of gravity information corresponding to a user's posture using a weight balance calculated based on the center of gravity (S140).

[0085] At this time, the weight balance may include front-rear weight balance and left-right weight balance.

[0086] At this time, the output value of the heat map can be normalized to extract the center of gravity.

[0087] For example, as shown in equation (2), the weighted Arg softmax function, which is one of the softmax functions, is applied to the heat map to normalize it, and the center of gravity (X p ,Y p ) can be output.

[0088]

number

[0089] At this time, a center of gravity correction value is generated, and the left and right weight balance for the user's left and right feet is calculated based on the center of gravity correction value. The values ​​corresponding to the left and right feet in the heat map are normalized to extract the center of gravity of the left and right feet, and the extracted center of gravity points are corrected to calculate the front and rear weight balance corresponding to each of the left and right feet.

[0090] For example, if the center of gravity correction value is (X p ,Y p ), the left and right weight balance can be calculated as shown in equation (3).

[0091]

number

[0092] As another example, the center of gravity of the left foot and the center of gravity of the right foot can be calculated by applying a weighted Argsoftmax function to each of the left foot and the right foot in the heat map, and the front-to-back weight balance corresponding to each of the user's left foot and right foot can be calculated based on the corrected values ​​of the center of gravity of the left foot and the center of gravity of the right foot.

[0093] If the left foot center of gravity correction value is Y L_p , the center of gravity correction value for the right foot is Y R_p If this assumption is made, then it is possible to calculate the front / rear weight balances R / L_front and R / L_rear corresponding to the left foot and the right foot, respectively, as shown in equation (4).

[0094]

number

[0095] Although not shown in FIG. 1, the method for providing center of gravity information using an image according to an embodiment of the present invention outputs center of gravity information corresponding to the image based on a user interface.

[0096] 11 and 12, for example, a user interface is provided on a user's mobile terminal such as a smartphone, and the user interface can provide center of gravity information 1120 to be matched with the image being played back while playing back an image 1110 used to output center of gravity information.

[0097] That is, by providing center of gravity information regarding the golf swing posture as shown in Fig. 11, the user can objectively grasp the timing and degree of center of gravity movement during the golf swing. Also, by analyzing body parts with slightly incorrect posture through the extracted joint information and providing the information to the user, it can be used for swing coaching.

[0098] Alternatively, as shown in FIG. 12, it may be used as a basis for judging whether the movements of bodyweight training are being performed correctly, or when the user assumes a home training posture, the extracted joint information may be used to analyze the posture for home training coaching.

[0099] At this time, center of gravity information corresponding to at least one of the heat map, weight balance, center of gravity point, and moving line of the center of gravity point can be outputted through the user interface.

[0100] For example, as shown in FIG. 8, a heat map, a center of gravity 810 displayed on the heat map, and a movement line of the center of gravity 820 may be output, and the left and right weight balance as well as the front and rear weight balance of each foot may be quantified and output.

[0101] In this way, the center of gravity information provided by the present invention can be provided as a service using a PC or smartphone-based application system without using a special output device.

[0102] Also, although not shown in FIG. 1, the method for providing center of gravity information using an image according to an embodiment of the present invention can store various pieces of information generated in the above-mentioned center of gravity information providing process in a separate storage module.

[0103] This method of providing center of gravity information makes it possible to provide center of gravity information regarding a user's posture and balance using only images captured by a camera in everyday locations where no other measuring equipment is provided, without any restrictions on location or equipment.

[0104] In addition, the center of gravity information service can be provided in the form of an application that can be freely operated in the user's personal PC, smartphone, or other environment.

[0105] Furthermore, by providing center of gravity information corresponding to the user's movements contained in the video, a service can be provided to assist users who enjoy sports that are affected by exercise posture and shifts in the center of gravity to correct their posture and move more efficiently.

[0106] 13 and 14 are diagrams illustrating in detail a process of providing center of gravity information according to an embodiment of the present invention.

[0107] First, referring to FIG. 13, the present invention can obtain a front image (f) 1310 in which a user is photographed from the front, and a side image (s) 1320 in which a user is photographed from the side, based on images of the user photographed from multiple angles.

[0108] Thereafter, two-dimensional poses 1311, 1321 that indicate the positions of multiple joints can be generated based on each image.

[0109] At this time, each image is converted into array data consisting of 8-bit integers, and the array data is input into a deep learning-based joint estimation model to generate 2D poses 1311, 1321 in which multiple joints are displayed in a 2D coordinate system.

[0110] Thereafter, the 2D coordinates 1312, 1322 for each of the multiple joints are combined to obtain 3D coordinates 1330 for each of the multiple joints, and the 3D coordinates 1330 are scaled based on the head and toes to generate a 3D pose corresponding to the corrected 3D coordinates 1331.

[0111] Thereafter, the corrected 3D coordinates 1331 can be input to a center of gravity estimation model 1340, which corresponds to an artificial neural network for generating a sole heat map, to output the heat map.

[0112] Now, referring to FIG. 14, the heat map 1410 can be normalized to output the center of gravity point 1420.

[0113] Thereafter, the left and right weight balance 1430 for the left and right feet can be calculated based on the corrected value of the center of gravity 1420.

[0114] In addition, the values ​​corresponding to the left foot and the right foot in the heat map can be normalized to extract the center of gravity of the left foot and the center of gravity of the right foot, and the extracted center of gravity points can be corrected to calculate the front and rear weight balance 1440 corresponding to the left foot and the right foot, respectively.

[0115] Then, center of gravity information 1450 to be provided to a user can be generated based on the generated heat map and the calculated center of gravity point and weight balance.

[0116] FIG. 15 is a block diagram showing a center of gravity information providing device according to an embodiment of the present invention.

[0117] 15, the apparatus for providing center of gravity information according to an embodiment of the present invention includes a communication unit 1510, a processor 1520, and a memory 1530.

[0118] The communication unit 1510 serves to transmit and receive information required for providing center of gravity information via a communication network such as a network.

[0119] At this time, the network is a concept that includes all networks that have been used in the past and networks that can be developed in the future. For example, the network may be one of a wired / wireless short-range communication network that provides communication between various information devices within a limited area, a mobile communication network that provides communication between mobile objects and between mobile objects and the outside world, a satellite communication network that provides communication between earth stations using satellites, or a wired / wireless communication network, or a combination of two or more of them. Meanwhile, the transmission method standard of the network is not limited to the existing transmission method standard, but may include all transmission method standards that will be developed in the future.

[0120] The processor 1520 generates multi-angle two-dimensional poses corresponding to the user's posture based on images of the user captured from multiple angles.

[0121] In other words, the present invention can generate a 2D pose used to generate center of gravity information using only images captured by a camera even in everyday locations where separate center of gravity measurement equipment is not installed.

[0122] For example, if a user wants to know whether their posture and balance are correct while doing home training at home, they can take a picture of their home training posture using a camera on their smartphone, and input the home training video via a user interface provided on the smartphone according to one embodiment of the present invention.

[0123] In this case, the images captured from multiple angles may include a front image captured from the front of the user and a side image captured from the side of the user.

[0124] Therefore, the multi-angle 2D poses can include 2D poses generated corresponding to the user's front posture extracted from the front image and 2D poses generated corresponding to the user's side posture extracted from the side image.

[0125] In this case, in the present invention, the left and right weight balance of the user can be estimated using a two-dimensional pose generated corresponding to the front posture of the user.

[0126] For example, as shown in Fig. 2, when a user is photographed using one camera 200 positioned in front of the user, an image 310 including only the frontal pose of the user can be acquired as shown in Fig. 3. In this case, a 2D pose 330 corresponding to the frontal pose of the user can be generated through a process of converting the image 310 into array data 320, which will be described later.

[0127] Using the two-dimensional pose 330 thus generated, the left and right weight balance 400 for the frontal posture of the user can be estimated, as shown in Fig. 4. The process of estimating the left and right weight balance using the two-dimensional pose for the frontal posture of the user will be described in detail later.

[0128] At this time, the left and right weight balance 400 shown in FIG. 4 corresponds to a value estimated based on both feet of the user captured in the image 310 shown in FIG. 3, and the balance of the right foot and the balance of the left foot are estimated to be values ​​inversely proportional to each other.

[0129] For example, if the user's body is analyzed to be leaning to the right based on the 2D pose, the balance value of the right foot increases and the balance value of the left foot decreases. Conversely, if the user's body is analyzed to be leaning to the left based on the 2D pose, the balance value of the left foot increases and the balance value of the right foot decreases.

[0130] In addition, in the present invention, the front-to-back weight balance of the user can also be estimated using a two-dimensional pose generated corresponding to the user's side posture.

[0131] For example, as shown in Fig. 5, when a user is photographed simultaneously using a camera 510 positioned in front of the user and a camera 520 positioned to the side of the user, an image 610 corresponding to the user's side pose at the same time can be obtained as shown in Fig. 6 along with an image 310 corresponding to the user's front pose as shown in Fig. 3. In this case, when the image 610 corresponding to the user's side pose is converted into array data 620, which will be described later, a two-dimensional pose 630 corresponding to the user's side pose can be generated.

[0132] That is, the multi-angle two-dimensional pose in the present invention can be a concept that includes both the two-dimensional pose 330 in the front pose shown in FIG. 3 and the two-dimensional pose 630 in the side pose shown in FIG.

[0133] In order to generate a multi-angle 2D pose, the images captured from multiple angles must be input into a deep learning-based joint estimation model to estimate the position of each of the user's joints. To do this, the images captured from multiple angles can first be converted into array data for input into the deep learning-based joint estimation model.

[0134] At this time, the joint estimation model can correspond to a deep learning-based model.

[0135] For example, a joint estimation model according to an embodiment of the present invention may correspond to a deep learning model designed using the TensorFlow Framework.

[0136] In this case, the array data may correspond to data consisting of 8-bit integers.

[0137] At this time, the image converted into array data is input into a deep learning-based joint estimation model, and multiple joints corresponding to the user's posture in the image can be displayed in a two-dimensional coordinate system.

[0138] In this case, the multiple joints can correspond to nine joints corresponding to the head, right shoulder, left shoulder, right buttock, left buttock, right knee, left knee, right foot, and left foot.

[0139] In other words, according to the present invention, simply by inputting a captured image of a user into a joint estimation model, the positions of nine joints corresponding to the user's posture in the image can be estimated and displayed in a two-dimensional coordinate system.

[0140] In this case, the multi-angle two-dimensional pose can correspond to data expressed by connecting multiple joints on a two-dimensional coordinate system.

[0141] For example, as shown in Figures 3 and 6, two-dimensional poses 330 and 630 can be generated by representing the nine joints estimated by the joint estimation model as two-dimensional points and connecting each point according to the user's body structure.

[0142] In this case, the video input for generating multi-angle 2D poses is not video edited by an expert, but video taken by a user in their daily life, so that the beginning and end of the video contain unnecessary frames 1021 and 1022, as shown in FIG. 10.

[0143] For example, the video contains unnecessary parts, such as a part that is shot when the user presses the camera's start shooting button to adjust the camera angle, and a part that moves to press the camera's end shooting button.

[0144] In the present invention, in order to prevent calculations from being performed on such unnecessary frames, which would cause unnecessary load on the system, unnecessary frames 1021 and 1022 are deleted from the entire frames corresponding to the image, and an effective frame 1010 is extracted, and only the effective frame 1010 is converted into array data.

[0145] For example, the present invention may provide a video editing function that allows a user to directly select only valid frames via a user interface provided to the user.

[0146] As another example, any portion of the video in which the user's entire body is not captured may be determined as an unnecessary frame and deleted.

[0147] At this time, the method of deleting unnecessary frames from the image and extracting only valid frames may use various methods applicable to the present invention, and is not limited to a specific method.

[0148] In addition, processor 1520 generates a three-dimensional pose by combining two-dimensional poses at multiple angles based on the attitude information of the camera that captured the video.

[0149] For example, the 2D coordinates (X f ,Y f ) and obtain 2D coordinates (X s ,Y s ) can be obtained. The 2D coordinates obtained in this way are combined for each joint position according to the camera orientation to obtain 3D coordinates (X, Y, Z = X f ,Y f ,X s ) can be obtained. At this time, the Z value in the 3D coordinates corresponds to the X value of the 2D coordinates (X s ) can be used.

[0150] At this time, the acquired 3D coordinates are scaled based on the user's head and toes, and expressed as vector values ​​on the 3D coordinates as shown in Figure 7, thereby generating a 3D pose corresponding to the user's posture in the video.

[0151] For example, the scaling of the three-dimensional coordinates is performed according to equation (1).

[0152]

number

[0153] The processor 1520 also generates a heat map (HEATMAP) that visually indicates the pressure distribution on the sole of the foot based on the three-dimensional pose, and extracts the center of gravity based on the heat map.

[0154] At this time, the heat map visually indicates the pressure distribution on the soles of the feet according to the user's posture, and is generally provided in a manner that measures the center of gravity based on a computing device connected to the pressure sensor when the user stands directly on a step equipped with a pressure sensor.

[0155] However, in the present invention, even without a pressure sensor, a heat map can be generated by inputting the 3D pose generated based on the video into a deep learning-based center of gravity estimation model.

[0156] In this case, the center of gravity estimation model can correspond to a deep learning-based artificial neural network.

[0157] For example, the center of gravity estimation model according to an embodiment of the present invention may correspond to a deep learning model-based artificial neural network designed using the TensorFlow Framework.

[0158] At this time, the generated 3D pose makes it possible to know all the front and side poses of the user in the video, so that the 3D pose can be input to the center of gravity estimation model to estimate the front-rear and left-right weight balance for the user's posture in the video. The estimated front-rear and left-right weight balance can be expressed as the pressure received by the soles of the user's feet, and a heat map such as that shown in FIG. 8 can be provided.

[0159] At this time, the heat map is displayed so that the greater the pressure applied to the sole of the foot, the more visually emphasized it is.

[0160] For example, if you check the weight balance between the left and right sides shown numerically on the heat map in Figure 8, you can see that the weight balance is biased to the left, with LEFT at 89 and RIGHT at 11. Therefore, the left foot can be emphasized more than the right foot on the heat map to clearly see at a glance that higher pressure is applied to the left foot than the right foot.

[0161] FIG. 9 illustrates the process described above so that it can be clearly understood at a glance. Based on images captured from multiple angles, 2D poses are generated for each of the front and side images of the user in the images, and a heat map can be generated using the 3D poses generated by combining the 2D poses.

[0162] The processor 1520 also provides center of gravity information corresponding to the user's posture using the weight balance calculated based on the center of gravity.

[0163] At this time, the weight balance may include front-rear weight balance and left-right weight balance.

[0164] At this time, the output value of the heat map can be normalized to extract the center of gravity.

[0165] For example, as shown in equation (2), the weighted Arg softmax function, which is one of the softmax functions, is applied to the heat map to normalize it, and the center of gravity (X p ,Y p ) can be output.

[0166]

number

[0167] At this time, a center of gravity correction value is generated, and the left and right weight balance corresponding to the user's left and right feet is calculated based on the center of gravity correction value. The values ​​corresponding to the left and right feet in the heat map are normalized to extract the center of gravity of the left and right feet, and the extracted center of gravity points are corrected to calculate the front and rear weight balance corresponding to the left and right feet.

[0168] For example, if the center of gravity correction value is (X p ,Y p ), the left and right weight balance can be calculated as shown in equation (3).

[0169]

number

[0170] As another example, the center of gravity of the left foot and the center of gravity of the right foot can be calculated by applying a weighted Argsoftmax function to each of the left foot and the right foot in the heat map, and the front-to-back weight balance corresponding to each of the user's left foot and right foot can be calculated based on the corrected values ​​of the center of gravity of the left foot and the center of gravity of the right foot.

[0171] If the left foot center of gravity correction value is Y L_p , the center of gravity correction value for the right foot is Y R_p If this assumption is made, then it is possible to calculate the front / rear weight balances R / L_front and R / L_rear corresponding to the left foot and the right foot, respectively, as shown in equation (4).

[0172]

number

[0173] The processor 1520 also outputs center of gravity information to correspond to the image based on the user interface.

[0174] 11 and 12, for example, a user interface is provided on a user's mobile terminal such as a smartphone, and the user interface can provide center of gravity information 1120 to be matched with the image being played back while playing back an image 1110 used to output center of gravity information.

[0175] That is, by providing center of gravity information regarding the golf swing posture as shown in Fig. 11, the user can objectively grasp the timing and degree of center of gravity movement during the golf swing. Also, by analyzing body parts with slightly incorrect posture through the extracted joint information and providing the information to the user, it can be used for swing coaching.

[0176] Alternatively, as shown in FIG. 12, it may be used as a basis for judging whether the movements of bodyweight training are being performed correctly, or when the user assumes a home training posture, the extracted joint information may be used to analyze the posture for home training coaching.

[0177] At this time, center of gravity information corresponding to at least one of the heat map, weight balance, center of gravity point, and moving line of the center of gravity point can be outputted through the user interface.

[0178] For example, as shown in FIG. 8, a heat map, a center of gravity 810 shown on the heat map, and a line of movement of the center of gravity 820 may be output, and the left and right weight balance as well as the front and rear weight balance of each of the feet may be quantified and output.

[0179] The memory 1530 stores at least one of an image, a heat map, and center of gravity information.

[0180] In addition, the memory 1530 can store various information generated during the process of providing the center of gravity information.

[0181] Through this center of gravity information providing device, center of gravity information regarding a user's posture and balance can be provided without restrictions on location or equipment, using only images captured by a camera in everyday locations where no separate measuring equipment is installed.

[0182] In addition, the center of gravity information service can be provided in the form of an application that can be freely operated in the user's personal PC, smartphone, or other environment.

[0183] Furthermore, by providing center of gravity information corresponding to the user's movements contained in the video, a service can be provided to assist users who enjoy sports that are affected by exercise posture and shifts in the center of gravity to correct their posture and move more efficiently.

[0184] As described above, the method and apparatus for providing center of gravity information using an image according to the present invention are not limited to the configurations and methods of the embodiments described above, and the above embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

Claims

1. generating a multi-angle two-dimensional pose corresponding to a posture of a user based on images of the user captured from multiple angles; generating a three-dimensional pose by combining the two-dimensional poses of the multiple angles; generating a heat map (HEATMAP) visually showing a pressure distribution of the sole of the foot based on the three-dimensional pose, and extracting a center of gravity based on the heat map; providing center of gravity information corresponding to a posture of the user using a weight balance calculated based on the center of gravity point; A method for providing center of gravity information, comprising:

2. The weight balance is 2. The method for providing center of gravity information according to claim 1, further comprising the step of: providing a center of gravity information including a front-rear weight balance and a left-right weight balance.

3. The method of claim 2 , further comprising the step of outputting the center of gravity information in accordance with the image based on a user interface.

4. The outputting step includes: The center-of-gravity information providing method according to claim 3 , further comprising the step of outputting at least one of the heat map, the weight balance, the center of gravity point, and a line of movement of the center of gravity point.

Images