Video-based motion assessment method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a video-based motion evaluation method, device, computer equipment and storage medium, the method comprising: obtaining a first motion video of a user and a second motion video of a coach; identifying the user's motion type according to the first motion video; extracting a first key frame image in the first motion video and extracting a second key frame image in the second motion video according to the motion type; detecting a first human key point in the first key frame image and detecting a second human key point in the second key frame image; calculating the difference between the first human key point and the second human key point; and evaluating the user's motion score according to the difference and the motion type. The present invention can adaptively extract key frame images and calculate the difference according to the motion type, and evaluate the user's motion in combination with the motion type and the difference, with a higher accuracy. In addition, the present invention also relates to blockchain technology, and the motion score can be stored in a blockchain node.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology in artificial intelligence, and in particular to a video-based motion assessment method, device, computer equipment and storage medium. Background Art

[0002] With the progress of society, people's awareness of fitness has gradually increased. However, due to the limitations of fitness venues and time, the demand for taking fragmented time at home to do fitness exercises has gradually increased, and more and more teaching videos of body movements have been launched on various sharing platforms.

[0003] In order to evaluate the body movements of home users and allow them to objectively know their own exercise effects, the prior art usually extracts key frames from the user's exercise video and standard exercise video, and then compares the key frames through a body recognition model to calculate the user's exercise score.

[0004] However, regardless of the type of motion, key frames are extracted in the same or random manner, so the calculated motion score cannot effectively evaluate whether the user's body movements are standard, and the evaluation accuracy is low. Summary of the invention

[0005] In view of the above, it is necessary to propose a video-based motion assessment method, device, computer equipment and storage medium, which can adaptively extract key frame images and calculate the difference according to the motion type, and evaluate the user's motion by combining the motion type and the difference, with higher accuracy.

[0006] A first aspect of the present invention provides a video-based motion assessment method, the video-based motion assessment method comprising:

[0007] Obtaining a first exercise video of the user and a second exercise video of the coach;

[0008] identifying the user's exercise type according to the first exercise video;

[0009] Extracting a plurality of first key frame images in the first motion video and extracting a plurality of second key frame images in the second motion video according to the motion type;

[0010] Detecting a plurality of first human key points in each first key frame image and detecting a plurality of second human key points in each second key frame image;

[0011] Calculating the difference between the plurality of first human body key points and the plurality of second human body key points;

[0012] An exercise score of the user is evaluated according to the difference and the exercise type.

[0013] According to an optional embodiment of the present invention, identifying the movement type of the user according to the first movement video includes:

[0014] Continuously extracting a first frame image, a second frame image, and a third frame image from the first motion video;

[0015] Calculating a first pixel difference between the first frame image and the second frame image and calculating a second pixel difference between the second frame image and the third frame image;

[0016] comparing an average difference between the first pixel difference and the second pixel difference with a plurality of preset difference ranges;

[0017] Matching a target preset difference range corresponding to the average difference;

[0018] The exercise type corresponding to the target preset difference range is determined as the exercise type of the user.

[0019] According to an optional embodiment of the present invention, after extracting a plurality of first key frame images in the first motion video and extracting a plurality of second key frame images in the second motion video according to the motion type, the video-based motion assessment method further includes:

[0020] Performing grayscale processing on each first key frame image to obtain a first grayscale image, and performing grayscale processing on each second key frame image to obtain a second grayscale image;

[0021] The first grayscale image is compressed to obtain a first compressed image, and the second grayscale image is compressed to obtain a second compressed image.

[0022] According to an optional embodiment of the present invention, the calculating the difference between the plurality of first human body key points and the plurality of second human body key points comprises:

[0023] Acquire a first coordinate point of each first human key point relative to the first key frame image and acquire a second coordinate point of each second human key point relative to the second key frame image;

[0024] Obtaining the resolution of the display screen of the computer device;

[0025] Convert each of the first coordinate points to obtain a first standard coordinate point and convert each of the second coordinate points to obtain a second standard coordinate point according to the resolution;

[0026] The difference is calculated based on the first standard coordinate point and the second standard coordinate point.

[0027] According to an optional embodiment of the present invention, the resolution is the first coefficient*the second coefficient, and converting the first coordinate point according to the resolution to obtain the first standard coordinate point and converting the second coordinate point to obtain the second standard coordinate point includes:

[0028] Calculating a first ratio of a length of the first key frame image to the first coefficient and calculating a second ratio of a width of the first key frame image to the second coefficient;

[0029] Calculating a third ratio of the length of the second key frame image to the first coefficient and calculating a fourth ratio of the width of the second key frame image to the second coefficient;

[0030] Scaling the first coordinate point by the first ratio in the vertical direction and scaling the first coordinate point by the second ratio in the horizontal direction to obtain a first standard coordinate point;

[0031] The second coordinate point is scaled by the third ratio in the vertical direction and the second coordinate point is scaled by the fourth ratio in the horizontal direction to obtain a second standard coordinate point.

[0032] According to an optional embodiment of the present invention, calculating the difference degree according to the first standard coordinate point and the second standard coordinate point includes:

[0033] Associating the first standard coordinate point with the corresponding second standard coordinate point according to the order of the time axis;

[0034] The distance between each associated coordinate point is calculated to obtain the difference.

[0035] According to an optional embodiment of the present invention, evaluating the user's exercise score according to the difference and the exercise type includes:

[0036] Calculate the variance of all differences;

[0037] Obtaining a score corresponding to the variance;

[0038] Obtaining a score weight corresponding to the sport type;

[0039] The user's sports score is obtained according to the product of the score and the score weight, and the sports score is stored in the blockchain node.

[0040] A second aspect of the present invention provides a video-based motion assessment device, the video-based motion assessment device comprising:

[0041] A video acquisition module, used to acquire a first motion video of the user and a second motion video of the coach;

[0042] a type identification module, configured to identify the user's exercise type according to the first exercise video;

[0043] A key frame extraction module, used for extracting a plurality of first key frame images in the first motion video and extracting a plurality of second key frame images in the second motion video according to the motion type;

[0044] A key point detection module, used to detect a plurality of first human key points in each first key frame image and to detect a plurality of second human key points in each second key frame image;

[0045] A difference calculation module, used for calculating the difference between the plurality of first human body key points and the plurality of second human body key points;

[0046] The motion evaluation module is used to evaluate the user's motion score according to the difference and the motion type.

[0047] A third aspect of the present invention provides a computer device, the computer device comprising a processor, the processor being configured to implement the video-based motion assessment method when executing a computer program stored in a memory.

[0048] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the video-based motion assessment method when executed by a processor.

[0049] In summary, the video-based motion assessment method, device, computer equipment and storage medium described in the present invention obtain the user's motion type by identifying the user's first motion video, and extract key frames from the first motion video and the second motion video according to the motion type, thereby realizing the adaptive extraction of key frame images according to the user's motion type. The extracted key frame images can more effectively reflect the user's body movements, and then detect the human body key points in the key frame images and calculate the difference between the human body key points. Finally, the user's motion score is evaluated in combination with the difference and the motion type, which has higher accuracy and is more practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of a video-based motion assessment method provided in Embodiment 1 of the present invention.

[0051] Figure 2 It is a structural diagram of a video-based motion assessment device provided in Embodiment 2 of the present invention.

[0052] Figure 3 It is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0055] Figure 1 The flowchart of the video-based motion assessment method provided by the first embodiment of the present invention is as follows. The video-based motion assessment method specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0056] S11, obtaining a first exercise video of the user and a second exercise video of the coach.

[0057] An image acquisition device may be installed on the computer device, and the image acquisition device is used to capture and obtain the first motion video of the user and the second motion video of the coach.

[0058] The computer device may be provided with a communication module, through which the first motion video of the user and the second motion video of the coach sent by other electronic devices are received.

[0059] The manner in which the computer device obtains the user's first motion video and the coach's second motion video is not limited to the above-listed ones.

[0060] S12: Identify the exercise type of the user according to the first exercise video.

[0061] Since each user has a different learning situation, the user's movement type is identified through the first movement video, and key frame images are extracted according to the movement type, so as to improve the accuracy of evaluating the user's movement.

[0062] In an optional embodiment, identifying the user's motion type according to the first motion video includes:

[0063] Continuously extracting a first frame image, a second frame image, and a third frame image from the first motion video;

[0064] Calculating a first pixel difference between the first frame image and the second frame image and calculating a second pixel difference between the second frame image and the third frame image;

[0065] comparing an average difference between the first pixel difference and the second pixel difference with a plurality of preset difference ranges;

[0066] Matching a target preset difference range corresponding to the average difference;

[0067] The exercise type corresponding to the target preset difference range is determined as the exercise type of the user.

[0068] In this optional embodiment, the pixel difference between the two frames can effectively reflect the type of exercise of the user. When the pixel difference is large, it indicates that the user's movement amplitude is large or the movement changes more, which is a violent exercise or exercise for young people. When the pixel difference is small, it indicates that the user's movement amplitude is small or the movement changes less, which is a soothing exercise or exercise for the elderly.

[0069] A plurality of difference ranges are pre-stored in the computer device, and each difference range corresponds to a motion type. Three frames of images can be continuously extracted from the middle part of the first motion video, and the pixel differences of the two frames before and after the images are calculated respectively, and then the average difference of the pixel differences is calculated, and finally it is determined in which difference range the average difference falls, and the motion type of the user can be determined according to the corresponding relationship between the difference range and the motion type.

[0070] S13: extracting a plurality of first key frame images from the first motion video and a plurality of second key frame images from the second motion video according to the motion type.

[0071] Exemplarily, assuming that the user's exercise type is strenuous exercise, it means that the user's movements change quickly, and the information expressed by the two frames of images before and after is quite different. Therefore, key frame images in the motion video can be extracted every preset first number of frames. If the user's exercise type is soothing exercise, it means that the user's movements change slowly, and the information expressed by the two frames of images before and after is slightly different. Therefore, key frame images in the motion video can be extracted every preset second number of frames.

[0072] The preset first frame number is less than the preset second frame number, for example, the preset first frame number is 1, and the preset second frame number is 4. By using different frame numbers to extract key frame images according to the motion type, when the motion change is fast, as many key frame images as possible can be extracted for motion evaluation to improve the accuracy of the evaluation; when the motion change is slow, a small number of key frame images can be extracted for motion evaluation to reduce the amount of calculation.

[0073] In an optional embodiment, after extracting a plurality of first key frame images in the first motion video and extracting a plurality of second key frame images in the second motion video according to the motion type, the video-based motion assessment method further includes:

[0074] Performing grayscale processing on each first key frame image to obtain a first grayscale image, and performing grayscale processing on each second key frame image to obtain a second grayscale image;

[0075] The first grayscale image is compressed to obtain a first compressed image, and the second grayscale image is compressed to obtain a second compressed image.

[0076] This optional embodiment, by gray-scaling the key frame image, removes the color and texture information in the key frame image, which can improve the efficiency of the human key point detection model in detecting the human key points in the key frame image; and compressing the gray-scale image can further improve the efficiency of the human key point detection model in detecting the human key points in the key frame image.

[0077] S14, detecting a plurality of first human key points in each first key frame image and detecting a plurality of second human key points in each second key frame image.

[0078] An open source TensorFlow model can be downloaded, and a human key point detection model can be trained based on the open source TensorFlow model, and a plurality of human key points in a key frame image can be detected by the human key point detection model. The training process of the human key point detection model is prior art, and the present invention will not be elaborated in detail.

[0079] The key points of the human body include: 0 nose, 1 midpoint of clavicle, 2 right shoulder, 3 right elbow, 4 right wrist, 5 left shoulder, 6 left elbow, 7 left wrist, 8 right hip, 9 right knee, 10 right ankle, 11 left hip, 12 left knee, 13 left ankle, 14 right eye, 15 left eye, 16 right ear, 17 left ear and other 18 points.

[0080] S15, calculating the difference between the plurality of first human body key points and the plurality of second human body key points.

[0081] After detecting multiple first human body key points in the user's first motion video and multiple second human body key points in the coach's second motion video, the computer device can calculate the difference based on the multiple first human body key points and the multiple second human body key points to evaluate the difference between the user's movement and the coach's movement.

[0082] In an optional embodiment, the calculating the difference between the plurality of first human body key points and the plurality of second human body key points comprises:

[0083] Acquire a first coordinate point of each first human key point relative to the first key frame image and acquire a second coordinate point of each second human key point relative to the second key frame image;

[0084] Obtaining the resolution of the display screen of the computer device;

[0085] Convert the first coordinate point to obtain a first standard coordinate point and convert the second coordinate point to obtain a second standard coordinate point according to the resolution;

[0086] The difference is calculated based on the first standard coordinate point and the second standard coordinate point.

[0087] In this optional embodiment, since there are physical differences between the user and the coach in height, size, etc., the first coordinate point and the second coordinate point are relative positions. In order to facilitate the subsequent calculation of the difference, the first coordinate point and the second coordinate point at the relative position are converted on the same display screen, so that the first standard coordinate point and the second standard coordinate point obtained after the conversion have the same dimension and are comparable, and the calculated difference has a higher accuracy rate.

[0088] In an optional embodiment, the resolution is the first coefficient*the second coefficient.

[0089] In an optional embodiment, converting the first coordinate point according to the resolution to obtain a first standard coordinate point and converting the second coordinate point to obtain a second standard coordinate point includes:

[0090] Calculating a first ratio of a length of the first key frame image to the first coefficient and calculating a second ratio of a width of the first key frame image to the second coefficient;

[0091] Calculating a third ratio of the length of the second key frame image to the first coefficient and calculating a fourth ratio of the width of the second key frame image to the second coefficient;

[0092] Scaling the first coordinate point by the first ratio in the vertical direction and scaling the first coordinate point by the second ratio in the horizontal direction to obtain a first standard coordinate point;

[0093] The second coordinate point is scaled by the third ratio in the vertical direction and the second coordinate point is scaled by the fourth ratio in the horizontal direction to obtain a second standard coordinate point.

[0094] In this optional embodiment, the coordinate points in the key frame image can be mapped onto the display screen by calculating the ratio between the key frame image and the resolution and then scaling the coordinate points according to the ratio.

[0095] In an optional embodiment, calculating the difference degree according to the first standard coordinate point and the second standard coordinate point includes:

[0096] Associating the first standard coordinate point with the corresponding second standard coordinate point according to the order of the time axis;

[0097] The distance between each associated coordinate point is calculated to obtain the difference.

[0098] In this optional embodiment, the first standard coordinate point and the second standard coordinate point are matched in the order of the time axis, and then the Euclidean distance between every two coordinate points is calculated to obtain the difference between the user and the coach.

[0099] S16: Evaluate the user's exercise score according to the difference and the exercise type.

[0100] The present invention relates to artificial intelligence, which obtains the user's motion type by identifying a first motion video of the user, extracts key frames from the first motion video and a second motion video according to the motion type, and realizes adaptive extraction of key frame images according to the user's motion type, wherein the extracted key frame images can more effectively reflect the user's body movements; then, a human key point detection model is used to detect human key points in the key frame images and calculate the difference between the human key points, and finally, the user's motion score is evaluated in combination with the difference and the motion type, which has higher accuracy and is more in line with reality.

[0101] In an optional embodiment, evaluating the user's exercise score according to the difference and the exercise type includes:

[0102] Calculate the variance of all differences;

[0103] Obtaining a score corresponding to the variance;

[0104] Obtaining a score weight corresponding to the sport type;

[0105] The sports score of the user is obtained according to the product of the score and the score weight.

[0106] In this optional embodiment, a first correspondence between the variance and the score can be preset, for example, a variance of 0.1 corresponds to a score of 95, and a variance of 0.2 corresponds to a score of 85. A second correspondence between the type of exercise and the score weight can be preset, for example, if the type of exercise is elderly, the score weight is 1.1, and if the type of exercise is young, the score weight is 0.9. The score of the variance is obtained according to the first correspondence, and the score weight corresponding to the type of exercise is obtained according to the second correspondence. Finally, the user's exercise score can be calculated according to the score and the score weight. Since the calculated exercise score not only takes into account the difference between the user's body movements and the coach's body movements, but also takes into account the type of exercise, for elderly exercises, since the elderly have bent limbs as they grow older, the score can be expanded, and the calculated exercise score is more accurate, and the scoring of the user is more practical.

[0107] In an optional embodiment, the video-based motion assessment method further includes:

[0108] Acquire a first target difference whose difference is greater than a preset first difference threshold;

[0109] Acquire a second target difference whose difference is less than a preset second difference threshold;

[0110] Playing the first motion video and the second motion video on a display screen;

[0111] Prompting a human body key point corresponding to the first target difference in the first motion video according to a preset first prompt; and / or

[0112] According to the preset second prompt, the human body key points corresponding to the second target difference are prompted in the first motion video.

[0113] The coach's motion video and the user's motion video are displayed on the same display screen at the same time. For example, the user's first motion video is displayed on the right side of the computer device, and the coach's second motion video is displayed on the left side of the computer device, so that the user can not only watch the standard body movements, but also watch his own body movements.

[0114] The preset prompt may include a text prompt, a voice prompt or an animation prompt, etc. The preset first difference threshold and the preset second difference threshold may be the same or different.

[0115] The preset first prompt is used to prompt the user of the key points of the human body where the action difference occurs, and indicates the correct movement direction of the key points of the human body according to the position of the key points where the action difference occurs (for example, indicated by a red arrow, and voice reminders can also be used synchronously). The preset second prompt is used to prompt the key points of the human body where the action standard occurs, and the position of the key points where the action standard occurs is displayed by changing color or changing into the limbs of a cartoon character. While playing the sports video on the same screen, according to the preset prompt method, not only prompts are given when there are incorrect limb movements, but also voice praise or rewards are given when the limb movements are correct, so that the user can get dual feedback in both vision and hearing.

[0116] In an optional embodiment, the coach's second sports video in this embodiment can be stored on the blockchain node, thereby taking advantage of the distributed data storage and reading on the blockchain.

[0117] It should be emphasized that in order to further ensure the privacy and security of the above sports scores, the above sports scores can also be stored in a blockchain node.

[0118] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0119] Figure 2 It is a structural diagram of a video-based motion assessment device provided in Embodiment 2 of the present invention.

[0120] In some embodiments, the video-based motion assessment device 20 may include a plurality of functional modules composed of program code segments. The program code of each program segment in the video-based motion assessment device 20 may be stored in a memory of a computer device and executed by the at least one processor to perform (see Figure 1 Description) Functionality for video-based motion assessment.

[0121] In this embodiment, the video-based motion evaluation device 20 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: a video acquisition module 201, a type recognition module 202, a key frame extraction module 203, an image processing module 204, a key point detection module 205, a difference calculation module 206, a motion evaluation module 207 and a video playback module 208. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0122] The video acquisition module 201 is used to acquire a first motion video of a user and a second motion video of a coach.

[0123] An image acquisition device may be installed on the computer device, and the image acquisition device is used to capture and obtain the first motion video of the user and the second motion video of the coach.

[0124] The computer device may be provided with a communication module, through which the first motion video of the user and the second motion video of the coach sent by other electronic devices are received.

[0125] The manner in which the computer device obtains the user's first motion video and the coach's second motion video is not limited to the above-listed ones.

[0126] The type identification module 202 is used to identify the exercise type of the user according to the first exercise video.

[0127] Since each user has a different learning situation, the user's movement type is identified through the first movement video, and key frame images are extracted according to the movement type, so as to improve the accuracy of evaluating the user's movement.

[0128] In an optional embodiment, the type identification module 202 identifies the user's exercise type according to the first exercise video, including:

[0129] Continuously extracting a first frame image, a second frame image, and a third frame image from the first motion video;

[0130] Calculating a first pixel difference between the first frame image and the second frame image and calculating a second pixel difference between the second frame image and the third frame image;

[0131] comparing an average difference between the first pixel difference and the second pixel difference with a plurality of preset difference ranges;

[0132] Matching a target preset difference range corresponding to the average difference;

[0133] The exercise type corresponding to the target preset difference range is determined as the exercise type of the user.

[0134] In this optional embodiment, the pixel difference between the two frames can effectively reflect the type of exercise of the user. When the pixel difference is large, it indicates that the user's movement amplitude is large or the movement changes more, which is a violent exercise or exercise for young people. When the pixel difference is small, it indicates that the user's movement amplitude is small or the movement changes less, which is a soothing exercise or exercise for the elderly.

[0135] A plurality of difference ranges are pre-stored in the computer device, and each difference range corresponds to a motion type. Three frames of images can be continuously extracted from the middle part of the first motion video, and the pixel differences of the two frames before and after the images are calculated respectively, and then the average difference of the pixel differences is calculated, and finally it is determined in which difference range the average difference falls, and the motion type of the user can be determined according to the corresponding relationship between the difference range and the motion type.

[0136] The key frame extraction module 203 is used to extract a plurality of first key frame images in the first motion video and a plurality of second key frame images in the second motion video according to the motion type.

[0137] Exemplarily, assuming that the user's exercise type is strenuous exercise, it means that the user's movements change quickly, and the information expressed by the two frames of images before and after is quite different. Therefore, key frame images in the motion video can be extracted every preset first number of frames. If the user's exercise type is soothing exercise, it means that the user's movements change slowly, and the information expressed by the two frames of images before and after is slightly different. Therefore, key frame images in the motion video can be extracted every preset second number of frames.

[0138] The preset first frame number is less than the preset second frame number, for example, the preset first frame number is 1, and the preset second frame number is 4. By using different frame numbers to extract key frame images according to the motion type, when the motion change is fast, as many key frame images as possible can be extracted for motion evaluation to improve the accuracy of the evaluation; when the motion change is slow, a small number of key frame images can be extracted for motion evaluation to reduce the amount of calculation.

[0139] The image processing module 204 is used to process the multiple first key frame images and the multiple second key frame images.

[0140] In an optional embodiment, the image processing module 204 processes the plurality of first key frame images and the plurality of second key frame images, including:

[0141] Performing grayscale processing on each first key frame image to obtain a first grayscale image, and performing grayscale processing on each second key frame image to obtain a second grayscale image;

[0142] The first grayscale image is compressed to obtain a first compressed image, and the second grayscale image is compressed to obtain a second compressed image.

[0143] This optional embodiment, by gray-scaling the key frame image, removes the color and texture information in the key frame image, which can improve the efficiency of the human key point detection model in detecting the human key points in the key frame image; and compressing the gray-scale image can further improve the efficiency of the human key point detection model in detecting the human key points in the key frame image.

[0144] The key point detection module 205 is used to detect a plurality of first human key points in each first key frame image and to detect a plurality of second human key points in each second key frame image.

[0145] An open source TensorFlow model can be downloaded, and a human key point detection model can be trained based on the open source TensorFlow model, and a plurality of human key points in a key frame image can be detected by the human key point detection model. The training process of the human key point detection model is prior art, and the present invention will not be elaborated in detail.

[0146] The key points of the human body include: 0 nose, 1 midpoint of clavicle, 2 right shoulder, 3 right elbow, 4 right wrist, 5 left shoulder, 6 left elbow, 7 left wrist, 8 right hip, 9 right knee, 10 right ankle, 11 left hip, 12 left knee, 13 left ankle, 14 right eye, 15 left eye, 16 right ear, 17 left ear and other 18 points.

[0147] The difference calculation module 206 is used to calculate the difference between the first plurality of human body key points and the second plurality of human body key points.

[0148] After detecting multiple first human body key points in the user's first motion video and multiple second human body key points in the coach's second motion video, the computer device can calculate the difference based on the multiple first human body key points and the multiple second human body key points to evaluate the difference between the user's movement and the coach's movement.

[0149] In an optional embodiment, the difference calculation module 206 calculates the difference between the plurality of first human body key points and the plurality of second human body key points including:

[0150] Acquire a first coordinate point of each first human key point relative to the first key frame image and acquire a second coordinate point of each second human key point relative to the second key frame image;

[0151] Obtaining the resolution of the display screen of the computer device;

[0152] Convert the first coordinate point to obtain a first standard coordinate point and convert the second coordinate point to obtain a second standard coordinate point according to the resolution;

[0153] The difference is calculated based on the first standard coordinate point and the second standard coordinate point.

[0154] In this optional embodiment, since there are physical differences between the user and the coach in height, size, etc., the first coordinate point and the second coordinate point are relative positions. In order to facilitate the subsequent calculation of the difference, the first coordinate point and the second coordinate point at the relative position are converted on the same display screen, so that the first standard coordinate point and the second standard coordinate point obtained after the conversion have the same dimension and are comparable, and the calculated difference has a higher accuracy rate.

[0155] In an optional embodiment, the resolution is the first coefficient*the second coefficient.

[0156] In an optional embodiment, converting the first coordinate point according to the resolution to obtain a first standard coordinate point and converting the second coordinate point to obtain a second standard coordinate point includes:

[0157] Calculating a first ratio of a length of the first key frame image to the first coefficient and calculating a second ratio of a width of the first key frame image to the second coefficient;

[0158] Calculating a third ratio of the length of the second key frame image to the first coefficient and calculating a fourth ratio of the width of the second key frame image to the second coefficient;

[0159] Scaling the first coordinate point by the first ratio in the vertical direction and scaling the first coordinate point by the second ratio in the horizontal direction to obtain a first standard coordinate point;

[0160] The second coordinate point is scaled by the third ratio in the vertical direction and the second coordinate point is scaled by the fourth ratio in the horizontal direction to obtain a second standard coordinate point.

[0161] In this optional embodiment, the coordinate points in the key frame image can be mapped onto the display screen by calculating the ratio between the key frame image and the resolution and then scaling the coordinate points according to the ratio.

[0162] In an optional embodiment, calculating the difference degree according to the first standard coordinate point and the second standard coordinate point includes:

[0163] Associating the first standard coordinate point with the corresponding second standard coordinate point according to the order of the time axis;

[0164] The distance between each associated coordinate point is calculated to obtain the difference.

[0165] In this optional embodiment, the first standard coordinate point and the second standard coordinate point are matched in the order of the time axis, and then the Euclidean distance between every two coordinate points is calculated to obtain the difference between the user and the coach.

[0166] The motion evaluation module 207 is used to evaluate the user's motion score according to the difference and the motion type.

[0167] The user's motion type is obtained by identifying the user's first motion video, and key frames are extracted from the first motion video and the second motion video according to the motion type. The key frame images are adaptively extracted according to the user's motion type. The extracted key frame images can more effectively reflect the user's body movements. The human body key points in the key frame images are detected and the difference between the human body key points is calculated. Finally, the user's motion score is evaluated in combination with the difference and the motion type, which has higher accuracy and is more realistic.

[0168] In an optional embodiment, the movement evaluation module 207 evaluates the movement score of the user according to the difference and the movement type, including:

[0169] Calculate the variance of all differences;

[0170] Obtaining a score corresponding to the variance;

[0171] Obtaining a score weight corresponding to the sport type;

[0172] The sports score of the user is obtained according to the product of the score and the score weight.

[0173] In this optional embodiment, a first correspondence between the variance and the score can be preset, for example, a variance of 0.1 corresponds to a score of 95, and a variance of 0.2 corresponds to a score of 85. A second correspondence between the type of exercise and the score weight can be preset, for example, if the type of exercise is elderly, the score weight is 1.1, and if the type of exercise is young, the score weight is 0.9. The score of the variance is obtained according to the first correspondence, and the score weight corresponding to the type of exercise is obtained according to the second correspondence. Finally, the user's exercise score can be calculated according to the score and the score weight. Since the calculated exercise score not only takes into account the difference between the user's body movements and the coach's body movements, but also takes into account the type of exercise, for elderly exercises, since the elderly have bent limbs as they grow older, the score can be expanded, and the calculated exercise score is more accurate, and the scoring of the user is more practical.

[0174] The video playing module 208 is used to play the first sports video and the second sports video according to a preset prompt.

[0175] In an optional embodiment, the video playing module 208 plays the first motion video and the second motion video according to the preset prompt, including:

[0176] Acquire a first target difference whose difference is greater than a preset first difference threshold;

[0177] Acquire a second target difference whose difference is less than a preset second difference threshold;

[0178] Playing the first motion video and the second motion video on a display screen;

[0179] Prompting a human body key point corresponding to the first target difference in the first motion video according to a preset first prompt; and / or

[0180] According to the preset second prompt, the human body key points corresponding to the second target difference are prompted in the first motion video.

[0181] The coach's motion video and the user's motion video are displayed on the same display screen at the same time. For example, the user's first motion video is displayed on the right side of the computer device, and the coach's second motion video is displayed on the left side of the computer device, so that the user can not only watch the standard body movements, but also watch his own body movements.

[0182] The preset prompt may include a text prompt, a voice prompt or an animation prompt, etc. The preset first difference threshold and the preset second difference threshold may be the same or different.

[0183] The preset first prompt is used to prompt the user of the key points of the human body where the action difference occurs, and indicates the correct movement direction of the key points of the human body according to the position of the key points where the action difference occurs (for example, indicated by a red arrow, and voice reminders can also be used synchronously). The preset second prompt is used to prompt the key points of the human body where the action standard occurs, and the position of the key points where the action standard occurs is displayed by changing color or changing into the limbs of a cartoon character. While playing the sports video on the same screen, according to the preset prompt method, not only prompts are given when there are incorrect limb movements, but also voice praise or rewards are given when the limb movements are correct, so that the user can get dual feedback in both vision and hearing.

[0184] In an optional embodiment, the coach's second sports video in this embodiment can be stored on the blockchain node, thereby taking advantage of the distributed data storage and reading on the blockchain.

[0185] It should be emphasized that in order to further ensure the privacy and security of the above sports scores, the above sports scores can also be stored in a blockchain node.

[0186] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0187] See also Figure 3 FIG. 2 is a schematic diagram of the structure of a computer device according to Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the computer device 3 includes a memory 31 , at least one processor 32 , at least one communication bus 33 and a transceiver 34 .

[0188] Those skilled in the art should understand that Figure 3The structure of the computer device shown does not constitute a limitation of the embodiments of the present invention, and may be a bus structure or a star structure. The computer device 3 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components.

[0189] In some embodiments, the computer device 3 is a computer device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The computer device 3 may also include client devices, which include but are not limited to any electronic product that can interact with the client through a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0190] It should be noted that the computer device 3 is only an example, and other existing or future electronic products that are suitable for the present invention should also be included in the protection scope of the present invention and included here by reference.

[0191] In some embodiments, the memory 31 stores program codes, and the at least one processor 32 can call the program codes stored in the memory 31 to perform related functions. For example, the modules described in the above embodiments are program codes stored in the memory 31 and are executed by the at least one processor 32 to implement the functions of the modules. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0192] In some embodiments, the at least one processor 32 is the control core (Control Unit) of the computer device 3, and uses various interfaces and lines to connect various components of the entire computer device 3, and executes various functions and processes data of the computer device 3 by running or executing programs or modules stored in the memory 31, and calling data stored in the memory 31. For example, when the at least one processor 32 executes the program code stored in the memory, all or part of the steps of the video-based motion assessment method described in the embodiment of the present invention are implemented. The at least one processor 32 can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same function or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0193] In some embodiments, the at least one communication bus 33 is configured to implement connection and communication between the memory 31 and the at least one processor 32, etc.

[0194] Although not shown, the computer device 3 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 32 through a power management device, so that the power management device can manage charging, discharging, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 3 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0195] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute the part of the video-based motion assessment method described in various embodiments of the present invention.

[0196] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0197] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0199] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A video-based motion assessment method, characterized in that: The video-based motion assessment method comprises: Obtaining a first exercise video of the user and a second exercise video of the coach; Identifying the motion type of the user according to the first motion video includes continuously extracting a first frame image, a second frame image, and a third frame image in the first motion video; calculating a first pixel difference between the first frame image and the second frame image and calculating a second pixel difference between the second frame image and the third frame image; comparing an average difference between the first pixel difference and the second pixel difference with a plurality of preset difference ranges; matching a target preset difference range corresponding to the average difference; and determining that the motion type corresponding to the target preset difference range is the motion type of the user; Extracting a plurality of first key frame images in the first motion video and extracting a plurality of second key frame images in the second motion video using different frame numbers according to the motion type, including: if the motion type is an intense motion, extracting a plurality of first key frame images in the first motion video and extracting a plurality of second key frame images in the second motion video every a preset first frame number; if the motion type is a relaxing motion, extracting a plurality of first key frame images in the first motion video and extracting a plurality of second key frame images in the second motion video every a preset second frame number, wherein the preset first frame number is less than the preset second frame number; Detecting a plurality of first human key points in each first key frame image and detecting a plurality of second human key points in each second key frame image; Calculating the difference between the plurality of first human body key points and the plurality of second human body key points; An exercise score of the user is evaluated according to the difference and the exercise type.

2. The video-based motion assessment method according to claim 1, characterized in that: After extracting a plurality of first key frame images from the first motion video and extracting a plurality of second key frame images from the second motion video according to the motion type, the video-based motion assessment method further includes: Performing grayscale processing on each first key frame image to obtain a first grayscale image, and performing grayscale processing on each second key frame image to obtain a second grayscale image; The first grayscale image is compressed to obtain a first compressed image, and the second grayscale image is compressed to obtain a second compressed image.

3. The video-based motion assessment method according to claim 1, characterized in that: The calculating the difference between the plurality of first human body key points and the plurality of second human body key points comprises: Acquire a first coordinate point of each first human key point relative to the first key frame image and acquire a second coordinate point of each second human key point relative to the second key frame image; Get the resolution of the display screen of the computer device; Convert each of the first coordinate points to obtain a first standard coordinate point and convert each of the second coordinate points to obtain a second standard coordinate point according to the resolution; The difference is calculated based on the first standard coordinate point and the second standard coordinate point.

4. The video-based motion assessment method according to claim 3, characterized in that: The resolution is the first coefficient*the second coefficient, and the converting the first coordinate point to obtain the first standard coordinate point and the converting the second coordinate point to obtain the second standard coordinate point according to the resolution includes: Calculating a first ratio of a length of the first key frame image to the first coefficient and calculating a second ratio of a width of the first key frame image to the second coefficient; Calculating a third ratio of the length of the second key frame image to the first coefficient and calculating a fourth ratio of the width of the second key frame image to the second coefficient; Scaling the first coordinate point by the first ratio in the vertical direction and scaling the first coordinate point by the second ratio in the horizontal direction to obtain a first standard coordinate point; The second coordinate point is scaled by the third ratio in the vertical direction and the second coordinate point is scaled by the fourth ratio in the horizontal direction to obtain a second standard coordinate point.

5. The video-based motion assessment method according to claim 4, characterized in that: The calculating the difference degree according to the first standard coordinate point and the second standard coordinate point comprises: Associating the first standard coordinate point with the corresponding second standard coordinate point according to the order of the time axis; The distance between each associated coordinate point is calculated to obtain the difference.

6. The video-based motion assessment method according to claim 5, characterized in that: The evaluating the user's exercise score according to the difference and the exercise type comprises: Calculate the variance of all differences; Obtaining a score corresponding to the variance; Obtaining a score weight corresponding to the sport type; The user's sports score is obtained according to the product of the score and the score weight, and the sports score is stored in the blockchain node.

7. A video-based motion assessment device, characterized in that: The video-based motion assessment device comprises: A video acquisition module, used to acquire a first motion video of the user and a second motion video of the coach; A type recognition module is used to recognize the motion type of the user according to the first motion video, including continuously extracting a first frame image, a second frame image and a third frame image in the first motion video; calculating a first pixel difference between the first frame image and the second frame image and calculating a second pixel difference between the second frame image and the third frame image; comparing an average difference between the first pixel difference and the second pixel difference with a plurality of preset difference ranges; matching a target preset difference range corresponding to the average difference; and determining that the motion type corresponding to the target preset difference range is the motion type of the user; A key frame extraction module, for extracting a plurality of first key frame images in the first motion video and a plurality of second key frame images in the second motion video by using different frame numbers according to the motion type, including: if the motion type is an intense motion, extracting a plurality of first key frame images in the first motion video and a plurality of second key frame images in the second motion video at intervals of a preset first frame number; if the motion type is a relaxing motion, extracting a plurality of first key frame images in the first motion video and a plurality of second key frame images in the second motion video at intervals of a preset second frame number, wherein the preset first frame number is less than the preset second frame number; A key point detection module, used to detect a plurality of first human key points in each first key frame image and to detect a plurality of second human key points in each second key frame image; A difference calculation module, used for calculating the difference between the plurality of first human body key points and the plurality of second human body key points; The motion evaluation module is used to evaluate the user's motion score according to the difference and the motion type.

8. A computer device, characterized in that: The computer device comprises a processor, and the processor is configured to implement the video-based motion assessment method according to any one of claims 1 to 6 when executing a computer program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video-based motion assessment method according to any one of claims 1 to 6 is implemented.

Images