Vision-based motion video fine analysis method and device

Through the detailed analysis method of vision-based motion video, using action recognition and human posture estimation models, the existing motion scoring system has been solved, and more objective and fine-grained action scoring and auxiliary suggestions are achieved. , highly popular.

CN114550027BActive Publication Date: 2024-12-06TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210052785.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-18
Publication Date
2024-12-06
Estimated Expiration
2042-01-18

AI Technical Summary

Technical Problem

The existing sports scoring system has problems such as high cost, complex operation, poor generalization and interference with the assessee, which makes it difficult to promote and use on a large scale.

Method used

Using the fine analysis method of motion video based on vision, by obtaining video data and inputting it into the preset action recognition model and human body pose estimation model, action category recognition and key point information matching, action scores are calculated, and visual analysis diagrams and actions are generated. Auxiliary advice.

Benefits of technology

It lowers the threshold for technical application, realizes more objective motion scores and fine-grained auxiliary suggestions, avoids interference with the movement process, and has the advantages of being easy to promote and popularize.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114550027B_ABST
    Figure CN114550027B_ABST
Patent Text Reader

Abstract

The present application proposes a vision-based fine analysis method for motion video, including: obtaining video data of the motion video, and inputting the video data into a preset action recognition model to obtain the corresponding action category; according to the action category, obtaining target standard data matching the action category from pre-collected standard video data; inputting the video data into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video; inputting the target standard data into the human posture estimation model to obtain the second key point information of each frame of the standard video image; matching the first key point information and the second key point information, determining the similarity of the first key point information and the second key point information, and determining the overall action score of the motion video according to the similarity. The present application can objectively score actions, and generate action auxiliary suggestions in a more fine-grained manner, and is easy to promote and implement, and has good application prospects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a vision-based motion video fine analysis method and device. Background Art

[0002] The traditional scoring method is mainly based on human subjective judgment, which requires the scorer to have certain experience and professionalism. The problems with manual scoring mainly include strong subjectivity, low efficiency, low accuracy, and vague scoring standards.

[0003] Existing technical implementation solutions:

[0004] One of the core technical points of the existing sports scoring system is the method of motion capture modeling of the athlete's movements. The main method currently used for motion modeling is to capture motion through sensor equipment. Sensors are installed on various joints of the human body. The data returned by the sensors are simulated through a certain algorithm model. According to the joint connection relationship of the human body, it is abstracted into a simple rigid entity, and the human skeleton point motion model can be further obtained.

[0005] The second core technical point of the existing motion scoring system is the motion evaluation method for the modeled motion model. The main method currently used is to obtain the skeleton point model, and judge the difference between the mutual motion relationship between its various joints, mainly the difference in the angular rotation range, relative position offset, etc. between each other, and then conduct scoring and evaluation based on multiple skeleton point data.

[0006] Disadvantages of existing technology:

[0007] The threshold for hardware equipment is relatively high: sensors need to be installed on various joints of the human body. Such a set of equipment is usually very expensive, which poses a certain obstacle to the promotion of sports scoring technology. At the same time, there may be certain problems with the adaptability of such equipment, and there may be accuracy deviations in data collection for individuals with large differences in body shape.

[0008] For competitive sports events, the use of hardware such as sensors will place unnecessary burdens on athletes and affect their competitive state.

[0009] Overall, the existing technology has high requirements for equipment and sites, and has the defects of high cost, complex operation, poor versatility, and interference with the movements of the person being evaluated, which means it is difficult to promote and use on a large scale. Summary of the invention

[0010] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0011] To this end, the first purpose of this application is to propose a vision-based motion video fine analysis method, which solves the problems of existing methods such as high cost, complex operation, poor versatility, and interference with the movements of the person being evaluated, which makes it difficult to promote and use them on a large scale. By using ordinary equipment to obtain video data, there is almost no interference to the movement process, and at the same time, the threshold for technical application is lowered, and it has the advantages of easy promotion and popularization. According to the analysis of the acquired video data, a more objective action score can be obtained, and this application can give more fine-grained action assistance suggestions through scoring in the time and space dimensions.

[0012] The second objective of this application is to propose a vision-based motion video fine analysis device.

[0013] A third object of the present application is to provide a non-transitory computer-readable storage medium.

[0014] To achieve the above-mentioned objectives, the first aspect of the present application proposes a vision-based method for fine analysis of motion videos, including: obtaining video data of the motion video, and inputting the video data into a preset action recognition model to obtain the corresponding action category; according to the action category, obtaining target standard data matching the action category from pre-collected standard video data, wherein the standard video data includes standard paradigms of different categories of actions; inputting the video data into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video; inputting the target standard data into the human posture estimation model to obtain the second key point information of each frame of the standard video image; matching the first key point information and the second key point information to determine the similarity between the first key point information and the second key point information, and determining the overall action score of the motion video based on the similarity.

[0015] Optionally, in one embodiment of the present application, before inputting the video data into a preset action recognition model, the method further includes:

[0016] Preprocess the video data, including:

[0017] adjusting the resolution, format and encoding of the video images in the video data to match the resolution, format and encoding of the standard video images in the standard video data; and / or

[0018] Normalizing pixel values ​​of the video image in the video data; and / or

[0019] Filling the pixel values ​​of the video image in the video data; and / or

[0020] The video images in the video data are cut into video segments according to the standard video images, so that the video images in the video data are aligned with the standard video images in time.

[0021] Optionally, in one embodiment of the present application, matching the first key point information and the second key point information to determine the similarity between the first key point information and the second key point information, and determining the overall action score of the motion video according to the similarity includes:

[0022] The similarity between the first key point information and the corresponding second key point information of each frame of video image is calculated respectively, and the average value of the similarity of all frames of video image is calculated, and the average value is used as the overall action score of the motion video.

[0023] Optionally, in one embodiment of the present application, before respectively calculating the similarity between the first key point information and the corresponding second key point information of each frame of video image, the method further includes:

[0024] The first key point information and the corresponding second key point information of each frame of video image are respectively scaled to a standard scale.

[0025] Optionally, in one embodiment of the present application, after determining the overall action score of the motion video according to the similarity, the method further includes:

[0026] A visual analysis diagram is generated according to the overall action score of the motion video, wherein the visual analysis diagram includes a timeline of video images in the motion video and action scores corresponding to the video images at different moments on the timeline.

[0027] Optionally, in one embodiment of the present application, after generating a visual analysis diagram according to the overall action score of the sports video, the method further includes:

[0028] Based on the visual analysis diagram, the action score corresponding to the body part is obtained. When the action score of the body part is lower than the preset threshold, the action assistance suggestion is generated according to the degree of deviation between the body part and the target body part.

[0029] Optionally, in one embodiment of the present application, the video data is input into a preset action recognition model to obtain a corresponding action category, including:

[0030] The input video data is sampled at high frame rate and low frame rate respectively, corresponding to the input fast branch and slow branch channels, to obtain different types of motion features;

[0031] Fusion of different types of action features to obtain feature value data;

[0032] Use the softmax method to calculate the feature value data and obtain the corresponding action category.

[0033] Optionally, in one embodiment of the present application, the video data is input into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video, including:

[0034] Detect the input video data and obtain all key points;

[0035] Use PAF prediction to obtain the vector field between all key points, and calculate the weights of the association degree of all key points based on the vector field;

[0036] Use the weights predicted by PAF to match all key points and obtain the optimal matching relationship, thereby obtaining the first key point information.

[0037] Input the target standard data into the human posture estimation model to obtain the second key point information of each frame of standard video image, including:

[0038] Detect the input target standard data and obtain all key points;

[0039] Use PAF prediction to obtain the vector field between all key points, and calculate the weights of the association degree of all key points based on the vector field;

[0040] Use the weights predicted by PAF to match all key points and obtain the optimal matching relationship, thereby obtaining the second key point information.

[0041] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a vision-based motion video fine analysis device, comprising:

[0042] The recognition module is used to obtain video data of the sports video and input the video data into a preset action recognition model to obtain the corresponding action category;

[0043] A matching module, used for acquiring target standard data matching the action category from pre-collected standard video data according to the action category, wherein the standard video data includes standard paradigms of different categories of actions;

[0044] A data processing module, used to input the video data into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video; and input the target standard data into the human posture estimation model to obtain the second key point information of each frame of the standard video image;

[0045] The scoring module is used to match the first key point information with the second key point information to determine the similarity between the first key point information and the second key point information, and determine the overall action score of the motion video according to the similarity.

[0046] In order to achieve the above-mentioned objectives, the third aspect embodiment of the present application proposes a non-temporary computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor, can execute a vision-based motion video fine analysis method.

[0047] The vision-based motion video fine analysis method, vision-based motion video fine analysis device and non-temporary computer-readable storage medium of the embodiments of the present application solve the problems of high cost, complex operation, poor versatility and interference with the movements of the person being evaluated in the existing methods, which makes it difficult to promote and use them on a large scale. By using ordinary equipment to obtain video data, there is almost no interference with the movement process, and at the same time, the threshold for technical application is lowered. It has the advantages of easy promotion and popularization. According to the analysis of the obtained video data, a more objective action score can be obtained, and the present application can give more fine-grained action assistance suggestions through scoring in the time dimension and the space dimension.

[0048] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0050] Figure 1 A flowchart of a method for fine-grained analysis of motion videos based on vision provided in Example 1 of the present application;

[0051] Figure 2 This is an overall block diagram of the vision-based motion video fine analysis method according to an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the structure of an action recognition model of a method for fine analysis of motion videos based on vision according to an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the structure of a human body posture estimation model according to an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the structure of an improved human posture estimation model of the vision-based motion video fine analysis method according to an embodiment of the present application;

[0055] Figure 6 This is an example diagram of a visual analysis diagram of a visual-based motion video fine analysis method according to an embodiment of the present application;

[0056] Figure 7A schematic diagram of the structure of a vision-based motion video fine analysis device provided in Example 2 of the present application. DETAILED DESCRIPTION

[0057] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0058] The traditional scoring method is mainly based on subjective human judgment, which requires the scorer to have certain experience and professionalism. The problems of human scoring mainly include strong subjectivity, low efficiency, low accuracy, and vague scoring standards. With the development of artificial intelligence, it is possible to implement an action scoring system based on video data. The system can score actions based on the obtained video data. This method is easy to promote and implement, and has good application prospects.

[0059] The following describes the vision-based motion video fine analysis method and device according to the embodiments of the present application with reference to the accompanying drawings.

[0060] Figure 1 This is a flowchart of a vision-based motion video fine analysis method provided in Example 1 of the present application.

[0061] like Figure 1 As shown, the vision-based motion video fine analysis method includes the following steps:

[0062] Step 101, obtaining video data of a sports video, and inputting the video data into a preset action recognition model to obtain a corresponding action category;

[0063] Step 102, according to the action category, obtaining target standard data matching the action category from pre-collected standard video data, wherein the standard video data includes standard paradigms of different action categories;

[0064] Step 103, inputting the video data into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video;

[0065] Step 104, inputting the target standard data into a human posture estimation model to obtain the second key point information of each frame of the standard video image;

[0066] Step 105 : Match the first key point information with the second key point information to determine the similarity between the first key point information and the second key point information, and determine the overall action score of the motion video according to the similarity.

[0067] The vision-based motion video fine analysis method of the embodiment of the present application obtains video data of the motion video and inputs the video data into a preset action recognition model to obtain the corresponding action category; according to the action category, obtains target standard data matching the action category from pre-collected standard video data, wherein the standard video data includes standard paradigms of different categories of actions; inputs the video data into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video; inputs the target standard data into the human posture estimation model to obtain the second key point information of each frame of the standard video image; matches the first key point information with the second key point information to determine the similarity between the first key point information and the second key point information, and determines the overall action score of the motion video according to the similarity. As a result, it is possible to solve the problems of existing methods such as high cost, complex operation, poor versatility, and interference with the movements of the person being evaluated, which make it difficult to promote and use them on a large scale. By using ordinary equipment to obtain video data, there is almost no interference with the movement process, and at the same time, the threshold for technical application is lowered, and it has the advantages of easy promotion and popularization. By analyzing the obtained video data, a more objective action score can be obtained, and the present application can give more fine-grained action assistance suggestions by scoring in the time and space dimensions.

[0068] Compared with the existing technology of obtaining data through sensors, this application directly uses low-threshold devices such as cameras to obtain video data, which has greater advantages at the audience level and greatly reduces the threshold for technical application. At the same time, since no hardware facilities attached to the body of the person being scored are used, there is almost no interference with the movement process. In order to ensure the final detection effect, there must be certain requirements for the consistency of the input data, such as lighting environment, shooting angle, etc. At the same time, this application combines the knowledge of experts in the field and proposes an objective, fine-grained, and highly accurate fine scoring method, which can objectively analyze the movements, and can also provide movement suggestions that are helpful in assisting movement training in related fields, further improving the skill level of personnel. This application can be applied to scenarios where movement quality assessment is required, such as dance scoring, diving movement scoring, etc., and movement analysis results and suggestions are given while scoring.

[0069] In an embodiment of the present application, a single mobile device with a video recording function can be used to shoot video and collect motion video data and standard video data, wherein the shooting environment needs to ensure a certain degree of stability and ensure sufficient lighting and an unobstructed single-person environment for collection.

[0070] Furthermore, in the embodiment of the present application, before the video data is input into the preset action recognition model, the method further includes:

[0071] Preprocess the video data, including:

[0072] Adjusting the resolution, format and encoding of the video image in the video data to match the resolution, format and encoding of the standard video image in the standard video data, wherein the video image and the standard video image may be encoded in the h.264 specification, in the mp4 format, and the resolution size may be uniformly set to 256; and / or

[0073] Normalizing the pixel values ​​of the video image in the video data, where the original value range is 0-255, and the pixel values ​​can be uniformly normalized to the range of 0-1; and / or

[0074] Filling the pixel values ​​of the video image in the video data may fill the pixels of the video image to a size of 256*256, and the filling value may be set to 0; and / or

[0075] The video images in the video data are cut into video segments according to the standard video images, so that the video images in the video data are aligned with the standard video images in time.

[0076] In the embodiment of the present application, the video data of the motion video uses the matched target standard data as a comparison benchmark.

[0077] In the present application, video data is input into a preset human posture estimation model to obtain the first key point information of each frame of video image in the motion video, wherein the first key point information is the human skeleton point data in the video data, and the skeleton point data usually includes: ankle joint, knee joint, hip joint, shoulder, neck, elbow joint, wrist joint, eye, ear, nose, etc.

[0078] In this application, the target standard data is input into the human body posture estimation model to obtain the second key point information of each frame of the standard video image, wherein the second key point information is the human body skeleton point data in the target standard data, and the skeleton point data usually includes: ankle joint, knee joint, hip joint, shoulder, neck, elbow joint, wrist joint, eye, ear, nose, etc.

[0079] Further, in the embodiment of the present application, the first key point information and the second key point information are matched to determine the similarity between the first key point information and the second key point information, and the overall action score of the motion video is determined according to the similarity, including:

[0080] The similarity between the first key point information and the corresponding second key point information of each frame of video image is calculated respectively, and the average value of the similarity of all frames of video image is calculated, and the average value is used as the overall action score of the motion video.

[0081] In the present application, the similarity calculation is performed on the first key point information of a single frame video image and the corresponding second key point information, wherein:

[0082] The similarity calculation formula is expressed as:

[0083]

[0084] Among them, OKS represents the similarity between the first key point information of a single frame video image and the corresponding second key point information. The closer OKS is to 1, the better the similarity is, and the closer it is to 0, the worse the similarity is. i represents the key point at position i, and d i represents the Euclidean distance between key points at position i, S represents the scale factor, which can be taken as the neck-hip center distance, δ i Indicates the normalization factor of the key point at position i. The normalization factor can use the standard value given in the COCO key point dataset ((0.026, nose), (0.025, eyes), (0.035, ears), (0.079, shoulders), (0.072, elbows), (0.062, wraps), (0.107, hips), (0.087, knees), (0.089, ankles), nums of keypoint indicates the number of key points,

[0085] Among them, d i The calculation formula is expressed as:

[0086]

[0087] Among them, the coordinate value of the i position of the video image is (x it ,y it ), the coordinate value of position i of the standard video image is (x is ,y is ).

[0088] The present application performs similarity calculation on a single position in the spatial dimension to obtain a characterization of the situation where the position deviates from the target standard data. By performing similarity calculation on all positions of a single-frame video image, the similarity of the single-frame video image is obtained, and the average value of the similarities of all frames of video images is calculated, and the average value is used as the overall action score of the motion video.

[0089] Furthermore, in the embodiment of the present application, before respectively calculating the similarity between the first key point information and the corresponding second key point information of each frame of the video image, the method further includes:

[0090] The first key point information and the corresponding second key point information of each frame of video image are respectively scaled to a standard scale.

[0091] The present application can perform standard scale scaling on the first key point information and the corresponding second key point information of each frame of video image according to their respective neck-hip center point distances, thereby avoiding deviations due to body shape problems in the video image and the standard video image.

[0092] Furthermore, in the embodiment of the present application, after determining the overall action score of the motion video according to the similarity, the method further includes:

[0093] A visual analysis diagram is generated according to the overall action score of the motion video, wherein the visual analysis diagram includes a timeline of video images in the motion video and action scores corresponding to the video images at different moments on the timeline.

[0094] In an embodiment of the present application, a visual analysis graph is generated based on the overall action score of the motion video. The time axis of the video image in the motion video is the horizontal axis of the visual analysis graph, and the action scores corresponding to the video images at different times on the time axis are the vertical axis, so that the action scores of the motion videos at different times can be intuitively understood.

[0095] Furthermore, in the embodiment of the present application, after generating a visual analysis diagram according to the overall action score of the sports video, the method further includes:

[0096] Based on the visual analysis diagram, the action score corresponding to the body part is obtained. When the action score of the body part is lower than the preset threshold, the action assistance suggestion is generated according to the degree of deviation between the body part and the target body part.

[0097] In an embodiment of the present application, the action score corresponding to each body part can be obtained based on the visual analysis diagram to observe whether the action score corresponding to each body part is lower than a preset threshold. The action score is the value of the similarity OKS. The value range calculated by OKS is between 0-1, and the preset threshold range is also between 0-1, which can be set to 0.8.

[0098] Furthermore, in an embodiment of the present application, the video data is input into a preset action recognition model to obtain corresponding action categories, including:

[0099] The input video data is sampled at high frame rate and low frame rate respectively, corresponding to the input fast branch and slow branch channels, to obtain different types of motion features;

[0100] Fusion of different types of action features to obtain feature value data;

[0101] Use the softmax method to calculate the feature value data and obtain the corresponding action category.

[0102] The backbone of the action recognition model of this application uses ResNet-50 to achieve a balance between effect and processing speed, and this application uses direct addition or Concat of feature maps to perform feature fusion.

[0103] The action recognition model based on Slow-fast Networks deep learning in the embodiment of the present application is used to perform action recognition tasks, is pre-trained on the Kinetic-400 dataset, and then collects motion video data for training, wherein the collection of motion video data for training specifically includes: firstly giving category labels to the collected motion video data, and then using the stochastic gradient descent method to perform fine-tune training based on the pre-trained model, and training for multiple epochs until the recognition accuracy cannot be significantly improved.

[0104] Furthermore, in an embodiment of the present application, the video data is input into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video, including:

[0105] Detect the input video data and obtain all key points;

[0106] Use PAF prediction to obtain the vector field between all key points, and calculate the weights of the association degree of all key points based on the vector field;

[0107] Use the weights predicted by PAF to match all key points and obtain the optimal matching relationship, thereby obtaining the first key point information.

[0108] Input the target standard data into the human posture estimation model to obtain the second key point information of each frame of standard video image, including:

[0109] Detect the input target standard data and obtain all key points;

[0110] Use PAF prediction to obtain the vector field between all key points, and calculate the weights of the association degree of all key points based on the vector field;

[0111] Use the weights predicted by PAF to match all key points and obtain the optimal matching relationship, thereby obtaining the second key point information.

[0112] This application first detects all key points, and then uses the weights predicted by PAF (part affinity fields) to match all key points to find the optimal matching relationship. The matching method is: PAF predicts the vector field between two key points. The weight of the association between the two key points can be calculated through the vector field. Then, the Hungarian algorithm with the maximum weight can be used to obtain the matching relationship between all key points.

[0113] Figure 2 This is an overall block diagram of the vision-based motion video fine analysis method according to an embodiment of the present application.

[0114] like Figure 2 As shown in the figure, the vision-based motion video fine analysis method can be divided into three parts: data processing, data analysis, and data visualization. Among them, data processing includes collecting motion video data and preprocessing the video data; data analysis includes analyzing the video data using a motion recognition model and a human posture estimation model to obtain a motion score; and generating a visual analysis diagram based on the motion score.

[0115] Figure 3 Schematic diagram of the structure of the action recognition model of the vision-based motion video fine analysis method in an embodiment of the present application.

[0116] like Figure 3 As shown in the figure, the upper branch is a low frame rate sampling branch that extracts spatial semantic information, and the lower branch is a high frame rate sampling branch that extracts more fine-grained temporal information. The feature dimension differences between the branches are as follows Figure 3 As shown in the figure, feature fusion is required between each stage, and feature fusion can be performed by directly adding or concat feature maps.

[0117] Figure 4 Schematic diagram of the structure of the human body posture estimation model according to an embodiment of the present application.

[0118] like Figure 4 As shown in the figure, the human posture estimation model uses VGG-19 as the backbone, and the feature map generated is F. It is then trained in multiple steps, with two independent branches in each step. One branch is the heatmap branch, which is used to predict key points, and the other branch is the PAF branch, which is used to predict the PAF map for matching between key points. Each branch is trained separately. The feature map obtained by the two branches will be combined with F as the input of F in the next step.

[0119] Figure 5 Schematic diagram of the structure of the improved human posture estimation model of the vision-based motion video fine analysis method according to the embodiment of the present application.

[0120] like Figure 5 As shown in the figure, in order to further reduce the computational consumption, the human pose estimation model is lightweighted and improved. Backbone is replaced from VGG-19 to MobileNet v1, and the multiple refinement stages in the original human pose estimation model are streamlined to only the initial stage and one refinement stage. At the same time, the dual-branch structure is further weighted.

[0121] Figure 6This is an example diagram of a visual analysis diagram of the vision-based motion video fine analysis method according to an embodiment of the present application.

[0122] like Figure 6 As shown, an example diagram of a visualization analysis diagram corresponding to each body part is shown. In the diagram, the timeline of the motion video is the horizontal axis of the visualization analysis diagram, and the action scores corresponding to the video images at different times on the timeline are the vertical axis. The action scores corresponding to each body part are obtained based on the visualization analysis diagram, and whether the action scores corresponding to each body part are lower than a preset threshold is observed. When the action score of a body part is lower than the preset threshold, action assistance suggestions are generated according to the degree of deviation between the body part and the target body part.

[0123] Figure 7 A schematic diagram of the structure of a vision-based motion video fine analysis device provided in Example 2 of the present application.

[0124] like Figure 7 As shown, the vision-based motion video fine analysis device includes:

[0125] The recognition module 10 is used to obtain video data of the sports video and input the video data into a preset action recognition model to obtain a corresponding action category;

[0126] A matching module 20 is used to obtain target standard data matching the action category from pre-collected standard video data according to the action category, wherein the standard video data includes standard paradigms of different categories of actions;

[0127] The data processing module 30 is used to input the video data into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video; and input the target standard data into the human posture estimation model to obtain the second key point information of each frame of the standard video image;

[0128] The scoring module 40 is used to match the first key point information with the second key point information to determine the similarity between the first key point information and the second key point information, and determine the overall action score of the motion video according to the similarity.

[0129] The vision-based motion video fine analysis device of the embodiment of the present application includes a recognition module, which is used to obtain video data of the motion video and input the video data into a preset action recognition model to obtain the corresponding action category; a matching module, which is used to obtain target standard data matching the action category from pre-collected standard video data according to the action category, wherein the standard video data includes standard paradigms of different categories of actions; a data processing module, which is used to input the video data into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video; and input the target standard data into the human posture estimation model to obtain the second key point information of each frame of the standard video image; a scoring module, which is used to match the first key point information with the second key point information to determine the similarity between the first key point information and the second key point information, and determine the overall action score of the motion video according to the similarity. As a result, it is possible to solve the problems of existing methods such as high cost, complex operation, poor versatility, and interference with the movements of the person being evaluated, which make it difficult to promote and use them on a large scale. By using ordinary equipment to obtain video data, there is almost no interference with the movement process, and at the same time, the threshold for technical application is lowered, and it has the advantages of easy promotion and popularization. By analyzing the obtained video data, a more objective action score can be obtained, and the present application can give more fine-grained action assistance suggestions by scoring in the time and space dimensions.

[0130] In order to implement the above-mentioned embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the vision-based motion video fine analysis method of the above-mentioned embodiment is implemented.

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0132] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0133] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0134] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0135] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0136] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0138] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A vision-based motion video fine analysis method, characterized in that: include: Acquire video data of a sports video, and input the video data into a preset action recognition model to obtain a corresponding action category; According to the action category, acquiring target standard data matching the action category from pre-collected standard video data, wherein the standard video data includes standard paradigms of different categories of actions; Inputting the video data into a preset human posture estimation model to obtain first key point information of each frame of video image in the motion video; Inputting the target standard data into the human body posture estimation model to obtain the second key point information of each frame of standard video image; Matching the first key point information with the second key point information to determine the similarity between the first key point information and the second key point information, and determining the overall action score of the motion video according to the similarity; The first key point information and the second key point information are matched to determine the similarity between the first key point information and the second key point information, and the overall action score of the motion video is determined according to the similarity, including: The similarity between the first key point information and the corresponding second key point information of each frame of video image is calculated respectively, and the average value of the similarity of all frames of video images is calculated, and the average value is used as the overall action score of the motion video, wherein the similarity calculation adopts the OKS indicator, which is expressed as: Among them, d i represents the Euclidean distance between key points at position i, S represents the scale factor, S is the neck-hip center distance, δ i Represents the key point normalization factor at position i; Before respectively calculating the similarity between the first key point information and the corresponding second key point information of each frame of video image, the method further includes: The first key point information and the corresponding second key point information of each frame of video image are respectively scaled according to the respective neck-hip center point distances; The human body posture estimation model uses MobileNet v1 as a backbone, the multiple refinement stages in the human body posture estimation model include an initial stage and a refinement stage, and the dual-branch structure of the human body posture estimation model shares weights; The step of inputting the video data into a preset human posture estimation model to obtain first key point information of each frame of video image in the motion video includes: Detect the input video data and obtain all key points; Use PAF prediction to obtain the vector field between all key points, and calculate the weights of the association degree of all key points based on the vector field; The weights predicted by PAF are used to match all key points to obtain the optimal matching relationship, thereby obtaining the first key point information. Specifically, the Hungarian algorithm with the maximum weight is used to obtain the matching relationship between all key points.

2. The method according to claim 1, characterized in that Before inputting the video data into a preset action recognition model, the method further includes: Preprocessing the video data, wherein the preprocessing includes: adjusting the resolution, format and encoding of the video image in the video data to match the resolution, format and encoding of the standard video image in the standard video data; and / or Normalizing the pixel values ​​of the video image in the video data; and / or Filling the pixel values ​​of the video image in the video data; and / or The video image in the video data is subjected to a video segment cutting process according to the standard video image, so that the video image in the video data is aligned with the standard video image in time.

3. The method according to claim 1, characterized in that After determining the overall action score of the sports video according to the similarity, the method further includes: A visual analysis diagram is generated according to the overall action score of the motion video, wherein the visual analysis diagram includes a timeline of video images in the motion video and action scores corresponding to the video images at different moments on the timeline.

4. The method according to claim 3, characterized in that After generating a visual analysis diagram according to the overall action score of the sports video, the method further includes: Based on the visual analysis diagram, the action score corresponding to the body part is obtained, and when the action score of the body part is lower than a preset threshold, an action assistance suggestion is generated according to the degree of deviation between the body part and the target body part.

5. The method according to claim 1, characterized in that The step of inputting the video data into a preset action recognition model to obtain a corresponding action category includes: The input video data is sampled at high frame rate and low frame rate respectively, corresponding to the input fast branch and slow branch channels, to obtain different types of motion features; Fusion of different types of action features to obtain feature value data; Use the softmax method to calculate the feature value data and obtain the corresponding action category.

6. The method according to claim 1, characterized in that The target standard data is input into the human body posture estimation model to obtain the second key point information of each frame of the standard video image, including: Detect the input target standard data and obtain all key points; Use PAF prediction to obtain the vector field between all key points, and calculate the weights of the association degree of all key points based on the vector field; Use the weights predicted by PAF to match all key points and obtain the optimal matching relationship, thereby obtaining the second key point information.

7. A vision-based motion video fine analysis device, characterized in that: include: A recognition module, used to obtain video data of a sports video and input the video data into a preset action recognition model to obtain a corresponding action category; A matching module, configured to obtain target standard data matching the action category from pre-collected standard video data according to the action category, wherein the standard video data includes standard paradigms of different categories of actions; A data processing module, used for inputting the video data into a preset human posture estimation model to obtain the first key point information of each frame of the video image in the motion video; and Inputting the target standard data into the human body posture estimation model to obtain the second key point information of each frame of standard video image; A scoring module, used for matching the first key point information with the second key point information to determine the similarity between the first key point information and the second key point information, and determining the overall action score of the motion video according to the similarity; The first key point information and the second key point information are matched to determine the similarity between the first key point information and the second key point information, and the overall action score of the motion video is determined according to the similarity, including: The similarity between the first key point information and the corresponding second key point information of each frame of video image is calculated respectively, and the average value of the similarity of all frames of video images is calculated, and the average value is used as the overall action score of the motion video, wherein the similarity calculation adopts the OKS indicator, which is expressed as: Among them, d i represents the Euclidean distance between key points at position i, S represents the scale factor, S is the neck-hip center distance, δ i Represents the key point normalization factor at position i; Before respectively calculating the similarity between the first key point information and the corresponding second key point information of each frame of video image, the method further includes: The first key point information and the corresponding second key point information of each frame of video image are respectively scaled according to the respective neck-hip center point distances; The human body posture estimation model uses MobileNet v1 as a backbone, the multiple refinement stages in the human body posture estimation model include an initial stage and a refinement stage, and the dual-branch structure of the human body posture estimation model shares weights; The step of inputting the video data into a preset human posture estimation model to obtain first key point information of each frame of video image in the motion video includes: Detect the input video data and obtain all key points; Use PAF prediction to obtain the vector field between all key points, and calculate the weights of the association degree of all key points based on the vector field; The weights predicted by PAF are used to match all key points to obtain the optimal matching relationship, thereby obtaining the first key point information. Specifically, the Hungarian algorithm with the maximum weight is used to obtain the matching relationship between all key points.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.