Motion video analysis method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a motion video analysis method and device, electronic equipment, a computer storage medium and a computer program product, which are applied to the technical field of image processing, and the method comprises the following steps: obtaining a motion video of a target object; the motion video comprises a plurality of sub-videos; performing human body key point detection on each frame of human body image in each sub-video to obtain human body key points of each frame of human body image and confidence of the human body key points; determining the coordinates of the human body key points of each frame of human body image under the condition that the confidence is determined to meet the preset condition, and correcting the coordinates of the human body key points of the current frame of human body image according to the coordinates of the human body key points of the plurality of frames of human body images before and after to obtain the corrected coordinates; and determining a key frame image of the motion video according to the corrected coordinate corresponding to each sub-video, so as to perform motion evaluation on the target object through the key frame image. Therefore, the accuracy of the evaluation result can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, including but not limited to a method, device, electronic device, computer storage medium, and computer program product for analyzing motion videos. Background Art

[0002] Currently, with the reduction of video acquisition and storage costs and the rapid development of computer vision technology, the form of capturing motion videos with cameras for evaluation and analysis has gradually become the mainstream in various school physical fitness tests and physical education teaching.

[0003] However, for some sports events where a single moving object moves quickly, such as standing long jump, in the actual shooting scenario, due to factors such as the jitter of acquisition devices such as cameras, the rapid movement, violent shaking, and self-occlusion of the moving object, it is easy to cause the aliasing of different spatial position information, resulting in the blurring and distortion of video frames, seriously losing the action detail information of the video frames. If the detection results of the video frames are directly evaluated and analyzed, the accuracy of the evaluation results will be low. Summary of the Invention

[0004] Based on the problems in the related art, the embodiments of the present application provide a method, device, electronic device, computer storage medium, and computer program product for analyzing motion videos.

[0005] The technical solution of the embodiments of the present application is realized as follows:

[0006] The embodiments of the present application provide a method for analyzing motion videos, including:

[0007] Obtain the motion video of the target object; the motion video includes multiple sub-videos, and each sub-video includes the current frame human body image and multiple adjacent front and rear frame human body images;

[0008] Perform human key point detection on each frame of human body image in each sub-video to obtain the human key points of each frame of human body image and the confidence of the human key points;

[0009] When it is determined that the confidence meets the preset conditions, determine the coordinates of the human key points of each frame of human body image, and correct the coordinates of the human key points of the current frame of human body image according to the coordinates of the human key points of the multiple adjacent front and rear frame human body images to obtain the corrected coordinates;

[0010] Determine the key frame image of the motion video according to the corrected coordinates corresponding to each sub-video, so as to perform motion evaluation on the target object through the key frame image.

[0011] The embodiments of the present application provide a device for analyzing motion videos, including:

[0012] An acquisition module, configured to acquire a motion video of a target object; the motion video includes a plurality of sub-videos, and each sub-video includes a current-frame human body image and multiple adjacent front and rear human body images;

[0013] A detection module, configured to perform human key point detection on each frame of human body image in each sub-video to obtain the human key points of each frame of human body image and the confidence of the human key points;

[0014] A correction module, configured to determine the coordinates of the human key points of each frame of human body image when it is determined that the confidence meets a preset condition, and correct the coordinates of the human key points of the current-frame human body image according to the coordinates of the human key points of the multiple adjacent front and rear human body images to obtain corrected coordinates;

[0015] An evaluation module, configured to determine a key frame image of the motion video according to the corrected coordinates corresponding to each sub-video, so as to perform motion evaluation on the target object through the key frame image.

[0016] An embodiment of the present application provides an electronic device, the device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the motion video analysis method provided by the foregoing one or more technical solutions.

[0017] An embodiment of the present application provides a computer storage medium, the computer storage medium stores a computer program; after the computer program is executed, it can implement the motion video analysis method provided by the foregoing one or more technical solutions.

[0018] An embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the motion video analysis method provided by the foregoing one or more technical solutions.

[0019] The motion video analysis method, device, electronic device, computer storage medium, and computer program product provided by the embodiments of the present application obtain a motion video of a target object; the motion video includes a plurality of sub-videos, and each sub-video includes a current-frame human body image and multiple adjacent front and rear human body images; perform human key point detection on each frame of human body image in each sub-video to obtain the human key points of each frame of human body image and the confidence of the human key points; when it is determined that the confidence meets the preset conditions, determine the coordinates of the human key points of each frame of human body image, and correct the coordinates of the human key points of the current-frame human body image according to the coordinates of the human key points of the multiple front and rear human body images to obtain the corrected coordinates; determine the key frame image of the motion video according to the corrected coordinates corresponding to each sub-video, so as to evaluate the motion of the target object through the key frame image.

[0020] It can be seen that in the embodiments of the present application, after obtaining the human key points of each frame of human body image in each sub-video, the human key point information of the current frame is corrected according to the human key point information of multiple front and rear frames to eliminate the influence of factors such as motion blur or occlusion on the human key points. On this basis, the key frame image is determined, which can ensure the accuracy of the key frame image. Since the key frame image contains the key actions and related details in the motion process of the target object, evaluating the motion of the target object according to the key frame image can improve the accuracy of the evaluation result while ensuring the evaluation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic flowchart of a motion video analysis method provided by the embodiments of the present application;

[0022] Figure 2 is a schematic diagram of one frame of human body image during a standing long jump provided by the embodiments of the present application;

[0023] Figure 3 is a schematic flowchart of correcting the coordinates of human key points provided by the embodiments of the present application;

[0024] Figure 4 is a schematic flowchart of another motion video analysis method provided by the embodiments of the present application;

[0025] Figure 5 is a schematic diagram of the composition structure of a motion video analysis device provided by the embodiments of the present application;

[0026] Figure 6 is a schematic diagram of the composition structure of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] To make the objectives, technical solutions and advantages of this application clearer, the following will further describe this application in detail with reference to the accompanying drawings. The described embodiments should not be regarded as limitations of this application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0028] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the technical field to which the embodiments of this application belong. The terms used in the embodiments of this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] In some embodiments of this application, the motion video analysis method can be implemented according to a processor in a motion video analysis device. The above-mentioned processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0030] Figure 1 is a schematic flowchart of a motion video analysis method provided by an embodiment of this application. As Figure 1 shown, the process may include the following steps:

[0031] Step 100: Obtain a motion video of a target object.

[0032] In the embodiments of this application, the motion video analysis method can be applied to scenarios for evaluating motion states. Here, the type of scenario is not specifically limited. For example, it can be a sports examination scenario, a sports training scenario, etc.; the type of motion to be evaluated is also not specifically limited. For example, it can be a highly explosive motion such as a standing long jump, a triple jump, or a high jump.

[0033] Here, the target object refers to the object for which motion assessment is required; exemplarily, in the case where the motion video analysis method is applied to a sports examination scenario, the target object can be a student taking the examination.

[0034] Exemplarily, there is no limitation on the method for obtaining the motion video. For example, it can be collected by a video capture device such as a camera or a camcorder and sent to the motion video analysis device; or the motion video analysis device can obtain a pre-stored motion video from other storage devices.

[0035] In the embodiments of the present application, the motion video may include multiple sub-videos, and each sub-video includes a current frame human body image and multiple adjacent front and rear human body images. Exemplarily, after obtaining the motion video of the target object, a sliding time window can be constructed, and the motion video can be divided according to the sliding time window to obtain multiple sub-videos; each sub-video may include multiple consecutive human body images.

[0036] Exemplarily, the sliding time window refers to a data statistics method that can frame a time series according to a specified time length to calculate the statistical metrics within the frame. It is equivalent to a slider with a specified time length sliding on a scale, and the data within the slider can be fed back every time a time point is slid; for example, the sliding time window can be a window with a time length of 5, or a window with a time length of 7 frames; here, if the time length of the sliding time window is 5, it means that the sliding time window slides in units of 5 image frames; it should be noted that the size of the sliding time window is specifically determined by custom according to the actual application scenario, and the embodiments of the present application do not make specific limitations on this.

[0037] Exemplarily, assume that the motion video includes 6 consecutive human body images, namely human body image 1 to human body image 6. If the time length of the sliding time window is 5, then according to this sliding time window, the motion video can be divided into two sub-videos: human body image 1 to human body image 5 and human body image 2 to human body image 6; at this time, each sub-video includes 5 consecutive human body images.

[0038] In the embodiments of the present application, each sub-video may include a current frame human body image and multiple adjacent front and rear human body images; exemplarily, the current frame human body image can be represented as (I, t), where I represents the current frame human body image and t represents the time point corresponding to the current frame human body image. Each sub-video may include the current frame human body image (I, t) and the front and rear L frames of human body images adjacent in time, and can form a sub-video with a total of 2L + 1 frames of human body images; where L is a value greater than zero, and according to experience, it can be taken as 2.

[0039] Step 101: Perform human key point detection on each frame of human body image in each sub-video to obtain the human key points and the confidence levels of the human key points for each frame of human body image.

[0040] Exemplarily, human key points are points for positioning and identifying key parts (usually major human joints) in the movement process of the target object. The number of human key points in each frame of human body image is the same and can be multiple. Here, the types of human key points in each frame of human body image are not limited. For example, human key points can include but are not limited to left and right ankle key points, left and right knee key points, left and right hip key points, left and right shoulder key points, left and right elbow key points, and left and right wrist key points, etc.

[0041] In the embodiments of the present application, after obtaining multiple sub-videos after dividing the motion video, a human key point detection algorithm can be used to perform human key point detection on each frame of human body image in each sub-video to obtain the corresponding key point detection results. The key point detection results include the human key points and the confidence levels of the human key points of the target object in each frame of human body image. Here, the type of the human key point detection algorithm is not specifically limited. For example, the AlphaPose algorithm, the OpenPose algorithm, or the MobilePose algorithm, etc. can be used.

[0042] It should be noted that in the process of performing human key point detection on each frame of human body image using the human key point detection algorithm, heat maps corresponding to each frame of human body image will be obtained first. By post-processing the heat maps corresponding to each frame of human body image, the coordinates of the human key points and the confidence levels of the human key points of the corresponding target object in each frame of human body image can be obtained.

[0043] Exemplarily, the confidence level represents the probability that the human key point is a real human key point. Specifically, if the confidence level of a certain human key point is higher, it means that the accuracy of this human key point is higher. On the contrary, if the confidence level of a certain human key point is lower, it means that the accuracy of this human key point is lower. It should be noted that the factors causing the low confidence level of human key points can include but are not limited to occlusion, motion blur, etc. The embodiments of the present application do not limit this.

[0044] Step 102: When it is determined that the confidence level meets the preset conditions, determine the coordinates of the human key points of each frame of human body image, and correct the coordinates of the human key points of the current frame of human body image according to the coordinates of the human key points of multiple frames of human body image before and after, to obtain the corrected coordinates.

[0045] Here, the confidence level represents the confidence level corresponding to each sub-video. In the embodiments of the present application, before determining that the confidence level corresponding to each sub-video meets the preset conditions, a corresponding confidence level threshold can be set. Furthermore, based on this confidence level threshold, it is determined whether the confidence level meets the preset conditions. Here, the number and values of the confidence level thresholds can be set according to actual needs and are not specifically limited herein.

[0046] Exemplarily, according to the above content, there can be multiple factors causing low confidence levels of human body key points. If these factors are considered comprehensively without distinction, only one confidence level threshold needs to be set. For example, the confidence level threshold can be set to 0.7 or 0.8, etc. If these factors are distinguished, multiple confidence level thresholds can be set correspondingly. Since occlusion and motion blur are the main factors causing low confidence levels, therefore, in the case of considering these two factors of occlusion and motion blur, two confidence level thresholds can be set correspondingly. For example, the confidence level threshold corresponding to the occlusion factor can be set to 0.4 or 0.5, etc., and the confidence level threshold corresponding to the motion blur factor can be set to 0.7 or 0.8, etc.

[0047] It should be understood that regardless of whether the number of confidence level thresholds is one or multiple, the implementation manner of determining whether the confidence level meets the preset conditions based on the confidence level threshold is similar. For ease of description, the following takes one confidence level threshold as an example for illustration.

[0048] Exemplarily, for the implementation manner of determining that the confidence level meets the preset conditions, it can be: for each sub-video, the confidence levels of the same human body key point of each frame of human body image are respectively compared with the preset confidence level threshold to obtain a comparison result. If, according to the comparison result, it is determined that the confidence level of the first human body key point of the current frame of human body image is less than or equal to the confidence level threshold, and the number of second human body key points with a confidence level greater than the confidence level threshold in multiple frames of human body images before and after is within the preset range, then it is determined that the confidence level meets the preset conditions.

[0049] Here, the first key point and the second key point are the same human body key point of each frame of human body image. Exemplarily, the first key point can be regarded as a low-confidence point, and the second key point can be regarded as a high-confidence point.

[0050] Exemplarily, the number of human key points in each frame of the human body image in each sub-video is the same and corresponds one by one; that is, each human key point in the current frame of the human body image in each sub-video has a corresponding human key point in multiple frames of human body images before and after; among them, each human key point in each frame of the human body image has a corresponding confidence level. For example, a certain sub-video includes 3 frames of human body images, namely human body image 1 to human body image 3. Assuming that human body image 1 includes human key point A, human key point B, and human key point C, then human body image 2 and human body image 3 also include human key point A, human key point B, and human key point C, and human key point A, human key point B, and human key point C in each frame of the human body image all have corresponding confidence levels.

[0051] Exemplarily, for each sub-video, after obtaining the confidence level of each human key point in each frame of the human body image, the confidence levels of the same human key point in each frame of the human body image can be respectively compared with a preset confidence level threshold to obtain a comparison result; if it is determined according to the comparison result that the confidence level of the first human key point in the current frame of the human body image is less than or equal to the confidence level threshold, and the number of second human key points with a confidence level greater than the confidence level threshold in multiple frames of human body images before and after is within a preset range (equivalent to determining that the low-confidence points in the current frame of the human body image are high-confidence points in multiple frames of human body images before and after, and the number of high-confidence points is within a preset range), then it is determined that the confidence level of this human key point meets the preset conditions; if the number of second human key points is not within the preset range, then it is determined that the confidence level of this human key point does not meet the preset conditions, and so on, and the above processing is performed on other human key points in the current frame of the human body image.

[0052] Here, the value of the preset range can be set according to the actual situation, and an exemplary description will be given below in combination with formula (1).

[0053]

[0054] Among them, n represents the number of time points, that is, the number of frames of the human body image in each sub-video, k represents a low-confidence point in the current frame of the human body image, S k (t) represents that the low-confidence point k is a high-confidence point at the time point t, Denote the number of high-confidence points that the low-confidence point k is in the time points from 1 to n, that is, the number of high-confidence points that the low-confidence point k is in multiple frames of human body images before and after. Combining with formula (1), it can be known that the value range of the preset range is [n / a, n / c], that is, if it is determined that the number of high-confidence points of each low-confidence point in the current frame of human body image in multiple frames of human body images before and after is greater than n / a and less than n / c, it is determined that the confidence meets the preset conditions; a is used to control the lower limit of the high-confidence points of each frame of human body image in each sub-video, and c is used to control the upper limit of the high-confidence points of each frame of human body image in each sub-video; here, the values of a and c can be set according to the actual situation, and the embodiments of the present application do not limit this.

[0055] Further, when it is determined that the number of the second human body key points is within the preset range, it is also possible to determine whether the number of each frame of human body images is greater than a set value, and when it is determined to be yes, it is determined that the confidence of the human body key point meets the preset conditions.

[0056] Exemplarily, for each frame of human body image of each sub-video, if it is determined that the number of the second human body key points is within the preset range, it is also necessary to determine whether the number of each frame of human body images is greater than the set value b, that is, to determine whether b < n is satisfied. b is used to control the length of the consecutive frames of human body images in each sub-video, and can be set to 4 according to experience.

[0057] It should be noted that there is no specific limitation on the execution order of the above two judgment conditions; that is, it is also possible to first determine whether the number of each frame of human body images is greater than the set value, and then determine whether the number of the second human body key points is within the preset range; for each sub-video, if both of the above conditions are satisfied, it is determined that the confidence of the human body key point meets the preset conditions.

[0058] In some embodiments, when it is determined that the confidence meets the preset conditions, determining the coordinates of the human body key points of each frame of human body image may include: when it is determined that the confidence meets the preset conditions, processing the human body key points of each frame of human body image through a bidirectional GRU network to obtain the coordinates of the human body key points of each frame of human body image.

[0059] In the embodiments of the present application, if it is determined according to the above steps that the confidence of a certain human body key point in the current frame of human body image meets the preset conditions, the bidirectional GRU network is started, and the bidirectional GRU network is used to process the same human body key point of each frame of human body image to obtain the coordinates of the same human body key point of each frame of human body image. On the contrary, if it is determined according to the above steps that the confidence of a certain human body key point in the current frame of human body image does not meet the preset conditions, the bidirectional GRU network is not started.

[0060] Here, the bidirectional GRU network is a recurrent neural network with two directions. This network combines the outputs of the forward GRU layer and the backward GRU layer, and can better capture the context information in time series data. Exemplarily, since each frame of the human body image corresponds to a time point, the same human key points in each frame of the human body image in each sub-video can be regarded as time series data. By analyzing the context information in the time series data through the Bidirectional Gated Recurrent Unit network, the coordinates of the human key points in each frame of the human body image can be obtained.

[0061] It can be seen that in the embodiments of the present application, whether the confidence meets the preset conditions is used as the opening strategy of the bidirectional GRU network, and only when it is ensured that the confidence meets the preset conditions, the bidirectional GRU network will be opened for processing. In this way, the processing efficiency can be improved while ensuring the data accuracy.

[0062] Exemplarily, after obtaining the coordinates of the human key points in each frame of the human body image, according to the coordinates of the human key points in multiple frames of the human body image before and after, the coordinates of the human key points in the current frame of the human body image can be corrected, which may include: obtaining the coordinate coefficients of the human key points in each frame of the human body image; determining the first product of the coordinates of the human key points in multiple frames of the human body image before and after and the coordinate coefficients, and the second product of the coordinates of the human key points in the current frame of the human body image and the coordinate coefficients; correcting the coordinates of the human key points in the current frame of the human body image according to the first product and the second product.

[0063] In the embodiments of the present application, for each sub-video, the coordinate coefficients of each human key point in each frame of the human body image can be preset; among them, the coordinate coefficients of each human key point in each frame of the human body image can be the same or different; for the same human key point, the coordinate coefficient corresponding to the current frame of the human body image should be greater than that of multiple frames before and after, and the sum of the coordinate coefficients of the same human key point in each frame of the human body image is 1; for example, a certain sub-video includes 5 frames of human body images, namely human body image 1 to human body image 5, and human body image 3 is the current frame of the human body image. The coordinate coefficients of the human key points in human body image 1 to human body image 5 can be set to 0.1, 0.2, 0.4, 0.2, 0.1 respectively.

[0064] Exemplarily, the coordinate coefficients of the human key points in each frame of the human body image can be obtained, then the first product of the coordinates of the human key points in multiple frames of the human body image before and after and the coordinate coefficients, and the second product of the coordinates of the human key points in the current frame of the human body image and the coordinate coefficients can be determined. By accumulating these two products, the corrected coordinates can be obtained.

[0065] It should be noted that the human key point for correcting the human body image of the current frame is the first human key point mentioned above; that is, it is only necessary to correct the coordinates of each low-confidence point in the human body image of the current frame that is less than or equal to the confidence threshold; the following is an exemplary explanation in conjunction with formula (2).

[0066] V' k (m)=0.1*V k (m-2)+0.2*V k (m-1)+0.4*V k (m)+0.2*V k (m+1)+0.1*V k (m+2) (2)

[0067] Among them, m represents the time point corresponding to the current frame human image, V k (m) is the coordinate of low confidence point k in the human body image of the current frame, V' k (m) is the corrected coordinate of the low confidence point k in the current frame of the human body image, and 0.1, 0.2, 0.4, 0.2, and 0.1 are the coordinate coefficients of the low confidence point k in different frames of the human body image; here, 0.1*V k (m-2), 0.2*V k (m-1), 0.2*V k (m+1), 0.1*V k (m+2) corresponds to the first product above, 0.4*V k (m) corresponds to the second product mentioned above.

[0068] Step 103: Determine the key frame image of the motion video according to the corrected coordinates corresponding to each sub-video, so as to perform motion evaluation on the target object through the key frame image.

[0069] Exemplarily, for each sub-video, after correcting the coordinates of the low-confidence points of the human body image in the current frame, key frame images can be extracted from the motion video to obtain key frame images; here, the number of key frame images is usually multiple, which is related to the type of motion, and the embodiments of the present application are not limited to this. For example, in the case where the motion type is standing long jump, the key frame images may include a backswing frame, a take-off frame, a highest air frame, and a landing frame.

[0070] In some embodiments, determining the key-frame images of a motion video according to the corrected coordinates corresponding to each sub-video may include: obtaining the time points of each human body image in the motion video; determining the coordinate time series of the same human body key points in the motion video according to the corrected coordinates corresponding to each sub-video and the time points of each human body image in the motion video; dividing the motion video according to the coordinate time series to obtain a plurality of image intervals; and obtaining the key-frame images of the motion video from the plurality of image intervals.

[0071] In the embodiments of the present application, each human body image in the motion video has a corresponding time point. After obtaining the corrected coordinates corresponding to each sub-video, the coordinate time series of the same human body key points in the motion video can be determined in chronological order by combining the time points of each human body image; wherein, the coordinate time series represents the coordinate change of the same human body key point over time.

[0072] Exemplarily, it is assumed that the motion video includes 5 human body images, and each human body image includes 3 identical human body key points. Then, 3 coordinate time series can be correspondingly obtained, and each coordinate time series includes the coordinates of the same human body key point at 5 different time points.

[0073] Further, after obtaining the coordinate time series of the same human body key points in the motion video, the motion process of the target object in the motion video can be divided according to the coordinate time series of one of the human body key points to obtain a plurality of image intervals; the key-frame images of the motion video can be obtained by extracting key frames from each of the plurality of image intervals; wherein, each image interval includes multiple consecutive human body images. Taking the standing long jump as an example, the process of determining the key-frame images will be described exemplarily below.

[0074] For example, first, all the key points of the human body that are focused on during the entire standing long jump process are organized into a coordinate time sequence in the order of time points. At this time, the same key point of different frames of human body images can correspond to a coordinate time sequence; then, according to the coordinate time sequence of one of the key points of the human body of the target object, for example, the coordinate change of the key point x of the foot of the target object, the entire standing long jump process is divided into three image intervals before the start of the jump, during the jump, and after the feet land. These three image intervals are divided by two division points, where the highest frame in the air is the division point between the jump process and after the feet land, and the frame corresponding to the highest point of the center of gravity of the target object during the jump process is taken as the highest frame in the air. In addition, before the start of the take-off, there is a backswing frame, that is, the frame when the angle between the arms and the vertical line of the body reaches the maximum, and the last maximum value frame of the angle before the first division point is obtained as the backswing frame; the jump process includes the take-off frame and the highest flying frame, the take-off frame is the frame when the toes are about to cross or are crossing the take-off line, the image interval before the start of the take-off is intercepted by the arms crossing the vertical line of the body and the flying height, and the extraction range is gradually narrowed. Finally, the body stretch angle is represented by the angle composed of ankles, knees, and hips, and the angle composed of knees, hips, and shoulders as the abstraction of the body stretch, and the corresponding frame of their maximum value is determined as the take-off frame; after the feet land, it includes the landing frame, that is, the frame with the lowest body center of gravity and the maximum body curling state, which can be determined by the height value of the hip key point. Based on the above steps, the four key frame images of the standing long jump process can be determined. These four key frame images can reflect the action state of each key moment of the standing long jump in detail, providing a basis for subsequent action specification recognition and analysis and judgment.

[0075] It can be seen that the embodiment of the present application adopts a stage division method to confirm the image interval to which each key frame belongs, which can effectively reduce the confusion in key frame image capture during the entire movement process and provide a basis for the evaluation and analysis of subsequent movement actions.

[0076] In some embodiments, performing motion evaluation on a target object through a key frame image may include: obtaining a standard action corresponding to the key frame image from a standard action library; performing a similarity analysis on the motion action of the key frame image and the standard action to obtain a similarity score; and performing motion evaluation on the target object based on the similarity score.

[0077] In an embodiment of the present application, a standard action library can be established in advance; the standard action library can include multiple standard actions and related parameters, and the standard actions can be established according to the standard action key frames of different age groups, different genders and different height-to-weight ratios.

[0078] Exemplarily, after obtaining the key-frame image, the standard action corresponding to the key-frame image can be obtained from the standard action library according to the age group, gender, and height-weight ratio to which the target object in the key-frame image belongs; then, a similarity analysis is performed on the motion action of the key-frame image and the standard action to obtain a similarity score.

[0079] It can be understood that in the embodiment of the present application, the establishment of the standard action library does not need to take the entire motion process as a unit, and can be constructed only by storing each key-frame image and its parameters in the motion process. Taking the standing long jump as an example, only 4 key-frame images of the standing long jump and their parameters need to be stored in the standard action library, effectively saving storage space and improving the efficiency of action comparison.

[0080] Here, the cosine similarity algorithm can be used to perform a similarity analysis on the motion action of the key-frame image and the standard action; specifically, an independent action analysis is performed on the motion action of the target object in each key-frame image; assuming that the vector formed by the human key point i and the human key point j of the target object in the key-frame image is {KP ij}, and the vector formed by the human key point i and the human key point j in the standard action corresponding to the target object in the standard action library is {GP ij}, if the cosine similarity algorithm is used to determine the similarity between these two vectors, the similarity score Score ij of the corresponding actions of the human key point i and the human key point j in the key-frame image can be obtained, as shown in formula (3):

[0081]

[0082] Exemplarily, assuming that the number of actions to be compared in the key-frame image is m, the similarity score corresponding to the key-frame image can be expressed as

[0083] In some embodiments, the above method may further include: after determining the key-frame image of the motion video, determining whether the key-frame image is a motion-blurred image, and if it is determined to be so, performing a de-blurring process on the key-frame image to obtain a de-blurred key-frame image; correspondingly, based on the similarity score, performing a motion evaluation on the target object may include: obtaining a human rating of the de-blurred key-frame image; and performing a motion evaluation on the target object based on the similarity score and the human rating.

[0084] Exemplarily, the implementation method for determining whether a key-frame image is a motion-blurred image can be as follows: First, determine whether it is a blurred image according to the characteristics of the heavy-tailed distribution of the key-frame image, and then determine whether it is a motion-blurred image according to whether the gradient spectrum of the key-frame image contains parallel dark stripes; here, if the gradient spectrum of the key-frame image contains parallel dark stripes, it indicates that the key-frame image is a motion-blurred image; conversely, if the gradient spectrum of the key-frame image does not contain parallel dark stripes, it indicates that the key-frame image is not a motion-blurred image.

[0085] Exemplarily, motion blur is the aliasing of different spatial position information caused by the rapid movement of the target person during video shooting. The essential reason is that the movement distance of the target object within a single-frame exposure time far exceeds the size of a single pixel. The more it exceeds, the blurrier the target object in the image.

[0086] Exemplarily, since the key-frame images of a motion video may have motion blur, in order to facilitate the target person and the instructor to clearly and intuitively understand the details during the occurrence of key actions and facilitate the evaluation and analysis of motion postures, it is necessary to perform de-blurring processing on the key-frame images. Here, de-blurring processing refers to a technology that makes the original unclear key-frame images clear, improves the image quality, and enhances the image interpretation and recognition effects.

[0087] Exemplarily, a de-blurring algorithm related to motion video can be used to perform de-blurring processing on the key-frame images to obtain de-blurred key-frame images; here, the type of de-blurring algorithm is not limited; for example, a motion video de-blurring algorithm based on interpolation calculation can be used, or other types of de-blurring algorithms can be used.

[0088] Exemplarily, in the case where it is determined that the key-frame image is a motion-blurred image, if a motion video de-blurring algorithm based on interpolation calculation is used to perform de-blurring processing on the key-frame image, the process can include: estimating the relative motion parameters between the key-frame image and multiple frames of images before and after, that is, the relative motion offset between two images, registering the key-frame image with multiple frames of images before and after based on the relative motion offset, obtaining the blur degree measure between the key-frame image and multiple frames of images before and after, using this as a parameter for weighted operation, and performing interpolation calculation on the pixel points corresponding to the key-frame image, so as to obtain a clear image after de-blurring processing of the key-frame image. This method removes the motion blur of the key-frame image through multiple frames of images before and after, facilitating the target object and the coach to recall and judge the motion actions through the clear key-frame image.

[0089] It can be seen that in the embodiments of the present application, when deblurring the key-frame image of a motion video, not only the information contained in the key-frame image itself is used, but also the information contained in the front and rear frame images of the key-frame image is combined. In this way, the motion of pixel points in the key-frame image over time can be comprehensively considered to achieve the effects of reducing motion blur and noise interference, and obtaining a clear key-frame image.

[0090] It can be seen that in the embodiments of the present application, the front and rear frames are used to deblur the key-frame image, effectively removing the limb blur caused by the rapid movement of the target object, facilitating the subsequent clear backtracking of the key action moments of the standing long jump by the guiding personnel and the target object, and discovering the action details that can be judged by non-skeletal key points in the action.

[0091] Further, after obtaining the deblurred key-frame image, the human score of the deblurred key-frame image can be obtained; furthermore, based on the similarity score and the human score, the motion of the target object can be jointly evaluated to obtain the final evaluation result.

[0092] Exemplarily, the weight coefficients of the similarity score and the human score can be set respectively, and the sum of the weight coefficients of the two is 1; after obtaining the similarity score and the human score, the similarity score and the human score can be multiplied by the corresponding weight coefficients respectively and then accumulated to obtain the final score of the key-frame image; the following will give an exemplary description of this process with reference to formula (4).

[0093]

[0094] Among them, Score T represents the final score of one of the key-frame images, T represents the time point corresponding to the key-frame image, Score p represents the human score of the deblurred key-frame image, e is the weight coefficient of the similarity score, and (1 - e) is the weight coefficient of the human score.

[0095] Further, in the case where there are multiple key-frame images corresponding to the target object, the corresponding weight coefficient θ t can be set according to the importance of the actions in each key-frame image, and the total score Score of the target object can be obtained by weighted summation through the weight coefficient θ t , as shown in formula (5):

[0096] Score = ∑θ t *Score T

[0097] It should be understood that after obtaining the final scores of each key-frame image during the movement of the target object and the total score of the target object according to the above formulas (4) and (5), the instructor of the target object, such as a teacher or a coach, can refer to the de-blurred key-frame images and, in combination with the similarity of the actions in each key-frame image, clearly discover the action details at critical moments during the entire movement process, so as to formulate an effective improvement plan, issue a reliable exercise prescription, and guide the target object to improve the sports performance.

[0098] Exemplarily, taking the take-off frame of a standing long jump as an example for illustration, the take-off frame can be defined as the frame with the largest body stretching angle when the feet of the target object are about to cross or are crossing the take-off line; referring to Figure 2 , the human key points in the take-off frame may include the ankle key point A, the knee key point B, the hip key point C, the shoulder key point D, the elbow key point E, and the wrist key point F. By calculating the similarity between the vectors formed by the ankle key point A and the knee key point of the target object (AB), the vector formed by the knee key point B and the hip key point C (BC), the vector formed by the hip key point C and the shoulder key point D (CD), etc. and the vectors formed by the corresponding key points of the standard actions in the standard action library, in addition to obtaining the similarity scores of the corresponding actions, it is also possible to rely on the detailed analysis of the key-frame images by the instructor to determine the details that cannot be analyzed by the relevant human key points, issue improvement suggestions for the standing long jump take-off, and guide the target object to improve the sports performance; for example, the swinging direction of the head of the target object during the take-off process is inconsistent with the body, and the degree of body torsion is too large, etc., which may all affect the final result of the standing long jump, and the target object needs to be assisted to correct this subsequently.

[0099] It can be seen that in the embodiment of the present application, not only can the similarity scores of the actions at each critical moment be efficiently evaluated, but also the subjective personal scores of professionals can be added based on the de-blurred key-frame images, effectively improving the detail level and reliability of the exercise prescription plan.

[0100] In order to better reflect the purpose of the present application, based on the above embodiments of the present application, a further description is made for the video conferencing scenario.

[0101] Figure 3 is a schematic flow chart for correcting the coordinates of human key points provided by the embodiment of the present application, as Figure 3As shown in the figure, the process may include the following steps: After obtaining the motion video of the target object, where the motion video includes consecutive human body images from the (t-1)-th frame to the (t+n)-th frame, the motion video can be divided to obtain multiple sub-videos; The human body key point detection algorithm is used to detect the human body key points of the n-frame human body images, and the human body key points from the (t-1)-th frame human body image to the (t+n)-th frame human body image and the confidence of each human body key point can be obtained; Here, taking the human body image of the t-th frame as the current frame image in a certain sub-video as an example, the confidence of the human body key points in the human body image of the t-th frame and its front and back frame human body images in the sub-video is compared with a preset confidence threshold (here, the confidence threshold may include the confidence threshold corresponding to the occlusion factor and the confidence threshold corresponding to the motion blur factor, and the two can be set to 0.4 and 0.7 respectively), and according to the comparison result, the identification of each low-confidence point can be obtained; The identification includes at least one of occlusion and motion blur; Then, it is judged whether the confidence corresponding to the sub-video meets the preset conditions, and when it meets, the bidirectional GRU network is started, and the human body key points of the human body image of the t-th frame and its front and back frame human body images in the sub-video are input into the forward GRU layer and the reverse GRU layer of the bidirectional GRU network to obtain the coordinates of the human body key points of each frame of human body image. After obtaining the coordinates of the human body key points of each frame of human body image, combined with the coordinates of the high-confidence points in the front and back frame human body images for joint analysis, the coordinates of the low-confidence points in the current frame human body image are corrected to obtain the corrected coordinates of the human body key points.

[0102] It can be seen that in the embodiment of the present application, the bidirectional GRU network is used to correct the low-confidence key points of the current frame through the high-confidence key points of adjacent frames, effectively eliminating the inaccurate key point detection caused by external factors such as occlusion and motion video blur, and through a certain opening strategy, fully considering the effectiveness and timeliness of network analysis, and making a trade-off between the algorithm speed performance and the accuracy performance flexibly through parameter control, improving the method applicability.

[0103] Figure 4 is a schematic flowchart of another motion video analysis method provided by the embodiment of the present application, as Figure 4 shown, the process may include the following steps:

[0104] Step 200: Use the bidirectional GRU network to correct the coordinates of the human body key points.

[0105] Step 201: Determine the key frame image.

[0106] Step 202: Perform de-blurring processing on the key frame image.

[0107] Step 203: Perform similarity scoring on the key frame image based on the standard action library.

[0108] Step 204: Obtain the human score of the de-blurred key frame image, and determine the evaluation result based on the similarity score and the human score.

[0109] It should be noted that the implementation manners of the processes in this flowchart have been described in the foregoing embodiments, and will not be elaborated herein.

[0110] Figure 5 is a schematic structural diagram of the composition of the motion video analysis device provided by an embodiment of the present application. As Figure 5 shown, the motion video analysis device 300 includes: an acquisition module 301, a detection module 302, a correction module 303, and an evaluation module 304, where:

[0111] The acquisition module 301 is configured to acquire a motion video of a target object; the motion video includes a plurality of sub-videos, and each sub-video includes a current frame human body image and multiple adjacent front and rear human body images;

[0112] The detection module is configured to perform human key point detection on each frame of human body image in each sub-video to obtain the human key points of each frame of human body image and the confidence of the human key points;

[0113] The correction module 303 is configured to determine the coordinates of the human key points of each frame of human body image when it is determined that the confidence meets a preset condition, and correct the coordinates of the human key points of the current frame of human body image according to the coordinates of the human key points of the multiple adjacent front and rear human body images to obtain corrected coordinates;

[0114] The evaluation module 304 is configured to determine the key frame image of the motion video according to the corrected coordinates corresponding to each sub-video, and perform motion evaluation on the target object through the key frame image.

[0115] In some embodiments, the correction module 303 is further configured to:

[0116] For each sub-video, compare the confidence of the same human key point of each frame of human body image with a preset confidence threshold respectively to obtain a comparison result;

[0117] If it is determined according to the comparison result that the confidence of the target human key point of the current frame of human body image is less than or equal to the confidence threshold, and the confidence of the target human key point of at least one of the front and rear frames of human body image is greater than the confidence threshold, then determine whether the number of target human key points greater than the confidence threshold is within a preset range. If so, determine that the confidence meets the preset condition.

[0118] In some embodiments, the correction module 303 is further configured to:

[0119] When it is determined that the confidence level meets the preset conditions, the human key points of each frame of human body image are processed through a bidirectional GRU network to obtain the coordinates of the human key points of each frame of human body image.

[0120] In some embodiments, the correction module 303 is further configured to:

[0121] Obtain the coordinate coefficients of the human key points of each frame of human body image;

[0122] Determine the first product of the coordinates of the human key points of the multi-frame human body images before and after and the coordinate coefficients, and the second product of the coordinates of the human key points of the current frame of human body image and the coordinate coefficients;

[0123] Correct the coordinates of the human key points of the current frame of human body image according to the first product and the second product.

[0124] In some embodiments, the correction module 303 is further configured to:

[0125] Obtain the time points of each frame of human body image in the motion video;

[0126] According to the corrected coordinates corresponding to each sub-video and the time points of each frame of human body image in the motion video, determine the coordinate time series of the same human key point in the motion video; the coordinate time series represents the coordinate change of the same human key point over time;

[0127] Divide the motion video according to the coordinate time series to obtain a plurality of image intervals; obtain the key frame images of the motion video from the plurality of image intervals.

[0128] In some embodiments, the evaluation module 304 is further configured to:

[0129] Obtain the standard action corresponding to the key frame image from the standard action library;

[0130] Perform a similarity analysis on the motion action of the key frame image and the standard action to obtain a similarity score;

[0131] Perform a motion evaluation on the target object based on the similarity score.

[0132] In some embodiments, the device method further includes a deblurring module, and the deblurring module is used for:

[0133] After determining the key frame image of the motion video, determine whether the key frame image is a motion blurred image. If it is determined to be the case, perform de-blurring processing on the key frame image to obtain a de-blurred key frame image;

[0134] Correspondingly, the evaluation module 304 is further configured to:

[0135] Obtain the human score of the de-blurred key frame image;

[0136] Perform motion evaluation on the target object based on the similarity score and the human score.

[0137] In practical applications, the above-mentioned acquisition module 301, detection module 302, correction module 303, evaluation module 304 and de-blurring module can all be implemented by a processor located in an electronic device, and the processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0138] In addition, each functional module in this embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0139] If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the related technology, or all or part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0140] Specifically, the computer program instructions corresponding to a motion video analysis method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer program instructions corresponding to a motion video analysis method in the storage medium are read or executed by an electronic device, any one of the motion video analysis methods of the foregoing embodiments is implemented.

[0141] Based on the same technical concept as the foregoing embodiments, refer to Figure 6 , an embodiment of the present application further provides an electronic device 400, which may include: a memory 401 and a processor 402; wherein,

[0142] The memory 401 is used to store computer programs and data;

[0143] The processor 402 is configured to execute the computer program stored in the memory to implement any one of the motion video analysis methods of the foregoing embodiments.

[0144] In practical applications, the above-mentioned memory 401 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk (Hard Disk Drive, HDD) or solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memories, and provide instructions and data to the processor 402.

[0145] The above-mentioned processor 402 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It can be understood that for different motion video analysis devices, the electronic devices for implementing the above-mentioned processor functions may also be others, and the embodiments of the present application do not make specific limitations.

[0146] In some embodiments, an embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements any one of the motion video analysis methods of the foregoing embodiments.

[0147] In some embodiments, the functions or modules included in the device provided by the embodiments of the present application may be used to execute the methods described in the method embodiments above. The specific implementation may refer to the description of the method embodiments above. For the sake of brevity, it will not be repeated here.

[0148] The descriptions of the above embodiments tend to emphasize the differences between the embodiments. The same or similar parts can be referred to each other. For the sake of brevity, they will not be repeated here.

[0149] The methods disclosed in the method embodiments provided by the present application can be arbitrarily combined without conflict to obtain new method embodiments.

[0150] The features disclosed in the product embodiments provided by the present application can be arbitrarily combined without conflict to obtain new product embodiments.

[0151] The features disclosed in the method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0152] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0153] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0155] The above is only the preferred embodiment of the present application, and is not used to limit the protection scope of the present application.

Claims

1. A method for analyzing sports videos, characterized in that, The method includes: Obtaining a motion video of a target object; the motion video includes a plurality of sub-videos, and each sub-video includes a current-frame human body image and multiple adjacent front and rear frames of human body images; Performing human key point detection on each frame of human body image in each sub-video to obtain the human key points of each frame of human body image and the confidence levels of the human key points; When it is determined that the confidence levels meet the preset conditions, determining the coordinates of the human key points of each frame of human body image, and correcting the coordinates of the human key points of the current-frame human body image according to the coordinates of the human key points of the multiple adjacent front and rear frames of human body images to obtain corrected coordinates; Determining key frame images of the motion video according to the corrected coordinates corresponding to each sub-video, so as to perform motion evaluation on the target object through the key frame images.

2. The method according to claim 1, wherein The determination that the confidence levels meet the preset conditions includes: For each sub-video, comparing the confidence levels of the same human key point of each frame of human body image with a preset confidence level threshold respectively to obtain comparison results; If it is determined according to the comparison results that the confidence level of the first human key point of the current-frame human body image is less than or equal to the confidence level threshold, and the number of second human key points greater than the confidence level threshold in the multiple adjacent front and rear frames of human body images is within a preset range, it is determined that the confidence levels meet the preset conditions; the first key point and the second key point are the same human key point of each frame of human body image.

3. The method according to claim 1, wherein The determination of the coordinates of the human key points of each frame of human body image when it is determined that the confidence levels meet the preset conditions includes: When it is determined that the confidence levels meet the preset conditions, processing the human key points of each frame of human body image through a bidirectional GRU network to obtain the coordinates of the human key points of each frame of human body image.

4. The method according to any one of claims 1 to 3, characterized in that, The correction of the coordinates of the human key points of the current-frame human body image according to the coordinates of the human key points of the multiple adjacent front and rear frames of human body images includes: Obtaining the coordinate coefficients of the human key points of each frame of human body image; Determining a first product of the coordinates of the human key points of the multiple adjacent front and rear frames of human body image and the coordinate coefficients, and a second product of the coordinates of the human key points of the current-frame human body image and the coordinate coefficients; Correcting the coordinates of the human key points of the current-frame human body image according to the first product and the second product.

5. The method according to claim 1, wherein The determination of the key frame images of the motion video according to the corrected coordinates corresponding to each sub-video includes: Obtaining the time points of each frame of human body image in the motion video; Determining a coordinate time series of the same human key point in the motion video according to the corrected coordinates corresponding to each sub-video and the time points of each frame of human body image in the motion video; the coordinate time series represents the coordinate change of the same human key point over time; Dividing the motion video according to the coordinate time series to obtain multiple image intervals; obtaining the key frame images of the motion video from the multiple image intervals.

6. The method according to claim 1, wherein The motion evaluation of the target object through the key frame images includes: Obtain the standard action corresponding to the key frame image from the standard action library; Perform similarity analysis on the motion action of the key frame image and the standard action to obtain a similarity score; Perform motion evaluation on the target object based on the similarity score.

7. The method according to claim 6, wherein The method further includes: After determining the key frame image of the motion video, determine whether the key frame image is a motion blurred image. If it is determined to be the case, perform de-blurring processing on the key frame image to obtain a de-blurred key frame image; Correspondingly, the performing motion evaluation on the target object based on the similarity score includes: Obtain the human rating of the de-blurred key frame image; Perform motion evaluation on the target object based on the similarity score and the human rating.

8. A sports video analysis device, characterized in that, The device includes: An acquisition module, configured to acquire a motion video of a target object; the motion video includes a plurality of sub-videos, and each sub-video includes a current frame human body image and multiple adjacent front and rear human body images; A detection module, configured to perform human key point detection on each frame of human body image in each sub-video to obtain the human key points of each frame of human body image and the confidence levels of the human key points; A correction module, configured to determine the coordinates of the human key points of each frame of human body image when it is determined that the confidence levels meet a preset condition, and correct the coordinates of the human key points of the current frame human body image according to the coordinates of the human key points of the multiple adjacent front and rear human body images to obtain corrected coordinates; An evaluation module, configured to determine the key frame image of the motion video according to the corrected coordinates corresponding to each sub-video, so as to perform motion evaluation on the target object through the key frame image.

9. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images