Video analysis method and device, computing equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a video analysis method and device, computing equipment and a computer readable storage medium. The method comprises the steps that a to-be-analyzed motion video is acquired, and the motion video is a video that a target object executes a target action; based on an action index corresponding to the target action, analyzing the target object in the motion video to obtain target information of the action index; wherein the action index is represented by at least one physical quantity related to the target object; and determining an analysis result of the motion video based on the target information and the reference information of the motion index. According to the method, the video analysis accuracy can be improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a video analysis method and apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of computer technology, computers have been widely used to analyze various data in order to improve data analysis efficiency.

[0003] In sports and fitness scenarios, users can follow certain sports videos or train according to a set exercise plan. During exercise, users can use terminal devices to record their exercise videos, and the terminal devices can provide guidance on action correction and action problem assessment based on the exercise videos to help users achieve better exercise results.

[0004] However, the efficiency and accuracy of terminal devices' analysis of users' motion videos still need to be improved. Summary of the invention

[0005] The embodiments of this specification provide a video analysis method, which can improve the accuracy of video analysis. One or more embodiments of this specification also relate to a video analysis device, a computing device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of an embodiment of this specification, a video analysis method is provided, the method comprising: Acquire a motion video to be analyzed, wherein the motion video is a video of a target object performing a target action; Based on the action index corresponding to the target action, the target object in the motion video is analyzed to obtain target information of the action index; wherein the action index is represented by at least one physical quantity related to the target object; Based on the target information and the reference information of the action index, an analysis result of the motion video is determined.

[0007] According to another aspect of an embodiment of this specification, a video analysis device is provided, including: An acquisition module, used for acquiring a motion video to be analyzed, wherein the motion video is a video of a target object performing a target action; A first analysis module is used to analyze the target object in the motion video based on the action index corresponding to the target action to obtain target information of the action index; wherein the action index is represented by at least one physical quantity related to the target object; The first determination module is used to determine the analysis result of the motion video based on the target information and the benchmark information of the action index.

[0008] According to another aspect of the embodiments of this specification, there is provided a computing device, including: a memory and a processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above method are implemented.

[0009] According to another aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the steps of the above method are implemented when the computer program / instruction is executed by a processor.

[0010] According to another aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, and when the computer program / instruction in the computer program product is executed in a processor, the steps of the above method are implemented.

[0011] In one embodiment of the present specification, a corresponding action index can be set for a target action, and the action index can be quantified into at least one physical quantity, and benchmark information of each action index is also provided, that is, information of the action index under a standard target action. For a motion video to be analyzed, the target information of these action indicators can be determined for the target object therein, and the target information of each action index can be compared with the benchmark information to determine whether the target object's execution of the target action meets the requirements, thereby obtaining an analysis result of the motion video. In this way, video analysis can be achieved by determining the information of each physical quantity related to the target object in the motion video, and the determined information can better represent the state of the target object without complicated processing of the video, thereby improving the efficiency and accuracy of video analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of a video analysis method provided by an embodiment of this specification; Figure 2 is a curve diagram of knee angle variation provided by an embodiment of this specification; Figure 3 It is a human body segmentation map provided in an embodiment of this specification; Figure 4 It is another human body segmentation map provided in an embodiment of this specification; Figure 5 It is another human body segmentation map provided in an embodiment of this specification; Figure 6 is a human body segmentation map provided by another embodiment of this specification; Figure 7 is another human body segmentation map provided by another embodiment of this specification; Figure 8 It is a simplified flow chart of another video analysis method provided in an embodiment of this specification; Fig. 9 It is a structural schematic diagram of a video analysis device provided in an embodiment of this specification; Fig.10 It is a structural block diagram of a computing device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0013] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0014] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms of "one", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term "at least one" in one or more embodiments of this specification refers to "one or more", and "multiple" refers to "two or more". The term "including" is an open description and should be understood as "including but not limited to", and may include other content on the basis of the content already described.

[0015] It should be understood that although the terms "first", "second", etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, "first" may also be referred to as "second", and similarly, "second" may also be referred to as "first". Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0016] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant standards and requirements, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0017] With the development of computer technology, the use of various videos is becoming more and more extensive. For example, the execution of certain tasks can rely on the analysis of related videos. In sports and fitness scenarios, users can shoot videos of their movements and then analyze the videos to determine whether there are problems in their movements and how to make subsequent improvements and adjustments. Users can use mobile devices to evaluate sports problems, and usually detect in real time during the user's movement process to identify whether the user's current movement is standard, and then provide guidance on action correction. However, this real-time error correction guidance requires high computing and analysis capabilities of the device, so the guidance capabilities of current mobile devices are limited, and the guidance results are not accurate and detailed enough. In another way, the user's action can also be collected based on a somatosensory device. In this way, the user needs to be equipped with a somatosensory device, and the action collection requirements are high, and the collection time is long, which is not suitable for promotion to ordinary users. Therefore, the solution of relying on mobile devices (such as mobile phones) to collect user's motion data for detailed action problem evaluation still needs to be further improved.

[0018] The embodiments of this specification provide a video analysis method that can automatically mark sports videos to improve the accuracy and efficiency of video analysis. This specification also relates to a video analysis device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail one by one in the following embodiments.

[0019] The video analysis method provided in this specification can be applied to a video analysis device and the computing device. The video analysis device and the computing device can be a server device, and the terminal device can send the video to the server device so that the server device can execute the video analysis method provided in the embodiment of this specification. The terminal device can be a terminal device with video acquisition capabilities such as a smart phone, a laptop computer, a tablet computer, a virtual reality display device or an augmented reality display device. In some embodiments, if the data analysis capability of the terminal device is relatively strong, the video analysis device and the computing device can also be a terminal device.

[0020] Figure 1 1 is a flowchart of a video analysis method provided in an embodiment of this specification. The method can be applied to a video analysis device. In this embodiment of this specification, the video analysis device is taken as a server device as an example. Figure 1 As shown, the video analysis method includes the following steps: Step 102: Obtain a motion video to be analyzed, wherein the motion video is a video of a target object performing a target action.

[0021] The video analysis device can obtain a motion video from a video acquisition device, and the motion video can be generated by the video acquisition device. Accordingly, step 102 may include receiving a motion video to be analyzed sent by the video acquisition device. The video acquisition device can be a terminal device with a video shooting function such as a smart phone, a tablet computer, or a laptop computer. In some embodiments, the video analysis device can also obtain the motion video to be analyzed from the motion video stored therein. In some embodiments, the motion video can be a motion video of a virtual object generated by a computer.

[0022] For example, the video analysis device can be a background server device of the target application, which is used to support the normal operation of the target application. For example, the target application can be a sports and fitness program, or it can be other video analysis programs. The target application can be installed in the terminal (such as the aforementioned video acquisition device). During the process of sports and fitness, the user can start the target application and trigger a training course for the target action. Accordingly, the terminal can automatically capture the user's video and generate a sports video of the user performing the target action. Optionally, the user can also click on the video acquisition control to trigger the terminal to perform video acquisition. During this process, the terminal can guide the user to perform the target action and can also prompt the user to ensure that he is within the video acquisition range. The moving object (that is, the user) in the sports video is referred to as the target object below, and the target object is usually a real person. In some special cases, the moving object can also be an animal or some virtual object.

[0023] The terminal can send the collected video stream to the video analysis device in real time during the video collection process; or the terminal can also send the complete video to the video analysis device after collecting the video of the target object performing the target action. For example, an execution time can be set for each action, and the terminal can collect the video and generate the motion video corresponding to the action within the execution time. The user can perform the action multiple times within the execution time. Optionally, the number of executions can also be set for each action to guide the user to exercise according to the set number of executions.

[0024] The terminal can generate different motion videos for different actions, and each generated motion video can carry information about its corresponding action, so that after the video analysis device obtains the motion video to be analyzed, it can directly know the target action corresponding to the motion video. For example, after training for one action, the user wants to switch to training for another action, then the user can end the training recording process corresponding to the current action on the terminal and trigger the training recording of another action, so that the terminal can re-collect the motion video corresponding to the other action.

[0025] In some embodiments, the sports video to be analyzed may not carry information about the corresponding action. In this case, the video analysis device may first perform action recognition on the video to determine the target action corresponding to the video, and then perform subsequent analysis. In some embodiments, a sports video may also be a video of a user performing multiple actions. In this case, the video analysis device may perform action recognition on the video to split it into multiple video segments corresponding to the multiple actions, and then perform subsequent analysis.

[0026] Step 104: Analyze the target object in the motion video based on the action index corresponding to the target action to obtain target information of the action index; wherein the action index is represented by at least one physical quantity related to the target object.

[0027] In the embodiment of this specification, corresponding action indexes can be pre-configured for various motions, and the number of action indexes corresponding to each action of the action index can be one or more. Each action index can also be pre-quantized into at least one physical quantity, such as physical quantities such as angle, radian, distance, displacement and speed, and different action indexes can be quantized into these physical quantities. If the angle includes relative angle and absolute angle, distance calculation includes relative distance, absolute distance, x-direction distance, y-direction distance and z-direction distance, etc. This can facilitate the flexible definition and configuration of action indexes for different actions.

[0028] For example, the action indicators corresponding to the squat action may include the squat amplitude, the knee sway amplitude, the back bending condition, etc. The squat amplitude can be quantified as the knee angle, such as taking the knee point as the vertex, and determining the angle of the angle formed by the line connecting the hip point and the knee point, the line connecting the ankle point and the knee point, and the vertex as the knee angle. The knee sway amplitude can be quantified as: the angle formed by the vector formed by the knee point and the ankle point and the horizontal ground, and the difference in change during the movement. The back bending condition can be quantified as the angle between the back curve and the reference line, which is the line connecting the neck point and the hip point. In the embodiment of this specification, the action indicators can be configured for various actions using a JSON file. The video analysis device can adapt the JSON file to parse the action indicator values ​​required in the motion video. There is no need to write a set of algorithms for each action, which can achieve efficient reuse and can quickly add new actions that require motion problem assessment.

[0029] The information that can be collected from an object in different orientations is different, and the corresponding information of some action indicators may not be obtained. It is difficult to accurately determine the actual execution of the object on the action using the action indicator. The orientation refers to the orientation of the object relative to the video acquisition device. In some embodiments, the action indicator corresponding to the target action can also be related to the orientation of the object performing the target action, and different action indicators can be configured for the target action in advance for different orientations. For example, when the object is facing away from the video acquisition device, the information of the object's face cannot be collected, and then the action indicator related to the face will not be set in this orientation. Accordingly, step 104 may include: determining the orientation of the target object in the motion video; based on the action indicator corresponding to the target action under the orientation, analyzing the target object in the motion video to obtain the target information of the action indicator. After the video analysis device identifies the target object in the motion video, it can first analyze the orientation of the target object, and then determine the action indicator corresponding to the target action based on the orientation, and then perform subsequent analysis based on the action indicator.

[0030] For each action, the video analysis device can determine the information of the corresponding action index by calculating the information of the physical quantity corresponding to each action index of the action, and the information of each action index can be the information of the physical quantity corresponding to it. The information of the action index corresponding to the action can represent the execution status of the action by the user. The video analysis device can determine whether there is a problem in the execution of the action according to whether each action index corresponding to the action meets the requirements.

[0031] In an embodiment of the present specification, after the video analysis device obtains the motion video to be analyzed and determines its corresponding target action, it can determine the corresponding action indicators based on the target action. Then, the physical quantities corresponding to these action indicators in the motion video can be calculated to obtain the target information of each action indicator. In an embodiment of the present specification, in a motion video in which a target object performs a target action, the action indicator of the target action is characterized by at least one physical quantity related to the target object, and accordingly, it is necessary to calculate the information of these physical quantities for the target object to obtain the information of each action indicator. The video analysis device can first identify the target object in the motion video, and then calculate the physical quantity for the target object.

[0032] The video analysis device analyzes the motion video, which can actually be an analysis of multiple frames of images in the motion video. Identifying the target object for the video actually means identifying the target object for each frame of image. For example, the video analysis device can first determine the multiple frames of images to be analyzed in the motion video. The multiple frames of images can be all the frames in the motion video, or they can also be part of the images in the motion video, which can reduce the consumption of computing resources and improve the analysis efficiency. The analysis of the target object in the motion video can include analyzing the area where the target object is located in each frame of the motion video, and can also include analyzing the relationship between each position in the area where the target object is located in the image and other areas.

[0033] In one embodiment, frame extraction processing may be performed on the motion video to determine the multiple frames of images, so as to reduce the amount of calculation. Accordingly, before step 104, the video analysis method provided in the embodiment of this specification may include performing frame extraction processing on the motion video to determine the multiple frames of images to be analyzed in the motion video. For example, since the information difference in adjacent frame images is small, the video analysis device may extract once every frame to double the calculation rate. Optionally, the video analysis device may also extract once every two frames or other number of frames. The video analysis device may also determine the number of image frames spaced apart by the extraction based on the frame rate of the motion video, such as the higher the frame rate, the more image frames are spaced apart.

[0034] In some embodiments, after determining the multiple frames of images and identifying the target object for each frame of image, the video analysis device can detect multiple skeleton points of the target object in each frame of image. Accordingly, in step 104, based on the action index corresponding to the target action and the multiple skeleton points of the target object in each frame of image, the target object in the motion video can be analyzed to obtain information on the action index. The target object can be a person, and the process of identifying the target object for each frame of image is also the process of performing human body detection. For example, the detection frame where the human body is located in the image can be obtained through this detection. The skeleton points detected for the target object in the image here can be two-dimensional skeleton points, and the information of the two-dimensional skeleton points can be obtained through this detection, such as the two-dimensional coordinate information. The video analysis device can analyze all the target objects in the multiple frames of image in the motion video, or it can only analyze the target objects in some images in the multiple frames of image.

[0035] For example, multiple skeletal points to be detected may be configured in advance for the target action, and the skeletal points to be detected corresponding to different actions may be the same, or may be adaptively configured for different actions. For example, the multiple skeletal points may include ankle skeletal points, knee skeletal points, hip skeletal points, neck skeletal points, elbow skeletal points, wrist skeletal points, finger skeletal points, and head skeletal points. For the target action, only the information of these skeletal points corresponding to it may be detected.

[0036] In the embodiments of this specification, different image frames may be analyzed to determine different action index information. For example, for some action indexes, only a single frame or a few frames in the motion video may be analyzed, while for some action indexes, more frames within a period of time may need to be analyzed.

[0037] Step 106: Determine the analysis result of the motion video based on the target information of the motion index and the reference information of the motion index.

[0038] In the embodiments of this specification, when configuring the action index for each motion action, the benchmark information for each action index may also be configured. The benchmark information may refer to the information of the action index when it meets the corresponding motion requirements, and the benchmark information may reflect the standard state of the action. The benchmark information may be determined based on offline data or based on personnel experience.

[0039] After determining the action index corresponding to the target motion, the video analysis device can also obtain its benchmark information. After determining the target information of each action index, the target information of each action index is compared with the corresponding benchmark information to obtain the analysis results of each action index (such as the difference between the target information and the benchmark information). The analysis results can reflect whether there are problems corresponding to each action index in the process of the target object performing the target action. Then, based on the analysis results of each action index, the analysis results of the motion video are determined. The analysis results of the motion video may include the analysis results of each action index. Optionally, it may also include guidance opinions, recommended actions or training plans, etc. for the target object.

[0040] After generating the analysis result of the motion video, the video analysis device may send the analysis result to the video acquisition device, so that the user can obtain the analysis result based on the video acquisition device, and then adjust and improve the motion state or motion plan based on the analysis result. In some embodiments, the video analysis device may also send at least one representative frame of the motion video and / or at least one video clip to the video acquisition device. The analysis result may be presented by annotating the image or video clip.

[0041] The video analysis device can perform certain simple analysis on the received video in real time and provide real-time feedback to the terminal; and after the video stream is sent, it can also perform detailed analysis on the video to send the analysis result to the terminal after the analysis. In the embodiment of this specification, since the video analysis device can analyze the motion video after the target object completes the entire process of performing the target action, the video analysis device has sufficient time to analyze the video, which can ensure the detail and accuracy of the video analysis.

[0042] In summary, in the video analysis method provided by the embodiment of this specification, a corresponding action index can be set for the target action, and the action index can be quantified into at least one physical quantity, and benchmark information of each action index is also provided, that is, the information of the action index under the standard target action. For the motion video to be analyzed, the target information of these action indicators can be determined for the target object therein, and the target information of each action index can be compared with the benchmark information to determine whether the execution of the target action by the target object meets the requirements, and the analysis result of the motion video can be obtained. In this way, video analysis can be achieved by determining the information of each physical quantity related to the target object in the motion video. The determined information can better represent the state of the target object, without the need for complex processing of the video, and can improve the efficiency and accuracy of video analysis.

[0043] Some optional implementation methods of the video analysis method provided in the embodiments of this specification are introduced below.

[0044] In some embodiments, the motion video may be a video of a target object performing multiple target actions. The video analysis device may segment the motion video based on the action count to obtain video segments corresponding to each target action performed by the target object, and then analyze the action index for the obtained video segments. The step of analyzing the target object in the motion video based on the action index corresponding to the target action in the above step 104 to obtain the target information of the action index may include the following steps 1042 to 1046.

[0045] Step 1042: Based on the segmentation index corresponding to the target action, analyze the target object in the motion video to obtain information on the segmentation index at multiple moments in the motion video.

[0046] In the embodiments of this specification, a corresponding segmentation index can be pre-configured for each action, and the segmentation index can also be an action index of the action, and the segmentation index can also be quantified as a certain physical quantity. During the progress of the action, the information of the segmentation index corresponding to the action changes sequentially and has peak values ​​and valley values. The segmentation index can be a core indicator of the action. For example, the segmentation index corresponding to the squat action can be the knee angle, and during the squat process, the knee angle changes from large to small and then becomes large again. If the target object in the motion video performs multiple target actions, the information of the segmentation index can change periodically, and each cycle corresponds to the execution process of a target action.

[0047] The video analysis device can determine the information of the segmentation index at each moment based on the information of the target object in the image frame at the moment for multiple moments in the motion video, so as to facilitate the subsequent segmentation of the video. For example, the multiple frames of images to be analyzed determined by the video analysis device for the motion video in the above introduction may be the image frames at the multiple moments, and the video analysis device can analyze the target object in the multiple frames of images to determine the information of the segmentation index corresponding to each image frame. Optionally, the video analysis device can also determine multiple additional moments in the motion video, and then determine the information of the segmentation index at the multiple moments accordingly. The video analysis device can determine the information of the segmentation index based on the information of the two-dimensional skeleton points of the target object determined for each image frame. The information of each indicator in this manual can be determined based on the information of the two-dimensional skeleton points of the target object in each image frame.

[0048] Step 1044, based on the segmentation moments in which the information of the segmentation indicators in the multiple moments meets the segmentation conditions, the motion video is divided into multiple video segments, and the target image frame in the motion video is determined based on the multiple video segments; wherein one video segment corresponds to a process in which the target object performs a target action, and the target image frame is an image frame corresponding to the extreme value information of the segmentation indicator in the video segment.

[0049] After determining the information of the segmentation index at the multiple moments, the video analysis device can determine the segmentation moment based on the information according to the segmentation conditions, and then divide the motion video into multiple video segments based on the determined segmentation moment, each video segment corresponding to an execution process of the target motion. The segmentation condition that needs to be satisfied at the segmentation moment can be that the change trend of the information of the segmentation index before and after the moment changes, such as the segmentation moment can be the moment when the information of the segmentation index is at a peak or valley. For example, for a squatting action, the segmentation moment can be the moment when the knee angle changes from increasing to decreasing, or from decreasing to increasing.

[0050] For example, the video analysis device may form a variation curve of the segmentation index information based on the segmentation index information obtained at multiple moments, and further determine the segmentation moment based on the variation curve to achieve segmentation of the running video. Figure 2 This is a curve diagram of knee angle changes provided in one embodiment of this specification. Figure 2 Two change curves are shown, which can correspond to different target objects performing squatting actions. The two curves each include three change cycles, and accordingly, it can be intuitively determined that the two target objects have performed three squats. The video analysis device can use the first peak in each cycle as a segmentation moment to divide the motion video and obtain three video segments corresponding to the three squats.

[0051] The video analysis device can also determine the image frame corresponding to the extreme value information of the segmentation index in the video segment based on the divided video segment to obtain the target image frame. The target image frame can also be called an extreme value frame. The condition that the target image frame needs to meet can also be pre-configured for the target action. The condition can be that the target image frame corresponds to a maximum value or a minimum value. The conditions that the target image frame needs to meet are different for different actions. The target image frame can correspond to the moment when the sense of force is the strongest during the execution of the action. For example, the target image frame corresponding to the squat action corresponds to the minimum value, that is, the image frame when the knee angle in the video segment is the smallest is determined as the target image frame.

[0052] In many cases, the target image frame can reflect the execution of the target action by the target object, such as reflecting the execution of an action indicator. The video analysis device can determine the target image frame so as to perform corresponding analysis based on the target image frame later. Through the target image frame, a more accurate analysis can be performed on the execution of the target action by the target object. In a first embodiment, the video analysis device can determine a corresponding target image frame for each video segment obtained by division. In a second embodiment, the video analysis device can select a target video segment from each video segment obtained by division, and determine a corresponding target image frame only for the target video segment. The target video segment can be a video segment that can better reflect the execution effect of the target object on the target action.

[0053] For the second embodiment, a segment selection index and a segment selection condition may be pre-configured for the target action to determine the target video segment based on whether the information of the index in each video segment meets the segment selection condition. The segment selection index may also be an action index corresponding to the target action, and the segment selection index may be the same as the aforementioned segment index, or may also be different. Continuing with the above example about the squat action, the segment selection index corresponding to the squat action may also be the knee angle, or the segment selection index corresponding to the squat action may also be the degree of back curvature. For example, the segment selection condition corresponding to the squat action may be that the rate of change of the knee angle is minimal.

[0054] Accordingly, in the second embodiment, in step 1044, the video analysis device can analyze the target object in the motion video based on the segmentation selection index corresponding to the target action, and obtain information on the segmentation selection index at multiple moments in the motion video; determine a target video segment in which the information on the segmentation selection index meets the segmentation selection condition among multiple video segments; and determine a target image frame in the target video segment.

[0055] The video analysis device can determine the information of the segment selection index at each moment in the entire motion video, and then determine the information of the segment selection index in each video segment. For the information of the segment selection index in each video segment, it is determined whether it meets the segment selection conditions, or the information of the segment selection index in each video segment is compared to determine the target video segment that meets the segment selection conditions. Still using the above example about the squat action, the video analysis device can determine the change speed of the knee angle based on multiple knee angles in each video segment, and then compare the knee change speed in each video segment to determine the video segment with the smallest speed as the target video segment.

[0056] Step 1046: Analyze the target object in the video segment and the target image frame based on the action index corresponding to the target action to obtain target information of the action index.

[0057] If the action index corresponding to the target action includes the aforementioned determined segmentation index and segmentation selection index, the target information of the two indexes can be directly obtained here. For other action indexes, the target object in the divided video segment and the determined target image frame can be further analyzed to obtain the target information of these action indexes. Optionally, the segmentation index and segmentation selection index can also be re-analyzed based on the determined video segment and target image frame.

[0058] For the other action indicators, if the corresponding target image frames are determined for each video segment in step 1044, the video analysis device in step 1046 can select the corresponding video segments and target image frames for analysis based on the different requirements of each action indicator, or can also analyze each video segment and target image frame. If the target video segment is determined in the previous step 1044, and the target image frame is only determined in the target video segment, then in step 1046, the video analysis device can analyze only the target video segment and the target image frame in the target video segment based on the action indicator corresponding to the target action, so as to save computing resources and improve analysis efficiency.

[0059] In some embodiments, each action index in the embodiment of this specification can be divided into a static index and a dynamic index. The static index can determine the corresponding information based on only one frame of image, while the dynamic index needs to determine the corresponding information based on multiple frames of image. In this case, the above step 1046 may include: based on the static index in the action index corresponding to the target action, analyzing the target image frame to obtain the target information of the static index; based on the dynamic index in the action index corresponding to the target action, analyzing the video segment to obtain the target information of the dynamic index. For the static index corresponding to the target action, the video analysis device can directly analyze the determined target image frame, such as only analyzing the target image frame in the target video segment, and the target information of the static index in the motion video can be obtained. For the dynamic index corresponding to the target action, the video analysis device can analyze the determined target video segment to obtain the target information of the dynamic index in the motion video. For example, the squat amplitude and whether the waist is arched or collapsed at the lowest point of the squat are static indicators, and whether the knees shake during the squat process is a dynamic indicator. In this way, a clearer definition of each action indicator can be achieved. At the same time, for different action indicators, it is only necessary to analyze the corresponding part of the images or videos in the motion video, without having to analyze all the images in the motion video. This can save computing resources and improve analysis efficiency while ensuring the analysis effect.

[0060] The foregoing content introduces that the video analysis device can identify the target object and detect the two-dimensional skeleton points for the motion video, and determine the information of each action indicator based on the information of the detected two-dimensional skeleton points. For different action indicators, the video analysis device may also need to perform other processing on the motion video, such as detecting the information of the three-dimensional skeleton points and segmenting the area covered by the target object in the video. The target object can be a person, and segmenting the area covered by the target object is also called human body segmentation. In the embodiments of this specification, the target object in the motion video is processed, that is, the target object in the image frame in the motion video is processed.

[0061] In some implementations, in step 104, analyzing the target object in the motion video based on the action index corresponding to the target action to obtain target information of the action index may include: Based on a first action indicator corresponding to the target action, a target three-dimensional skeleton point of a target object in a motion video is detected, and based on information of the target three-dimensional skeleton point, target information of the first action indicator is determined; wherein the first action indicator is related to the depth of the target object in the motion video; Based on a second action indicator corresponding to the target action, an area covered by the target object is segmented from the motion video, and target information of the second action indicator is determined based on information of the area; wherein the second action indicator is related to a contour of the target object.

[0062] If the target action corresponds to a first action indicator, the first action indicator may be related to the depth of the target object in the motion video, and the information of the indicator cannot be directly determined based on the two-dimensional skeleton point information; then the video analysis device may perform three-dimensional skeleton point detection on the target object in the motion video. The first action indicator may also be pre-configured with target three-dimensional skeleton points that need to be detected in the object, and the video analysis device only needs to detect the target three-dimensional skeleton points to obtain the information of the target three-dimensional skeleton points. For some first action indicators, the information of the target three-dimensional skeleton points can be directly used as the target information of the first action indicator; for some first action indicators, the video analysis device may also need to further process the information of the target three-dimensional skeleton points to obtain the target information of the first action indicator.

[0063] For example, if the first action indicator is the forward and backward movement of the character's pelvis, and the orientation of the target object in the motion video is positive, then the information of the first action indicator cannot be determined directly based on the two-dimensional skeleton points in the target object in the motion video. Furthermore, the video analysis device can perform depth detection on the target object in the motion video to determine the three-dimensional position information of the three-dimensional skeleton points of the target object's pelvis. Since the forward and backward movement is a dynamic indicator, the video analysis device determines the target information of the first action indicator based on the position change of the three-dimensional skeleton points in multiple frames of images.

[0064] If the target action corresponds to the second action indicator, and the information of the indicator is related to the contour of the target object, the information of the indicator cannot be directly determined based on the two-dimensional skeleton point information; then the video analysis device can segment the target object in the motion video to extract the area covered by the target object, such as obtaining a mask image or a silhouette image of the target object. Then the video analysis device can determine the target information of the second action indicator based on the boundary of the area.

[0065] For example, the second action indicator is the degree of back curvature. When detecting two-dimensional skeleton points, generally only the neck skeleton points and the hip skeleton points are detected for the back. Only a straight line can be determined based on the two two-dimensional skeleton points, which cannot reflect the true curvature of the back. The degree of back curvature is related to the back contour of the human body, so the video analysis device can perform human body segmentation for the target object in the motion video, and then determine the degree of back curvature based on the obtained mask image that can reflect the human body contour, and obtain the target information of the second action indicator.

[0066] In the embodiments of this specification, some measures can also be taken to speed up the algorithm and ensure that the video analysis device analyzes the motion video at a faster speed. In one embodiment, the video analysis device can obtain a motion video with a lower resolution, and accordingly analyze it based on the image frames with a lower resolution to ensure that the amount of data processed is small and the processing efficiency is high. In this way, the video analysis device can use an analysis module that can obtain more accurate analysis results based on lower resolution images to analyze the motion video, so as to ensure the analysis effect without losing the analysis accuracy while ensuring the processing efficiency.

[0067] For example, obtaining the motion video to be analyzed in the above step 102 may include: obtaining the motion video to be analyzed whose resolution is lower than the target threshold. In the above step 104, analyzing the target object in the motion video based on the action index corresponding to the target action to obtain the target information of the action index may include: analyzing the target object in the motion video based on the action index corresponding to the target action using the target analysis module to obtain the target information of the action index; wherein the target analysis module is obtained by adjusting the auxiliary analysis model, and the auxiliary analysis model is used to analyze the motion video with a resolution higher than the target threshold. In some embodiments, the above-mentioned various analysis methods for the motion video (such as target object recognition, two-dimensional skeleton point detection, three-dimensional skeleton point detection, target object segmentation, and determination of various action index information) can be performed by the video analysis device using the target analysis module.

[0068] The motion video can be sent by a video acquisition device, so the video acquisition device can directly acquire motion videos with lower resolution, which can improve the upload speed of the video acquisition device to the video, and improve the download speed of the video analysis device to the motion video. In the embodiment of this specification, a target analysis module can be pre-constructed so that the target analysis module can obtain more accurate analysis results based on images with lower resolution. For example, the target analysis module can be obtained by adjusting the auxiliary analysis module, and the auxiliary analysis module can be used to analyze videos with higher resolution and determine the information of action indicators.

[0069] For example, the auxiliary analysis module can determine the specific bone points of the object and the results of object segmentation for a very high-definition image according to the algorithm set therein, and obtain a better image analysis result (such as information including action indicators). Then the resolution of the input image of the auxiliary analysis module is reduced to simulate the uploaded low-resolution image. The auxiliary analysis module processes the low-resolution image according to its algorithm to obtain an image analysis result. The image analysis result is compared with the image analysis result at the previous high resolution, and the algorithm is trained end-to-end, and the algorithm is adjusted until the analysis result of the auxiliary analysis module based on the low-resolution image is close to the analysis result of the high-resolution image. At this time, the auxiliary analysis module can be used as the target analysis module obtained after adjustment.

[0070] In some embodiments, the video analysis device also needs to send at least one of the representative image frames and video segments in the motion video to the video acquisition device, and can send image frames and video segments with higher resolution. The video analysis device can process the image frames and video segments directly obtained from the motion video to improve the resolution. For example, the resolution-improving process can be used in scenes where the resolution of the motion video uploaded by the video acquisition device is low. Accordingly, in the above step 106, the analysis result of the motion video is determined based on the target information of the action indicator and the benchmark information of the action indicator, including: determining the video content to be displayed in the motion video, wherein the video content includes the target image frame and / or target video segment in the motion video; performing resolution enhancement processing on the video content; obtaining the analysis result of the motion video based on the target information, the benchmark information of the action indicator, and the video content after the resolution enhancement process.

[0071] In the embodiments of the present specification, the target image frame and target video segment determined above can represent the execution of the target action by the target object, and the video analysis device can directly determine at least one of the target image frame and the target video segment as the video content to be displayed to the user. The video analysis device can perform super-resolution processing on the video content to improve the resolution. Afterwards, the video content with improved resolution can be combined with the analysis results of the action index to obtain the final analysis results of the motion video, which are then sent to the video acquisition device. For example, the analysis results of the action index can be marked on the video content so that the user can more easily understand the problems with his or her movement.

[0072] In some embodiments, the video analysis device can compare and analyze the analysis results of multiple motion videos to determine additional analysis results of the motion videos; wherein the multiple motion videos are videos of the target object performing the target action, and the multiple motion videos are generated at multiple times. In this way, the video analysis device can also compare the same object based on the multiple executions of the target object at different times to determine the change information of the target object's execution of the action, and provide more comprehensive motion problem analysis and guidance for the target object. For example, the additional analysis results can reflect the state change information of the target object performing the target action, and can also include the motion problem analysis and guidance information.

[0073] In the embodiment of this specification, a lower resolution can be used when the original motion video is collected and uploaded, so that the speed of overall video upload, download and algorithm analysis will be greatly improved. However, low resolution will lead to the problem of reduced accuracy of algorithm analysis. By adjusting the auxiliary analysis module, the accuracy is basically the same as that under high resolution. The algorithms involved in action video analysis include human body detection algorithm, human body two-dimensional and three-dimensional bone point detection, and human body segmentation algorithm. If these four algorithms need to be executed for each frame, the time consumption will be huge. In the embodiment of this specification, the action counting logic is first used to divide the video into several segments (wherein, the human body detection algorithm and human body two-dimensional bone point detection need to be used for each frame of the video to be analyzed), and the target image frame and target video segment are selectively detected and segmented according to whether the action index is a static index or a dynamic index. In this way, the detection and segmentation of human body three-dimensional bone points do not need to be performed for each frame, but are performed on demand. Because the information between video frames is relatively close, the algorithm is run in a frame extraction manner (the algorithm is run once every frame), so that the algorithm running speed can be directly doubled. Subsequently, the target image frame and the target video segment need to be returned to the user. Due to the low video resolution, the image and video effects are slightly lower. In the embodiment of this specification, an additional super-resolution algorithm with a resolution enlarged by 2 times is used to achieve the clarity improvement of the resulting image and video. In this way, while ensuring the clarity of the result, the performance improvement brought by the reduction of the original video resolution is also enjoyed.

[0074] For different target actions, the corresponding action indicators may be different, and the specific processing performed by the video analysis device in the process of determining the information of different action indicators may also be different. The following is an exemplary introduction to the video analysis method provided in the embodiment of this specification, taking the target action as a squat action.

[0075] The video analysis device can first segment the motion video of the squat, and split the motion video into action cycles (i.e. the aforementioned video segments) and action extreme value frames (i.e. the aforementioned target image frames). For the squat action, one squat is one action cycle, and when the squat reaches the bottom, the corresponding image frame is the action extreme value frame (generally the position with the maximum force of the action). The video analysis device can determine the required action cycle (i.e. the aforementioned target video segment) and action extreme value frame through a core indicator (such as knee angle, corresponding to the aforementioned segmentation indicator). When the knee angle is close to 180 degrees, it is a standing posture, and the smaller the value, the larger the squat. The video analysis device can obtain the 2D coordinates of the human skeleton points in each frame of the motion video, and calculate the knee angle in each frame based on the coordinates. By determining the knee angle in the motion video, the angle change curve can be obtained, as shown above Figure 2The curve shown. Through the periodicity of the curve, it can be determined that the target object did three squats and the frame with the largest squat range can be obtained. Then the image can be determined as the action extreme frame, and the period of the image can be determined as the required action period. In this way, the motion video has structured information, which is convenient for subsequent calculation based on the action extreme frame or the action period for different action indicators.

[0076] The following example uses the back concavity as an action indicator corresponding to the squat action. This action indicator is a static indicator. For this action indicator, the video analysis device can cut the action extreme value frame based on the bbox (i.e., human body detection frame) of the action extreme value frame to determine the human body segmentation map (mask). Figure 3 is a human body segmentation map provided in an embodiment of this specification. If the video analysis device cuts the extreme value frame, it can obtain Figure 3 The human body segmentation diagram shown in FIG. The video analysis device also calculates the angle between the vector from the neck point A to the hip point B and the y-axis (i.e., the vertical direction of the image) based on the keypoints (i.e., the 2D skeleton points of the human body). Then, the segmentation diagram can be rotated so that the vector from the neck point A to the hip point B is perpendicular to the x-axis (i.e., the horizontal direction of the image). Figure 4 is another human body segmentation map provided in an embodiment of this specification. If the video analysis device rotates the segmentation map so that the vector from the neck point to the hip point is perpendicular to the x-axis, it can be obtained Figure 4 Status shown.

[0077] After that, the segmentation map area between the y coordinate of the neck point A and the y coordinate of the hip point B can be taken out to obtain an auxiliary segmentation map (such as Figure 4 The curves on the left and right sides of the contour of region Q are searched, and the back curve is determined to be a left curve or a right curve according to the orientation of the human body. Figure 4 The back curve should be a right curve according to the human body orientation in , and then the back fitting curve (not marked in the figure) is obtained, which represents the back contour. After that, the curve can be affine transformed back to the coordinate system where the human body segmentation map is located when it is not rotated. In this process, the coordinates of each point in the back fitting curve are adjusted to correspond to the human back contour in the original coordinate system. Figure 5 is another human body segmentation map provided in an embodiment of this specification. Figure 6 is a human body segmentation map provided by another embodiment of this specification, Figure 7 is another human body segmentation diagram provided by another embodiment of this specification. Figures 5 to 7As shown in the figure, by connecting the first and last points of the back fitting curve, a reference line a can be obtained. By calculating the vertical distance of the back fitting curve straight line a, the maximum vertical distance can be obtained. Construct the back curve angle index, and construct an obtuse isosceles triangle with reference line a as the base and the maximum vertical distance as the height. The obtuse angle can characterize the degree of back curvature. According to whether the maximum vertical point is on the left or right side of the reference line a, it can be judged whether the back is convex or concave. Combined with the degree of back curvature, the information on the degree of back concave can be obtained. If the value of the index is distinguished by positive and negative, positive can indicate convexity and negative can indicate concave. Figures 5 to 7 Three states of back straightness, back convexity and back concaveness are shown respectively.

[0078] Figure 8 FIG. 1 is a simplified flow chart of another video analysis method provided in an embodiment of this specification. Figure 8 As shown, the video analysis device can first obtain a low-resolution motion video and its corresponding action ID. After that, the video can be downloaded and frame extraction can be performed on the video, such as extracting once every frame, and then the extracted frames are stored. Then, human body detection is performed on each extracted frame image, and two-dimensional skeleton point detection is performed on the detected human body to obtain multiple two-dimensional skeleton point information. Then, body orientation recognition and left and right limb recognition are performed on each frame image, and the action strategy corresponding to the action is determined in the action strategy file library in combination with the recognized orientation and the action ID corresponding to the video. The action strategy includes multiple action indicators, segmentation indicators, segmentation selection indicators and conditions corresponding to each indicator. Several action guidance items for multiple actions can be stored in the action strategy file library, each action guidance item corresponds to an action indicator, and these action indicators can be issued through JSON files. The video analysis device can adapt to JSON files to parse the action indicators required in the motion video.

[0079] After determining the action indicators required for analyzing the motion video, the video analysis device can combine the two-dimensional human skeleton point information detected in each frame image to calculate the core indicator of the target action (corresponding to the aforementioned segmentation indicator), segment the motion video based on this information, and obtain the required periodic intervals and extreme value frames. Then, the action indicator information is calculated based on the periodic intervals and extreme value frames to obtain the analysis results of the action indicators. The obtained low-resolution extreme value frames and periodic interval video segments need to be displayed, and they can be super-resolved to obtain high-resolution extreme value frames and periodic interval video segment outputs.

[0080] In the embodiments of this specification, detailed motion indicators can be analyzed for the motion video collected by the mobile terminal, such as: whether the back is slumped / arched, whether the knees shake when squatting, whether the amplitude is up to standard, etc. Several motion indicators for each action can be issued by configuring a JSON file, and the algorithm can adapt to the JSON file to parse the required motion indicator values ​​in the motion video. There is no need to write a set of algorithms for each action, which can achieve efficient reuse and quickly add new actions that require motion problem assessment. And there are not too many restrictions on the user's movement direction and dominant limbs. For example: squats require facing the video sideways, and the algorithm can adaptively process when the user faces the left side of the screen and the right side of the screen. If facing the screen, there will be an abnormal direction prompt; single-leg squats, users can squat with their left leg or their right leg, and the video analysis device can adaptively process and analyze whether it is a left-leg squat or a right-leg squat.

[0081] In the embodiment of this specification, video analysis is realized by the cooperation of the mobile terminal and the server. The mobile terminal guides the user to perform specified actions and collect motion data, and the server analyzes the motion data for video analysis, which not only ensures the convenience of use, but also ensures the accuracy of the results. The motion problem evaluation items are abstracted into various motion indicators, and the parameters required for the motion indicators are decoupled and turned into configurable parameter files, which realizes the high reusability of functions and the convenience of adding new actions and new motion indicators. The motion indicators are further refined into static extreme frame indicators and dynamic motion indicators, and the motion video is analyzed to obtain structured information (obtaining motion extreme frames and motion cycle intervals). While the definition of motion indicators is clearer, it also saves the calculation time of some complex motion indicators, such as only some motion indicators need to use human body segmentation map information and three-dimensional human body bone point information. Modules for adaptive user orientation and moving limbs are also added to give users a higher degree of freedom in motion without restricting the user's motion habits too much.

[0082] In addition, the method of using low-resolution video and frame extraction can reduce the amount of data that needs to be processed. The analysis module trained at low resolution is used to improve the accuracy of the motion index value, and the super-resolution algorithm is used to improve the clarity of the final output motion display image and motion display video, so as to offset the impact of low resolution and achieve a balance between speed and accuracy.

[0083] In the video analysis method provided in the embodiment of this specification, the motion video collection is convenient and the analysis result is highly accurate, which can more conveniently and accurately realize the evaluation of motion problems and the push of courses for correcting motion problems. If the motion analysis function can be provided to member users of the target application, it can help improve the conversion rate of member users.

[0084] Corresponding to the above-mentioned video analysis method embodiment, this specification also provides a video analysis device embodiment, Fig. 9Schematic diagram of a video analysis device provided in one embodiment of this specification. Fig. 9 As shown, the video analysis device comprises: An acquisition module 902 is used to acquire a motion video to be analyzed, wherein the motion video is a video of a target object performing a target action; A first analysis module 904 is used to analyze the target object in the motion video based on the action index corresponding to the target action, and obtain target information of the action index; wherein the action index is represented by at least one physical quantity related to the target object; The first determination module 906 is used to determine the analysis result of the motion video based on the target information and the reference information of the motion index.

[0085] Optionally, the first analysis module 904 is used to: Based on the segmentation index corresponding to the target action, the target object in the motion video is analyzed to obtain the information of the segmentation index at multiple moments in the motion video; Based on the segmentation moments at which the information of the segmentation indicators in the multiple moments satisfy the segmentation conditions, the motion video is divided into multiple video segments, and the target image frame in the motion video is determined based on the multiple video segments; wherein one video segment corresponds to the process of the target object performing a target action, and the target image frame is the image frame corresponding to the extreme value information of the segmentation indicator in the video segment; Based on the action index corresponding to the target action, the target object in the video segment and the target image frame is analyzed to obtain the target information of the action index.

[0086] Optionally, the action index corresponding to the target action includes a static index and / or a dynamic index; the first analysis module 904 is used to: Based on the static indicators in the action indicators corresponding to the target action, the target object in the target image frame is analyzed to obtain the target information of the static indicators; Based on the dynamic index in the action index corresponding to the target action, the target object in the video segment is analyzed to obtain the target information of the dynamic index.

[0087] Optionally, the video analysis device further includes: The second analysis module is used to analyze the target object in the motion video based on the segment selection index corresponding to the target action before analyzing the video segment and the target image frame based on the action index corresponding to the target action, so as to obtain information on the segment selection index at multiple moments in the motion video; A second determination module is used to determine a target video segment whose information of the segment selection index satisfies the segment selection condition among the multiple video segments; A third determination module is used to determine a target image frame in a target video segment; The first analysis module 904 is used for: Based on the action index corresponding to the target action, the target object in the target video segment and the target image frame is analyzed.

[0088] Optionally, the first analysis module 904 is used to: Based on a first action indicator corresponding to the target action, a target three-dimensional skeleton point of a target object in a motion video is detected, and based on information of the target three-dimensional skeleton point, target information of the first action indicator is determined; wherein the first action indicator is related to the depth of the target object in the motion video; Based on a second action indicator corresponding to the target action, an area covered by the target object is segmented from the motion video, and target information of the second action indicator is determined based on information of the area; wherein the second action indicator is related to a contour of the target object.

[0089] Optionally, the first analysis module 904 is used to: Determine the orientation of target objects in motion videos; Based on the action index corresponding to the target action in the direction, the target object in the motion video is analyzed to obtain target information of the action index.

[0090] Optionally, the video analysis device further includes: The third analysis module is used to compare and analyze the analysis results of multiple motion videos to determine additional analysis results of the motion videos; wherein the multiple motion videos are videos of the target object performing the target action, and the multiple motion videos are generated at multiple moments respectively.

[0091] Optionally, the video analysis device further includes: A processing module, used for performing frame extraction processing on the motion video to determine a plurality of frames of images to be analyzed in the motion video before analyzing the target object in the motion video based on the action index corresponding to the target action; A detection module, used for identifying a target object in multiple frames of images and detecting multiple skeleton points of the target object in each frame of images; The first analysis module 904 is used to analyze the target object in the motion video based on the action index corresponding to the target action and multiple skeleton points of the target object in each frame image to obtain information on the action index.

[0092] Optionally, the acquisition module 902 is used to: Obtaining a motion video to be analyzed with a resolution lower than a target threshold; The first analysis module 904 is used for: Based on the action index corresponding to the target action, the target object in the motion video is analyzed using the target analysis module to obtain the target information of the action index; wherein the target analysis module is adjusted according to the auxiliary analysis model, and the auxiliary analysis model is used to analyze the motion video with a resolution higher than the target threshold.

[0093] Optionally, the first determining module 906 is used to: Determining video content to be displayed in the motion video, wherein the video content includes a target image frame and / or a target video segment in the motion video; Perform resolution enhancement on video content; Based on the target information, the baseline information of the motion index, and the video content after the resolution enhancement processing, the analysis result of the motion video is obtained.

[0094] Optionally, the acquisition module 902 is used to: receive a motion video to be analyzed sent by a video acquisition device; The video analysis device also includes: The sending module is used to send the analysis result to the video acquisition device after determining the analysis result of the motion video based on the target information and the benchmark information of the action index.

[0095] In summary, in the video analysis device provided in the embodiments of this specification, a corresponding action index can be set for the target action, and the action index can be quantified into at least one physical quantity, and benchmark information of each action index is also provided, that is, the information of the action index under the standard target action. For the motion video to be analyzed, the target information of these action indicators can be determined for the target object therein, and the target information of each action indicator can be compared with the benchmark information to determine whether the execution of the target action by the target object meets the requirements, and the analysis result of the motion video can be obtained. In this way, video analysis can be achieved by determining the information of each physical quantity related to the target object in the motion video. The determined information can better represent the state of the target object, without the need for complex processing of the video, and can improve the efficiency and accuracy of video analysis.

[0096] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. As for the display device of the target image, since it is basically similar to the embodiment of the display method of the target image, the description is relatively simple, and the relevant parts can be referred to the partial description of the embodiment of the display method of the target image.

[0097] Fig.104 is a block diagram of a computing device provided in an embodiment of the present specification. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0098] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0099] In one embodiment of the present specification, the above components of the computing device 400 and Fig.10 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.10 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0100] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 400 may also be a mobile or stationary server.

[0101] The processor 420 is used to execute computer programs / instructions, which, when executed by the processor, implement the above Figure 1 or Figure 8 The method shown.

[0102] As for the computing device embodiment, since it is basically similar to the above-mentioned method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0103] One embodiment of the present specification also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the steps of the above-mentioned image processing method are implemented. The computer instructions include computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0104] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above method when the computer program / instruction is executed in a processor.

[0105] As for the computer-readable storage medium embodiment and the computer program product embodiment, since they are basically similar to the above-mentioned method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0106] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0107] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0108] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A video analysis method, characterized in that: The method comprises: Acquire a motion video to be analyzed, wherein the motion video is a video of a target object performing a target action; Based on the action index corresponding to the target action, the target object in the motion video is analyzed to obtain target information of the action index; wherein the action index is represented by at least one physical quantity related to the target object; Based on the target information and the reference information of the action index, an analysis result of the motion video is determined.

2. The method according to claim 1, characterized in that The step of analyzing the target object in the motion video based on the action indicator corresponding to the target action to obtain target information of the action indicator includes: Based on the segmentation index corresponding to the target action, analyzing the target object in the motion video to obtain information of the segmentation index at multiple moments in the motion video; Based on the segmentation moments at which the information of the segmentation index in the multiple moments meets the segmentation condition, the motion video is divided into multiple video segments, and the target image frame in the motion video is determined based on the multiple video segments; wherein one video segment corresponds to a process in which the target object performs the target action once, and the target image frame is an image frame corresponding to the extreme value information of the segmentation index in the video segment; Based on the action index corresponding to the target action, the target object in the video segment and the target image frame is analyzed to obtain target information of the action index.

3. The method according to claim 2, characterized in that The action indicator corresponding to the target action includes a static indicator and / or a dynamic indicator; the target object in the video segment and the target image frame is analyzed based on the action indicator corresponding to the target action to obtain target information of the action indicator, including: Based on the static indicators in the action indicators corresponding to the target action, analyzing the target object in the target image frame to obtain target information of the static indicators; Based on the dynamic index in the action index corresponding to the target action, the target object in the video segment is analyzed to obtain target information of the dynamic index.

4. The method according to claim 2, characterized in that: The determining the target image frame in the motion video based on the multiple video segments comprises: Based on the segment selection index corresponding to the target action, analyzing the target object in the motion video to obtain information of the segment selection index at multiple moments in the motion video; Determining, from among the plurality of video segments, a target video segment in which information of the segment selection index satisfies a segment selection condition; Determining a target image frame in the target video segment; The analyzing the target object in the video segment and the target image frame based on the action index corresponding to the target action includes: Based on the action index corresponding to the target action, the target object in the target video segment and the target image frame is analyzed.

5. The method according to any one of claims 1 to 4, characterized in that: The step of analyzing the target object in the motion video based on the action indicator corresponding to the target action to obtain target information of the action indicator includes: Based on a first action indicator corresponding to the target action, detecting a target three-dimensional skeleton point of the target object in the motion video, and determining target information of the first action indicator based on information of the target three-dimensional skeleton point; wherein the first action indicator is related to the depth of the target object in the motion video; Based on a second action indicator corresponding to the target action, the area covered by the target object is segmented from the motion video, and based on information of the area, target information of the second action indicator is determined; wherein the second action indicator is related to the outline of the target object.

6. The method according to any one of claims 1 to 4, characterized in that: The step of analyzing the target object in the motion video based on the action indicator corresponding to the target action to obtain target information of the action indicator includes: Determining the orientation of the target object in the motion video; Based on the action index corresponding to the target action in the direction, the target object in the motion video is analyzed to obtain target information of the action index.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: A comparative analysis is performed on the analysis results of the plurality of motion videos to determine an additional analysis result of the motion video; wherein the plurality of motion videos are all videos of the target object performing the target action, and the plurality of motion videos are generated at a plurality of moments respectively.

8. The method according to any one of claims 1 to 4, characterized in that: Before analyzing the target object in the motion video based on the action index corresponding to the target action, the method further includes: Performing frame extraction processing on the motion video to determine a plurality of frames of images to be analyzed in the motion video; Identify the target object in the multiple frames of image, and detect multiple skeleton points of the target object in each frame of image; The analyzing the target object in the motion video based on the action index corresponding to the target action includes: Based on the action index corresponding to the target action and the multiple skeleton points of the target object in each frame of the image, the target object in the motion video is analyzed.

9. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining the motion video to be analyzed includes: Obtaining a motion video to be analyzed with a resolution lower than a target threshold; The step of analyzing the target object in the motion video based on the action indicator corresponding to the target action to obtain target information of the action indicator includes: Based on the action index corresponding to the target action, a target analysis module is used to analyze the target object in the motion video to obtain target information of the action index; wherein the target analysis module is adjusted to obtain an auxiliary analysis model, and the auxiliary analysis model is used to analyze motion videos with a resolution higher than the target threshold.

10. The method according to any one of claims 1 to 4, characterized in that: The step of determining the analysis result of the motion video based on the target information and the reference information of the action index includes: Determining video content to be displayed in the motion video, wherein the video content includes a target image frame and / or a target video segment in the motion video; Performing resolution enhancement processing on the video content; Based on the target information, the reference information of the action index, and the video content after resolution enhancement processing, an analysis result of the motion video is obtained.

11. The method according to any one of claims 1 to 4, characterized in that: Applied to the server, the step of obtaining the motion video to be analyzed includes: Receiving a motion video to be analyzed sent by a video acquisition device; After determining the analysis result of the motion video based on the target information and the benchmark information of the action index, the method further includes: The analysis result is sent to the video acquisition device.

12. A video analysis device, characterized in that: The video analysis device comprises: An acquisition module, used for acquiring a motion video to be analyzed, wherein the motion video is a video of a target object performing a target action; A first analysis module is used to analyze the target object in the motion video based on the action index corresponding to the target action to obtain target information of the action index; wherein the action index is represented by at least one physical quantity related to the target object; The first determination module is used to determine the analysis result of the motion video based on the target information and the benchmark information of the action index.

13. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that: A computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the method according to any one of claims 1 to 11 when executed by a processor.

Images