Motion measurement method, device, storage medium and electronic device in video

Abstract

The present disclosure provides a method, device, computer-readable storage medium and electronic device for measuring motion in a video, and relates to the field of computer vision technology. The method for measuring motion in a video comprises: acquiring a target video; detecting key points of a human body in the target video, and determining a preset part of a human body in the target video according to the key points of the human body; determining at least two types of key frames corresponding to at least two reference states of a preset action in the target video based on the preset part of the human body in the target video; and counting and / or timing the preset action in the target video according to the at least two types of key frames. The present disclosure improves the accuracy of motion measurement.

Description

Background Art

[0002] As exercise becomes more popular in people's daily lives, more and more people use electronic means to assist in exercise, such as identifying people's movements through electronic devices, and then evaluating or guiding people's exercise process. Among them, exercise measurement is an important part of realizing the electronic exercise assistance function, including counting and timing people's movements.

[0003] In the related art, people are usually required to wear specific equipment (such as sports bracelets) when exercising, and use inertial sensors (or other sensors with similar principles) on the equipment to detect people's movements, thereby realizing motion measurement. However, there are many cases of misidentification when detecting movements through inertial sensors, resulting in low accuracy of motion measurement.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those skilled in the art. Summary of the invention

[0005] The present disclosure provides a method for measuring motion in a video, a device for measuring motion in a video, a computer-readable storage medium and an electronic device, thereby at least to a certain extent solving the problem of low accuracy of motion measurement in the related art.

[0006] According to a first aspect of the present disclosure, a method for measuring motion in a video is provided, comprising: acquiring a target video; detecting key points of a human body in the target video, and determining preset parts of a human body in the target video according to the key points of the human body; determining at least two types of key frames corresponding to at least two reference states of a preset action in the target video based on the preset parts of the human body in the target video; and counting and / or timing the preset action in the target video according to the at least two types of key frames.

[0007] According to a second aspect of the present disclosure, a motion measurement device in a video is provided, comprising: a video acquisition module, configured to acquire a target video; a human body detection module, configured to detect human body key points in the target video, and determine a preset human body part in the target video according to the human body key points; a key frame determination module, configured to determine at least two types of key frames corresponding to at least two reference states of a preset action in the target video based on the preset human body part in the target video; and a key frame processing module, configured to count and / or time the preset action in the target video according to the at least two types of key frames.

[0008] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for measuring motion in a video according to the first aspect and possible implementation methods thereof are implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the motion measurement method in the video of the above-mentioned first aspect and its possible implementation method by executing the executable instructions.

[0010] The technical solution disclosed in this disclosure has the following beneficial effects:

[0011] By comparing the preset parts of the human body with the baseline states of the preset actions, the key frames corresponding to the baseline states in the target video are obtained, thereby detecting the key time nodes when the user performs the preset actions to measure the preset actions. On the one hand, the preset parts of the human body include the body parts related to the preset actions, so that the comparison can be made as a whole, rather than just the detection of a single part, thereby improving the accuracy of key frame detection and motion measurement. On the other hand, this solution achieves a high degree of automation and reduces manual participation. On the other hand, this solution does not require users to wear equipment, which reduces hardware costs, reduces restrictions on user movements, and improves user experience.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0014] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment;

[0015] Figure 2 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0016] Figure 3 A flow chart showing a method for measuring motion in a video according to the exemplary embodiment is shown;

[0017] Figure 4 A schematic diagram showing key points of a human body in this exemplary embodiment;

[0018] Figure 5 A flowchart showing key frames corresponding to the initial state and the progress state in this exemplary embodiment is shown;

[0019] Figure 6 A flowchart showing key frames corresponding to a starting state and an ending state in this exemplary embodiment is shown;

[0020] Figure 7 A flow chart showing counting and timing of preset actions in this exemplary embodiment is shown;

[0021] Figure 8 A schematic structural diagram of a motion measurement device in a video according to this exemplary embodiment is shown. DETAILED DESCRIPTION

[0022] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0023] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0024] In the related art, people are usually required to wear specific equipment when exercising, such as wearing a sports bracelet on the wrist, wearing smart glasses on the head, etc., to detect people's movements through the changes in the posture of the equipment, and then realize sports measurement. However, the above equipment is only worn on a part of the body of a person and can only detect the movement of this part. For example, if a sports bracelet is worn on the left wrist and exercises, the sports bracelet can only detect the movement of the left hand. Obviously, it is difficult to accurately identify a person's whole body movements, and it is easy to misidentify, which affects the accuracy of sports measurement.

[0025] In view of the above problems, the exemplary embodiments of the present disclosure first provide a method for measuring motion in a video. Figure 1 The system architecture and application scenarios of the operating environment of this exemplary embodiment are exemplarily described.

[0026] Figure 1 1 shows a schematic diagram of a system architecture, which may include a video acquisition device 110 and a processing device 120. The video acquisition device 110 may be a camera, which is used to acquire a video of a user's motion (i.e., a target video); the processing device 120 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, etc., which is used to execute the method for measuring motion in a video in this exemplary embodiment to measure motion in the target video.

[0027] Figure 1 The video acquisition device 110 and the processing device 120 are shown as two independently arranged devices, which can be connected via a wired or wireless communication link to perform data exchange. For example, the video acquisition device 110 sends a target video to the processing device 120 via the communication link.

[0028] In one implementation, the video acquisition device 110 and the processing device 120 may also be integrated into one device. For example, a mobile phone has a built-in camera module, such as a camera module that may include a camera, and the camera module is equivalent to the video acquisition device 110; the mobile phone can capture the target video and execute the motion measurement method in the video to obtain the motion measurement result; and then the motion measurement result can be displayed to the user for reference.

[0029] The exemplary embodiments of the present disclosure also provide an electronic device for executing the above-mentioned motion measurement method in the video, and the electronic device may be the above-mentioned processing device 120. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned motion measurement method in the video by executing the executable instructions.

[0030] Below Figure 2The mobile terminal 200 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The construction in can also be applied to fixed type equipment.

[0031] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 201, a memory 202, a bus 203, a mobile communication module 204, an antenna 1, a wireless communication module 205, an antenna 2, a display screen 206, a sensor module 207, an audio module 208, a power module 209 and a camera module 210.

[0032] Processor 201 may include one or more processing units, for example: processor 210 may include AP (Application Processor), modem processor, GPU (Graphics Processing Unit), ISP (Image Signal Processor), controller, encoder, decoder, DSP (Digital Signal Processor), baseband processor and / or NPU (Neural-Network Processing Unit), etc. In this exemplary embodiment, the motion measurement method in the video may be executed by AP, GPU or DSP, etc. When the method involves neural network related processing, it may be executed by NPU, for example, NPU may load neural network parameters and execute neural network related algorithm instructions.

[0033] The encoder can encode (i.e. compress) an image or video to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data. The mobile terminal 200 can support one or more encoders and decoders. In this way, the mobile terminal 200 can process images or videos in a variety of encoding formats, such as: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) and other image formats, MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding) and other video formats.

[0034] The processor 201 may be connected to the memory 202 or other components via a bus 203 .

[0035] The memory 202 may be used to store computer executable program codes, which include instructions. The processor 201 executes various functional applications and data processing of the mobile terminal 200 by running the instructions stored in the memory 202. The memory 202 may also store application data, such as images, videos and other files.

[0036] The communication function of the mobile terminal 200 can be realized by the mobile communication module 204, the antenna 1, the wireless communication module 205, the antenna 2, the modulation and demodulation processor, and the baseband processor. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 204 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 200. The wireless communication module 205 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 200.

[0037] The display screen 206 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The sensor module 207 is used to implement various sensing detection functions, such as pressure sensors, gyroscope sensors, etc., to detect touch pressure and device posture. The audio module 208 is used to implement audio functions, such as playing audio and collecting voice. The power module 209 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc.

[0038] The camera module 210 may include components such as a lens 2101, an image sensor 2102, and an optical image stabilization system 2103. The lens 2101 is used for imaging and adjusting the optical path; the image sensor 2102 is used to collect optical signals and convert them into digital signals, which can be further transmitted to the ISP for processing; the optical image stabilization system 2103 is used to stabilize the lens 2101 to prevent imaging jitter and improve image quality. The camera module 210 can be used to capture target video.

[0039] Combine the following Figure 3 The method of measuring motion in a video according to the exemplary embodiment is described below. Figure 3 An exemplary process of the motion measurement method in the video is shown, which may include:

[0040] Step S310, obtaining a target video;

[0041] Step S320, detecting key points of the human body in the target video, and determining preset parts of the human body in the target video according to the key points of the human body;

[0042] Step S330, based on the preset human body part in the target video, determining at least two types of key frames corresponding to at least two reference states of the preset action in the target video;

[0043] Step S340: counting and / or timing the preset actions in the target video according to the at least two types of key frames.

[0044] Based on the above method, by comparing the preset parts of the human body with the baseline states of the preset actions, the key frames corresponding to the baseline states in the target video are obtained, thereby detecting the key time nodes when the user performs the preset actions to measure the preset actions. On the one hand, the preset parts of the human body include the body parts related to the preset actions, so that the comparison can be made as a whole, rather than just the detection of a single part, thereby improving the accuracy of key frame detection and motion measurement. On the other hand, this solution achieves a high degree of automation and reduces manual participation. On the other hand, this solution does not require users to wear equipment, which reduces hardware costs, reduces restrictions on user movements, and improves user experience.

[0045] Below Figure 3 Each step in the procedure is described in detail.

[0046] refer to Figure 3 , in step S310, the target video is obtained.

[0047] The target video refers to a video that needs to be measured for motion measurement, which is generally a video of the user exercising, such as the user's fitness video, dance video, or physical fitness test video.

[0048] The present disclosure does not limit the method for acquiring the target video. For example, the target video may be shot in real time, or a shot target video may be read from local storage, or a video stream of the target video may be received from the cloud.

[0049] Continue to refer Figure 3 In step S320, human body key points are detected in the target video, and preset human body parts in the target video are determined according to the human body key points.

[0050] The key points of the human body are the bone points that can represent the important parts of the human body. The present disclosure does not limit the number and types of key points of the human body. For example, Figure 4 As shown, 17 human body key points can be used, and the serial numbers are set to 0 to 16 respectively. The specific meaning of each human body key point can be referred to as shown in Table 1.

[0051] Table 1

[0052]

[0053] In this exemplary embodiment, human key points can be detected in image frames in the target video. The present disclosure does not limit the specific detection algorithm. For example, human key point detection can be performed in a top-down manner. Exemplarily, a simple baseline method can be used to collect relevant scene samples based on the algorithm, annotate and train a human key point recognition network model. The specific processing process can include training of the human key point recognition network model and feature extraction of the border area, human key point regression, human key point classification, data post-processing, and obtaining human key point recognition results.

[0054] In one implementation, the image frames in the target video may be screened to select frames containing human bodies. For example, a lightweight classification neural network may be pre-trained to classify whether a human body exists in an image, and the classification neural network may be used to process each frame of the target video to select frames containing human bodies, and then detect key points of human bodies in these frames.

[0055] In one embodiment, the image frames in the target video may be firstly detected for objects of interest, such as human body objects. Specifically, a single detection algorithm, such as the SSD (Single Shot MultiBoxDetector) algorithm, may be used to perform border selection, border normalization, feature extraction, border regression, classifier classification, and data post-processing on each frame of the target video to obtain a border region including the human body, wherein the feature extraction part may be obtained by convolution calculation using a trained convolutional neural network. Then, an image of the border region of the human body is cut out, and the image is subjected to detection of key points of the human body.

[0056] The preset human body parts are parts of the human body related to sports, and are usually formed by multiple human body key points. This exemplary embodiment can determine the preset human body parts according to the specific sports scene. For example, when running and jumping, head movements are not considered, and three preset human body parts of legs, hands, and upper body can be set. Among them, the legs include the left leg and the right leg, that is, the human body key points 0-1-2-3 and 0-4-5-6; the hands include the left arm and the right arm, that is, the human body key points 8-14-15-16 and 8-11-12-13; the upper body includes the part from the buttocks to the neck, that is, the human body key points 0-7-8-9.

[0057] In one embodiment, the preset human body part may include one or more limbs. A limb refers to a body part formed by connecting two adjacent human body key points. For example, according to the settings of the preset human body parts of the legs, hands, and upper body, the legs may include limbs 0-1 (right hip), 1-2 (right thigh), 2-3 (right calf), 0-4 (left hip), 4-5 (left thigh), 5-6 (left calf), the hands may include limbs 8-14 (right shoulder), 14-15 (right upper arm), 15-16 (right lower arm), 8-11 (left shoulder), 11-12 (left upper arm), 12-13 (left lower arm), and the upper body may include limbs 0-7 (waist), 7-8 (chest), 8-9 (neck).

[0058] After the key points of the human body are identified in the target video, the preset human body parts in the target video can be determined by connecting the key points of the human body. The preset human body parts can be determined for each frame of the target video, or a certain frame skipping strategy can be adopted, such as selecting one frame from every two or three consecutive frames to determine the preset human body parts in the frame image.

[0059] Continue to refer Figure 3 In step S330, based on the preset human body part in the target video, at least two types of key frames corresponding to at least two reference states of the preset action are determined in the target video.

[0060] A preset action refers to an action to be measured. For example, if two actions, jumping jacks and side leg raises, need to be measured in the target video, then jumping jacks and side leg raises are preset actions. For each preset action, at least two reference states can be determined. The reference state refers to the representative physical state of the user when doing the preset action. After determining the preset part of the human body in each frame or part of the frame of the target video, the state of the preset part of the human body can be compared with the reference state of the preset action. If the state of the preset part of the human body meets the reference state, the corresponding frame is determined as a key frame. It should be noted that for each reference state, one or more key frames corresponding thereto can be determined, and key frames corresponding to different reference states are recorded as different types of key frames. For example, if a preset action has two reference states, two types of key frames are obtained accordingly, and two different labels can be used to distinguish and mark the two types of key frames.

[0061] In one embodiment, the at least two reference states of the preset action include an initial state of the preset action and at least one ongoing state of the preset action. The initial state refers to the physical state of the user when the user starts to perform the preset action, and the ongoing state refers to the physical state of the user during the preset action. A preset action can have multiple ongoing states. Figure 5As shown, the above-mentioned determining at least two types of key frames corresponding to at least two reference states of the preset action in the target video based on the preset human body part may include the following steps S510 and S520:

[0062] Step S510, when it is determined that the state of the preset part of the human body in the current frame of the target video meets the initial state, the current frame is determined as a key frame corresponding to the initial state;

[0063] Step S520, when it is determined that the state of the preset part of the human body in the current frame satisfies the ongoing state, the current frame is determined as a key frame corresponding to the ongoing state.

[0064] The target video may be processed frame by frame, and the current frame refers to the frame currently being processed. In step S320, the position information of the preset part of the human body in the current frame may be obtained. Further, the state of the preset part of the human body may be determined according to the position information of the preset part of the human body to determine whether it satisfies the initial state or the ongoing state of the preset action.

[0065] In one embodiment, the preset human body part may include one or more limbs. The above-mentioned determination that the state of the preset human body part in the current frame of the target video satisfies the initial state may include the following steps:

[0066] It is determined whether the positional relationship of the limbs in the current frame satisfies the positional relationship corresponding to the initial state. If so, it is determined that the state of the preset part of the human body in the current frame satisfies the initial state.

[0067] Accordingly, the above-mentioned determination that the state of the preset part of the human body in the current frame satisfies the ongoing state may include the following steps:

[0068] It is determined whether the positional relationship of the limbs in the current frame satisfies the positional relationship corresponding to the proceeding state. If so, it is determined that the state of the preset part of the human body in the current frame satisfies the proceeding state.

[0069] Generally, for each preset human body part, several states can be set, and each state requires that the limbs of the preset human body part have a specific position relationship. For example, the hand can include the following states:

[0070] Vertical: The distance between the left and right arms is the same as the width of the shoulders, and the wrist, elbow and shoulder are in a straight line;

[0071] Unfold: There is an angle between the left arm and the right arm, and the wrist, elbow and shoulder are in a straight line;

[0072] Inward bend: the hand and shoulder are on the same side of the elbow;

[0073] Outward bend: The hand and shoulder are on opposite sides of the elbow.

[0074] For the initial state or the ongoing state of the preset action, the position relationship of the limbs in each preset part of the human body can be set accordingly. For example, the initial state of jumping jacks includes: the hands are spread, the legs are vertical, and the upper body is upright. The ongoing state of jumping jacks includes: the hands are vertical, the legs are spread, and the upper body is upright. Therefore, by comparing the position relationship of the limbs in the current frame with the position relationship corresponding to the initial state or the ongoing state of the preset action, it is determined whether the preset part of the human body in the current frame meets the above initial state or the ongoing state, so as to determine the key frame corresponding to the initial state or the key frame corresponding to the ongoing state.

[0075] In one embodiment, the at least two reference states of the preset action also include a starting state and an ending state of the preset action. The starting state refers to the physical state of the user when entering the preset action, and the ending state refers to the physical state of the user when exiting the preset action. Figure 6 As shown, the above-mentioned determining at least two types of key frames corresponding to at least two reference states of the preset action in the target video based on the preset human body part in the target video may also include the following steps S610 to S630:

[0076] Step S610, determining the posture parameters of the preset part of the human body in the current frame;

[0077] Step S620, when the posture parameters of the preset part of the human body in the current frame meet the parameter conditions corresponding to the starting state, and the posture parameters of the preset part of the human body in the previous frame of the current frame do not meet the parameter conditions corresponding to the starting state, the current frame is determined as a key frame corresponding to the starting state;

[0078] Step S630, when the posture parameters of the preset part of the human body in the current frame meet the parameter conditions corresponding to the end state, and the posture parameters of the preset part of the human body in the previous frame of the current frame do not meet the parameter conditions corresponding to the end state, the current frame is determined as the key frame corresponding to the end state.

[0079] The posture parameters of the preset human body parts include the position or angle parameters of the preset human body parts. For example, the posture parameters of the legs may include the angle between the two legs and the angles at the knees on the left and right sides, the posture parameters of the hands may include the angles at the armpits on the left and right sides and the angles at the wrists, and the posture parameters of the upper body may include the angle between the upper body and the horizontal direction.

[0080] In one embodiment, the preset human body part includes one or more limbs. The above-mentioned determination of the posture parameters of the preset human body part in the current frame may include the following steps:

[0081] The posture parameters of the preset parts of the human body in the current frame are determined according to the angle information of the limbs in the current frame.

[0082] The angle information of the limbs may include the angle between different limbs, or the angle of a limb relative to the horizontal direction or the vertical direction. The angle between different limbs may be used as the posture parameter of the preset part of the human body, or the relative position parameters between different limbs may be calculated to be used as the posture parameter of the preset part of the human body.

[0083] For the starting state or the ending state of the preset action, parameter conditions can be set accordingly to indicate the range of the posture parameters of the preset part of the human body that should be in when the user is in the starting state or the ending state of the preset action. For example, for the starting state or the ending state of the jumping jack, parameter conditions can be set for the angle between the legs of the legs, and parameter conditions can be set for the angle between the left armpit and the right armpit of the hands. If the posture parameters of the preset part of the human body in the current frame meet the parameter conditions corresponding to the starting state, and the posture parameters of the preset part of the human body in the previous frame of the current frame do not meet the parameter conditions corresponding to the starting state, it means that the user did not perform the preset action in the previous frame, and started to perform the preset action in the current frame. The current frame can be regarded as the starting frame of the preset action, so the current frame is determined as the key frame corresponding to the starting state. The posture parameters of the preset part of the human body in the current frame meet the parameter conditions corresponding to the end state, and the posture parameters of the preset part of the human body in the previous frame of the current frame do not meet the parameter conditions corresponding to the end state, which means that the user was doing the preset action in the previous frame and ended the preset action in the current frame. The current frame can be regarded as the end frame of the preset action, so the current frame is determined as the key frame corresponding to the end state.

[0084] In one embodiment, the method for measuring motion in a video may further include the following steps:

[0085] According to the state of a preset part of a human body in a target video, a motion start frame is determined in the target video.

[0086] Exemplarily, starting from the first frame of the target video, the state of the preset part of the human body in each frame can be detected frame by frame. If the preset part of the human body is static or dynamic, it can be judged by the change of the preset part of the human body in the current frame relative to the previous frame. If the degree of change is greater than a set threshold (the threshold can be determined based on the resolution of the target video and experience), the preset part of the human body is determined to be dynamic. When it is detected that the preset parts of the human body are all dynamic, the current frame is determined as the motion start frame. In one embodiment, different motion start frames can be determined for different preset actions. The motion start frame indicates that the user has finished preparation and started formal exercise. Further, in step S330, a key frame can be determined from the frame after the motion start frame in the target video. In this way, the preparation stage for the start of the movement can be eliminated, further improving the accuracy of motion measurement.

[0087] In one embodiment, when determining the state of a preset human body part in the current frame, the state of one or more frames in the previous n frames (n can be any positive integer) of the current frame can be combined to determine the change of state or the movement direction of the preset human body part. For example, the movement direction of the hips is determined based on the position of the hips in the current frame relative to the position of the hips in the previous frame, and the judgment of the movement direction is added on the basis of the state judgment, thereby improving the interference caused by the movement direction and the conflict problem when the action state is the same.

[0088] Table 2 shows the states or parameter conditions of the preset human body parts corresponding to the motion start frame, initial state, ongoing state, starting state, and ending state of the jumping jack action. Table 3 shows the states or parameter conditions of the preset human body parts corresponding to the motion start frame, initial state, ongoing state, starting state, and ending state of the side leg lift action. By comparing the preset human body parts in the target video with the states or parameter conditions in the table, the motion start frame and key frame corresponding to the jumping jack action, or the motion start frame and key frame corresponding to the side leg lift action can be detected from the target video.

[0089] Table 2

[0090]

[0091] Table 3

[0092]

[0093] Continue to refer Figure 3 In step S340, the preset actions in the target video are counted and / or timed according to the at least two types of key frames mentioned above.

[0094] The key frame represents a key time node when the user performs a preset action, thereby measuring the preset action, including at least one of counting and timing the preset action.

[0095] In one embodiment, reference Figure 7 As shown, the counting and / or timing of the preset actions in the target video according to at least two types of key frames may include the following steps S710 and S720:

[0096] Step S710, counting the preset actions in the target video according to the number of key frames corresponding to the initial state and the number of key frames corresponding to the ongoing state;

[0097] Step S720, timing the preset action in the target video according to the time interval between the key frame corresponding to the start state and the key frame corresponding to the end state.

[0098] Generally, the key frame corresponding to the initial state of a preset action and the key frame corresponding to the entry state of a preset action appearing in the target video indicate that the user has performed a preset action. Therefore, the key frames corresponding to the initial state and the key frames corresponding to the progress state can be combined in pairs, and each combination represents a preset action, thereby counting the preset actions.

[0099] The time interval between the key frame corresponding to the starting state and the key frame corresponding to the ending state, that is, the duration of the user's preset action. Therefore, the key frames corresponding to the starting state and the key frames corresponding to the ending state can be combined in pairs, and the time interval between the two frames in each combination can be counted, so as to time the preset action in the target video. During timing, in order to avoid the situation where the action stops, monitoring can be performed based on the motion count. For example, within the above time interval, if the number of preset actions is less than the set threshold (which can be determined based on specific actions and experience), it is considered that the action is paused and no timing is performed. Alternatively, the length of the time interval is limited. If the time interval exceeds the set duration (which can be determined based on specific actions and experience), it is considered that the action is paused and no timing is performed.

[0100] The exemplary embodiments of the present disclosure also provide a motion measurement device in a video. Figure 8 As shown, the motion measurement device 800 in the video may include:

[0101] The video acquisition module 810 is configured to acquire a target video;

[0102] The human body detection module 820 is configured to detect key points of the human body in the target video, and determine the preset parts of the human body in the target video according to the key points of the human body;

[0103] The key frame determination module 830 is configured to determine at least two types of key frames corresponding to at least two reference states of the preset action in the target video based on the preset human body part in the target video;

[0104] The key frame processing module 840 is configured to count and / or time the preset actions in the target video according to the at least two types of key frames mentioned above.

[0105] In one embodiment, the at least two reference states of the preset action include an initial state of the preset action and at least one ongoing state of the preset action. The at least two types of key frames corresponding to the at least two reference states of the preset action in the target video based on the preset human body part may include:

[0106] When it is determined that the state of the preset part of the human body in the current frame of the target video meets the initial state, the current frame is determined as a key frame corresponding to the initial state;

[0107] When it is determined that the state of the preset part of the human body in the current frame satisfies the ongoing state, the current frame is determined as a key frame corresponding to the ongoing state.

[0108] In one embodiment, the preset human body part includes one or more limbs. The above-mentioned determination that the state of the preset human body part in the current frame of the target video satisfies the initial state may include:

[0109] Determine whether the position relationship of the limbs in the current frame satisfies the position relationship corresponding to the initial state, and if so, determine whether the state of the preset part of the human body in the current frame satisfies the initial state;

[0110] Determining whether the state of the preset part of the human body in the current frame satisfies the ongoing state includes:

[0111] It is determined whether the positional relationship of the limbs in the current frame satisfies the positional relationship corresponding to the proceeding state. If so, it is determined that the state of the preset part of the human body in the current frame satisfies the proceeding state.

[0112] In one embodiment, the at least two reference states of the preset action further include a starting state and an ending state of the preset action. The above-mentioned determining at least two types of key frames corresponding to the at least two reference states of the preset action in the target video based on the preset human body part in the target video may also include:

[0113] Determine the posture parameters of the preset part of the human body in the current frame;

[0114] When the posture parameters of the preset part of the human body in the current frame meet the parameter conditions corresponding to the starting state, and the posture parameters of the preset part of the human body in the previous frame of the current frame do not meet the parameter conditions corresponding to the starting state, the current frame is determined as the key frame corresponding to the starting state;

[0115] When the posture parameters of the preset part of the human body in the current frame meet the parameter conditions corresponding to the end state, and the posture parameters of the preset part of the human body in the previous frame of the current frame do not meet the parameter conditions corresponding to the end state, the current frame is determined as the key frame corresponding to the end state.

[0116] In one embodiment, the preset human body part includes one or more limbs. The above-mentioned determination of the posture parameters of the preset human body part in the current frame may include:

[0117] The posture parameters of the preset parts of the human body in the current frame are determined according to the angle information of the limbs in the current frame.

[0118] In one implementation, counting and / or timing the preset actions in the target video according to at least two types of key frames may include:

[0119] Counting the preset actions in the target video according to the number of key frames corresponding to the initial state and the number of key frames corresponding to the ongoing state;

[0120] The preset action in the target video is timed according to the time interval between the key frame corresponding to the start state and the key frame corresponding to the end state.

[0121] In one implementation, the key frame determination module 830 is configured to:

[0122] Determine a motion start frame in the target video according to the state of a preset part of the human body in the target video;

[0123] The at least two types of key frames mentioned above are located after the motion start frame.

[0124] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0125] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute (more specifically, to enable the processor of the electronic device to execute) the steps of the various exemplary embodiments described in the above "Exemplary Method" section of this specification.

[0126] In one embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0127] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium includes, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0128] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0129] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0130] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0131] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0132] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0133] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for measuring motion in a video, characterized in that: include: Get the target video; Detecting key points of a human body in the target video, and determining preset parts of a human body in the target video according to the key points of the human body; Based on a preset human body part in the target video, determining at least two types of key frames corresponding to at least two reference states of a preset action in the target video; Counting and / or timing the preset action in the target video according to the at least two types of key frames; The at least two reference states of the preset action include: a starting state, an initial state, at least one ongoing state, and an end state of the preset action; the starting state refers to the physical state of the user when entering the preset action, the initial state refers to the physical state of the user when starting to do the preset action, the ongoing state refers to the physical state of the user when doing the preset action, and the end state refers to the physical state of the user when exiting the preset action; The counting and / or timing of the preset action in the target video according to the at least two types of key frames includes: counting the preset action in the target video according to the number of key frames corresponding to the initial state and the number of key frames corresponding to the ongoing state; timing the preset action in the target video according to the time interval between the key frame corresponding to the starting state and the key frame corresponding to the ending state; Among them, the timing of the preset action in the target video according to the time interval between the key frame corresponding to the starting state and the key frame corresponding to the ending state includes: within the time interval, if the number of the preset actions is less than a set threshold, the preset action is not timed.

2. The method according to claim 1, characterized in that: The determining, based on the preset human body part, at least two types of key frames corresponding to at least two reference states of the preset action in the target video respectively comprises: When it is determined that the state of the preset part of the human body in the current frame of the target video satisfies the initial state, determining the current frame as a key frame corresponding to the initial state; When it is determined that the state of the preset part of the human body in the current frame satisfies the ongoing state, the current frame is determined as a key frame corresponding to the ongoing state.

3. The method according to claim 2, characterized in that The preset human body part includes one or more limbs; and determining that the state of the preset human body part in the current frame of the target video satisfies the initial state includes: Determine whether the positional relationship of the limbs in the current frame satisfies the positional relationship corresponding to the initial state, and if so, determine whether the state of the preset part of the human body in the current frame satisfies the initial state; The determining that the state of the preset part of the human body in the current frame satisfies the ongoing state includes: It is determined whether the positional relationship of the limbs in the current frame satisfies the positional relationship corresponding to the ongoing state. If so, it is determined that the state of the preset part of the human body in the current frame satisfies the ongoing state.

4. The method according to claim 1, characterized in that: The determining, based on the preset human body part in the target video, at least two types of key frames corresponding to at least two reference states of the preset action in the target video, respectively, comprises: Determining the posture parameters of the preset part of the human body in the current frame of the target video; When the posture parameters of the preset part of the human body in the current frame meet the parameter conditions corresponding to the starting state, and the posture parameters of the preset part of the human body in the previous frame of the current frame do not meet the parameter conditions corresponding to the starting state, the current frame is determined as a key frame corresponding to the starting state; When the posture parameters of the preset part of the human body in the current frame meet the parameter conditions corresponding to the end state, and the posture parameters of the preset part of the human body in the previous frame of the current frame do not meet the parameter conditions corresponding to the end state, the current frame is determined as the key frame corresponding to the end state.

5. The method according to claim 4, characterized in that The preset human body part includes one or more limbs; and determining the posture parameter of the preset human body part in the current frame includes: The posture parameter of the preset part of the human body in the current frame is determined according to the angle information of the limb in the current frame.

6. The method according to claim 1, characterized in that The method further comprises: Determining a motion start frame in the target video according to a state of the preset part of the human body in the target video; The at least two types of key frames are located after the motion start frame.

7. A motion measurement device in a video, characterized in that: include: A video acquisition module is configured to acquire a target video; A human body detection module is configured to detect key points of a human body in the target video, and determine a preset part of a human body in the target video according to the key points of the human body; A key frame determination module is configured to determine at least two types of key frames corresponding to at least two reference states of a preset action in the target video based on a preset human body part in the target video; A key frame processing module, configured to count and / or time the preset action in the target video according to the at least two types of key frames; The at least two reference states of the preset action include: a starting state, an initial state, at least one ongoing state, and an end state of the preset action; the starting state refers to the physical state of the user when entering the preset action, the initial state refers to the physical state of the user when starting to do the preset action, the ongoing state refers to the physical state of the user when doing the preset action, and the end state refers to the physical state of the user when exiting the preset action; The counting and / or timing of the preset action in the target video according to the at least two types of key frames includes: counting the preset action in the target video according to the number of key frames corresponding to the initial state and the number of key frames corresponding to the ongoing state; timing the preset action in the target video according to the time interval between the key frame corresponding to the starting state and the key frame corresponding to the ending state; Among them, the timing of the preset action in the target video according to the time interval between the key frame corresponding to the starting state and the key frame corresponding to the ending state includes: within the time interval, if the number of the preset actions is less than a set threshold, the preset action is not timed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 6 by executing the executable instructions.

Images