A motion counting method, apparatus, and computing device

Abstract

This invention discloses an action counting method, comprising: acquiring an image sequence of a moving object performing actions, the image sequence including multiple image frames; obtaining a reference action sequence and a action sequence to be matched from the image sequence; matching the action sequence to be matched with the reference action sequence to obtain a matching result corresponding to the moving object; and updating the number of actions of the moving object based on the matching result. This invention also discloses corresponding apparatus, computing devices, and storage media.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an action counting method, apparatus, and computing device. Background Technology

[0002] More and more people are improving their health through exercise. Because exercise requires repetitive physical movements to stimulate or train desired body parts, these repetitive movements need to be counted to achieve better training results.

[0003] Some counting schemes require wearing special equipment (such as various accelerometer devices) to perform the counting, which is costly and limited to specific application scenarios, making it difficult to promote on a large scale.

[0004] Therefore, a more advanced motion counting scheme is desired. Summary of the Invention

[0005] Therefore, embodiments of the present invention provide an action counting method, apparatus, and computing device in an attempt to solve or at least alleviate the problems mentioned above.

[0006] According to one aspect of the present invention, an action counting method is provided, comprising: acquiring an image sequence of a moving object performing an action, the image sequence including multiple image frames; obtaining a reference action sequence and a action sequence to be matched from the image sequence; matching the action sequence to be matched with the reference action sequence to obtain a matching result corresponding to the moving object; and updating the number of actions of the moving object based on the matching result of the moving object.

[0007] Optionally, in the method according to an embodiment of the present invention, the step of obtaining a reference action sequence from an image sequence includes: arranging a predetermined window at the first image frame of the image sequence; and obtaining a plurality of image frames contained within the predetermined window as a reference action sequence.

[0008] Optionally, in the method according to an embodiment of the present invention, the step of obtaining the action sequence to be matched from the image sequence includes: moving a predetermined window with a predetermined step size; and obtaining a plurality of image frames contained within the moved predetermined window as the action sequence to be matched.

[0009] Optionally, the method according to an embodiment of the present invention further includes: repeating the steps of obtaining the action sequence to be matched from the image sequence, matching the action sequence to be matched with the reference action sequence, and updating the number of actions of the moving object based on the matching result of the moving object, until the predetermined window moves to the last image frame of the image sequence.

[0010] Optionally, the method according to an embodiment of the present invention further includes: using a pose evaluation model to obtain a set of key points of a moving object in an image frame; and the step of matching the action sequence to be matched with a reference action sequence includes: matching the action sequence to be matched with the reference action sequence based on the set of key points of the moving object to obtain a matching result corresponding to the moving object.

[0011] Optionally, in the method according to an embodiment of the present invention, the step of matching the action sequence to be matched with the reference action sequence based on the key point set of the moving object to obtain the matching result corresponding to the moving object includes: calculating the sequence distance between the action sequence to be matched and the reference action sequence based on the key point set of the moving object; comparing the sequence distance with a distance threshold; if the sequence distance is less than the distance threshold, the matching result of the moving object is a successful match.

[0012] Optionally, in the method according to an embodiment of the present invention, the step of calculating the distance between the action sequence to be matched and the reference action sequence based on the key point set of the moving object includes: for each image frame to be matched in the sequence to be matched, calculating the distance between the image frame to be matched and its corresponding reference image frame in the reference action sequence based on the key point set of the moving object; and calculating the sequence distance between the action sequence to be matched and the reference action sequence based on the distance between the image frame to be matched and the corresponding reference image frame.

[0013] Optionally, in the method according to an embodiment of the present invention, the step of calculating the distance between a target image frame and its corresponding reference image frame in a reference action sequence based on the set of key points of the moving object includes: constructing a target matrix of the moving object in the target image frame based on the set of key points of the moving object in the target image frame; constructing a reference matrix of the moving object in the corresponding reference image frame based on the set of key points of the moving object in the corresponding reference image frame; calculating a transformation matrix from the target matrix of the moving object to its reference matrix; converting the target matrix into an approximate reference matrix based on the transformation matrix; and calculating the distance between the target image frame and the corresponding reference image frame based on the reference matrix and the approximate reference matrix.

[0014] Optionally, in the method according to an embodiment of the present invention, the step of calculating the distance between the image frame to be matched and the corresponding reference image frame based on the reference matrix and the approximate reference matrix includes: calculating the distance between each row of the reference matrix and each row of the approximate reference matrix; and obtaining the distance between the image frame to be matched and the corresponding reference image frame based on the distance between each row.

[0015] Optionally, in the method according to an embodiment of the present invention, the step of calculating the transformation matrix from the matching matrix of the moving object to its reference matrix includes: obtaining the transformation matrix using the least squares method.

[0016] Optionally, in the method according to an embodiment of the present invention, the step of updating the number of actions of the moving object based on the matching result of the moving object includes: if the matching result of the moving object is a successful match, then incrementing the number of actions of the moving object by one.

[0017] Optionally, in the method according to an embodiment of the present invention, the distance includes Euclidean distance.

[0018] Optionally, in the method according to an embodiment of the present invention, the pose evaluation model includes: a feature extraction module, adapted to extract image features of an image frame; a key point localization module, adapted to predict multiple key points contained in the image frame and their confidence levels based on the image features; and a key point connection module, adapted to predict the connections and affinity fields between pairs of key points based on the image features.

[0019] Optionally, in the method according to an embodiment of the present invention, the attitude evaluation model further includes: a connection matching module, adapted to determine target connections based on the affinity field of connections between multiple key points and multiple key point pairs; and further adapted to obtain a set of key points for each moving object based on the target connections.

[0020] According to another aspect of the present invention, an action counting method is provided, comprising: detecting a user's predetermined operation; in response to detecting the predetermined operation, acquiring an image sequence of the user's action via an image acquisition unit; obtaining a reference action sequence and a action sequence to be matched from the image sequence; matching the action sequence to be matched with the reference action sequence to obtain a matching result corresponding to the user; updating the number of user actions based on the user's matching result; and displaying the number of user actions via a display unit.

[0021] According to another aspect of the present invention, an action counting device is provided, comprising: an image acquisition unit adapted to acquire an image sequence of a moving object performing an action, the image sequence including multiple image frames; a sequence acquisition unit adapted to acquire a reference action sequence and a action sequence to be matched from the image sequence; an action matching unit adapted to match the action sequence to be matched with the reference action sequence to obtain a matching result corresponding to the moving object; and a counting update unit adapted to update the number of actions of the moving object based on the matching result of the moving object.

[0022] According to another aspect of the present invention, an action counting device is provided, comprising: a detection unit adapted to detect a predetermined operation of a user; an image acquisition unit adapted to acquire an image sequence of actions performed by the user in response to the detection of the predetermined operation; a sequence acquisition unit adapted to acquire a reference action sequence and a action sequence to be matched from the image sequence; an action matching unit adapted to match the action sequence to be matched with the reference action sequence to obtain a matching result corresponding to the user; a count update unit adapted to update the number of actions performed by the user based on the matching result; and a display unit adapted to display the number of actions performed by the user.

[0023] According to another aspect of the present invention, a computing device is provided, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to an embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform a method according to an embodiment of the present invention.

[0025] The action counting scheme according to embodiments of the present invention, considering the repetitive nature of actions, directly obtains a reference action sequence from the image sequence of the moving object performing the action as a template for subsequent matching. This enables real-time and automatic counting of various types of actions, eliminating the need for additional specialized equipment or template design for different types of actions. It is simple to operate, has few limitations (e.g., it can be performed on mobile terminals), and improves the user experience. Furthermore, the use of a posture evaluation model for action matching improves matching accuracy.

[0026] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more obvious and understandable, specific implementation methods of the embodiments of the present invention are described below. Attached Figure Description

[0027] To achieve the foregoing and related objectives, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings. These aspects indicate various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The foregoing and other objectives, features, and advantages of this disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings. Throughout this disclosure, the same reference numerals generally refer to the same parts or elements.

[0028] Figure 1 A schematic diagram of an action counting system 100 according to an embodiment of the present invention is shown;

[0029] Figure 2 A schematic diagram of a computing device 200 according to an embodiment of the present invention is shown;

[0030] Figure 3 A flowchart of an action counting method 300 according to an embodiment of the present invention is shown;

[0031] Figure 4 A schematic diagram of an attitude evaluation model 400 according to an embodiment of the present invention is shown;

[0032] Figure 5 A schematic diagram of a key point localization module 420 and a key point connection module 430 according to an embodiment of the present invention is shown;

[0033] Figure 6 A schematic diagram illustrating the matching of a sequence of actions to be matched with a reference sequence of actions according to an embodiment of the present invention is shown; and

[0034] Figure 7 A schematic diagram of an action counting device 700 according to an embodiment of the present invention is shown. Detailed Implementation

[0035] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0036] During exercise, users desire the ability to conveniently and in real-time count their repetitive movements in order to quantify the exercise process. To this end, embodiments of the present invention provide a movement counting device that can acquire image sequences of user movements and count the movements based on these image sequences.

[0037] Figure 1 A schematic diagram of an action counting system 100 according to an embodiment of the present invention is shown. Figure 1 As shown, the motion counting system 100 includes an image acquisition unit 110 and a motion counting device 700.

[0038] The image acquisition unit 110 can acquire image sequences of a moving object performing actions, such as video of the moving object performing actions. The image sequence includes multiple consecutive or non-consecutive image frames, each of which records the motion posture of the moving object. In various embodiments, the image acquisition unit 110 can be any camera capable of acquiring image sequences, such as a smartphone camera, a laptop camera, a tablet camera, a webcam, or other suitable cameras.

[0039] The motion counting device 700 is coupled to the image acquisition unit 110 and can count repetitive movements performed by a moving object based on the image sequence acquired by the image acquisition unit 110. In some embodiments, the image acquisition unit 110 can acquire image sequences in real time, and the motion counting device 700 can also count them in real time. The motion counting device 700 can also be coupled to the display unit 120 and display the updated number of movements via the display unit 120 so that the user can understand their exercise status.

[0040] It should be noted that although the image acquisition unit 110 and / or the display unit 120 are shown as separate from the motion counting device 700, they can also be integrated into the motion counting device 700 as part of the motion counting device 700.

[0041] The action counting device 700 according to an embodiment of the present invention can be implemented by the following computing device 200. Figure 2 A schematic diagram of a computing device 200 according to an embodiment of the present invention is shown. The computing device 200 is an electronic device capable of acquiring and / or displaying images, such as a personal computer, a mobile terminal (e.g., a smart device), a tablet computer, or other suitable device that may have image acquisition and / or display functions.

[0042] like Figure 2 As shown, computing device 200 may include a memory interface 202, one or more processors 204, and a peripheral interface 206. The memory interface 202, one or more processors 204, and / or the peripheral interface 206 may be discrete components or integrated into one or more integrated circuits. In computing device 200, various components can be coupled via one or more communication buses or signal lines. Sensors, devices, and subsystems can be coupled to the peripheral interface 206 to facilitate various functionalities.

[0043] For example, motion sensor 210, light sensor 212, and distance sensor 214 can be coupled to peripheral interface 206 to facilitate functions such as orientation, illumination, and ranging. Other sensors 216 can also be connected to peripheral interface 206, such as positioning systems (e.g., GPS receivers), temperature sensors, biometric sensors, or other sensing devices, thereby helping to implement related functions.

[0044] The camera subsystem 220 and the optical sensor 222 can be used to facilitate the implementation of camera functions such as image acquisition, wherein the camera subsystem 220 and the optical sensor 222 can be, for example, a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) optical sensor.

[0045] The computing device 200 may facilitate communication functions through one or more wireless communication subsystems 224, wherein the wireless communication subsystems 224 may include radio frequency receivers and transmitters and / or optical (e.g., infrared) receivers and transmitters. The specific design and implementation of the wireless communication subsystems 224 may depend on one or more communication networks supported by the computing device 200. For example, the computing device 200 may include communication subsystems 224 designed to support GSM networks, GPRS networks, EDGE networks, Wi-Fi or WiMax networks, and Bluetooth™ networks.

[0046] The audio subsystem 226 can be coupled to the speaker 228 and the microphone 230 to help implement voice-enabled functions such as speech recognition, speech copying, digital recording, and telephone functions.

[0047] To display an image, the I / O subsystem 240 may include a display controller 242 and / or one or more other input controllers 244. The display controller 242 may be coupled to a display 246. The display 246 may be a liquid crystal display (LCD), a touchscreen, or other type of display. In some implementations, the display 246 and the display controller 242 may use any of a variety of touch sensing technologies to detect contact and movement or pauses, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. One or more other input controllers 244 may be coupled to other input / control devices 248, such as one or more buttons, joystick switches, thumbwheels, infrared ports, USB ports, and / or pointing devices like pens. One or more of these buttons (not shown) may include up / down buttons for controlling the volume of the speaker 228 and / or microphone 230.

[0048] The memory interface 202 may be coupled to the memory 250. The memory 250 may include high-speed random access memory and / or non-volatile memory, such as one or more disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR).

[0049] Memory 250 can store program 254, which runs on the operating system 252 stored in memory 250. When the computing device is running, the operating system 252 is loaded from memory 250 and executed by processor 204. Program 254, when running, is also loaded from memory 250 and executed by processor 204. Among the various programs 254, one type is an action counting device 700 according to an embodiment of the present invention, and includes instructions configured to execute the action counting method 300 according to an embodiment of the present invention.

[0050] Figure 3 A flowchart of an action counting method 300 according to an embodiment of the present invention is shown, which is adapted to be executed in an action counting device 700.

[0051] like Figure 3 As shown, the action counting method 300 begins with step S310. In step S310, an image sequence of a moving object performing an action can be acquired via the image acquisition unit 110. This image sequence includes multiple image frames. In some embodiments, an image sequence of one moving object performing an action can be acquired; that is, the image frame records only one moving object. In other embodiments, an image sequence of multiple moving objects performing an action can be acquired; that is, the image frame records multiple moving objects simultaneously. The embodiments of the present invention do not limit the number of moving objects recorded in an image frame.

[0052] According to embodiments of the present invention, for each image frame in an image sequence, a set of key points for each moving object in that image frame is obtained. In some embodiments, a pose evaluation model can be used to obtain the set of key points for each moving object in the image frame. That is, the image frame is input into the pose evaluation model so that the pose evaluation model outputs the set of key points for each moving object in the image frame. The set of key points for the moving object includes the position coordinates of each key point of the moving object in the image frame.

[0053] Any pose evaluation model in this field can be used to obtain the key point set, such as pose evaluation models based on DeepPose, pose evaluation models based on CPN, pose evaluation models based on OpenPose, pose evaluation models based on HRNet, etc.

[0054] Figure 4A schematic diagram of an attitude evaluation model 400 according to an embodiment of the present invention is shown. Figure 4 As shown, the posture evaluation model 400 includes a feature extraction module 410, a key point localization module 420, a key point connection module 430, and a connection matching module 440.

[0055] The feature extraction module 410 is adapted to receive image frames and extract image features from the image frames. The feature extraction module 410 can be implemented as a partial layer in a convolutional neural network (e.g., a layer in VGG-19, or other lightweight network structures), and the extracted image features can be multiple feature maps.

[0056] In some embodiments, the size and / or pixel values ​​of the image frame may be preprocessed before the image frame is input into the pose evaluation model 400. For example, the image frame may be converted to a predetermined size (typically 256x256), or the pixel values ​​of the image frame may be converted from [0,255] to [-1,1].

[0057] After extracting image features, the feature extraction module 410 inputs the image features into two branches, namely the keypoint localization module 420 and the keypoint connection module 430, to simultaneously predict keypoints (parts) and connections (pairs) between keypoint pairs. Specifically, the keypoint localization module 420 can predict the confidence level of each keypoint, while the keypoint connection module 430 can predict the connection and affinity field between each keypoint pair.

[0058] Understandably, key points are body parts of a moving object, usually corresponding to joints with a certain degree of freedom, such as the neck, shoulder, elbow, wrist, waist, knee, and ankle. A connection is the link between a pair of key points. This connection can constitute a limb (e.g., the connection between the left shoulder and the neck is a limb) or it may not be a limb (e.g., the connection between the left shoulder and the left eye is not a limb).

[0059] The keypoint localization module 420 and the keypoint connection module 430 can be implemented as a two-branch, multi-stage convolutional neural network, such as... Figure 5 As shown. In Figure 5 In the diagram, F represents the feature map of the image frame, which is input to branch 1 and branch 2, respectively. Each branch includes multiple stages, Stage1, ..., Staget. For example, it can include two stages, Stage1 and Stage2. Each branch in each stage has a corresponding loss function, LOSS. Branch 1 is the keypoint localization module 420, and branch 2 is the keypoint connection module 430.

[0060] Branch 1 ultimately outputs multiple confidence maps S. t(Heatmaps) are used, with each confidence map corresponding to a keypoint of a type (e.g., left shoulder, left wrist) and indicating the confidence level of the keypoint. Understandably, the extreme points of the confidence map are the keypoints. The output of branch L(t) can also be processed using a non-maximum suppression (NMS) algorithm to obtain the discretized keypoint locations and their confidence maps.

[0061] Branch 2 finally outputs the affinity field L between multiple keypoint pairs. t (Part Affinity Fields, PAFs) are a collection of two-dimensional vector fields that include the location and orientation of connections between keypoint pairs. They can be used to calculate the confidence level of connections between keypoint pairs.

[0062] Each keypoint can have multiple connections, meaning it can have multiple possible limbs. Therefore, the connection matching module 440 needs to determine the correct connections (hereinafter referred to as target connections) based on the locations of multiple keypoints and the affinity of the connections between multiple keypoint pairs.

[0063] For example, the confidence of the connection between each keypoint pair can be calculated based on the location of multiple keypoints and the affinity of the connections between multiple keypoint pairs. Then, target connections are matched to maximize the total confidence of multiple target connections. Various maximum bipartite graph matching methods can be used to determine the target connections, such as the Hungarian algorithm.

[0064] The connection matching module 440 can also obtain a set of key points for each moving object based on the determined target connection, and each key point includes the position coordinates of the key point in the corresponding confidence map.

[0065] For example, suppose each target connection belongs to a different motion object, and each motion object is represented by two keypoints corresponding to a target connection. We can determine whether two motion objects are the same motion object. Specifically, if two motion objects have the same keypoints, then these two motion objects are the same motion object. Therefore, the keypoint sets of the two motion objects are merged. For every pair of motion objects, we determine whether the two motion objects are the same motion object, until there are no longer two motion objects with the corresponding keypoints.

[0066] Finally, the position coordinates of each keypoint in the image frame can be obtained based on the position coordinates of each keypoint in the corresponding confidence map. Specifically, the scaling ratio between the size of the output confidence map and the size of the image frame of the input pose evaluation model 400 (hereinafter referred to as the input image frame) can be determined, and the position coordinates of each keypoint in the corresponding confidence map can be transformed based on this scaling ratio. For example, if the width and height of the confidence map are both 1 / 8 of the input image frame, then multiplying the position coordinates of each keypoint in the corresponding confidence map by 8 will give the position coordinates of each keypoint in the input image frame.

[0067] It should be noted that when an image frame records only one moving object, the pose evaluation model 400 may not include the aforementioned connection matching module 440, and the multiple keypoints obtained by the keypoint localization module 420 constitute the keypoint set of that moving object. However, when an image frame records multiple moving objects, the pose evaluation model 400 should include the aforementioned connection matching module 440 to obtain the keypoint set of each moving object.

[0068] According to step S320, a reference action sequence and a target action sequence can be obtained from the acquired image sequence. In some embodiments, a predetermined window is arranged at the first image frame of the image sequence, and multiple image frames contained within the predetermined window are obtained as a reference action sequence. It is understood that for repetitive actions, the actions performed by the moving object at the start of exercise can serve as a template for subsequent actions.

[0069] The width of the predetermined window can be set based on the actual situation. For example, the window width can be set to 10 image frames, and 10 image frames within the predetermined window can be obtained as a reference action sequence.

[0070] Furthermore, considering the minimal differences in actions between adjacent image frames, multiple image frames contained within the predetermined window can be acquired using a frame-skipping method as a reference action sequence. For example, for image frames contained within the predetermined window, the next image frame is acquired after a predetermined number of image frames have elapsed.

[0071] After obtaining the reference action sequence, a predetermined window can be moved by a predetermined step size to obtain multiple image frames contained within the moved window as the action sequence to be matched. The method for obtaining the action sequence to be matched is the same as that for obtaining the reference action sequence, and both contain the same number of image frames.

[0072] The predetermined step size is typically less than the number of image frames contained in the reference action sequence, but greater than half of that number of image frames. The specific step size can be set based on the actual situation.

[0073] After obtaining the action sequence to be matched, in step S330, the action sequence to be matched is matched with the reference action sequence to obtain the matching result corresponding to each moving object. Specifically, for each moving object, the action sequence to be matched is matched with the reference action sequence based on the key point set of the moving object to obtain the matching result corresponding to the moving object.

[0074] The set of key points for each moving object in each image frame to be matched within the action sequence to be matched can be obtained, which serves as the set of key points to be matched for that moving object in that image frame. The set of key points for each moving object in each reference image frame within the reference action sequence can also be obtained, which serves as the set of reference key points for that moving object in that reference image frame.

[0075] When an image frame records only one moving object, all key points in the image frame to be matched are the set of key points to be matched for that moving object in that image frame, and all key points in the reference image frame are the set of reference key points for that moving object in the reference image frame.

[0076] When multiple moving objects are recorded in an image frame, the region of each moving object in the image frame to be matched can be determined, and the key points located in the corresponding region are the set of key points to be matched for the corresponding moving object. Similarly, the region of each moving object in a reference image frame can be determined, and the key points located in the corresponding region are the set of reference key points for the corresponding moving object.

[0077] The following describes the process of matching the action sequence to be matched with the reference action sequence based on the key point set of the moving object, using a moving object as an example.

[0078] The sequence distance between the action sequence to be matched and the reference action sequence can be calculated based on the keypoint set of the moving object. Specifically, for each image frame to be matched in the sequence to be matched, the distance between the image frame to be matched and its corresponding reference image frame in the reference action sequence is calculated based on the keypoint set of the moving object.

[0079] In some embodiments, a reference matrix A for the moving object in the corresponding reference image frame is constructed based on a set of reference keypoints of the moving object in the corresponding reference image frame. Simultaneously, a matching matrix B for the moving object in the image frame to be matched is constructed based on a set of keypoints to be matched of the moving object in the image frame to be matched. Each row of matrix A or B corresponds to a keypoint and includes at least the position coordinates of that keypoint. For example, the constructed matrices A and B may include three columns, with all elements in the third column being 1. That is, each row of the keypoint matrix A or B can be represented as [xy 1], where x is the horizontal coordinate of the corresponding keypoint and y is the vertical coordinate of the corresponding keypoint.

[0080] Calculate the transformation matrix T from the target matrix to its reference matrix for the moving object. For example, use the least squares method to solve for the transformation matrix T such that |B*TA| is minimized.

[0081] Based on the obtained transformation matrix T, the matrix to be matched is transformed into an approximate reference matrix B', where B' = B*T. Based on the reference matrix and the approximate reference matrix, the distance between the image frame to be matched and the corresponding reference image frame is calculated.

[0082] In some embodiments, the distance between each row of the reference matrix and each row of the approximate reference matrix can be calculated, and then the distance between the image frame to be matched and the corresponding reference image frame can be obtained based on the distance between each row. For example, the distance can be Euclidean distance. Assuming that the reference matrix and the approximate reference matrix have M rows, for each row of the reference matrix A, its corresponding row in the approximate reference matrix B' is determined, and the Euclidean distance between the two rows is calculated according to the following formula:

[0083]

[0084] Where, d row x is the distance between rows. a y a Let x be the x and y coordinates of the key points corresponding to each row of reference matrix A. b y b' Let B' be the x-axis and y-axis coordinates of the key points corresponding to each row of the approximate reference matrix B'.

[0085] Then, calculate the distance between the image frame to be matched and the corresponding reference image frame using the following formula:

[0086]

[0087] Where, d image d represents the distance between the image frame to be matched and the reference image frame. row,i Let be the distance between the i-th row of the reference matrix A and the i-th row of the approximate reference matrix B'.

[0088] After obtaining the distance between each image frame to be matched in the action sequence and its corresponding reference image frame in the reference action sequence, the sequence distance between the action sequence to be matched and the reference action sequence can be calculated based on the distance between the image frame to be matched and its corresponding reference image frame. For example, the sequence distance can be obtained by adding the distances between each image frame to be matched and its corresponding reference image frame.

[0089] The obtained sequence distance is compared with the distance threshold. If the sequence distance is less than the distance threshold, the matching result of the moving object is a successful match; otherwise, the matching result of the moving object is a failed match.

[0090] Then, in step S340, the number of actions of each moving object is updated based on the matching results of each moving object.

[0091] For example, if the matching result for the moving object is successful, the number of actions performed by the moving object is incremented by one. If the matching result for the moving object is unsuccessful, the number is not incremented. However, considering that the reference action sequence should also be counted as one action, the number of actions performed by the moving object can be initially set to one.

[0092] The steps described above—obtaining the action sequence to be matched from the image sequence, matching the action sequence to be matched with the reference action sequence, and updating the number of actions of each moving object based on the matching results of each moving object—can be repeated until the predetermined window moves to the last image frame of the image sequence.

[0093] Figure 6 A schematic diagram illustrating the matching of a sequence of actions to be matched with a reference sequence of actions according to an embodiment of the present invention is shown. Figure 6 As shown, an image sequence of a moving object performing an action is acquired. A predetermined window (marked with a dashed line) is placed at the first image frame of the image sequence to obtain a reference action sequence 620. The predetermined window is then moved two image frames to obtain an action sequence 640 to be matched. Matching the action sequence 640 with the reference action sequence 620 results in a match failure for the moving object, and the count of the action is not incremented. The predetermined window is then moved two more image frames to obtain an action sequence 660 to be matched. Matching the action sequence 660 with the reference action sequence 620 results in a match success for the moving object, and the count of the action is incremented.

[0094] It should be noted that one or more steps in the action counting method 300 according to embodiments of the present invention can also be performed by other devices, such as the server 130 communicating with the action counting device 700. In some embodiments, the action counting device 600 may send an image sequence of one or more moving objects performing actions to the server 130 after acquiring the image sequence. The server 130 obtains a reference action sequence and a motion sequence to be matched from the image sequence, matches the motion sequence to be matched with the reference action sequence to obtain the matching result corresponding to each moving object, and returns the matching result corresponding to each moving object to the action counting device 700. The action counting device 700 then updates the number of actions of each moving object based on the matching result of each moving object.

[0095] Furthermore, according to some embodiments of the present invention, the matching success rate of moving objects can be calculated, for example, by dividing the number of successful matches by the total number of matches. If the matching failure rate reaches a predetermined threshold, the user can be prompted, for example, by displaying a prompt message via a display unit.

[0096] According to other embodiments of the present invention, motion suggestions for a moving object can be generated based on the object's actions. For example, pose suggestions can be generated based on action posture (a set of key points in an image frame can characterize the motion posture), frequency suggestions can be generated based on action frequency, and so on.

[0097] Figure 7 A schematic diagram of an action counting device 700 according to an embodiment of the present invention is shown. Figure 7 As shown, the action counting device 700 includes an image acquisition unit 710, a sequence acquisition unit 720, an action matching unit 730, and a counting update unit 740.

[0098] The image acquisition unit 710 is adapted to acquire image sequences of one or more moving objects performing actions, the image sequence comprising multiple image frames. The sequence acquisition unit 720 is coupled to the image acquisition unit 710 and is adapted to acquire a reference action sequence and a sequence of actions to be matched from the acquired image sequence. The action matching unit 730 is coupled to the sequence acquisition unit 720 and is adapted to match the action sequence to be matched with the reference action sequence to obtain the matching result corresponding to each moving object. The counting update unit 740 is coupled to the action matching unit 730 and is adapted to update the number of actions of each moving object based on the matching result of each moving object.

[0099] In some embodiments, the motion counting device 700 may further include a display unit 750, which is coupled to the counting update unit 740 and is adapted to display the number of motions of each moving object.

[0100] For detailed processing logic and implementation procedures of each unit in the motion counting device 700, please refer to the preceding text. Figures 1-6 The descriptions of the motion counting system 100 and the motion counting method 300 will not be repeated here.

[0101] This invention also provides an action counting method. Specifically, it can detect a user's predetermined operation, such as clicking a specified application icon or selecting a specified button on a mobile terminal. Then, in response to the predetermined operation, an image acquisition unit acquires an image sequence of the user's actions. A reference action sequence and a sequence to be matched are obtained from the image sequence. The sequence to be matched is then matched with the reference action sequence to obtain the matching result for the user. Based on the user's matching result, the user's action count is updated. Finally, the user's action count is displayed via a display unit. For detailed processing logic and implementation of each step in this action counting method, please refer to the preceding text. Figures 1-6 The descriptions of the motion counting system 100 and the motion counting method 300 will not be repeated here.

[0102] Accordingly, embodiments of the present invention may also provide an action counting device. This action counting device may include a detection unit adapted to detect a user's predetermined operation; an image acquisition unit adapted to acquire an image sequence of the user's actions in response to the detection of the predetermined operation; a sequence acquisition unit adapted to acquire a reference action sequence and a sequence of actions to be matched from the image sequence; an action matching unit adapted to match the sequence of actions to be matched with the reference action sequence to obtain a matching result corresponding to the user; a count update unit adapted to update the number of user actions based on the user's matching result; and a display unit adapted to display the number of user actions. For detailed processing logic and implementation of each unit in this action counting device, please refer to the foregoing description. Figures 1-6 The descriptions of the motion counting system 100 and the motion counting method 300 will not be repeated here.

[0103] Understandably, the motion counting scheme according to embodiments of the present invention can be applied to many scenarios, including but not limited to private exercise scenarios, public exercise scenarios, competition scenarios, etc. In a private exercise scenario, a user can start counting their exercise movements by clicking the motion counting application icon on a mobile terminal, and the number of movements is displayed in real time on the mobile terminal screen so that the user can understand their exercise progress. In a competition scenario, the motion counting scheme according to embodiments of the present invention can count the exercise movements performed by multiple individuals, such as two athletes or exercisers, and display the number of movements performed by both athletes in real time on the mobile terminal screen, thereby comparing which athlete has more movements. The present invention does not limit these application scenarios in any way.

[0104] In summary, the action counting scheme according to the embodiments of the present invention, considering the repetitive nature of the actions themselves, directly obtains a reference action sequence from the image sequence of the moving object performing the action as a template for subsequent matching, which can realize real-time and automatic counting of various types of actions without the need to wear special equipment for counting, nor is it necessary to design templates for different types of actions. It is easy to operate, has few limitations (for example, it can be performed on a mobile terminal), and improves the user experience.

[0105] The various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of the embodiments of the present invention, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, USB flash drive, floppy disk, CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, that machine becomes an apparatus for practicing the embodiments of the present invention.

[0106] When the program code is executed on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute the method of the embodiments of the present invention according to instructions in the program code stored in the memory.

[0107] By way of example, and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Any combination of the above is also included within the scope of readable media.

[0108] In the specification provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the examples of the embodiments of the present invention. Based on the above description, the required structure for constructing such systems is apparent. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the contents of the embodiments of the present invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing preferred embodiments of the present invention.

[0109] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0110] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this method of disclosure should not be construed as reflecting an intention that the claimed embodiments of the invention require more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0111] Those skilled in the art will understand that modules, units, or components of the devices disclosed in the examples herein can be arranged in the devices described in this embodiment, or alternatively, can be located in one or more devices different from the devices in this example. The modules in the foregoing examples can be combined into a single module or, in addition, can be divided into multiple sub-modules.

[0112] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0113] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the embodiments of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0114] Furthermore, some of the embodiments described herein are methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing the above functions. Therefore, a processor having the necessary instructions for implementing the above methods or method elements forms means for implementing the method or method elements. Furthermore, the elements described herein in the apparatus embodiments are examples of means for implementing the functions performed by elements for the purpose of carrying out the invention.

[0115] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

[0116] Although embodiments of the invention have been described with respect to a limited number of examples, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the embodiments described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of interpreting or limiting the subject matter of the embodiments of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of embodiments of the invention is illustrative rather than restrictive, and the scope of the embodiments is defined by the appended claims.

Claims

1. A method for counting actions, comprising: Acquire an image sequence of a moving object performing an action, the image sequence comprising multiple image frames; Obtain a reference action sequence and a target action sequence from the image sequence; A pose evaluation model is used to obtain the set of key points of the moving object in the image frame; Based on the set of key points of the moving object, the action sequence to be matched is matched with the reference action sequence to obtain the matching result corresponding to the moving object; as well as Based on the matching results of the moving object, the number of actions of the moving object is updated; The step of matching the action sequence to be matched with the reference action sequence based on the key point set of the moving object to obtain the matching result corresponding to the moving object includes: calculating the sequence distance between the action sequence to be matched and the reference action sequence based on the key point set of the moving object; comparing the sequence distance with a distance threshold; if the sequence distance is less than the distance threshold, the matching result of the moving object is a successful match.

2. The method as described in claim 1, wherein, The steps for obtaining the reference action sequence from the image sequence include: A predetermined window is arranged at the first image frame of the image sequence; Multiple image frames contained within the predetermined window are acquired as the reference action sequence.

3. The method as described in claim 2, wherein, The steps for obtaining the action sequence to be matched from the image sequence include: The predetermined window is moved by a predetermined step size; Multiple image frames contained within a predetermined window after the movement are acquired as the sequence of actions to be matched.

4. The method of claim 3, wherein, Also includes: Repeat the steps described above: obtaining the action sequence to be matched from the image sequence, matching the action sequence to be matched with the reference action sequence, and updating the number of actions of the moving object based on the matching result of the moving object, until the predetermined window moves to the last image frame of the image sequence.

5. The method of claim 1, wherein, The step of calculating the distance between the action sequence to be matched and the reference action sequence based on the set of key points of the moving object includes: For each image frame to be matched in the action sequence to be matched, the distance between the image frame to be matched and its corresponding reference image frame in the reference action sequence is calculated based on the key point set of the moving object. Based on the distance between the image frame to be matched and the corresponding reference image frame, the sequence distance between the action sequence to be matched and the reference action sequence is calculated.

6. The method of claim 5, wherein, The step of calculating the distance between the image frame to be matched and its corresponding reference image frame in the reference action sequence based on the set of key points of the moving object includes: Based on the set of key points of the moving object in the image frame to be matched, construct the matching matrix of the moving object in the image frame to be matched; Based on the set of key points of the moving object in the corresponding reference image frame, a reference matrix of the moving object in the corresponding reference image frame is constructed. Calculate the transformation matrix from the target matrix of the moving object to its reference matrix; Based on the transformation matrix, the matrix to be matched is converted into an approximate reference matrix; Based on the reference matrix and the approximate reference matrix, the distance between the image frame to be matched and the corresponding reference image frame is calculated.

7. The method of claim 6, wherein, The step of calculating the distance between the image frame to be matched and the corresponding reference image frame based on the reference matrix and the approximate reference matrix includes: Calculate the distance between each row of the reference matrix and each row of the approximate reference matrix; Based on the distance between each row, the distance between the image frame to be matched and the corresponding reference image frame is obtained.

8. The method of claim 6, wherein, The steps for calculating the transformation matrix from the target matrix to its reference matrix of the moving object include: The transformation matrix is ​​obtained by using the least squares method.

9. The method of claim 1, wherein, The step of updating the number of actions of the moving object based on the matching result of the moving object includes: If the matching result of the moving object is successful, then the number of actions of the moving object is incremented by one.

10. The method of claim 7, wherein, The distances mentioned include Euclidean distances.

11. The method of claim 1, wherein, The attitude evaluation model includes: The feature extraction module is adapted to extract image features from the image frame; A keypoint localization module is adapted to predict multiple keypoints and their confidence levels contained in the image frame based on the image features; and The key point connection module is adapted to predict the connections and affinity fields between key point pairs based on the image features.

12. The method of claim 11, wherein, The attitude evaluation model also includes: The connection matching module is adapted to determine a target connection based on the affinity field of the connections between multiple key points and multiple key point pairs; it is also adapted to obtain a set of key points for each moving object based on the target connection.

13. A method for counting actions, comprising: Detect the user's scheduled actions; In response to detecting the predetermined operation, an image sequence of the user's actions is acquired via the image acquisition unit; A reference action sequence and a target action sequence are obtained from the image sequence, wherein the image sequence includes multiple image frames; A pose evaluation model is used to obtain the set of key points of the moving object in the image frame; Based on the set of key points of the moving object, the action sequence to be matched is matched with the reference action sequence to obtain the matching result corresponding to the moving object; Based on the user's matching results, the number of actions performed by the user is updated; as well as The number of user actions is displayed via a display unit; The step of matching the action sequence to be matched with the reference action sequence based on the key point set of the moving object to obtain the matching result corresponding to the moving object includes: calculating the sequence distance between the action sequence to be matched and the reference action sequence based on the key point set of the moving object; comparing the sequence distance with a distance threshold; if the sequence distance is less than the distance threshold, the matching result of the moving object is a successful match.

14. An action counting device, comprising: An image acquisition unit is adapted to acquire image sequences of moving objects performing actions, the image sequence including multiple image frames; The sequence acquisition unit is adapted to acquire a reference action sequence and a matchable action sequence from the image sequence; The action matching unit is adapted to use a pose evaluation model to obtain a set of key points of the moving object in the image frame, and based on the set of key points of the moving object, match the action sequence to be matched with the reference action sequence to obtain the matching result corresponding to the moving object. as well as The counting update unit is adapted to update the number of actions of the moving object based on the matching result of the moving object; The step of matching the action sequence to be matched with the reference action sequence based on the key point set of the moving object to obtain the matching result corresponding to the moving object includes: calculating the sequence distance between the action sequence to be matched and the reference action sequence based on the key point set of the moving object; comparing the sequence distance with a distance threshold; if the sequence distance is less than the distance threshold, the matching result of the moving object is a successful match.

15. An action counting device, comprising: The detection unit is suitable for detecting the user's pre-defined operations; The image acquisition unit is adapted to acquire an image sequence of user actions in response to the detection of the predetermined operation; A sequence acquisition unit is adapted to acquire a reference action sequence and a matchable action sequence from the image sequence, wherein the image sequence includes multiple image frames; The action matching unit is adapted to use a pose evaluation model to obtain a set of key points of the moving object in the image frame, and based on the set of key points of the moving object, match the action sequence to be matched with the reference action sequence to obtain the matching result corresponding to the moving object. The counting update unit is adapted to update the number of actions of the user based on the user's matching results; as well as A display unit is adapted to display the number of actions performed by the user. The step of matching the action sequence to be matched with the reference action sequence based on the key point set of the moving object to obtain the matching result corresponding to the moving object includes: calculating the sequence distance between the action sequence to be matched and the reference action sequence based on the key point set of the moving object; comparing the sequence distance with a distance threshold; if the sequence distance is less than the distance threshold, the matching result of the moving object is a successful match.

16. A computing device, comprising: One or more processors; Memory; and A program, wherein the program is stored in the memory and configured to be executed by the one or more processors, the program comprising instructions for performing the action counting method according to any one of claims 1-13.

17. A computer-readable storage medium storing a program, the program comprising instructions that, when executed by a computing device, cause the computing device to perform the action counting method according to any one of claims 1-13.

Images