Action recognition method, action recognition device, and electronic device
The motion recognition method leverages key point and joint angle data in image frames with a shallow learning network to overcome the limitations of deep learning networks, ensuring high accuracy and scalability in motion recognition.
Patent Information
- Application Number
- JP2020169125
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-10-15
- Filing Date
- 2020-10-06
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-10-06
AI Technical Summary
Existing motion recognition methods using deep learning networks require large amounts of training data, are computationally intensive, and suffer from scalability issues due to interference from other image information, while methods based on key points are inaccurate and difficult to scale.
A motion recognition method that utilizes key point feature extraction to determine the positions and angles of joints in an image frame, employing a shallow learning network for accurate classification with reduced computational load, and adjusts recognition results based on historical motion data and time relationships.
This approach achieves high accuracy and scalability by using key point and joint angle information without deep learning networks, allowing for flexible adaptation to new motions without extensive retraining, and enhances recognition precision through temporal adjustments.
Smart Images

Figure 0007581753000002 
Figure 0007581753000003 
Figure 0007581753000004
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of electronic and information technology. [Background technology]
[0002] Human action recognition has a wide range of potential applications. For example, human action recognition can be applied to scenarios such as consumer behavior analysis, healthcare, and smart factories.
[0003] In the prior art, a typical action recognition method includes a method for performing human body action recognition based on image frames captured by a camera. For example, a classifier based on a deep learning network is used to classify the image frames and recognize the human body action in the image frames. The action recognition method also includes a method for extracting human body key points from the image frames captured by a camera and recognizing the human body action based on the key points.
[0004] It should be noted that the above description of the technical background is for the purpose of making the technical solutions of the present invention clear and complete, and is described for the purpose of making those skilled in the art understand. These technical solutions are merely described as part of the background technology of the present invention, and are not well known to those skilled in the art. Summary of the Invention [Problem to be solved by the invention]
[0005] According to the discovery of the inventor of the present invention, the general action recognition method has some limitations. For example, the method using a classifier based on a deep learning network needs a large amount of training data to train the deep learning network, and the method directly classifies the image signal, so the amount of calculation is large and it is interfered by other information in the image, and the recognition efficiency is not high. In addition, when recognizing a new action, the deep learning network needs to be retrained for the new action, so the scalability of the method is low. In the method of recognizing human body actions based on human body keypoints, the parameters used are relatively single, so it is difficult to ensure the accuracy of the recognition result.
[0006] Embodiments of the present invention provide a gesture recognition method, a gesture recognition device and an electronic device, which recognizes human body gestures based on the positions of key points of the human body and joint angles in an image frame, thereby achieving low computational complexity and high accuracy. [Means for solving the problem]
[0007] In a first aspect of an embodiment of the present invention, there is provided a motion recognition device including: a keypoint feature extraction unit that processes information of keypoints of a target human body in an image frame and calculates keypoint features of the target human body, the keypoint features including positions of predetermined keypoints of the target human body and angles of predetermined joints; and a first recognition unit that recognizes motions of the target human body based on the keypoint features and outputs a first recognition result.
[0008] In a second aspect of an embodiment of the present invention, there is provided a motion recognition method, comprising: a step of processing information of key points of a target human body in an image frame and calculating key point features of the target human body, the key point features including positions of predetermined key points of the target human body and angles of predetermined joints; and a step of recognizing a motion of the target human body based on the key point features and outputting a first recognition result.
[0009] In a third aspect of the present invention, there is provided an electronic device comprising the action recognition device according to the first aspect of the present invention.
[0010] The advantageous effects of the embodiment of the present invention are as follows: Recognizing the motion of the human body based on the positions of the key points of the human body and the angles of the joints in the image frames has low computational complexity and high accuracy.
[0011] Certain embodiments of the invention are disclosed in detail, as set forth in the following description and in the drawings, which show how the principles of the invention may be employed, but are not intended to be limited in scope, and all such modifications, alterations, and equivalents are within the spirit and content of the appended claims.
[0012] Features described and / or shown in one embodiment may be used in the same or similar manner in one or more other embodiments, may be combined with features in the other embodiments, or may substitute for features in the other embodiments.
[0013] It should be noted that the term "comprising" when used in the present text means the presence of a feature, element, step or component and does not exclude the presence or addition of one or more other features, elements, steps or components. [Brief description of the drawings]
[0014] The drawings included herein are for understanding the embodiments of the present invention, constitute a part of this specification, and are for illustrating the embodiments of the present invention, and together with the description of the text, explain the principles of the present invention. Note that the drawings described herein are merely for illustrating the embodiments of the present invention, and those skilled in the art can easily obtain other drawings based on these drawings. [Figure 1] 1 is a schematic diagram of an example of an action recognition device according to a first embodiment of the present invention. [Diagram 2]FIG. 2 is a schematic diagram of an example of key points of a target body in a current image frame; [Diagram 3] FIG. 2 is a schematic diagram of an example of predefined key points; [Figure 4] FIG. 4 is a schematic diagram of an example of how the keypoint feature extractor calculates keypoint features; [Diagram 5] 11 is a schematic diagram of an example of adjustment of a first recognition result by a second recognition unit. FIG. [Figure 6] FIG. 11 is a schematic diagram of an example of a motion recognition method according to a second embodiment of the present invention. [Figure 7] FIG. 11 is a schematic diagram illustrating an example of a configuration of an electronic device according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] The above and other features of the present invention will become apparent from the drawings and the following description. The specification and drawings disclose specific embodiments of the present invention, i.e., some embodiments according to the principles of the present invention. It is to be understood that the present invention is not limited to the described embodiments, and the present invention includes all modifications, variations, and equivalents within the scope of the appended claims.
[0016] In the embodiments of the present invention, the terms "first" and "second" are used to distinguish different elements by name and do not imply the spatial arrangement or temporal order of these elements, and the elements are not limited to these terms. The term "and / or" includes any one or more of the listed terms and combinations thereof. The terms "including," "including," and "having" refer to the presence of stated features, elements, components or members but do not exclude the presence or addition of one or more other features, elements, components or members.
[0017] In the present embodiment, the singular forms "a", "the", "the" and the like include the plural, meaning "a kind" or "a class", and are not limited to "one". Additionally, the term "said" includes both the singular and the plural, unless the context clearly dictates otherwise. Additionally, the term "according to" means "at least in part depending on" and the term "based on" means "at least in part based on", unless the context clearly dictates otherwise.
[0018] <Example 1> A first embodiment of the present invention provides an action recognition device.
[0019] 1 is a schematic diagram of an example of an action recognition device according to a first embodiment of the present invention. As shown in FIG. 1, the action recognition device 1 includes a keypoint feature extraction unit 11 and a first recognition unit 12.
[0020] Here, the keypoint feature extraction unit 11 processes information of the keypoints of the target human body in the image frame and calculates the keypoint features of the target human body. The keypoint features include the positions of predetermined keypoints of the target human body and angles of predetermined joints. The first recognition unit 12 recognizes the motion of the target human body in the image frame based on the keypoint features calculated by the keypoint feature extraction unit 11, and outputs a first recognition result.
[0021] According to the first embodiment of the present invention, the action recognition device 1 recognizes the action of the human body based on the positions of the key points of the human body and the angles of the joints in the image frame. Since the positions of the key points and the angles of the joints are part of the information in the image frame, the first recognition unit 12 does not need to use a deep learning network and can perform accurate classification with a small amount of calculation. Furthermore, compared with a method of performing action recognition based only on the positions of the key points, the action recognition device 1 of the present invention further performs action recognition based on the angles of the joints of the human body, thereby increasing the accuracy of recognition. Furthermore, when it is necessary to add or change an action to be recognized, it is only necessary to adjust the predetermined key points and / or predetermined joints without requiring a large amount of training, so that the action recognition device 1 of the present invention has high scalability and high flexibility.
[0022] In at least one embodiment, the image frames may be image frames in a video captured by, for example, a camera, and each image frame may have a chronological order, for example, an nth image frame corresponds to a time Tn on the chronological order, where n is a natural number.
[0023] In the following description, when recognizing the motion of the target human body in the n-th image frame, the n-th image frame is referred to as a current image frame, and an image frame preceding the current image frame in the time series is referred to as a historical image frame, for example, the historical image frame corresponds to time T(nk) in the time series.
[0024] In at least one embodiment, information of key points of the target human body in the current frame image may be input to the key point feature extractor 11 .
[0025] Here, the information of the key points of the target human body in the current image frame may be, for example, the coordinate values of each key point of the target human body in the current image frame, which may be expressed by pixel values in two mutually perpendicular directions, where the coordinate value of the pixel in the upper left corner of the current image frame may be set to (0,0), and the two mutually perpendicular directions may be the width direction and height direction of the current image frame.
[0026] FIG 2 is a schematic diagram of an example of key points of a target human body in a current image frame. As shown in FIG 2, the key points may be, for example, a nose 21, a left shoulder 22, a right shoulder 23, a left thigh 24, a right thigh 25, a left knee 26, a right knee 27, a left ankle 28, a right ankle 29, a left elbow 30, a right elbow 31, a left wrist 32, a right wrist 33, a left eye 34, a right eye 35, a left ear 36, and a right ear 37 of the target human body. The distribution of key points shown in FIG 2 is merely an example, and the present invention is not limited thereto.
[0027] In at least one embodiment, the information of each key point in Fig. 2 may be obtained by performing object detection on the current image frame. For example, a template method or the like is used to detect the object image and the body part of the object body in the current frame image, a key point is set on each detected part to represent the part, and the coordinate value of each key point is output as the information of the key point of the object body in the current image frame. Also, for example, a sensor may be set on the human body that is the subject of the camera, and the information of the key point of the object body in the current image frame may be determined based on the information sensed by the sensor.
[0028] In at least one embodiment, the keypoint feature extraction unit 11 may select at least some keypoints from the information of the keypoints of the target human body in the input current image frame as predetermined keypoints, and process the information of the predetermined keypoints.
[0029] Fig. 3 is a schematic diagram of an example of the predetermined key points. As shown in Fig. 3, the predetermined key points may be, for example, the nose 21, the left shoulder 22, the right shoulder 23, the left thigh 24, the right thigh 25, the left knee 26, the right knee 27, the left ankle 28, and the right ankle 29 of the target human body. Also, as shown in Fig. 3, the joint 241 at the left thigh 24, the joint 251 at the right thigh 25, the joint 261 at the left knee 26, and the joint 271 at the right knee 27 may be the predetermined joints.
[0030] The predetermined key points and predetermined joints shown in Fig. 3 are merely examples, and the present invention is not limited thereto, and the predetermined key points and predetermined joints may be set according to the motion to be recognized. For example, when recognizing motions related to the trunk and legs of a human body, nine predetermined key points and four predetermined joints may be set as shown in Fig. 2, and the nine key points may not be located on parts such as the arms. Also, when recognizing other motions of the human body, such as lifting and opening the arms, key points on the arms (for example, the left elbow 30, right elbow 31, left wrist 32, and right wrist 33 in Fig. 2) may be set as predetermined key points, and the joints at the left elbow 30 and right elbow 31 may be set as predetermined joints.
[0031] In at least one embodiment, the keypoint features of the target body in an image frame acquired by the keypoint feature extractor 11 include positions of predetermined keypoints and angles of predetermined joints of the target body in the image frame.
[0032] 4 is a schematic diagram of an example of a method for calculating keypoint features by the keypoint feature extractor 11. As shown in FIG. 4, the method includes the following steps.
[0033] In step 401, the coordinate values of predetermined key points of the target body in the image frame are adjusted according to the height of the target body in the image frame.
[0034] In step 402, angles of certain joints are calculated based on the adjusted coordinate values of certain key points of the target human body in the image frames.
[0035] In step 401, the height of the target human body in the current image frame may be that shown in H of FIG. 2, i.e., the difference between the height coordinate value of the key point where the eyes of the target human body are located in the current image frame (e.g., left eye 34 or right eye 35 in FIG. 2) and the height coordinate value of the key point where the ankles of the target human body are located in the current image frame (e.g., left ankle 28, right ankle 29 in FIG. 2).
[0036] In step 401, the ratio between the height H of the target human body in the current image frame and a predetermined standard height H0 may be calculated, and the coordinate values (height coordinate value and width coordinate value) of each predetermined pixel of the target human body in the current image frame may be divided by this ratio to obtain adjusted coordinate values.
[0037] According to step 401, a normalization process can be performed on the coordinate values of each predetermined key point of the target human body in the current image frame, so that changes in the coordinate values of each predetermined key point due to differences in the distance between the subject and the camera or the viewing angle can be avoided, and the accuracy of recognition can be improved.
[0038] In addition, the present invention is not limited thereto, and other methods may be used to adjust the coordinate values of each of the predetermined key points in step 401. In addition, the coordinate values of each of the predetermined key points may be adjusted according to other sizes of the target human body.
[0039] In step 402, the angles of the joint 241, the joint 251, the joint 261, and the joint 271 may be calculated for the adjusted coordinate values. Here, the angle of the joint 241 is, for example, the angle between the connecting line L1 between the left shoulder 22 and the left hip 24 and the connecting line L2 between the left hip 24 and the left knee 26 in Fig. 3, and the calculation method for the angles of the other joints is similar.
[0040] In at least one embodiment, the keypoint features of the target human body in the current image frame extracted by the keypoint feature extraction unit 11 are input to the first recognition unit 12, which recognizes the target human body based on the keypoint feature information and outputs a first recognition result. The first recognition unit 12 may perform recognition using a shallow learning network, so that accurate recognition can be performed with a small amount of calculation.
[0041] For example, the first recognition unit 12 may perform recognition by a multilayer perceptron model, which may include one input layer, two hidden layers, and one output layer. Here, the input layer may receive one 1*22 vector, which may include, for example, height and width coordinate values of nine predetermined key points and angles of four predetermined joints as shown in FIG. 3. Each hidden layer may have, for example, 100 nodes. The output device may output the probability of six types of motions, which may be, for example, walking, standing, sitting, squatting, lying, and bending. Here, the motion with the highest probability is the first recognition result of the first recognition unit 12.
[0042] In the present invention, the multilayer perceptron model is merely an example, and the first recognition unit 12 may perform recognition using other models, such as a support vector machine (SVM) model.
[0043] In at least one embodiment, the action recognition device 1 may further include a second recognition unit 13 as shown in FIG.
[0044] Here, the second recognition unit 13 corrects the first recognition result about the movement of the target body in the current image frame of the first recognition unit 12 based on the movement parameters of predetermined key points of the target body and / or the second recognition result (i.e., the historical second recognition result) output by the second recognition unit 13 about the movement of the target body in the historical image frame preceding the current image frame, and outputs the second recognition result about the movement of the target body in the current image frame.
[0045] In at least one embodiment, the motion parameters of the predetermined key points of the target human body may be obtained based on the adjusted positions of the predetermined key points of the target human body in the current image frame and the adjusted positions of the predetermined key points of the target human body in the past image frames. For example, as shown in Fig. 1, the action recognition device 1 may further include a key point motion parameter calculation unit 14, which calculates a motion vector between each of the predetermined key points of the target human body in the current image frame and each of the predetermined key points of the target human body in the past image frames, and calculates the motion parameters of the predetermined key points of the target human body based on the motion vector of each of the predetermined key points.
[0046] Here, the keypoint feature extraction unit 11 may perform processing of the steps of FIG. 4 on the information of the keypoints on the target human body in the history image frame, and obtain the adjusted positions of the predetermined keypoints on the target human body in the history image frame.
[0047] In at least one embodiment, the motion parameters of the predetermined key points of the target human body include the mean of the motion vectors of each of the predetermined key points, and / or the standard deviations of the coordinate values of each of the predetermined key points, and / or the motion direction of each of the predetermined key points, and / or the number of the predetermined key points moving along the predetermined direction. Moreover, this embodiment is not limited thereto, and the motion parameters of the predetermined key points of the target human body may be other parameters.
[0048] In addition, in the second recognition unit 13 and the key point movement parameter calculation unit 14, the target human body in the current image frame and the target human body in the history image frames correspond to the same subject person, and the target human body corresponding to the same subject person in the current image frame and the history image frames may be determined by using an object tracking technology.
[0049] Furthermore, the second recognition result output by the second recognition unit 13 for the current image frame may be input to the second recognition unit 13 for performing motion recognition on the target human body in a subsequent image frame after the current image frame.
[0050] Fig. 5 is a schematic diagram of an example of adjustment of the first recognition result by the second recognition unit 13. As shown in Fig. 5, 51 represents the first recognition result, 52 represents the second recognition result, and 53 represents the basis for the second recognition unit 13 to generate the second recognition result 53.
[0051] As shown in FIG. 5, the second recognition result may be, for example, walking, standing, sitting, squatting, lying, bending, running, crawling, getting up, falling down, jumping, etc.
[0052] As shown in FIG. 5, each of the grounds 531 to 539 may be as follows.
[0053] Rationale 531: If the first recognition result is "walking" and the average value of the motion vectors of each predetermined key point is smaller than a first threshold (thread_walk), the second recognition result is "standing".
[0054] Reason 532: If the first recognition result is "standing" and the average value of the motion vectors of each predetermined key point is equal to or greater than the first threshold (thread_walk), the second recognition result is "walking".
[0055] Reason 533: If the first recognition result is "bending" or "lying down", the second recognition result is identical to the first recognition result.
[0056] Reason 534: If the first recognition result is “standing” or “walking” and the average value of the motion vectors of each specified key point is greater than or equal to the second threshold (thread_run), the second recognition result is “running”.
[0057] Reason 535: if the first recognition result is "lying down" and the average value of the motion vectors of each predetermined key point is equal to or greater than the third threshold (thread_crawl), the second recognition result is "crawling".
[0058] Reason 536: If the first recognition result is “standing” or “walking” or “bending” and the second recognition result of the history image frame is “sitting” or “crawling” or “lying” and the number of predetermined key points moving upward is greater than 9, then the second recognition result is “standing up”.
[0059] Reason 537: If the first recognition result is “lying down” and the second recognition result of the historical image frame is “standing” or “walking” or “bending”, and the number of predetermined key points moving upward is 3 or less, then the second recognition result is “falling”.
[0060] Reason 538: If the first recognition result is “standing” or “walking” or “bending”, and the number of predetermined key points moving upward is equal to 13 or 0, and the average value of the motion vector of each predetermined key point is greater than or equal to the fourth threshold (thread_jump), then the second recognition result is “jumping”.
[0061] Reason 539: If the first recognition result is "sitting" or "squatting", the second recognition result is identical to the first recognition result.
[0062] The first recognition result 51, the second recognition result 52, and the bases 531 to 539 shown in FIG. 5 are merely examples, and the embodiments of the present invention are not limited to these.
[0063] According to the second recognition unit 13, the result of the motion recognition can be adjusted based on the relationship between the motion of the human body and time, so that the motion of the human body can be recognized more accurately.
[0064] 1, the action recognition device 1 may further include an image frame interval determination unit 15. Here, the image frame interval determination unit 15 determines the number of image frames intervening between the history image frame and the current image frame based on the image frame acquisition rate (e.g., the frame rate of the image frames captured by the camera, i.e., the number of frames per second (FPS)).
[0065] Table 1 shows the correspondence between the frame rates of the image frames and the history image frames. [Table 1]
[0066] In Table 1, if the current image frame is the nth frame, when the frame rate is 3 to 8, the previous image frame is the n-1th frame; when the frame rate is 9 to 14, the previous image frame is the n-2th frame; when the frame rate is 15 to 20, the previous image frame is the n-3th frame; when the frame rate is 21 to 26, the previous image frame is the n-4th frame; and when the frame rate is 27 to 32, the previous image frame is the n-5th frame.
[0067] In this way, by providing the image frame interval determination unit 15, when the frame rate of the image frames changes, the number of image frames intervening between the history image frame and the current image frame can be selected, so that the action recognition device 1 can be used while maintaining the above-mentioned set thresholds (for example, the first to fourth thresholds related to FIG. 5 ). This makes it possible to expand the application range of the action recognition device 1 and improve its scalability.
[0068] According to the first embodiment of the present invention, the action recognition device 1 recognizes the action of the human body based on the positions of the key points of the human body and the angles of the joints in the image frame. Since the positions of the key points and the angles of the joints are part of the information in the image frame, the first recognition unit 12 does not need to use a deep learning network and can perform accurate classification with a small amount of calculation. Furthermore, compared to a method of performing action recognition based only on the positions of the key points, the action recognition device 1 of the present invention further performs action recognition based on the angles of the joints of the human body, thereby increasing the accuracy of recognition. Furthermore, when it is necessary to add or change an action to be recognized, it is only necessary to adjust the predetermined key points and / or predetermined joints without requiring a large amount of training, so that the action recognition device 1 of the present invention has high scalability and high flexibility. Furthermore, the action recognition device 1 of the present invention can adjust the result of action recognition based on the relationship between the movement of the human body and time, so that the action of the human body can be recognized more accurately.
[0069] <Example 2> A second embodiment of the present invention provides an action recognition method corresponding to the action recognition device 1 of the first embodiment of the present invention.
[0070] 6 is a schematic diagram of an example of a behavior recognition method according to the second embodiment of the present invention. As shown in FIG. 6, the method includes the following steps:
[0071] In step 601, information of key points of a target body in an image frame is processed to calculate key point features of the target body, including positions of certain key points and angles of certain joints of the target body.
[0072] In step 602, recognize the target human body motion based on the keypoint features, and output a first recognition result.
[0073] Here, step 601 may be realized by, for example, steps 401 and 402 in FIG.
[0074] As shown in FIG. 6, the method further includes the following steps.
[0075] In step 603, based on the motion parameters of the predetermined key points of the target body and / or the second recognition result output for the motion of the target body in the previous history image frame, the first recognition result is corrected, and a second recognition result for the motion of the target body in the image frame is output.
[0076] As shown in FIG. 6, the method further includes the following steps.
[0077] In step 604, a motion vector between each predetermined key point of the target body in the image frame and each predetermined key point of the target body in the history image frame is calculated, and the motion parameters are calculated based on the motion vector of each predetermined key point.
[0078] In step 604, the motion parameters include an average value of the motion vectors of each predefined keypoint, and / or a standard deviation of the motion vectors of each predefined keypoint, and / or a motion direction of each predefined keypoint, and / or a number of predefined keypoints moving along a predefined direction.
[0079] As shown in FIG. 6, the method further includes the following steps.
[0080] In step 605, the number of intervening image frames between the current image frame and the previous image frame is determined based on the acquisition rate of the current image frames.
[0081] According to the second embodiment of the present invention, the action recognition method recognizes the action of the human body based on the positions of the key points of the human body and the angles of the joints in the image frame, and since the positions of the key points and the angles of the joints are part of the information in the image frame, there is no need to use a deep learning network, and accurate classification can be performed with a small amount of calculation. In addition, compared with a method of performing action recognition based only on the positions of the key points, the action recognition method of the present invention further performs action recognition based on the angles of the joints of the human body, thereby increasing the accuracy of recognition. Furthermore, when it is necessary to add or change an action to be recognized, it is only necessary to adjust the predetermined key points and / or predetermined joints without requiring a large amount of training, so that the action recognition method of the present invention has high scalability and high flexibility. In addition, the action recognition method of the present invention can adjust the result of action recognition based on the relationship between the movement of the human body and time, so that the action of the human body can be recognized more accurately.
[0082] <Example 3> An embodiment of the present invention provides an electronic device, and the electronic device includes the action recognition device of the first embodiment.
[0083] Fig. 7 is a schematic diagram of an example of the configuration of an electronic device according to a third embodiment of the present invention. As shown in Fig. 7, an electronic device 700 may include a central processing unit (CPU) 701 and a memory 702, and the memory 702 is connected to the central processing unit 701. The memory 702 may store various data and programs for control, and executes the programs under the control of the central processing unit 701.
[0084] In one embodiment, the functionality of the action recognition device 1 may be integrated into the central processing unit 701 .
[0085] Here, the central processing unit 701 may be configured to execute the action recognition method of the second embodiment.
[0086] 7, the electronic device 700 may further include an input / output unit 703 and a display unit 704. The functions of the above units are similar to those of the conventional technology, and the description thereof will be omitted here. The electronic device 700 may not include all of the components shown in FIG. 7. The electronic device 700 may include components not shown in FIG. 7, and may refer to the conventional technology.
[0087] An embodiment of the present invention provides a computer-readable program that, when executed in a motion recognition device or an electronic device, causes the motion recognition device or the electronic device to execute the motion recognition method of the second embodiment.
[0088] The embodiment of the present invention further provides a storage medium that stores a computer-readable program for causing the action recognition device or electronic device to execute the action recognition method of the second embodiment.
[0089] The methods / apparatuses described with reference to the embodiments of the present invention may be implemented in hardware, software modules executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in the drawings, or one or more combinations of the functional block diagrams, may correspond to each software module in a computer program flow, or each hardware module. These software modules may correspond to each step of the first embodiment. These hardware modules may be realized by implementing these software modules in hardware, for example, using a field programmable gate array (FPGA).
[0090] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor so that the processor reads information from the storage medium and writes information to the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal, or in a memory card inserted in the mobile terminal. For example, if the electronic device uses a MEGA-SIM card or a large-capacity flash memory device with a relatively large capacity, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.
[0091] One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the figures may be implemented in a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any suitable combination thereof to perform the functions described herein. One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the figures may be implemented in, for example, a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with a DSP communication, or any other configuration.
[0092] Although the present invention has been described above with reference to specific embodiments, the above description is merely illustrative and does not limit the scope of protection of the present invention. Various modifications and changes may be made to the present invention without departing from the spirit and principles of the present invention, and these modifications and changes also fall within the scope of the present invention.
[0093] Furthermore, the following supplementary notes are disclosed regarding the embodiments including the above-mentioned examples. (Appendix 1) A motion recognition device, comprising: a keypoint feature extracting unit for processing information of keypoints of a target human body in an image frame and calculating keypoint features of the target human body, the keypoint features including positions of predetermined keypoints of the target human body and angles of predetermined joints; a first recognition unit that recognizes a motion of the target human body based on the keypoint features and outputs a first recognition result. (Appendix 2) The keypoint feature extraction unit, when processing the information of the keypoints of the target human body in the image frame, Adjusting coordinate values of predetermined key points of the target body in the image frame according to a height of the target body in the image frame; The motion recognition device according to claim 1, further comprising: calculating an angle of the predetermined joint based on adjusted coordinate values of predetermined key points of the target human body in the image frame. (Appendix 3) 3. The action recognition device according to claim 2, wherein the height of the target human body in the image frame is a difference between a height coordinate value of a key point where the eyes of the target human body in the image frame are located and a height coordinate value of a key point where the ankles of the target human body in the image frame are located. (Appendix 4) The action recognition device of claim 1, further comprising: a second recognition unit that corrects the first recognition result based on motion parameters of the predetermined key points of the target human body and / or a second recognition result output by the second recognition unit for the action of the target human body in a history image frame previous to the image frame, and outputs a second recognition result for the action of the target human body in the image frame. (Appendix 5) 5. The action recognition device of claim 4, further comprising a key point motion parameter calculation unit that calculates motion vectors between each of the predetermined key points of the target human body in the image frames and each of the predetermined key points of the target human body in the history image frames, and calculates the motion parameters based on the motion vectors of each of the predetermined key points. (Appendix 6) 6. The action recognition apparatus of claim 5, wherein the motion parameters include an average value of a motion vector of each of the predetermined key points, and / or a standard deviation of a motion vector of each of the predetermined key points, and / or a motion direction of each of the predetermined key points, and / or a number of the predetermined key points moving along a predetermined direction. (Appendix 7) 5. The action recognition device according to claim 4, further comprising an image frame interval determination unit that determines a number of image frames intervening between a history image frame and the image frame based on an acquisition rate of the image frames. (Appendix 8) 8. An electronic device comprising the motion recognition device according to any one of claims 1 to 7. (Appendix 9) A motion recognition method, comprising: processing information of key points of a target body in an image frame and calculating key point features of the target body, the key point features including positions of predetermined key points of the target body and angles of predetermined joints; recognizing a motion of the target human body based on the keypoint features, and outputting a first recognition result. (Appendix 10) The step of processing information of key points of the target human body in the image frame includes: adjusting coordinate values of predetermined key points of the target body in the image frame according to a height of the target body in the image frame; and calculating angles of the predetermined joints based on adjusted coordinate values of predetermined key points of the target human body in the image frames. (Appendix 11) 11. The action recognition method according to claim 10, wherein the height of the target human body in the image frame is a difference between a height coordinate value of a key point where the eyes of the target human body in the image frame are located and a height coordinate value of a key point where the ankles of the target human body in the image frame are located. (Appendix 12) 10. The action recognition method of claim 9, further comprising a step of correcting the first recognition result based on motion parameters of the predetermined key points of the target body and / or a second recognition result output for the action of the target body in a previous history image frame to the image frame, and outputting a second recognition result for the action of the target body in the image frame. (Appendix 13) 13. The action recognition method of claim 12, further comprising the steps of: calculating motion vectors between each of the predetermined key points of the target body in the image frames and each of the predetermined key points of the target body in the history image frames; and calculating the motion parameters based on the motion vectors of each of the predetermined key points. (Appendix 14) 14. The method of claim 13, wherein the motion parameters include an average value of a motion vector of each of the predetermined key points, and / or a standard deviation of a motion vector of each of the predetermined key points, and / or a motion direction of each of the predetermined key points, and / or a number of the predetermined key points moving along a predetermined direction. (Appendix 15) 13. The action recognition method of claim 12, further comprising: determining a number of image frames intervening between the history image frame and the image frame based on an acquisition rate of the image frames.
Claims
1. A motion recognition device, comprising: a keypoint feature extracting unit for processing information of keypoints of a target human body in an image frame and calculating keypoint features of the target human body, the keypoint features including positions of predetermined keypoints of the target human body and angles of predetermined joints; a first recognition unit that recognizes a motion of the target human body based on the keypoint features and outputs a first recognition result; The keypoint feature extraction unit, when processing the information of the keypoints of the target human body in the image frame, Adjusting coordinate values of predetermined key points of the target body in the image frame according to a height of the target body in the image frame; Calculate the angle of the predetermined joint based on the adjusted coordinate value of the predetermined key point of the target human body in the image frame; a height of the target human body in the image frame being a difference between a height coordinate value of a key point where the eyes of the target human body are located in the image frame and a height coordinate value of a key point where the ankles of the target human body are located in the image frame.
2. 2. The action recognition device according to claim 1, further comprising: a second recognition unit that corrects the first recognition result based on motion parameters of the predetermined key points of the target body and / or a second recognition result output by the second recognition unit for the action of the target body in a history image frame preceding the image frame, and outputs a second recognition result for the action of the target body in the image frame.
3. 3. The action recognition device according to claim 2, further comprising a key point motion parameter calculation unit that calculates motion vectors between each of the predetermined key points of the target human body in the image frame and each of the predetermined key points of the target human body in the history image frame, and calculates the motion parameters based on the motion vectors of each of the predetermined key points.
4. 4. The action recognition device according to claim 3, wherein the motion parameters include an average value of a motion vector of each of the predetermined key points, and / or a standard deviation of a motion vector of each of the predetermined key points, and / or a motion direction of each of the predetermined key points, and / or a number of the predetermined key points moving along a predetermined direction.
5. The action recognition device according to claim 2 , further comprising an image frame interval determination unit that determines a number of image frames intervening between a history image frame and the image frame based on an acquisition rate of the image frames.
6. An electronic device comprising the action recognition device according to any one of claims 1 to 5.
7. A motion recognition method, comprising: processing information of key points of a target body in an image frame and calculating key point features of the target body, the key point features including positions of predetermined key points of the target body and angles of predetermined joints; recognizing a motion of the target human body based on the keypoint features, and outputting a first recognition result; When processing the keypoint information of the target human body in the image frame, Adjusting coordinate values of predetermined key points of the target body in the image frame according to a height of the target body in the image frame; Calculate the angle of the predetermined joint based on the adjusted coordinate value of the predetermined key point of the target human body in the image frame; a height of the target human body in the image frame is a difference between a height coordinate value of a key point where the eyes of the target human body are located in the image frame and a height coordinate value of a key point where the ankles of the target human body are located in the image frame.
8. 8. The action recognition method of claim 7, further comprising: correcting the first recognition result based on motion parameters of the predetermined key points of the target body and / or a second recognition result output for the action of the target body in a history image frame preceding the image frame, and outputting a second recognition result for the action of the target body in the image frame.
Citation Information
Patent Citations
Program, device, and method for recognizing actions of persons using a plurality of recognition engines
JP2019144830A
