Motion recognition device, method, and electronic device

Abstract

To provide a motion recognition device, a method and an electronic device.SOLUTION: The method includes performing key-point recognition on an object in a video frame using a neural network, obtaining key-point information and a PAF score of the object, making key-point connections based on the key-point information and the PAF score, generating a plurality of key-point connection candidates based on the key-point connection results, determining whether one of at least two key-point connection candidates of the plurality of key-point connection candidates is valid to perform selection among the plurality of key-point connection candidates, as well as recognizing the motion of the object based on the selected key-point connection candidates.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to the technical field of video detection.

Background Art

[0002] Currently, when performing motion recognition (also referred to as pose estimation) on one or more objects in a video frame, two strategies of "from top to bottom" and "from bottom to top" can be adopted. In the "from top to bottom" strategy, first, an object (for example, a human body) is detected, and then the pose of each object is independently estimated in each detected image region. In the "from bottom to top" strategy, first, information on a plurality of key points (or key parts) is detected, and then these key points are connected to generate connection candidates, and based on these connection candidates, the pose of each object is estimated.

[0003] Among them, the "from bottom to top" scheme includes, for example, the open-source OpenPose, and by using the Part Affinity Field (PAF), an associated score can be used. The PAF encodes the position and direction of the parts (for example, limbs) of an object in the image domain and the confidence map (CMAP), and in the CMAP, the peak value corresponds to each visible part of each object (for example, a human body).

[0004] Note that the introduction of the above background art is for clearly and completely explaining the technical solution of the present invention and for facilitating understanding by those skilled in the art. These technical solutions should not be construed as well-known to those skilled in the art just because they are described in the background art of the present invention.

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the inventors discovered the following: In the conventional "bottom-to-top" scheme, it is possible to generate duplicate or incorrect keypoint connection candidates for the same part of an object based on the results of keypoint connections. Performing motion recognition based on these duplicate or incorrect keypoint connection candidates may reduce the accuracy of the object's motion recognition results.

[0006] To solve at least one of the above-mentioned problems, embodiments of the present invention aim to provide an action recognition device, method, and electronic device that can improve the accuracy of action recognition results in a "bottom-to-top" scheme. [Means for solving the problem]

[0007] According to one aspect of the embodiments of the present invention, a motion recognition device is provided, which is, A keypoint recognition unit that uses a neural network to recognize keypoints in objects within video frames and obtains keypoint information and PAF scores for those objects; A keypoint connection unit that performs keypoint connection based on the aforementioned keypoint information and the PAF score; A connection candidate generation unit that generates multiple keypoint connection candidates based on the results of the aforementioned keypoint connection; A connection candidate determination unit that makes a selection among the plurality of keypoint connection candidates by determining whether at least two of the plurality of keypoint connection candidates are valid; and The system includes a motion recognition unit that performs motion recognition of the object based on selected keypoint connection candidates.

[0008] According to another aspect of the embodiments of the present invention, a motion recognition method is provided, which is, Using a neural network, keypoint recognition is performed for objects in video frames, and the keypoint information and PAF score of the said objects are obtained; Keypoint connections are made based on the aforementioned keypoint information and the PAF score; Based on the results of the aforementioned keypoint connections, multiple keypoint connection candidates are generated; A selection is made for the plurality of keypoint connection candidates by determining whether at least two of the plurality of keypoint connection candidates are valid; and This includes recognizing the movement of the object based on the selected keypoint connection candidates.

[0009] According to another aspect of the embodiments of the present invention, an electronic device is provided which includes a memory and a processor, wherein the memory contains a computer program, and the processor is configured to realize the above-described operation recognition method by executing the computer program. [Effects of the Invention]

[0010] The advantageous effects of the embodiment of the present invention are at least as follows: a selection is made from a plurality of keypoint connection candidates by determining whether at least two of the keypoint connection candidates are valid; and the motion of the object is recognized based on the selected keypoint connection candidate. By making a selection from the generated keypoint connection candidates in this way, the accuracy of the motion recognition result in the "bottom-to-top" scheme can be improved.

[0011] Furthermore, features described and / or shown in one embodiment may be used in the same or similar manner in one or more other embodiments, combined with or substituting features in other embodiments.

[0012] When used herein, terms such as “contains / have” refer to the presence of a feature, element, step, or assembly, but do not exclude the presence or addition of one or more other features, elements, steps, or assemblies. [Brief explanation of the drawing]

[0013] Elements and features described in one drawing or one embodiment of the present invention can be combined with elements and features shown in one or more other drawings or embodiments. Furthermore, similar reference numerals in the drawings are used to indicate corresponding parts in several drawings and to indicate corresponding parts used in multiple embodiments. [Figure 1] This figure shows a motion recognition method in an embodiment of the present invention. [Figure 2] This figure shows a candidate keypoint connection in an embodiment of the present invention. [Figure 3] This figure shows another keypoint connection candidate in an embodiment of the present invention. [Figure 4] This figure shows yet another keypoint connection candidate in an embodiment of the present invention. [Figure 5] This figure shows other keypoint connection candidates in an embodiment of the present invention. [Figure 6] This figure shows other keypoint connection candidates in an embodiment of the present invention. [Figure 7] This figure shows the adjustment to the bounding box in an embodiment of the present invention. [Figure 8] This figure shows a motion recognition device according to an embodiment of the present invention. [Figure 9] This figure shows an electronic device in an embodiment of the present invention. [Modes for carrying out the invention]

[0014] By referring to the attached drawings and the following description, the aforementioned and other features of the present invention will become apparent. In the specification and drawings, specific embodiments of the present invention are disclosed, but they only show some of the embodiments that can adopt the principles of the present invention. It should be understood that the present invention is not limited to the described embodiments, that is, the present invention also includes all changes, modifications, and alternatives belonging to the scope of the attached patent claims.

[0015] In an embodiment of the present invention, the object as the detection target may be a human body of each age group, for example, an elderly person, a child, an elderly person and / or a caregiver, a child and / or a protector, etc. It should be noted that the present invention is not limited to these. The object as the detection target may also be a human body with biological characteristics, a robot without biological characteristics, etc.

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0017] <Embodiment of the first side> In an embodiment of the present invention, an action recognition method is provided. FIG. 1 is a diagram showing the action recognition method in an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps (operations).

[0018] 101: Use a neural network to perform keypoint recognition on the object in the video frame, and obtain the keypoint information and PAF score of the object; 102: Perform keypoint connection based on the keypoint information and PAF score; 103: Generate a plurality of keypoint connection candidates based on the result of keypoint connection; 104: For at least two of the plurality of keypoint connection candidates, determine whether one of the keypoint connection candidates is valid, and perform selection on the plurality of keypoint connection candidates; and 105: Perform action recognition of the object based on the selected keypoint connection candidate.

[0019] Figure 1 above is provided to illustrate an embodiment of the present invention, but the present invention is not limited thereto. For example, the execution order between each operation can be appropriately adjusted, or some operations can be added or removed. Those skilled in the art may make appropriate modifications based on the above description, not limited to the description in Figure 1.

[0020] In several embodiments, keypoint recognition is performed using neural networks based on ResNet, DenseNet, etc., to obtain object PAF information and confidence map information; keypoint information is obtained based on the confidence map information; and the PAF score between any two keypoints can be calculated based on the PAF information and keypoint information. For specific details regarding neural networks, PAF, confidence maps, PAF scores, etc., please refer to related technologies.

[0021] In an embodiment of the present invention, keypoint connections can be made based on keypoint information and PAF scores, and multiple keypoint connection candidates can be generated based on the results of the keypoint connections.

[0022] For example, using a keypoint clustering algorithm, a binary image can be generated after removing low-confidence keypoints by applying a threshold (e.g., 0.1) to the confidence map. Relationship pairs (connections) between one keypoint and another can be found, and the relationship with the minimum affinity between the two keypoints can be selected as the final relationship pair. After checking (traversing) all relationship pairs one by one, keypoints of the same person are connected to draw a skeleton. By connecting and combining the skeletons of multiple people, multiple keypoint connection candidates can be generated.

[0023] Figure 2 shows keypoint connection candidates in an embodiment of the present invention. As shown in Figure 2, for example, three people appear in the video frame (as shown in 201, 202, and 203), and after processing such as keypoint recognition, one or more keypoint connection candidates are generated for each person. One or more connected (communicated) keypoints may be referred to as a single keypoint connection candidate, but is not limited to this.

[0024] In embodiments of the present invention, object motion recognition can be performed based on keypoint connection candidates. For example, motion recognition (pose detection) of multiple people can be performed based on OpenPose. However, if duplicate or incorrect keypoint connection candidates are generated, the accuracy of motion recognition can be negatively affected. For example, in Figure 2, four keypoint connection candidates are generated for the human body shown at 203, but some of these four keypoint connection candidates are duplicates, and some are even incorrect. Therefore, subsequent motion recognition becomes inaccurate.

[0025] In embodiments of the present invention, the accuracy of motion recognition can be further improved by making a selection from multiple keypoint connection candidates before motion recognition. Below, the selection from multiple keypoint connection candidates will be explained illustratively, using the example that at least two keypoint connection candidates include a first keypoint connection candidate and a second keypoint connection candidate.

[0026] In some embodiments, the validity of a first keypoint connection candidate can be determined based on the position and size of the boundary frame of the first keypoint connection candidate, and the position and size of the boundary frame of the second keypoint connection candidate.

[0027] For example, based on the position and size of the boundary frame, it can be determined whether the boundary frame of the first keypoint connection candidate is covered by the boundary frame of the second keypoint connection candidate. If the boundary frame of the first keypoint connection candidate is covered by the boundary frame of the second keypoint connection candidate, it can be determined that the first keypoint connection candidate is a subset of the second keypoint connection candidate. Therefore, since the first keypoint connection candidate is a redundant part of the second keypoint connection candidate, it can be determined that the first keypoint connection candidate is invalid, and it can be dropped.

[0028] Furthermore, for example, if the area of ​​the boundary frame of the first keypoint connection candidate is smaller than a predetermined threshold, and the area of ​​the boundary frame of the second keypoint connection candidate is greater than or equal to the predetermined threshold, the first keypoint connection candidate may be a connection candidate resulting from a false detection, and the second keypoint connection candidate can be confirmed as the correct connection candidate. This allows the first keypoint connection candidate to be confirmed as invalid, and the first keypoint connection candidate can be dropped.

[0029] In several embodiments, it is possible to determine whether the first keypoint connection candidate is valid based on the number of keypoints in the first keypoint connection candidate and the number of keypoints in the second keypoint connection candidate.

[0030] For example, if the number of keypoints in the first keypoint connection candidate is smaller than the number of keypoints in the second keypoint connection candidate, it can be determined that the first keypoint connection candidate may be a subset of the second keypoint connection candidate. Therefore, since the first keypoint connection candidate is a redundant part of the second keypoint connection candidate, it can be determined that the first keypoint connection candidate is invalid, and it can be dropped.

[0031] Furthermore, for example, if the number of keypoints in the first keypoint connection candidate is smaller than a predetermined threshold, and the number of keypoints in the second keypoint connection candidate is equal to or greater than the predetermined threshold, it can be determined that the first keypoint connection candidate is a connection candidate resulting from a false detection, and that the second keypoint connection candidate is the correct connection candidate. As a result, it can be determined that the first keypoint connection candidate is invalid, and that first keypoint connection candidate can be dropped.

[0032] In some embodiments, the validity of a first keypoint connection candidate can be determined based on the number of keypoints and the position and size of the boundary frame of the first keypoint connection candidate, as well as the number of keypoints and the position and size of the boundary frame of the second keypoint connection candidate.

[0033] Specifically, if the boundary frame of the first keypoint connection candidate is covered by the boundary frame of the second keypoint connection candidate, it can be determined whether the number of keypoints of the first keypoint connection candidate is less than the first ratio of the total number of keypoints of the object, and whether the number of keypoints of the second keypoint connection candidate is greater than the second ratio of the total number of keypoints of the object.

[0034] If the number of keypoints in the first keypoint connection candidate is less than the first ratio of the total number of keypoints in the object, and the number of keypoints in the second keypoint connection candidate is greater than the second ratio of the total number of keypoints in the object, the first keypoint connection candidate is determined to be invalid, and the first keypoint connection candidate is dropped.

[0035] For example, as shown in Table 1 below. [Table 1]

[0036] Figure 3 is a diagram showing another keypoint connection candidate in an embodiment of the present invention, showing one keypoint connection candidate 301 (second keypoint connection candidate) generated for the human body 203 in Figure 2, and as shown in Figure 3, the keypoint connection candidate 301 has 15 keypoints connected to each other, and Figure 3 shows a boundary frame containing the 15 keypoints.

[0037] Figure 4 is a diagram showing yet another keypoint connection candidate in an embodiment of the present invention, showing yet another keypoint connection candidate 302 (first keypoint connection candidate) generated for the human body 203 in Figure 2, and as shown in Figure 4, the keypoint connection candidate 302 has three keypoints connected to each other, and Figure 4 shows a boundary frame containing the three keypoints.

[0038] Figure 5 is a diagram showing another keypoint connection candidate in an embodiment of the present invention, showing another keypoint connection candidate 303 (first keypoint connection candidate) generated for the human body 203 in Figure 2, and as shown in Figure 5, the keypoint connection candidate 303 has one keypoint, and the boundary frame containing the one keypoint is not shown in Figure 5.

[0039] Figure 6 is a diagram showing another keypoint connection candidate in an embodiment of the present invention, showing another keypoint connection candidate 304 (first keypoint connection candidate) generated for the human body 203 in Figure 2, and as shown in Figure 6, the keypoint connection candidate 304 has four keypoints connected to each other, and Figure 6 shows a boundary frame containing these four keypoints.

[0040] As shown in Figures 3 to 6, keypoint connection candidates 301 to 304 pertain to the same human body 203 and can be selected using the scheme described above. For example, K=18, α=1 / 3, and β=2 / 3. However, the present invention is not limited to these, and the specific values ​​of these parameters can be set according to the actual scene.

[0041] Regarding keypoint connection candidates 301 and 302, the boundary frame of keypoint connection candidate 301 covers the boundary frame of keypoint connection candidate 302. However, the number of keypoints in keypoint connection candidate 301 is 15, which is greater than 12 (18 * 2 / 3), and the number of keypoints in keypoint connection candidate 301 is 3, which is less than 6 (18 * 1 / 3). Therefore, keypoint connection candidate 302 is dropped.

[0042] Regarding keypoint connection candidates 301 and 303, the boundary frame of keypoint connection candidate 301 covers the boundary frame of keypoint connection candidate 303. However, the number of keypoints in keypoint connection candidate 301 is 15, which is greater than 12 (18 * 2 / 3), and the number of keypoints in keypoint connection candidate 303 is 1, which is less than 6 (18 * 1 / 3). Therefore, keypoint connection candidate 303 is dropped.

[0043] Regarding keypoint connection candidates 301 and 304, the boundary frame of keypoint connection candidate 301 covers the boundary frame of keypoint connection candidate 303. However, the number of keypoints in keypoint connection candidate 301 is 15, which is greater than 12 (18 * 2 / 3), and the number of keypoints in keypoint connection candidate 304 is 4, which is less than 6 (18 * 1 / 3). Therefore, keypoint connection candidate 304 is dropped.

[0044] Therefore, by making selections based on the number of keypoints and the position and size of the bounding box, duplicate or incorrect keypoint connection candidates can be removed more accurately, thereby further improving the accuracy of motion recognition.

[0045] In the embodiments of the present invention, the boundary frame can be directly acquired after processing such as keypoint detection and keypoint connection. To further improve the accuracy of selecting keypoint connection candidates, appropriate adjustments can also be made to the boundary frames of the keypoint connection candidates.

[0046] In some embodiments, the boundary frame of the first keypoint connection candidate is the smallest first rectangular frame that includes all the keypoints in the first keypoint connection candidate, and the boundary frame of the second keypoint connection candidate is the smallest second rectangular frame that includes all the keypoints in the second keypoint connection candidate.

[0047] In some embodiments, the boundary frame of the first keypoint connection candidate is the smallest first rectangular frame that includes all the keypoints in the first keypoint connection candidate, and the boundary frame of the second keypoint connection candidate is a rectangular frame that is enlarged by a predetermined ratio with respect to the long side of the second rectangular frame and / or enlarged by a predetermined ratio with respect to the short side of the second rectangular frame.

[0048] Figure 7 shows an example of adjusting the boundary frame in an embodiment of the present invention. As shown in Figure 7, 701 shows the smallest rectangular frame containing all keypoints, 702 shows the rectangular frame after being enlarged by a predetermined ratio with respect to the long side of the rectangular frame 701 and the short side of the rectangular frame 701, 703 shows the rectangular frame after being enlarged by a predetermined ratio with respect to the short side of the rectangular frame 701, and 704 shows the rectangular frame after being enlarged by a predetermined ratio with respect to the long side of the rectangular frame 701.

[0049] Although the steps or processes of the present invention have been described above, the present invention is not limited thereto. The motion recognition method may further include other steps or processes, and the specific details of these steps or processes can be found in the prior art. Furthermore, although embodiments of the present invention have been described above using several structures of motion recognition models as examples, the present invention is not limited to these structures, and appropriate modifications may be made to these structures, and all methods of implementing these modifications are also included within the scope of embodiments of the present invention.

[0050] The embodiments described above are for illustrative purposes to illustrate embodiments of the present invention, but the present invention is not limited thereto, and appropriate modifications may be made based on the embodiments described above. For example, the embodiments described above may be used individually, or a combination of several of the embodiments described above may be used.

[0051] As can be seen from the above embodiment, a selection is made from the multiple keypoint connection candidates by determining whether at least two of the keypoint connection candidates are valid, and the movement of the object is recognized based on the selected keypoint connection candidate. By making a selection from the generated keypoint connection candidates in this way, the accuracy of the object movement recognition result in the "bottom-to-top" scheme can be improved.

[0052] <Example of the second aspect> An embodiment of the present invention provides a motion recognition device, and the same description as in the embodiment of the first aspect is omitted here.

[0053] Figure 8 shows an action recognition device in an embodiment of the present invention, and as shown in Figure 8, the action recognition device 800 includes the following:

[0054] Keypoint recognition unit 801: Uses a neural network to recognize keypoints in objects within video frames, and acquires keypoint information and PAF scores for the objects; Keypoint connection unit 802: Performs keypoint connections based on keypoint information and PAF score; Connection candidate generation unit 803: Generates multiple keypoint connection candidates based on the results of keypoint connections; Connection candidate confirmation unit 804: For at least two of the multiple keypoint connection candidates, confirm whether one of them is valid, and make a selection for the multiple keypoint connection candidates; and Motion recognition unit 805: Performs motion recognition of an object based on the selected keypoint connection candidates.

[0055] In some embodiments, the keypoint recognition unit 801 is specifically used to recognize and acquire PAF information and confidence map information of an object based on the keypoints; to acquire the keypoint information based on the confidence map information; and to calculate the PAF score between any two keypoints based on the PAF information and the keypoint information.

[0056] In some embodiments, the at least two keypoint connection candidates include a first keypoint connection candidate and a second keypoint connection candidate.

[0057] In some embodiments, the connection candidate determination unit 804 determines whether the first keypoint connection candidate is valid based on the position and size of the boundary frame of the first keypoint connection candidate and the position and size of the boundary frame of the second keypoint connection candidate.

[0058] In some embodiments, the connection candidate determination unit 804 further determines whether the first key point connection candidate is valid based on the number of key points of the first key point connection candidate and the number of key points of the second key point connection candidate.

[0059] In some embodiments, the connection candidate determination unit 804 is used to determine whether the number of keypoints of the first keypoint connection candidate is less than a first ratio of the total number of keypoints of the object, and whether the number of keypoints of the second keypoint connection candidate is greater than a second ratio of the total number of keypoints of the object, if the boundary frame of the first keypoint connection candidate is covered by the boundary frame of the second keypoint connection candidate; and to determine that the first keypoint connection candidate is invalid and to drop the first keypoint connection candidate if the number of keypoints of the first keypoint connection candidate is less than a first ratio of the total number of keypoints of the object, and the number of keypoints of the second keypoint connection candidate is greater than a second ratio of the total number of keypoints of the object.

[0060] In some embodiments, the boundary frame of the first keypoint connection candidate is the smallest first rectangular frame containing all the keypoints in the first keypoint connection candidate, the boundary frame of the second keypoint connection candidate is the smallest second rectangular frame containing all the keypoints in the second keypoint connection candidate, or the boundary frame of the second keypoint connection candidate is a rectangular frame that is enlarged by a predetermined ratio with respect to the long side of the second rectangular frame and / or enlarged by a predetermined ratio with respect to the short side of the second rectangular frame.

[0061] Although the components or modules according to the present invention have been described above, the present invention is not limited thereto. The motion recognition device 800 may further include other components or modules, and the specific details of these components or modules can be found in the prior art.

[0062] For convenience, Figure 8 only shows the connection relationships or signal directions between each component or module; however, those skilled in the art should understand that various related technologies, such as buzz connections, may be employed. These components or modules may be implemented by hardware such as processors or memory units, but the embodiments of the present invention are not limited to this.

[0063] The embodiments described above are for illustrative purposes only, and the present invention is not limited thereto. Appropriate modifications may be made based on the embodiments described above. For example, the embodiments described above may be used individually, or a combination of several of the embodiments described above may be used.

[0064] As can be seen from the above embodiment, a selection is made from the multiple keypoint connection candidates by determining whether at least two of the keypoint connection candidates are valid, and the movement of the object is recognized based on the selected keypoint connection candidate. By making a selection from the generated keypoint connection candidates in this way, the accuracy of the object movement recognition result in the "bottom-to-top" scheme can be improved.

[0065] <Example of the third side> An embodiment of the present invention provides an electronic device, which includes the motion recognition device 800 described in the embodiment of the second aspect, and the contents thereof are incorporated herein by reference. This electronic device may be, for example, a computer, server, workstation, laptop computer, smartphone, etc., but the embodiments of the present invention are not limited to these.

[0066] Figure 9 shows an electronic device in an embodiment of the present invention. As shown in Figure 9, the electronic device 900 may include a processor (e.g., a central processor CPU) 910 and a memory unit 920, the memory unit 920 being connected to the central processor 910. The memory unit 920 can store various types of data, and can also store a program 921 for information processing, and can execute the program 921 under the control of the processor 910.

[0067] In some embodiments, the functions of the motion recognition device 800 are integrated into the processor 910. In some of these embodiments, the processor 910 is configured to implement the motion recognition method described in the first embodiment.

[0068] In some embodiments, the motion recognition device 800 is located independently of the processor 910. For example, the motion recognition device 800 is configured as a chip connected to the processor 910, and the functions of the motion recognition device 800 can be achieved by controlling the processor 910.

[0069] For example, the processor 910 may be configured to perform the following control: using a neural network to recognize keypoints in an object in a video frame and to obtain keypoint information and PAF scores for the object; to perform keypoint connections based on the keypoint information and PAF scores; to generate a plurality of keypoint connection candidates based on the results of the keypoint connections; to make a selection of the plurality of keypoint connection candidates by determining whether at least two of the plurality of keypoint connection candidates are valid; and to perform motion recognition of the object based on the selected keypoint connection candidates.

[0070] Furthermore, as shown in Figure 9, the electronic device 900 further includes an input / output (I / O) device 930, a display 940, and the functions of these components are the same as in the prior art, so a detailed explanation is omitted here. Note that the electronic device 900 does not need to include all the components shown in Figure 9. Also, the electronic device 900 may include components not shown in Figure 9, for which related technologies can be consulted.

[0071] In embodiments of the present invention, a computer-readable program is further provided, wherein when the program is executed in an electronic device, the program causes the computer to execute the operation recognition method described in the first embodiment in the electronic device.

[0072] In embodiments of the present invention, a storage medium storing a computer-readable program is further provided, wherein the computer-readable program causes a computer to execute the operation recognition method described in the first embodiment in an electronic device.

[0073] Furthermore, the above-described apparatus and method may be implemented by software or hardware, or by a combination of hardware and software. The present invention further relates to a computer-readable program as described below, that is, the program, when executed by a logic component, causes the logic component to implement the above-described apparatus or component, or to the logic component to implement the above-described various methods or steps. The logic component may be, for example, an FPGA (Field Programmable Gate Array), a microprocessor, or a processor used in a computer. The present invention further relates to a storage medium storing the above-described program, for example, a hard disk, a magnetic disk, an optical hard disk, a DVD, a flash memory, etc.

[0074] Furthermore, one or more combinations of the functional blocks shown in the drawings and / or one or more combinations of functional blocks may be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic component, discrete gate or transistor logic component, discrete hardware assembly or any other suitable combination for performing the functions described herein. Also, one or more combinations of the functional blocks shown in the drawings and / or one or more combinations of functional blocks may further be configured as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors connected to a DSP by communication or any other combination of any other configuration.

[0075] Furthermore, the following additional information is disclosed regarding the above-mentioned embodiments.

[0076] (Note 1) A method for recognizing motion, Using a neural network, keypoint recognition is performed for objects in video frames, and the keypoint information and PAF score of the said objects are obtained; Keypoint connections are made based on the aforementioned keypoint information and the PAF score; Based on the results of the aforementioned keypoint connections, multiple keypoint connection candidates are generated; A selection is made for the plurality of keypoint connection candidates by determining whether at least two of the plurality of keypoint connection candidates are valid; and A method comprising recognizing the motion of the object based on selected keypoint connection candidates.

[0077] (Note 2) The method described in Appendix 1, The recognition of the aforementioned key points is, Based on the aforementioned keypoints, the PAF information and confidence map information of the object are recognized and acquired; Based on the confidence map information, the key point information is acquired; and A method comprising calculating the PAF score between any two key points based on the PAF information and the key point information.

[0078] (Note 3) The method described in Appendix 1 or 2, The method wherein the at least two keypoint connection candidates include a first keypoint connection candidate and a second keypoint connection candidate.

[0079] (Note 4) The method described in Appendix 3, Selecting from the aforementioned multiple keypoint connection candidates is, A method comprising determining whether the first keypoint connection candidate is valid based on the position and size of the boundary frame of the first keypoint connection candidate and the position and size of the boundary frame of the second keypoint connection candidate.

[0080] (Note 5) The method described in Appendix 3 or 4, Selecting from the aforementioned multiple keypoint connection candidates is, A method comprising determining whether the first keypoint connection candidate is valid based on the number of keypoints of the first keypoint connection candidate and the number of keypoints of the second keypoint connection candidate.

[0081] (Note 6) The method described in any one of the appendices 3 to 5, Selecting from the aforementioned multiple keypoint connection candidates is, If the boundary frame of the first keypoint connection candidate is covered by the boundary frame of the second keypoint connection candidate, determine whether the number of keypoints of the first keypoint connection candidate is less than the first ratio of the total number of keypoints of the object, and determine whether the number of keypoints of the second keypoint connection candidate is greater than the second ratio of the total number of keypoints of the object; and A method comprising determining that the first keypoint connection candidate is invalid and dropping the first keypoint connection candidate if the number of keypoints of the first keypoint connection candidate is less than a first ratio of the total number of keypoints of the object, and the number of keypoints of the second keypoint connection candidate is greater than a second ratio of the total number of keypoints of the object.

[0082] (Note 7) A method according to any one of the appendices 3 to 6, The boundary frame of the first keypoint connection candidate is the smallest first rectangular frame that includes all the keypoints in the first keypoint connection candidate. A method wherein the boundary frame of the second keypoint connection candidate is the smallest second rectangular frame that includes all the keypoints in the second keypoint connection candidate, or the boundary frame of the second keypoint connection candidate is a rectangular frame that is enlarged by a predetermined ratio with respect to the long side of the second rectangular frame and / or enlarged by a predetermined ratio with respect to the short side of the second rectangular frame.

[0083] (Note 8) A storage medium that stores computer-readable programs, A storage medium that causes a computer to execute the operation recognition method described in any one of the appendices 1 to 7 within an electronic device, the computer-readable program.

[0084] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and any modification to the present invention falls within the technical scope of the present invention as long as it does not deviate from the spirit of the invention.

Claims

1. A motion recognition device, A keypoint recognition unit that uses a neural network to recognize keypoints in objects within video frames and acquires keypoint information and a PAF (Part Affinity Field) score for the objects; A keypoint connection unit that performs keypoint connection based on the aforementioned keypoint information and the aforementioned PAF score; A connection candidate generation unit that generates multiple keypoint connection candidates based on the results of the aforementioned keypoint connection; A connection candidate determination unit that makes a selection for the plurality of keypoint connection candidates by determining whether one of the at least two keypoint connection candidates is valid; and Includes a motion recognition unit that performs motion recognition of the object based on the selected keypoint connection candidates, The aforementioned at least two keypoint connection candidates include a first keypoint connection candidate and a second keypoint connection candidate. The aforementioned connection candidate confirmation unit is An action recognition device used to determine whether the first keypoint connection candidate is valid, based on the position and size of the boundary frame of the first keypoint connection candidate and the position and size of the boundary frame of the second keypoint connection candidate.

2. An action recognition device according to claim 1, The aforementioned key point recognition unit is Based on the aforementioned keypoints, the PAF information and confidence map information of the object are recognized and acquired; Based on the confidence map information, the key point information is acquired; and An motion recognition device used to calculate the PAF score between any two key points based on the PAF information and the key point information.

3. An action recognition device according to claim 1, The aforementioned connection candidate confirmation unit further, An action recognition device used to determine whether the first keypoint connection candidate is valid, based on the number of keypoints of the first keypoint connection candidate and the number of keypoints of the second keypoint connection candidate.

4. The motion recognition device according to claim 3, The aforementioned connection candidate confirmation unit is If the boundary frame of the first keypoint connection candidate is covered by the boundary frame of the second keypoint connection candidate, determine whether the number of keypoints of the first keypoint connection candidate is less than the first ratio of the total number of keypoints of the object, and determine whether the number of keypoints of the second keypoint connection candidate is greater than the second ratio of the total number of keypoints of the object; and An action recognition device used to determine that the first keypoint connection candidate is invalid and to drop the first keypoint connection candidate if the number of keypoints of the first keypoint connection candidate is less than the first ratio of the total number of keypoints of the object, and the number of keypoints of the second keypoint connection candidate is greater than the second ratio of the total number of keypoints of the object.

5. An action recognition device according to claim 1, The boundary frame of the first keypoint connection candidate is the smallest first rectangular frame that includes all the keypoints in the first keypoint connection candidate. An action recognition device wherein the boundary frame of the second keypoint connection candidate is the smallest second rectangular frame that includes all the keypoints in the second keypoint connection candidate, or the boundary frame of the second keypoint connection candidate is a rectangular frame that is enlarged by a predetermined ratio with respect to the long side of the second rectangular frame and / or enlarged by a predetermined ratio with respect to the short side of the second rectangular frame.

6. A method for recognizing motion, Using a neural network, keypoint recognition is performed for objects in video frames, and the keypoint information and PAF (Part Affinity Field) score of the said objects are obtained; Based on the aforementioned keypoint information and the PAF score, keypoint connections are made; Based on the results of the aforementioned keypoint connections, multiple keypoint connection candidates are generated; A selection is made for the plurality of keypoint connection candidates by determining whether at least two of the plurality of keypoint connection candidates are valid; and This includes recognizing the movement of the object based on the selected keypoint connection candidates, The aforementioned at least two keypoint connection candidates include a first keypoint connection candidate and a second keypoint connection candidate. Selecting from the aforementioned multiple keypoint connection candidates is, An operation recognition method that includes determining whether the first keypoint connection candidate is valid based on the position and size of the boundary frame of the first keypoint connection candidate and the position and size of the boundary frame of the second keypoint connection candidate.

7. The operation recognition method according to claim 6, Making a selection from the aforementioned multiple keypoint connection candidates further involves An operation recognition method that includes determining whether the first keypoint connection candidate is valid based on the number of keypoints of the first keypoint connection candidate and the number of keypoints of the second keypoint connection candidate.

8. Electronic equipment including memory devices and processors, The aforementioned memory contains a computer program, An electronic device configured to execute the computer program to realize the operation recognition method described in claim 6 or 7.

Images