Action recognition devices and methods and electronic devices

Abstract

This application provides an action recognition device, method, and electronic device. The method includes: obtaining keypoint information and partial affinity field information of an object in a video frame using a neural network; calculating a partial affinity field score corresponding to two adjacent keypoints based on the keypoint information, the partial affinity field information, and the distance between two adjacent keypoints, wherein the partial affinity field score is negatively correlated with the distance between the two adjacent keypoints; connecting keypoints based on the keypoint information and the partial affinity field score to generate multiple keypoint connection candidates; determining a skeleton map of the object based on the keypoint candidates; and performing action recognition of the object based on the skeleton map. This improves the accuracy of action recognition results in bottom-up approaches.

Description

Technical Field

[0001] This application relates to the field of video detection technology. Background Technology

[0002] Currently, action recognition (or pose estimation) of one or more objects in a video frame can be performed using two strategies: top-down and bottom-up. In the top-down strategy, objects (such as the human body) are first detected, and then the pose of each object is estimated independently in each detected image region. In the bottom-up strategy, multiple keypoints (or key parts) are first detected, and then these keypoints are connected to generate connection candidates. The pose of each object is estimated based on these connection candidates.

[0003] Bottom-up approaches include, for example, the open-source OpenPose; associated scores can be used through the Part Affinity Field (PAF). PAF encodes the location and orientation of object parts (e.g., limbs) in the image domain and confidence map (CMAP); in CMAP, peaks correspond to each visible part of each object (e.g., the human body).

[0004] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and for the convenience of those skilled in the art to understand them. It should not be assumed that the above technical solutions are known to those skilled in the art simply because these solutions have been described in the background section of this application. Summary of the Invention

[0005] However, the inventors discovered that current bottom-up approaches, when clustering keypoints, tend to associate keypoints belonging to different objects as keypoints of the same object. This can potentially generate incorrect keypoint connection candidates. If action recognition is performed based on these incorrect keypoint connection candidates, the accuracy of the object's action recognition results is reduced.

[0006] To address at least one of the aforementioned technical problems, embodiments of this application provide an action recognition device, method, and electronic device, with the aim of improving the accuracy of action recognition results in bottom-up approaches.

[0007] According to one aspect of the embodiments of this application, an action recognition device is provided, comprising:

[0008] The key point recognition unit uses a neural network to process objects in a video frame to obtain key point information and partial affinity field information of the objects;

[0009] The score calculation unit calculates the partial affinity field score corresponding to the two adjacent key points based on the key point information, the partial affinity field information, and the distance between the two adjacent key points. The partial affinity field score is negatively correlated with the distance between the two adjacent key points.

[0010] A key point connection unit connects key points based on the key point information and the partial affinity field score to generate multiple key point connection candidates.

[0011] A skeleton determination unit determines the skeleton map of the object based on the key point candidates; and

[0012] An action recognition unit performs action recognition on the object based on the skeletal diagram.

[0013] According to another aspect of the embodiments of this application, an action recognition method is provided, including:

[0014] A neural network is used to process objects in video frames to obtain key point information and partial affinity field information of the objects;

[0015] The partial affinity field score corresponding to the two adjacent key points is calculated based on the key point information, the partial affinity field information, and the distance between the two adjacent key points. The partial affinity field score is negatively correlated with the distance between the two adjacent key points.

[0016] Keypoints are connected based on the keypoint information and the partial affinity field score to generate multiple keypoint connection candidates;

[0017] Determine the skeleton map of the object based on the key point candidates; and

[0018] Action recognition of the object is performed based on the skeletal diagram.

[0019] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to implement the action recognition method as described above.

[0020] One of the beneficial effects of this application's embodiments is that: based on the distance between two adjacent key points, a partial affinity field score corresponding to those two key points is calculated, such that the partial affinity field score is negatively correlated with the distance between the two adjacent key points. This improves the accuracy of action recognition results in bottom-up approaches.

[0021] Referring to the following description and accompanying drawings, specific implementation methods of the embodiments of this application are disclosed in detail, indicating how the principles of the embodiments of this application can be adopted. It should be understood that the implementation methods of this application are not limited in scope. Within the spirit and scope of the appended claims, the implementation methods of this application include many changes, modifications, and equivalents. Attached Figure Description

[0022] The accompanying drawings, which form part of the specification, are used to provide a further understanding of the embodiments of this application and illustrate the implementation methods of this application, together with the textual description, to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other implementation methods based on these drawings without creative effort. In the drawings:

[0023] Figure 1 This is a schematic diagram of the action recognition method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the action recognition process according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an action recognition device according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] Referring to the accompanying drawings, the foregoing and other features of the embodiments of this application will become apparent from the following description. Specific embodiments of this application are specifically disclosed in the description and drawings, illustrating partial implementations in which the principles of the embodiments of this application can be adopted. It should be understood that this application is not limited to the described embodiments; rather, the embodiments of this application include all modifications, variations, and equivalents falling within the scope of the appended claims.

[0028] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish different elements by name, but do not indicate the spatial arrangement or chronological order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the terms listed in association and all combinations thereof. The terms "comprising," "including," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0029] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an." Furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context explicitly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context explicitly indicates otherwise.

[0030] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments. The term "comprising / including" as used herein means the presence of a feature, integral, step, or component, but does not exclude the presence or addition of one or more other features, integrals, steps, or components.

[0031] In the embodiments of this application, the object of detection can be a human body of various ages, such as the elderly, children, or elderly and / or caregivers, children and / or guardians. This application is not limited to these; the object of detection can be a human body or other animal body with vital characteristics, or a machine without vital characteristics, etc.

[0032] First aspect of the embodiments

[0033] This application provides an action recognition method. Figure 1 This is a schematic diagram of an action recognition method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0034] 101. Use neural networks to process objects in video frames to obtain key point information and partial affinity field information of the objects;

[0035] 102. Calculate the partial affinity field score corresponding to the two adjacent key points based on key point information, partial affinity field information, and the distance between the two adjacent key points. The partial affinity field score is negatively correlated with the distance between the two adjacent key points.

[0036] 103. Connect key points based on key point information and partial affinity field scores to generate multiple key point connection candidates;

[0037] 104. Determine the object's skeleton based on keypoint candidates; and

[0038] 105. Perform object action recognition based on the skeletal diagram.

[0039] It is worth noting that the above appendix Figure 1The embodiments of this application have only been illustrated schematically, and the application is not limited thereto. For example, the execution order between various operations can be appropriately adjusted, and other operations can be added or some operations can be removed. Those skilled in the art can make appropriate modifications based on the above description, and are not limited to the above-described embodiments. Figure 1 The records.

[0040] Figure 2 This is a schematic diagram of the action recognition process according to an embodiment of this application. In some embodiments, such as Figure 2 As shown, objects in video frames or images can be processed using neural networks based on ResNet or DenseNet to obtain the object's Part Affinity Field (PAF) and Confidence Map (CMAP) information. In some embodiments, parallel sub-networks can be used to obtain PAF and CMAP information separately. This application is not limited to this; other network models can also be used to generate PAF and CMAP information. For specific details regarding neural networks, please refer to related technologies.

[0041] In some embodiments, such as Figure 2 As shown, key point information of objects in a video frame or image can be obtained based on CMAP information. For details on obtaining key point information from CMAP information, please refer to relevant technologies.

[0042] In some embodiments, when there are multiple objects in a video frame or image, the obtained keypoint information can be a set of keypoint information corresponding to the multiple objects. Therefore, in order to identify the action of each object, the obtained keypoints need to be clustered to find the set of keypoints belonging to the same object.

[0043] In some embodiments, such as Figure 2 As shown, keypoints can be connected based on keypoint information and PAF scores to filter out redundant connections and generate keypoint connection candidates. Redundant connections can be those linking keypoints of different objects. Then, based on the principle that adjacent limb parts share common keypoints, keypoint connection candidates belonging to the same object can be combined through keypoint retrieval and other operations to obtain the object's skeleton map. Subsequently, actions of the object can be performed based on this skeleton map. For details on keypoint connection candidate generation, skeleton map generation, and action recognition, please refer to relevant technologies.

[0044] In some embodiments, the PAF score can be a value reflecting the probability that the two endpoints (keypoints) of a corresponding connection belong to the same object. For example, the higher the PAF score, the higher the probability that the two keypoints belong to the same object. Therefore, correctly calculating the PAF score will help improve the accuracy of keypoint connection candidates, and thus improve the accuracy of the skeleton map generated from the keypoint connection candidates and the accuracy of action recognition based on the skeleton map.

[0045] In some embodiments, when calculating the partial affinity field score, factors such as the distance between two adjacent keypoints can be considered, and the partial affinity field score corresponding to the two keypoints can be negatively correlated with the distance between them. Thus, when the distance between two adjacent keypoints is large, the partial affinity field scores of these two keypoints are small.

[0046] Since the distance between keypoints belonging to different objects is usually greater than the distance between adjacent keypoints belonging to the same object, when generating keypoint connection candidates using the aforementioned partial affinity field scores which are negatively correlated with distance, the possibility of incorrectly associating keypoints belonging to different objects as keypoints of the same object can be reduced. This can improve the accuracy of action recognition results in bottom-up approaches.

[0047] In some embodiments, the partial affinity field score can be calculated in various ways. For example, a first partial affinity field score can be calculated first, and then a partial affinity field score can be calculated based on the first partial affinity field score. Specifically, the first partial affinity field score between two adjacent keypoints is calculated based on keypoint information and partial affinity field information. For example, firstly, keypoints can be clustered according to the obtained keypoint information, and adjacent keypoints of two classes can be clustered into a bipartite graph; based on PAF information, pixel directions on all lines connecting the two classes of keypoints are extracted, and the direction vectors of all pixels on each line are integrated or discretely summed. The result represents the direction weight of this line, i.e., the first partial affinity field score.

[0048] For example, the affinity field score in the first part can be calculated as follows:

[0049]

[0050] or,

[0051]

[0052] Where, d j1 and d j2 For any two key points, PAF Score (d j1 dj2 ), To be related to key point d j1 and d j2 The corresponding first part of the affinity field score, p(u), p(u) i ) is d j1 and d j2 The point between, L C (p(u)), L C (p(u i Let p(u) and p(u) be points. i The ground truth part affinity vector is the truth label of the field. The specific calculation method for the first affinity field score can refer to existing techniques. This application is not limited to this; other methods can also be used to calculate the first affinity field score.

[0053] After calculating the first affinity field score, it can be multiplied by a function that negatively correlates with the distance between two adjacent keypoints to obtain the second affinity field score (the score corresponding to the connection between the two adjacent keypoints). In other words, the first affinity field score is corrected using a function that negatively correlates with the distance between two adjacent keypoints to obtain the PAF score used to generate keypoint connection candidates.

[0054] This application is not limited to this; other methods can also be used to calculate the partial affinity field score, as long as the partial affinity field score is negatively correlated with the distance between keypoints. For example, when calculating the partial affinity field score, the direction vector of the pixel on the line connecting keypoints can be multiplied by a function negatively correlated with the distance between keypoints, and then the direction vector weighted by the function can be integrated or discretely summed, and so on.

[0055] In some embodiments, the function that is negatively correlated with the distance between two adjacent keypoints can take various forms.

[0056] For example, the function can be represented as Where, d j1 and d j2 Let α be any two adjacent key points, β be a constant not equal to 0, β be a constant greater than 0, and γ be a constant.

[0057] For example, the function can be represented as: or And so on. In this case, the partial affinity field score can be represented, for example, in the following form:

[0058]

[0059] or,

[0060]

[0061] Where, d j1 and d j2 For any two key points, PAF′ Score (d j1 d j2 ), To be related to key point d j1 and d j2 The corresponding partial affinity field score, PAF Score (d j1 ,dj2) To be related to key point d j1 and d j2 The corresponding first part of the affinity field score, p(u), p(u) i ) is d j1 and d j2 The point between, L C (p(u)), L C (p(u i Let p(u) and p(u) be points. i The ground truth part affinity vector.

[0062] Furthermore, the function that is negatively correlated with the distance between two adjacent keypoints can also take other forms; for example, the function can also be expressed as... And so on. Among them, α i β is a non-zero constant. i γ is a constant greater than 0. i is a constant, and n is an integer greater than 1.

[0063] The above description only covers the steps or processes related to this application, but this application is not limited thereto. The action recognition method may also include other steps or processes; for details of these steps or processes, please refer to existing technologies. Furthermore, the above description only uses some structural examples of action recognition models to illustrate the embodiments of this application, but this application is not limited to these structures, and appropriate modifications can be made to these structures. All such modifications should be included within the scope of the embodiments of this application.

[0064] The above embodiments are merely illustrative examples of embodiments of this application, but this application is not limited thereto, and appropriate modifications can be made based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.

[0065] As can be seen from the above embodiments, based on the distance between two adjacent keypoints, a partial affinity field score corresponding to those two keypoints is calculated, such that the partial affinity field score is negatively correlated with the distance between the two adjacent keypoints. This improves the accuracy of action recognition results in bottom-up approaches.

[0066] Second aspect of the embodiments

[0067] This application provides an action recognition device, the contents of which are the same as those in the first aspect of the embodiment and will not be repeated here.

[0068] Figure 3 This is a schematic diagram of an action recognition device according to an embodiment of this application, as shown below. Figure 3 As shown, the motion recognition device 300 includes:

[0069] The key point recognition unit 301 uses a neural network to process video frames to obtain key point information and partial affinity field information of the object;

[0070] The score calculation unit 302 calculates the partial affinity field score between two adjacent key points based on the key point information, the partial affinity field information, and the distance between two adjacent key points. The partial affinity field score is negatively correlated with the distance between the two adjacent key points.

[0071] The key point connection unit 303 connects key points based on the key point information and the partial affinity field score to generate multiple key point connection candidates.

[0072] Skeleton determination unit 304 determines the skeleton map of the object based on the key point candidates; and

[0073] The action recognition unit 305 performs action recognition of the object based on the skeleton diagram.

[0074] In some embodiments, the score calculation unit 302 calculates a first partial affinity field score between two adjacent key points based on key point information and partial affinity field information, and multiplies the first partial affinity field score by a function that is negatively correlated with the distance between the two adjacent key points to obtain the partial affinity field score.

[0075] In some embodiments, the function that is negatively correlated with the distance between two adjacent keypoints can be expressed as: or Where, d j1 and d j2 Let be any two key points, and α be a constant that is not equal to 0.

[0076] In some embodiments, the score calculation unit 302 calculates the first part of the affinity field score as follows:

[0077]

[0078] or,

[0079]

[0080] Where, d j1 and d j2 For any two key points, PAF Score (d j1 d j2 ), To be related to key point d j1 and d j2 The corresponding first part of the affinity field score, p(u), p(u) i ) is d j1 and d j2 The point between, L C (p(u)), L C (p(u i Let p(u) and p(u) be points. i The ground truth part affinity vector.

[0081] In some embodiments, the key point recognition unit 303 obtains key point information in the following manner:

[0082] The key point recognition unit 303 uses a neural network to process objects in the video frame to obtain confidence map information, and obtains key point information based on the confidence map information.

[0083] In some embodiments, partial affinity field scores can be represented in the following form:

[0084]

[0085] or,

[0086]

[0087] Where, d j1 and d j2 For any two key points, PAF′ Score (d j1 d j2 ), To be related to key point d j1 and d j2 The corresponding partial affinity field score, PAF Score (d j1d j2 ), To be related to key point d j1 and d j2 The corresponding first part of the affinity field score, p(u), p(u) i ) is d j1 and d j2 The point between, L C (p(u)), L C (p(u i Let p(u) and p(u) be points. i The ground truth part affinity vector.

[0088] It is worth noting that the above description only covers the components or modules relevant to this application, but this application is not limited thereto. The motion recognition device 300 may also include other components or modules, and for details regarding these components or modules, please refer to related technologies.

[0089] For the sake of simplicity, Figure 3 The diagram only exemplifies the connection relationships or signal flow between various components or modules; however, those skilled in the art should understand that various related technologies, such as bus connections, can be employed. The aforementioned components or modules can be implemented using hardware facilities such as processors and memory; this application does not limit the scope of the embodiments.

[0090] The above embodiments are merely illustrative examples of embodiments of this application, but this application is not limited thereto, and appropriate modifications can be made based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.

[0091] As can be seen from the above embodiments, based on the distance between two adjacent keypoints, a partial affinity field score corresponding to those two keypoints is calculated, such that the partial affinity field score is negatively correlated with the distance between the two adjacent keypoints. This improves the accuracy of action recognition results in bottom-up approaches.

[0092] Third aspect of the embodiments

[0093] This application provides an electronic device including the motion recognition device 300 as described in the second aspect of the embodiment, the contents of which are incorporated herein by reference. This electronic device may be, for example, a computer, server, workstation, laptop computer, smartphone, etc.; however, this application is not limited thereto.

[0094] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of this application. For example... Figure 4 As shown, the electronic device 400 may include a processor (e.g., a central processing unit, CPU) 410 and a memory 420; the memory 420 is coupled to the central processing unit 410. The memory 420 can store various types of data; it also stores an information processing program 421, and executes the program 421 under the control of the processor 410.

[0095] In some embodiments, the functionality of the action recognition device 300 is integrated into the processor 410. The processor 410 is configured to implement the action recognition method as described in the first aspect of the embodiment.

[0096] In some embodiments, the motion recognition device 300 is configured separately from the processor 410. For example, the motion recognition device 300 can be configured as a chip connected to the processor 410, and the function of the motion recognition device 300 can be realized through the control of the processor 410.

[0097] For example, processor 410 is configured to perform the following control: process an object in a video frame using a neural network to obtain keypoint information and partial affinity field information of the object; calculate a partial affinity field score corresponding to two adjacent keypoints based on the keypoint information, the partial affinity field information, and the distance between two adjacent keypoints, wherein the partial affinity field score is negatively correlated with the distance between the two adjacent keypoints; perform keypoint connection based on the keypoint information and the partial affinity field score to generate multiple keypoint connection candidates; determine a skeleton map of the object based on the keypoint candidates; and perform action recognition of the object based on the skeleton map.

[0098] In addition, such as Figure 4 As shown, the electronic device 400 may also include: an input / output (I / O) device 430 and a display 440, etc.; the functions of the above components are similar to those in the prior art, and will not be described in detail here. It is worth noting that the electronic device 400 is not necessarily required to include... Figure 4 All components shown; in addition, the electronic device 400 may also include Figure 4 For components not shown, please refer to relevant technologies.

[0099] This application also provides a computer-readable program, wherein when the program is executed in an electronic device, the program causes the computer in the electronic device to perform the action recognition method as described in the first aspect of the embodiment.

[0100] This application also provides a storage medium storing a computer-readable program, wherein the computer-readable program causes a computer in an electronic device to perform the action recognition method as described in the first aspect of the embodiment.

[0101] The apparatus and methods described above in this application can be implemented in hardware or in combination with software. This application relates to a computer-readable program that, when executed by a logic component, enables the logic component to implement the apparatus or components described above, or to implement the various methods or steps described above. This application also relates to storage media for storing the above programs, such as hard disks, magnetic disks, optical disks, DVDs, flash memory, etc.

[0102] The methods / apparatus described in conjunction with the embodiments of this application can be directly embodied in hardware, software modules executed by a processor, or a combination of both. For example, one or more and / or combinations of one or more functional block diagrams shown in the figures can correspond to various software modules in a computer program flow, or to various hardware modules. These software modules can correspond to the various steps shown in the figures, respectively. These hardware modules can be implemented, for example, using a field-programmable gate array (FPGA) to embed these software modules.

[0103] The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor, enabling the processor to read information from and write information to the storage medium; or the storage medium can be an integral part of the processor. The processor and storage medium can reside in an ASIC. The software module can be stored in the memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the device (such as a mobile terminal) uses a high-capacity MEGA-SIM card or a high-capacity flash memory device, the software module can be stored in the MEGA-SIM card or the high-capacity flash memory device.

[0104] One or more and / or one or more combinations of functional blocks described in the accompanying drawings can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described herein. One or more and / or one or more combinations of functional blocks described in the accompanying drawings can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0105] The present application has been described above with reference to specific embodiments. However, those skilled in the art should understand that these descriptions are exemplary and not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles thereof, and these modifications and variations are also within the scope of the present application.

[0106] Regarding the implementation methods including the above embodiments, the following notes are also disclosed:

[0107] Appendix 1. An action recognition method, comprising:

[0108] A neural network is used to process objects in video frames to obtain key point information and partial affinity field information of the objects;

[0109] The partial affinity field score corresponding to the two adjacent key points is calculated based on the key point information, the partial affinity field information, and the distance between the two adjacent key points. The partial affinity field score is negatively correlated with the distance between the two adjacent key points.

[0110] Keypoints are connected based on the keypoint information and the partial affinity field score to generate multiple keypoint connection candidates;

[0111] Determine the skeleton map of the object based on the key point candidates; and

[0112] Action recognition of the object is performed based on the skeletal diagram.

[0113] Appendix 2. According to the method described in Appendix 1, the partial affinity field score is obtained in the following manner:

[0114] Calculate the first partial affinity field score corresponding to the two adjacent key points based on the key point information and the partial affinity field information; and

[0115] The first partial affinity field score is multiplied by a function that is negatively correlated with the distance between the two adjacent keypoints to obtain the partial affinity field score.

[0116] Appendix 3. According to the method described in Appendix 2, the function is represented as: or Where, d j1 and d j2 Let be any two key points, and α be a constant that is not equal to 0.

[0117] Appendix 4. According to the method described in Appendix 2, the first part of the affinity field score is calculated as follows:

[0118]

[0119] or,

[0120]

[0121] Where, d j1 and d j2 For any two key points, PAF Score (d j1 d j2 ), To be related to key point d j1 and d j2 The corresponding first part of the affinity field score, p(u), p(u) i ) is d j1 and d j2 The point between, L C (p(u)), L C (p(u i Let p(u) and p(u) be points. i The truth label part of the affinity vector field.

[0122] Appendix 5. According to the method described in Appendices 2 to 4, the partial affinity field score is expressed in the following form:

[0123] or,

[0124]

[0125] Where, d j1 and d j2 For any two key points, PAF′ Score (d j1 d j2 ), To be related to key point d j1 and d j2 The corresponding partial affinity field score, PAF Score (d j1 d j2 ), To be related to key point d j1 and d j2 The corresponding first part of the affinity field scores, p(u) and P(u) i ) is d j1 and d j2 The point between, L C (p(u)), L C (p(u i Let p(u) and p(u) be points. iThe ground truth part affinity vector.

[0126] Appendix 6. According to the method described in Appendix 1, the key point information is obtained in the following manner:

[0127] The video frame is processed using a neural network to obtain confidence map information, and the key point information is obtained based on the confidence map information.

[0128] Appendix 7. A storage medium storing a computer-readable program, wherein the computer-readable program causes a computer to perform the action recognition method as described in any one of Appendices 1 to 6 in an electronic device.

Claims

1. An action recognition apparatus, characterized by comprising: The apparatus comprises: a key point identification unit which processes an object in a video frame using a neural network to obtain key point information and part affinity field information of the object; a score calculation unit which calculates part affinity field scores corresponding to two adjacent key points according to the key point information, the part affinity field information and a distance between the two adjacent key points, the part affinity field scores being negatively related to the distance between the two adjacent key points; a key point connection unit which performs key point connection according to the key point information and the part affinity field scores to generate a plurality of key point connection candidates; a skeleton determination unit which determines a skeleton map of the object according to the key point candidates; and an action recognition unit which performs action recognition of the object according to the skeleton map, the score calculation unit calculates first part affinity field scores corresponding to the two adjacent key points according to the key point information and the part affinity field information, and multiplies the first part affinity field scores by a function negatively related to the distance between the two adjacent key points to obtain the part affinity field scores.

2. The apparatus according to claim 1, wherein The function is represented as: or wherein, and is any two key points, is a constant not equal to 0.

3. The apparatus of claim 1, wherein, the score calculation unit calculates the first part affinity field scores according to the following manner: ， or ， wherein, and are any two key points, , is a first partial affinity field score corresponding to a line connecting the key points and , is a second partial affinity field score corresponding to a line connecting the key points and , , is a true label partial affinity vector field of the points , .

4. The apparatus according to claim 1, wherein the key point identification unit processes the object in the video frame using a neural network to obtain confidence map information, and obtains the key point information according to the confidence map information.

5. A motion recognition method characterized by, The method comprises: processing an object in a video frame using a neural network to obtain key point information and part affinity field information of the object; calculating part affinity field scores corresponding to two adjacent key points according to the key point information, the part affinity field information and a distance between the two adjacent key points, the part affinity field scores being negatively related to the distance between the two adjacent key points; performing key point connection according to the key point information and the part affinity field scores to generate a plurality of key point connection candidates; determining a skeleton map of the object according to the key point candidates; and performing action recognition of the object according to the skeleton map, wherein the part affinity field scores are obtained in the following manner: calculating first part affinity field scores corresponding to the two adjacent key points according to the key point information and the part affinity field information; and multiplying the first part affinity field scores by a function negatively related to the distance between the two adjacent key points to obtain the part affinity field scores.

6. The method according to claim 5, wherein The function is represented as: or wherein, and are any two key points, is a constant not equal to 0.

7. The method of claim 5, wherein, the first part affinity field scores are calculated in the following manner: ， or ， wherein, and are any two key points, , is a first partial affinity field score corresponding to key points and , is a true label partial affinity vector field for a point and , , is a true label partial affinity vector field for a point , . 8.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, the processor is configured to execute the computer program to implement the action recognition method according to any one of claims 5 to 7.

Images