A method and apparatus for action recognition

Abstract

The application discloses a motion recognition method and device, wherein the method comprises the following steps: acquiring node features and edge weights of a previous frame image; wherein the nodes comprise first-type nodes and second-type nodes, the edge weights comprise first edge weights and second edge weights, the first-type nodes represent target objects in the previous frame image, and the second-type nodes represent overall motion categories of the previous frame image; the first edge weights represent relationships between any two first-type nodes, and the second edge weights represent relationships between any first-type node and the second-type node; performing feature initialization on node features and edge weights of a current frame image based on the node features and the edge weights of the previous frame image; and recognizing the initialized current frame image to obtain a motion recognition result of the current frame image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of intelligent security, and relate to, but are not limited to, a motion recognition method and device. BACKGROUND

[0002] The motion recognition scheme in a monitoring scenario is usually applied in the field of intelligent security. When an abnormal behavior is found, the abnormal behavior needs to be alarmed in time. Therefore, in addition to high accuracy, the scheme also needs to be real-time. The current scheme in the monitoring scenario mainly faces the following two problems:

[0003] Problem one: In the monitoring scenario, the proportion of people may be small, and the method of processing the entire picture cannot be used to recognize the motion of people in the picture.

[0004] Problem two: Because the intra-class difference of human motion is large, the accuracy of using only traditional algorithms to recognize motion is not high. SUMMARY

[0005] Therefore, embodiments of the present application provide a motion recognition method and device.

[0006] The technical scheme of the embodiments of the present application is as follows:

[0007] In a first aspect, the embodiments of the present application provide a motion recognition method, which comprises:

[0008] obtaining node features and edge weights of a previous frame image;

[0009] The node includes a first type of node and a second type of node, and the edge weight includes a first edge weight and a second edge weight. The first type of node represents a target object in the previous frame image, and the second type of node represents the overall motion category of the previous frame image. The first edge weight is the relationship between any two first type of nodes, and the second edge weight is the relationship between any first type of node and the second type of node.

[0010] performing feature initialization on the node features and edge weights of a current frame image based on the node features and edge weights of the previous frame image;

[0011] recognizing the initialized current frame image to obtain a motion recognition result of the current frame image.

[0012] In a second aspect, the embodiments of the present application provide a motion recognition device, which comprises:

[0013] An obtaining module is configured to obtain node features and edge weights of a previous frame image;

[0014] The node includes a first type of node and a second type of node, the edge weight includes a first edge weight and a second edge weight, the first type of node represents a target object in the previous frame image, and the second type of node represents a motion category of the target object in the previous frame image; the first edge weight is a relationship between any two first type of nodes, and the second edge weight is a relationship between any first type of node and the second type of node.

[0015] The first initialization module is configured to perform feature initialization on the node features and the edge weights of the current frame image based on the node features and the edge weights of the previous frame image.

[0016] The recognition module is configured to recognize the current frame image after the initialization to obtain a motion recognition result of the current frame image.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the above method when executing the program.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, which stores executable instructions for implementing the above method when executed by a processor.

[0019] In the embodiment of the present application, first, the node features and the edge weights of the previous frame image are acquired; then, the node features and the edge weights of the current frame image are initialized based on the node features and the edge weights of the previous frame image; finally, the current frame image after the initialization is recognized to obtain a motion recognition result of the current frame image. In this way, the node features and the edge weights of the constructed graph are utilized, and the information in the image is fused with prior knowledge, so that a more accurate recognition result can be obtained in real time. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 An implementation flowchart of a motion recognition method provided by an embodiment of the present application is shown in the figure;

[0021] Figure 2 An implementation flowchart of a graph construction method provided by an embodiment of the present application is shown in the figure;

[0022] Figure 3 An implementation flowchart of a graph construction method provided by an embodiment of the present application is shown in the figure;

[0023] Figure 4 An implementation flowchart of a motion recognition method provided by an embodiment of the present application is shown in the figure;

[0024] Figure 5 An implementation flowchart of a motion recognition method provided by an embodiment of the present application is shown in the figure;

[0025] Figure 6 A hardware entity schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the embodiments of the present application will be further described in detail below with reference to the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0027] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0028] In the following description, the terms “first\second\third” are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that “first\second\third” can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0030] The embodiments of the present application provide an action recognition method, as shown in the method, the method comprises: Figure 1

[0031] Step S110, obtaining node features and edge weights of a previous frame image;

[0032] The nodes include first type nodes and second type nodes, and the edge weights include first edge weights and second edge weights. The first type nodes represent target objects in the previous frame image, and the second type nodes represent overall action categories of the previous frame image. The first edge weights are relationships between any two first type nodes, and the second edge weights are relationships between any first type node and the second type node.

[0033] In the implementation process, the previous frame image can be an undirected weighted graph including nodes and edge weights.

[0034] ​Here, the first type of node represents a target object in the previous frame of image, and the feature of the first type of node can be a feature of the target object, which includes at least one of an image coding feature of the target object and an action coding feature corresponding to the target object.

[0035] The second type of node represents an overall action category of the previous frame of image, that is, the second type of node can determine a result of the action category of the previous frame of image.

[0036] The relationship between any two first type of nodes is defined as a first edge weight.

[0037] The relationship between any first type of node and the second type of node is defined as a second edge weight.

[0038] In step S120, the node features and edge weights of the current frame of image are initialized based on the node features and edge weights of the previous frame of image.

[0039] Here, the node features and edge weights of the current frame of image can be initialized based on the node features and edge weights of the previous frame of image.

[0040] For example, the node features and edge weights of the previous frame of image can be used as part of the node features and edge weights of the previous frame of image to initialize the node features and edge weights of the current frame of image, so as to fuse the information in the current frame of image with prior knowledge.

[0041] In step S130, the initialized current frame of image is recognized to obtain an action recognition result of the current frame of image.

[0042] In the implementation process, the constructed graph can be recognized by using a graph convolutional neural network (GCN) to obtain the action recognition result.

[0043] In the embodiments of the present application, the node features and edge weights of the previous frame of image are first obtained, then the node features and edge weights of the current frame of image are initialized based on the node features and edge weights of the previous frame of image, and finally the initialized current frame of image is recognized to obtain an action recognition result of the current frame of image. In this way, the node features and edge weights of the constructed graph are used, and the information in the image is fused with prior knowledge, so that a more accurate recognition result can be obtained in real time.

[0044] In some embodiments, the image is a constructed graph, and the constructed graph is an image constructed based on action recognition of objects in a collected image, that is, the aforementioned undirected and weighted graph. Before the above step S110, that is, before the node features and edge weights of the previous frame of image are obtained, the node features and edge weights of the previous frame of image can be obtained by using a trained model to recognize the action of the object in the image. Figure 2As shown, the method further comprises the following steps:

[0045] In step S210, object detection is performed on the acquired image of the previous frame to obtain at least one object detection frame, wherein each object detection frame comprises a target object.

[0046] In the implementation process, the acquired image can be obtained from the camera of the monitoring device. Since there can be multiple target objects in the acquired image, object detection can be performed on the acquired image to identify each target object in the acquired image with an object detection frame. Here, the method of obtaining the object detection frame is not limited.

[0047] In step S220, a fine-grained action detection algorithm is used to classify the fine-grained action of each target object to obtain the action category of each target object.

[0048] Here, the fine-grained action detection algorithm can detect the key parts of the human body, such as the limb action of the human body, at least including the action of the hand, the action of the leg, etc.

[0049] In the implementation process, the fine-grained action detection algorithm can be used to classify the fine-grained action of each target object to obtain the action category of each target object. For example, the target object can be classified into a punching action, a kicking action, a dodging action, and a jumping action, etc.

[0050] In step S230, the target object is determined as the first type of node, and the overall action category of the acquired image of the previous frame is initialized as the second type of node. The feature corresponding to the first type of node is obtained by fusing the original feature of the target object and the feature of the action category of the target object, and the feature corresponding to the second type of node is the feature of the overall action category of the acquired image.

[0051] Here, the constructed graph is a graph constructed after action recognition based on the objects in the acquired image. The constructed graph at least includes nodes, edges between nodes, i.e., the aforementioned undirected and weighted graph. The nodes include the first type of node and the second type of node.

[0052] The first type of node represents the target object in the image, i.e., each target object obtained can be determined as a first type of node. The feature corresponding to the first type of node can be obtained by fusing the original feature of the target object and the feature of the action category of the target object.

[0053] The overall action category of the acquired image is initialized as the second type of node, and the feature corresponding to the second type of node is the feature of the overall action category of the acquired image.

[0054] For example, Figure 3A flowchart for obtaining a constructed graph is provided in the embodiments of the present application, as shown in Figure 3 which includes three frames of continuously captured images 31, three frames of target images 32, and three constructed graphs 33, wherein,

[0055] The three frames of continuously captured images 31 can be obtained from images captured by a camera of a monitoring device.

[0056] The three frames of target images 32 are obtained by subjecting the three frames of continuously captured images 31 to a fine-grained human body and action detection algorithm. The boxes in the target images are object detection boxes, and each object detection box contains a target object.

[0057] The three constructed graphs 33 are obtained based on the three frames of target images 32. The constructed graph 33 includes first-type nodes and second-type nodes. The small circles enclosed by dashed lines in the graph are first-type nodes, and each small circle represents a target object detected in the target image 32. The large circles outside the dashed lines are second-type nodes, and the feature of the second-type nodes is the feature of the overall action category of the captured image 31.

[0058] Step S240, constructing the previous frame image based on the first-type nodes and the second-type nodes.

[0059] As shown in Figure 3 , an undirected and weighted graph, i.e., a previous frame image, can be constructed based on the first-type nodes and the second-type nodes.

[0060] In the embodiments of the present application, object detection is first performed on a previous frame of captured image to obtain at least one object detection box. Then, a fine-grained action detection algorithm is used to classify each target object in a fine-grained manner to obtain the action category of each target object. The target object is determined as the first-type node, and the overall action category of the previous frame of captured image is initialized as the second-type node. Finally, the previous frame image is constructed based on the first-type nodes and the second-type nodes. In this way, the previous frame image for identifying the overall action category can be obtained.

[0061] In some embodiments, the step of "initializing the overall action category of the previous frame of captured image as the second-type node" in the above step S230 can be implemented by the following steps:

[0062] Step 231, performing action recognition on the previous frame of captured image to obtain the overall action category corresponding to the previous frame of captured image.

[0063] In the implementation process, the action recognition algorithm can be used to recognize the collected image to obtain the action category of whether the collected image has the fighting behavior. For example, if it is recognized that any two target objects in the collected image have the fighting behavior, it can be determined that the overall action category of the collected image is fighting; if it is recognized that there is no body contact between any two target objects in the collected image, it can be determined that the overall action category of the collected image is non-fighting.

[0064] Step 232, encoding the overall action category to obtain the feature of the second type node, so as to form the second type node based on the feature of the second type node.

[0065] In the implementation process, the overall action category can be encoded by using the encoding mode used to obtain the feature of the first type node to obtain the feature of the second node; or other encoding modes can be used to encode the overall action category to obtain the feature of the second node.

[0066] In the embodiment of the application, the action recognition is first performed on the collected image of the previous frame to obtain the overall action category corresponding to the collected image of the previous frame; then the overall action category is encoded to obtain the feature of the second type node, so as to form the second type node based on the feature of the second type node. In this way, the feature corresponding to the second type node can be obtained to form the second type node.

[0067] In some embodiments, the above step S230 of "determining the target object as the first type node" can be implemented by the following steps:

[0068] Step 233, obtaining the original feature of the target object and the action category of the target object;

[0069] Here, the original feature of the target object can be obtained by using the encoder of the neural network to encode the target object. The action category of the target object is obtained by performing fine-grained action detection on the target object.

[0070] Step 234, encoding the action category to obtain the action feature corresponding to the action category;

[0071] In the implementation process, the action category can be encoded by using the encoder of the neural network to obtain the action feature; or other encoding algorithms can be used to encode the action category to obtain the action feature.

[0072] Step 235, fusing the original feature and the action feature to obtain the feature of the first type node, or splicing the original feature and the action feature to obtain the feature of the first type node, so as to form the first type node based on the feature of the first type node.

[0073] In some embodiments, when the dimensions of the original features and the action features are the same, the original features and the action features can be added to obtain the features of the first-type nodes.

[0074] In some embodiments, when the dimensions of the original features and the action features are different, the original features and the action features can be spliced to obtain the features of the first-type nodes.

[0075] In the embodiments of the present application, first, the original features of the target object and the action categories of the target object are obtained; then, the action categories are encoded to obtain the action features corresponding to the action categories; finally, the original features and the action features are fused to obtain the features of the first-type nodes, or the original features and the action features are spliced to obtain the features of the first-type nodes, so as to form the first-type nodes based on the features of the first-type nodes. In this way, the features of the first-type nodes obtained include the original features and the action categories, which can effectively improve the recognition accuracy in subsequent recognition of the constructed graph.

[0076] In some embodiments, the step of "obtaining the edge weight of the previous frame image" in the above step S110 can be implemented by the following steps:

[0077] Step 111, obtaining the similarity and the intersection over union between two first-type nodes;

[0078] Here, the similarity between two first-type nodes can be determined based on the features of the first-type nodes. The intersection over union between the corresponding two first-type nodes can be determined based on the detection boxes of the target objects.

[0079] Step 112, determining the first edge weight based on the similarity and / or the intersection over union.

[0080] Here, the first edge weight can be used to describe the relationship between two first-type nodes, that is, the relationship of interaction between the target objects in the collected images and the interaction between them. Based on the first edge weight, it can be determined whether the interaction between two target objects is relatively large. Since the similarity and the intersection over union can reflect the interaction between two target objects, the first edge weight can be determined based on the similarity; the first edge weight can also be determined based on the intersection over union; and the first edge weight can also be determined based on the similarity and the intersection over union.

[0081] In the embodiments of the present application, first, the similarity and the intersection over union between two first-type nodes are obtained; then, the first edge weight is determined based on the similarity and / or the intersection over union. In this way, the first edge weight used to represent the interaction between two objects can be determined.

[0082] In some embodiments, the step S110 of "obtaining the edge weight of the previous frame image" can be implemented by the following steps:

[0083] Step 113, obtaining the feature of the action category corresponding to the first type of node;

[0084] In the implementation process, the encoder in the neural network can be used to encode the action category corresponding to the first type of node to obtain the feature of the action category corresponding to the first type of node.

[0085] Step 114, obtaining the similarity between the feature of the action category corresponding to the first type of node and the feature corresponding to the second type of node;

[0086] Here, since the feature of the action category corresponding to the first type of node and the feature corresponding to the second type of node are both features for describing action categories, the two action categories can be mapped to the same space, and the similarity between the features corresponding to the two action categories is calculated. For example, in the case where the action category corresponding to the first type of node is a punch, and the feature of the second type of node is a fight, the similarity is high, that is, the feature contribution of the first type of node to the second type of node is large; in the case where the action category corresponding to the first type of node is standing and no fight occurs, and the feature of the second type of node is a fight, the similarity is low, that is, the feature contribution of the first type of node to the second type of node is small.

[0087] Step 115, determining the second edge weight based on the similarity;

[0088] In the implementation process, the second edge weight can be determined based on the similarity. The greater the similarity, the greater the contribution to the second edge weight, and the smaller the similarity, the smaller the contribution to the second edge weight.

[0089] Or,

[0090] Step 116, obtaining the confidence of the same action category corresponding to the first type of node and the second type of node;

[0091] Step 117, determining the second edge weight as the confidence.

[0092] In the implementation process, the confidence of the same action category corresponding to the first type of node and the second type of node can be obtained. For example, the confidence of the first type of node with a body action is 80%, and the category of the second type of node is a fight. Then the confidence of 80% can be determined as the second edge weight.

[0093] In the embodiments of the present application, the feature of the action category corresponding to the first type of node can be obtained, and the similarity between the feature of the action category corresponding to the first type of node and the feature corresponding to the second type of node can be obtained. In this way, the second edge weight can be determined based on the similarity.

[0094] In the embodiments of the present application, the confidence of the same action category corresponding to the first type of node and the second type of node can also be obtained, so that the confidence can be determined as the second edge weight.

[0095] In some embodiments, the step S120 of "performing feature initialization on the node features and edge weights of the current frame image based on the node features and edge weights of the previous frame image" can be implemented by the following steps:

[0096] In step S121, the node features of the previous frame image are applied to the node features of the current frame image in a sliding average manner to obtain the initialized node features of the current frame image.

[0097] Here, the moving average method is also called the moving average method. Based on the simple average method, the moving average value is calculated by sequentially adding and subtracting new and old data, so as to eliminate accidental variation factors, find out the development trend of things, and make predictions accordingly.

[0098] The node features of the previous frame image of the current frame image can be applied to the node features of the current frame image in a sliding average manner to obtain the initialized node features of the current frame image, so as to realize the fusion of the node features in the image and the prior node features.

[0099] In step S122, the edge weights of the previous frame image are applied to the edge weights of the current frame image in a sliding average manner to obtain the initialized edge weights of the current frame image.

[0100] Here, the edge weights of the previous frame image can also be applied to the edge weights of the current frame image in a sliding average manner to obtain the initialized edge weights of the current frame image, so as to realize the fusion of the edge weights in the image and the prior edge weights.

[0101] In the embodiments of the present application, the node features of the previous frame image are applied to the node features of the current frame image in a sliding average manner to obtain the initialized node features of the current frame image, and the edge weights of the previous frame image are applied to the edge weights of the current frame image in a sliding average manner to obtain the initialized edge weights of the current frame image. In this way, the information in the image can be fused with the prior knowledge, and the time sequence information can be encoded, so that more accurate recognition results can be obtained in real time when recognizing the current frame image.

[0102] In some embodiments, the step S121 of "applying the node features of the previous frame image to the node features of the current frame image in a sliding average manner to obtain the initialized node features of the current frame image" can be implemented by the following process:

[0103] The node feature of the previous frame image is weighted with the node of the current frame image to obtain the node feature of the current frame image after initialization.

[0104] In the implementation process, the node feature of the previous frame image can be weighted with the node of the current frame image by using the following formula (1) to obtain the node feature of the current frame image after initialization:

[0105] F i i = αF i-1 i-1 + (1-α)F i0 i (1);

[0106] Wherein, F i i represents the feature of the i-th frame after initialization, F i-1 i-1 represents the feature of the i-1-th frame, F i0 i represents the original feature extracted from the i-th frame image, and α is a weighting parameter.

[0107] The above step S122 "the edge weight of the previous frame image is applied to the edge weight of the current frame image in a sliding average manner to obtain the edge weight of the current frame image after initialization" can be realized by the following process:

[0108] The edge weight of the previous frame image is weighted with the edge weight of the current frame image to obtain the edge weight of the current frame image after initialization.

[0109] In the implementation process, the edge weight of the previous frame image can be weighted with the edge weight of the current frame image by using the following formula (2) to obtain the edge weight of the current frame image after initialization:

[0110] W i i = αW i-1 i-1 + (1-α)W i0 i (2);

[0111] Wherein, W i i represents the edge weight of the i-th frame after initialization, W i-1 i-1 represents the edge weight of the i-1-th frame, W i0 i represents the edge weight extracted from the i-th frame image, and α is a weighting parameter.

[0112] In the embodiments of the present application, the node features of the previous frame image are weighted with the nodes of the current frame image to obtain the node features of the current frame image after initialization. The edge weights of the previous frame image are weighted with the edge weights of the current frame image to obtain the edge weights of the current frame image after initialization. In this way, the node features and edge weights of the current frame image after initialization both fuse the node features and edge weights of the previous frame, encode the time sequence information, and can obtain more accurate recognition results.

[0113] In some embodiments, the constructed graph is an undirected weighted graph, and the step S130 of "identifying the current frame image after initialization to obtain the action recognition result of the current frame image" can be implemented by the following process:

[0114] The graph convolutional neural network is used to identify the current undirected weighted graph after initialization to obtain the action recognition result of the current undirected weighted graph.

[0115] Here, the graph convolutional neural network has the same function as the convolutional neural network (CNN), which can be a feature extractor, and the object of extraction is graph data, that is, the constructed graph, such as the constructed graph 33 shown in FIG. 3. Figure 3 The GCN designs a method for extracting features from graph data, so that these features can be used for node classification, graph classification, link prediction, and graph embedding of the graph.

[0116] In the implementation process, the GCN can be used to identify the current undirected weighted graph after initialization to obtain the action recognition result of the current undirected weighted graph, that is, the second type of node.

[0117] In the embodiments of the present application, the graph convolutional neural network is used to identify the current undirected weighted graph after initialization to obtain the action recognition result of the current undirected weighted graph. In this way, the action recognition result of the current frame image obtained is real-time and more accurate.

[0118] The action recognition scheme in the monitoring scene is usually applied in the intelligent security field. When an abnormal behavior is found, the abnormal behavior needs to be alarmed in time, so the scheme requires not only high accuracy but also real-time. The solution includes the following two types:

[0119] The first type is to locate the position of a person in a monitoring picture by using a prior detection algorithm to form a region of interest (ROI), and then to perform action recognition on the person in the ROI. This type of scheme has the problem that the algorithm running speed is related to the density of the crowd in the picture, and in the case of a large crowd density, the algorithm cannot achieve real-time.

[0120] The second type is to use a space-time action detection algorithm to detect the position of each person and the action category of each person in each frame of a video. This type of method has the disadvantages that the algorithm complexity is high, so the real-time performance is poor, and only the action of a single person can be recognized, and the action of a group cannot be recognized.

[0121] Figure 4 A flowchart of an action recognition method provided by an embodiment of the present application is shown in FIG. 1, which includes the following steps. Figure 4

[0122] In step S410, a fine-grained human body and action detection algorithm is used to detect a person in a picture and perform fine-grained action classification for each frame of a video.

[0123] Here, the fine-grained action (decomposition action of fighting) classification includes limb action categories such as punching and kicking. The fine-grained human body action detection algorithm needs a fine-grained human body action category database, and on this basis, a target detection algorithm can be used for learning.

[0124] In step S420, a graph is constructed after obtaining the detection result of each frame.

[0125] Here, the graph includes two types of nodes, real nodes (first type of nodes) and virtual nodes (second type of nodes). The real nodes are real persons in the picture, and the virtual nodes represent categories. The edges between the real nodes represent the relationship between each node. In addition to the features of the original bounding box (bbox), the features of the first step action detection algorithm are also included in the features of each real node, and the two types of features are encoded and fused; the weight of the edge describes the relationship between two real nodes or a real node and a virtual node. Each real node is connected to a virtual node by an edge, and the weight of the edge represents the degree of contribution to the action category.

[0126] The bbox is a bounding box that is used to mark a target when the target is detected.

[0127] In step S430, a graph convolutional neural network is used to update and learn the constructed graph.

[0128] In the implementation process, when the graph of the i-th frame is updated, the real nodes, virtual nodes, and edge weights in the graph of the i-1-th frame will affect the update of the graph of the i-th frame in a sliding average manner, so that the time sequence information is encoded to obtain a more accurate recognition result.​

[0129] In updating the constructed graph according to the detection result, the features of nodes and the weights of edges can be initialized respectively by using the following formulas (1) and (2), taking the weight of a node and an edge as an example:

[0130] F i i-1 i-1 i-1 i0 (1);

[0131] W i i-1 i-1 i-1 i0 (2);

[0132] Wherein, F i i represents the initialized feature of the i-th frame, F i-1 i-1 represents the feature of the i-1-th frame, F i0 i represents the original feature extracted from the i-th frame image, W i i represents the initialized edge weight of the i-th frame, W i-1 i-1 represents the edge weight of the i-1-th frame, W i0 i represents the edge weight extracted from the i-th frame image, and a is a weighting parameter.

[0133] In the embodiment of the application, a motion recognition method in a monitoring scene using a recurrent graph convolution network is provided, which can fuse the information in the image with prior knowledge, and obtain more accurate recognition results in real time.

[0134] Based on the foregoing embodiment, the embodiment of the application provides a motion recognition device, which includes various modules, each module includes various sub-modules, and can be realized by a processor in an electronic device; of course, it can also be realized by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processing (DSP) or a field programmable gate array (FPGA).

[0135] Figure 5 The composition structure diagram of the motion recognition device provided by the embodiment of the application is shown in FIG. 5, and the device 500 includes: Figure 5

[0136] The acquisition module 510 is configured to acquire the node features and edge weights of the previous frame image.

[0137] ​The nodes include first nodes and second nodes, and the edge weights include first edge weights and second edge weights. The first nodes represent target objects in the previous frame of image, and the second nodes represent overall action categories of the previous frame of image. The first edge weights represent relationships between any two first nodes, and the second edge weights represent relationships between any first node and the second node.

[0138] The first initialization module 520 is configured to perform feature initialization on the node features and the edge weights of the current frame of image based on the node features and the edge weights of the previous frame of image.

[0139] The recognition module 530 is configured to recognize the current frame of image after the initialization to obtain an action recognition result of the current frame of image.

[0140] In some embodiments, the image is a constructed graph, which is constructed based on action recognition of objects in a collected image. The device further includes a detection module, an action classification module, a determination module, a second initialization module, and a construction module. The detection module is configured to perform object detection on a previous frame of collected image to obtain at least one object detection frame, wherein each object detection frame includes a target object. The action classification module is configured to perform fine-grained action classification on each target object by using a fine-grained action detection algorithm to obtain an action category of each target object. The determination module is configured to determine the target object as the first node, wherein the feature corresponding to the first node is a feature fusion of the original feature of the target object and the action category of the target object. The second initialization module is configured to initialize an overall action category of the previous frame of collected image as the second node. The feature corresponding to the second node is a feature of the overall action category of the collected image. The construction module is configured to construct the previous frame of image based on the first node and the second node.

[0141] In some embodiments, the second initialization module includes an identification submodule and a first encoding submodule. The identification submodule is configured to perform action recognition on the previous frame of collected image to obtain an overall action category corresponding to the previous frame of collected image. The first encoding submodule is configured to encode the overall action category to obtain a feature of the second node, so as to form the second node based on the feature of the second node.

[0142] In some embodiments, the determining module comprises a first obtaining sub-module, a second encoding sub-module, a fusion sub-module or a splicing sub-module, wherein the first obtaining sub-module is configured to obtain the original feature of the target object and the action category of the target object; the second encoding sub-module is configured to encode the action category to obtain an action feature corresponding to the action category; the fusion sub-module is configured to fuse the original feature and the action feature to obtain the feature of the first type of node; and the splicing sub-module is configured to splice the original feature and the action feature to obtain the feature of the first type of node, so as to form the first type of node based on the feature of the first type of node.

[0143] In some embodiments, the obtaining module 510 comprises a second obtaining sub-module and a first determining sub-module, wherein the second obtaining sub-module is configured to obtain the similarity and / or the intersection over union between two first type of nodes; and the first determining sub-module is configured to determine the first edge weight based on the similarity and the intersection over union.

[0144] In some embodiments, the obtaining module 510 comprises a third obtaining sub-module, a fourth obtaining sub-module, a second determining sub-module, a fifth obtaining sub-module and a third determining sub-module, wherein the third obtaining sub-module is configured to obtain the feature of the action category corresponding to the first type of node; the fourth obtaining sub-module is configured to obtain the similarity between the feature of the action category corresponding to the first type of node and the feature corresponding to the second type of node; the second determining sub-module is configured to determine the second edge weight based on the similarity; the fifth obtaining sub-module is configured to obtain the confidence corresponding to the same action category of the first type of node and the second type of node; and the third determining sub-module is configured to determine the confidence as the second edge weight.

[0145] In some embodiments, the first initialization module 520 comprises a first sliding average sub-module and a second sliding average sub-module, wherein the first sliding average sub-module is configured to apply the node feature of the previous frame image to the node feature of the current frame image in a sliding average manner to obtain the node feature of the current frame image after completion of initialization; and the second sliding average sub-module is configured to apply the edge weight of the previous frame image to the edge weight of the current frame image in a sliding average manner to obtain the edge weight of the current frame image after completion of initialization.

[0146] In some embodiments, the first sliding average sub-module is further configured to perform weighted processing on the node feature of the previous frame image and the node of the current frame image to obtain the node feature of the current frame image after completion of initialization; and the second sliding average sub-module is further configured to perform weighted processing on the edge weight of the previous frame image and the edge weight of the current frame image to obtain the edge weight of the current frame image after completion of initialization.

[0147] In some embodiments, the construction graph is an undirected weighted graph, and the identification module 530 is further configured to identify the current undirected weighted graph that has been initialized by using a graph convolutional neural network to obtain an action recognition result of the current undirected weighted graph.

[0148] The above device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0149] It should be noted that, in the embodiments of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a plurality of instructions for causing an electronic device (which can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.) to execute all or part of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read Only Memory, ROM), a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific hardware and software combination.

[0150] Correspondingly, the embodiments of the present application provide a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the action recognition method provided in the above embodiments.

[0151] Correspondingly, the embodiments of the present application provide an electronic device, Figure 6 A hardware entity schematic diagram of the electronic device provided in the embodiments of the present application is shown in FIG. 6, which includes a memory 601 and a processor 602. The memory 601 stores a computer program executable on the processor 602, and the processor 602 implements the steps of the action recognition method provided in the above embodiments when executing the program. Figure 6 The hardware entity of the device 600 includes the memory 601 and the processor 602. The memory 601 stores a computer program executable on the processor 602, and the processor 602 implements the steps of the action recognition method provided in the above embodiments when executing the program.

[0152] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data (for example, image data, audio data, voice communication data and video communication data) to be processed by the processor 602 and modules in the electronic device 600, which can be implemented by a flash memory (FLASH) or a random access memory (Random Access Memory, RAM).

[0153] It should be noted that the description of the storage medium and device embodiments above is similar to the description of the method embodiments above, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0154] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that the size of the sequence number of the above processes in various embodiments of the present application does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The sequence number of the above embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments.

[0155] It should be noted that in this document, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0156] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0157] The units described above as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units; they can be located in one place or distributed on multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0158] In addition, each of the functional units in the embodiments of the present application can be integrated into one processing unit, each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in the form of hardware, or in the form of hardware plus software function units.

[0159] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes mobile storage devices, read only memory (ROM), magnetic discs or optical discs, and various storage medium that can store program codes.

[0160] Alternatively, the integrated units of the present application, if implemented in the form of software function modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a plurality of instructions for causing an electronic device (which can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.) to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROM, magnetic discs or optical discs, and various storage medium that can store program codes.

[0161] The methods disclosed in the several method embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0162] The features disclosed in the several product embodiments of the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0163] The features disclosed in the several method or device embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0164] The above is only an implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An action recognition method, the method comprising: performing object detection on a previous frame of captured images to obtain at least one object detection box, wherein each object detection box comprises a target object; performing fine-grained action classification on each target object using a fine-grained action detection algorithm to obtain an action category of each target object; determining the target object as a first type of node and initializing an overall action category of the previous frame of captured images as a second type of node; wherein a feature corresponding to the first type of node is obtained by fusing original features of the target object and features of the action category of the target object, and a feature corresponding to the second type of node is a feature of the overall action category of the captured images; constructing a previous frame of images based on the first type of node and the second type of node, wherein the images are constructed graphs, and the constructed graphs are images constructed after performing action recognition on objects in the captured images; obtaining node features and edge weights of the previous frame of images; wherein the nodes include the first type of node and the second type of node, the edge weights include first edge weights and second edge weights, the first type of node represents a target object in the previous frame of images, and the second type of node represents an overall action category of the previous frame of images; the first edge weights are relationships between any two first type of nodes, and the second edge weights are relationships between any first type of node and the second type of node; performing feature initialization on node features and edge weights of a current frame of images based on the node features and edge weights of the previous frame of images; recognizing the initialized current frame of images to obtain an action recognition result of the current frame of images.

2. The method of claim 1, wherein initializing the overall action category of the previous frame of captured images as the second type of node comprises: performing action recognition on the previous frame of captured images to obtain an overall action category corresponding to the previous frame of captured images; encoding the overall action category to obtain a feature of the second type of node, so as to form a second type of node based on the feature of the second type of node.

3. The method of claim 1, wherein determining the target object as the first type of node comprises: obtaining original features of the target object and an action category of the target object; encoding the action category to obtain an action feature corresponding to the action category; fusing the original features and the action feature to obtain a feature of the first type of node, or splicing the original features and the action feature to obtain the feature of the first type of node, so as to form a first type of node based on the feature of the first type of node.

4. The method of claim 1, wherein obtaining edge weights of the previous frame of images comprises: obtaining a similarity and an intersection over union between two first type of nodes; determining the first edge weights based on the similarity and / or the intersection over union.

5. The method of claim 1, wherein obtaining edge weights of the previous frame of images comprises: obtaining a feature of an action category corresponding to the first type of node; obtaining a similarity between a feature corresponding to an action category of the first type of node and a feature corresponding to the second type of node; determining the second edge weight based on the similarity; or, obtaining a confidence degree corresponding to the same action category of the first type of node and the second type of node; determining the confidence degree as the second edge weight. 6.The method of claim 1, wherein the initializing the node features and the edge weights of the current frame image based on the node features and the edge weights of the previous frame image comprises: applying the node features of the previous frame image to the node features of the current frame image in a sliding average manner to obtain the initialized node features of the current frame image; and applying the edge weights of the previous frame image to the edge weights of the current frame image in a sliding average manner to obtain the initialized edge weights of the current frame image. 7.The method of claim 6, wherein the applying the node features of the previous frame image to the node features of the current frame image in a sliding average manner to obtain the initialized node features of the current frame image comprises: performing weighted processing on the node features of the previous frame image and the nodes of the current frame image to obtain the initialized node features of the current frame image; and correspondingly, the applying the edge weights of the previous frame image to the edge weights of the current frame image in a sliding average manner to obtain the initialized edge weights of the current frame image comprises: performing weighted processing on the edge weights of the previous frame image and the edge weights of the current frame image to obtain the initialized edge weights of the current frame image. 8.The method of claim 1, wherein the constructing the graph is a directed weighted graph, and the identifying the initialized current frame image to obtain the action recognition result of the current frame image comprises: identifying the initialized current directed weighted graph by using a graph convolutional neural network to obtain the action recognition result of the current directed weighted graph. 9.An action recognition apparatus, comprising: a detection module configured to perform object detection on a previous frame of captured image to obtain at least one object detection frame, wherein each object detection frame includes a target object; an action classification module configured to perform fine-grained action classification on each target object by using a fine-grained action detection algorithm to obtain an action category of each target object; a determination module configured to determine the target object as a first type of node, wherein a feature corresponding to the first type of node is a feature obtained by fusing a raw feature of the target object and the action category of the target object; a second initialization module configured to initialize an overall action category of the previous frame of captured image as a second type of node, wherein a feature corresponding to the second type of node is a feature of the overall action category of the captured image; a construction module configured to construct a previous frame image based on the first type of node and the second type of node, wherein the previous frame image is a constructed graph, and the constructed graph is a graph constructed based on action recognition on objects in the captured image; and an obtaining module configured to obtain node features and edge weights of the previous frame image. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The nodes include the first type of nodes and the second type of nodes, the edge weights include first edge weights and second edge weights, the first type of nodes represent target objects in the previous frame of image, and the second type of nodes represent action categories of the target objects in the previous frame of image; the first edge weights are relationships between any two first type of nodes, and the second edge weights are relationships between any first type of node and the second type of node; The first initialization module is configured to perform feature initialization on the node features and the edge weights of the current frame of image based on the node features and the edge weights of the previous frame of image; The recognition module is configured to recognize the current frame of image after the initialization to obtain an action recognition result of the current frame of image.

Images