A behavior recognition method, apparatus, device, and medium

Abstract

The application discloses a behavior recognition method, device, equipment and medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring an image sequence of a to-be-recognized object in continuous time, and extracting a joint node of the image sequence by using a pose estimation algorithm to obtain target joint node information; extracting spatiotemporal feature information of the target joint node information, and constructing a multi-dimensional adjacency matrix with time dimension and space dimension by using the spatiotemporal feature information; wherein each joint node in the multi-dimensional adjacency matrix has a weight; dividing the target joint node information into different regions, and determining the association information between joint nodes in each region and the association information between joint nodes in different regions; performing graph convolution operation on the multi-dimensional adjacency matrix and the association information to obtain target convolution information, and classifying and recognizing the action of the to-be-recognized object by using the target convolution information. The accuracy of the behavior recognition result of the to-be-recognized object can be further improved by the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a behavior recognition method and device, equipment and medium. BACKGROUND

[0002] In the fields of public security, medical monitoring, traffic control, etc., a behavior recognition algorithm is often used to monitor the abnormal behavior of people. For example, in the field of public security, a behavior recognition algorithm is needed to detect whether there is a fight, theft, etc. in a public place; in the field of medical monitoring, a behavior recognition algorithm is needed to detect whether a patient has fallen, convulsed, etc.; in the field of traffic control, a behavior recognition algorithm is needed to detect whether there is a traffic accident on the road, etc.

[0003] In the prior art, an RGB (Red Green Blue) image-based behavior recognition model is generally used to detect the abnormal behavior of people, but this method is affected by factors such as the surrounding environment, background complexity, and image acquisition angle, which results in poor accuracy of the behavior recognition result. At present, there is no effective solution to this technical problem. SUMMARY

[0004] Therefore, the present application aims to provide a behavior recognition method, device, equipment and medium to further improve the accuracy of the behavior recognition result of a to-be-recognized object. The specific solutions are as follows:

[0005] A behavior recognition method comprises the following steps:

[0006] An image sequence of a to-be-recognized object at consecutive times is obtained, and a pose estimation algorithm is used to extract the joint nodes of the image sequence to obtain target joint node information;

[0007] The spatio-temporal feature information of the target joint node information is extracted, and a multi-dimensional adjacency matrix with time and space dimensions is constructed using the spatio-temporal feature information; each joint node in the multi-dimensional adjacency matrix has a weight;

[0008] The target joint node information is divided into different regions, and the association information between joint nodes in each region and the association information between joint nodes in different regions are determined to obtain target association information;

[0009] The multi-dimensional adjacency matrix and the target association information are subjected to graph convolution operation to obtain target convolution information, and the actions of the to-be-recognized object are classified and recognized using the target convolution information.

[0010] Preferably, the process of extracting the joint nodes of the image sequence by using the pose estimation algorithm to obtain the target joint node information comprises:

[0011] The joint nodes of the image sequence are extracted by using the Openpose algorithm to obtain the target joint node information.

[0012] Preferably, the process of extracting the spatio-temporal feature information of the target joint node information and constructing a multi-dimensional adjacency matrix with time dimension and space dimension by using the spatio-temporal feature information comprises:

[0013] The spatio-temporal feature information of the target joint node information is extracted by using a first model;

[0014] The expression of the first model is:

[0015]

[0016] In the formula, A(v ti ) represents the neighborhood node, v ti represents the convolution center point, v qi represents the neighborhood node in the space dimension, represents the distance between two joint nodes in the space dimension, L represents the threshold value for determining that two joint nodes are adjacent joint nodes, v ni represents the neighborhood node in the time dimension, represents the neighborhood node of the convolution center point of the nth image in the time dimension and the corresponding position joint nodes of the (n-1) th image and the (n+1) th image;

[0017] The multi-dimensional adjacency matrix with time dimension and space dimension is constructed by using the spatio-temporal feature information and a second model;

[0018] The expression of the second model is:

[0019]

[0020] In the formula, U ti represents the multi-dimensional adjacency matrix, U qi represents the adjacency matrix in the space dimension, U ni represents the adjacency matrix in the time dimension, L tq represents the distance between the convolution center point and the neighborhood node in the space domain, l tq represents the distance between the convolution center point and the neighborhood node in the time domain.

[0021] Preferably, the process of dividing the target joint node information into different regions, determining the association information between the joint nodes in each region and the association information between the joint nodes in different regions to obtain the target association information comprises:

[0022] divide the target joint information into a head region, a left upper limb region, a right upper limb region, a left lower limb region and a right lower limb region;

[0023] determine the association information between joints in the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region according to a third model, and determine the association information between joints in different regions of the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region according to a fourth model, to obtain the target association information;

[0024] wherein an expression of the third model is:

[0025]

[0026] wherein, represents a relationship between adjacent nodes in a region where v qi , 1 and b≠a, represents a reference vector, represents a vector between joint a and joint b in region x 1 , 1 represents the head region, x 2 represents the left upper limb region, x 3 represents the right upper limb region, x 4 represents the left lower limb region, x 5 represents the right lower limb region, i∈{1,2,3,4,5};

[0027] an expression of the fourth model is:

[0028]

[0029] wherein, represents a vector between a center node of region x i and a center node of region x j , i x 1 ∈{x 2 ,x 3 ,x 4 ,x 5} and x j ∈{x 1 ,x 2 ,x 3 ,x 4 ,x 5} and x i ≠x j .

[0030] Preferably, further comprising:

[0031] The fifth model is used to normalize the association information between the different joints of the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region.

[0032]

[0033] In the formula, The reference vector is represented as, The vector between the center node of the region x i and the center node of the region x j , x i ∈{x 1 ,x 2 ,x 3 ,x 4 ,x 5}, x j ∈{x 1 ,x 2 ,x 3 ,x 4 ,x 5} and x i ≠x j .

[0034] Preferably, the process of classifying and identifying the action of the to-be-identified object by using the target convolution information comprises:

[0035] The CNN is used to classify the target convolution information to obtain a target classification result, and the target classification result is used to classify and identify the action of the to-be-identified object.

[0036] Preferably, further comprising:

[0037] The target classification result is displayed in the form of probability by using the softmax.

[0038] Correspondingly, the application further discloses a behavior recognition device, comprising:

[0039] A joint extraction module is configured to acquire an image sequence of a to-be-identified object in continuous time, and extract joints of the image sequence by using a pose estimation algorithm to obtain target joint information.

[0040] A matrix construction module is configured to extract spatiotemporal feature information of the target joint information, and construct a multi-dimensional adjacency matrix with time dimension and space dimension by using the spatiotemporal feature information, wherein each joint in the multi-dimensional adjacency matrix has a weight.

[0041] The information determining module is configured to divide the target joint information into different areas, determine the association information between the joints in each area and the association information between the joints in different areas, and obtain target association information.

[0042] The behavior recognition module is configured to perform graph convolution operation on the multi-dimensional adjacency matrix and the target association information to obtain target convolution information, and use the target convolution information to classify and recognize the action of the object to be recognized.

[0043] Correspondingly, the application further discloses a behavior recognition device, which comprises:

[0044] The memory is configured to store a computer program.

[0045] The processor is configured to execute the computer program to realize the steps of the behavior recognition method.

[0046] Correspondingly, the application further discloses a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the behavior recognition method.

[0047] It can be seen that, in the application, firstly, the image sequence of the object to be recognized in continuous time is acquired, and the joint information of the image sequence is extracted by using a pose estimation algorithm to obtain target joint information; then, the spatiotemporal feature information of the target joint information is extracted, and a multi-dimensional adjacency matrix with time dimension and space dimension is constructed by using the spatiotemporal feature information; each joint in the adjacency matrix has a weight; after that, the target joint information is divided into different areas, and the association information between the joints in each area and the association information between the joints in different areas are determined to obtain target association information; finally, graph convolution operation is performed on the multi-dimensional adjacency matrix and the target association information to obtain target convolution information, and the action of the object to be recognized is classified and recognized by using the target convolution information. Compared with the prior art, the method provided by the application is equivalent to recognizing the behavior of the object to be recognized by using the joint information of the object to be recognized, so that the influence of background information and environmental information on the behavior recognition result is avoided. In the process of classifying and recognizing the behavior of the object to be recognized, not only the spatiotemporal feature information of the object to be recognized in the time and space dimensions is covered by the multi-dimensional adjacency matrix, but also the joint information of the object to be recognized in each area and between different areas is divided in coarse granularity and fine granularity, so that the accuracy of the behavior recognition result of the object to be recognized can be further improved. Correspondingly, the behavior recognition device, the device and the medium provided by the application also have the beneficial effects described above. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0049] Figure 1 A flow chart of a behavior recognition method provided by an embodiment of the present application;

[0050] Figure 2 A structural diagram of a behavior recognition device provided by an embodiment of the present application;

[0051] Figure 3 A structural diagram of a behavior recognition device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0053] Please refer to Figure 1 , Figure 1 A flow chart of a behavior recognition method provided by an embodiment of the present application, the method comprising:

[0054] Step S11: acquiring an image sequence of a to-be-recognized object in continuous time, and extracting a joint node of the image sequence by using a pose estimation algorithm to obtain target joint node information;

[0055] Step S12: extracting spatio-temporal feature information of the target joint node information, and constructing a multi-dimensional adjacency matrix with time dimension and space dimension by using the spatio-temporal feature information; each joint node in the multi-dimensional adjacency matrix has a weight;

[0056] Step S13: dividing the target joint node information into different regions, and determining the association information between joint nodes in each region and the association information between joint nodes in different regions to obtain target association information;

[0057] Step S14: performing graph convolution operation on the multi-dimensional adjacency matrix and the target association information to obtain target convolution information, and using the target convolution information to classify and recognize the action of the to-be-recognized object.

[0058] In the embodiment, a behavior recognition method is provided, which is used to recognize the behavior of a to-be-recognized object, and can further improve the accuracy of the behavior recognition result of the to-be-recognized object. In the method, first, an image sequence of the to-be-recognized object in continuous time is obtained, and a pose estimation algorithm is used to extract the joint nodes in the image sequence to obtain target joint node information. Specifically, the DeepPose algorithm, the convolution pose machine, the stacked hourglass network and the like can be used to extract the joint nodes in the image sequence.

[0059] It can be understood that the behavior recognition based on the human body joint nodes can not only effectively filter out the background information in the image sequence of the to-be-recognized object, but also effectively express the joint node information of the to-be-recognized object, so that the accuracy of the behavior recognition of the to-be-recognized object can be improved.

[0060] After the target joint node information of the to-be-recognized object is obtained, the spatio-temporal feature information of the target joint node information is extracted, and a multi-dimensional adjacency matrix with time dimension and space dimension is constructed by using the spatio-temporal feature information. Each joint node in the multi-dimensional adjacency matrix has a weight. It can be conceived that after the spatio-temporal feature information of the target joint node information is extracted and the multi-dimensional adjacency matrix with time dimension and space dimension is constructed by using the spatio-temporal feature information, not only the convolution field in the multi-dimensional adjacency matrix can be expanded, but also the important node information in the joint nodes of the to-be-recognized object is retained.

[0061] Then, the target joint node information is divided into different regions, and the association information between the joint nodes in each region and the association information between different regions is determined to obtain target association information. It can be understood that dividing the target joint node information into different regions is equivalent to dividing the target joint node information according to the human body structure, determining the association information between the joint nodes in each region of the to-be-recognized object is equivalent to representing the motion features of the to-be-recognized object in a coarse-grained form, and determining the association information between the joint nodes in different regions of the to-be-recognized object is equivalent to representing the motion features of the to-be-recognized object in a fine-grained form. Therefore, the target association information obtained from the association information between the joint nodes in each region and the association information between the joint nodes in different regions can express the joint node information of the to-be-recognized object from two aspects of coarse-grained and fine-grained.

[0062] Finally, the multi-dimensional adjacency matrix and the target association information are subjected to graph convolution operation to obtain target convolution information, and the target convolution information and the target association information are used to classify and identify the action of the object to be recognized. It is conceivable that since the graph convolution network (GCN) can process graph structure data with a generalized topology and has strong data space extraction capability, after the multi-dimensional adjacency matrix and the target association information are subjected to graph convolution operation, the posture behavior action features of the object to be recognized can be further enriched, and thus the action of the object to be recognized can be more accurately classified and identified by using the target convolution information.

[0063] It can be seen that in the embodiment, first, the image sequence of the object to be recognized in continuous time is obtained, and the pose estimation algorithm is used to extract the joint nodes of the image sequence to obtain target joint node information; then, the spatio-temporal feature information of the target joint node information is extracted, and the multi-dimensional adjacency matrix with time dimension and space dimension is constructed by using the spatio-temporal feature information; each joint node in the adjacency matrix has a weight; then, the target joint node information is divided into different regions, and the association information between the joint nodes in each region and the association information between the joint nodes in different regions are determined to obtain target association information; finally, the multi-dimensional adjacency matrix and the target association information are subjected to graph convolution operation to obtain target convolution information, and the action of the object to be recognized is classified and identified by using the target convolution information. Compared with the prior art, the method provided in the embodiment is equivalent to identifying the behavior of the object to be recognized by using the joint node information of the object to be recognized, which eliminates the influence of background information and environmental information on the behavior recognition result. In the process of classifying and identifying the behavior of the object to be recognized, not only the multi-dimensional adjacency matrix covers the spatio-temporal feature information of the object to be recognized in the time and space dimensions, but also the joint node information of the object to be recognized in each region and between different regions is divided into coarse-grained and fine-grained, so that the accuracy of the behavior recognition result of the object to be recognized can be further improved by the method.

[0064] Based on the above embodiment, the technical solutions are further described and optimized. As a preferred embodiment, the process of extracting the joint nodes of the image sequence by using the pose estimation algorithm to obtain the target joint node information comprises:

[0065] The joint nodes of the image sequence are extracted by using the Openpose algorithm to obtain the target joint node information.

[0066] It can be understood that, because the Openpose algorithm is not only a free and open-source pose estimation algorithm, but also can identify the joint nodes of multiple target objects, using the Openpose algorithm to extract the joint nodes in the image sequence can not only improve the convenience in extracting the joint nodes of the to-be-identified object in the image sequence, but also can improve the accuracy of the joint node recognition result.

[0067] Specifically, in the embodiment, the image sequence of the to-be-identified object at continuous time is sequentially input into the Openpose algorithm framework, and the joint nodes of the to-be-identified object on each image are extracted therefrom, wherein the joint nodes on each image include the neck, nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right hands, left and right waists, left and right knees, and left and right feet of the to-be-identified object, a total of 18 joint nodes. These joint nodes can be represented as:

[0068]

[0069] In the formula, represents the i-th joint node in the n-th image in the image sequence, T n represents the n-th image.

[0070] Based on the above embodiment, the technical solution is further described and optimized in the embodiment. As a preferred embodiment, the process of extracting the spatio-temporal feature information of the target joint node information and constructing a multi-dimensional adjacency matrix with time dimension and space dimension using the spatio-temporal feature information includes:

[0071] extracting the spatio-temporal feature information of the target joint node information using the first model;

[0072] wherein the expression of the first model is:

[0073]

[0074] In the formula, A(v ti ) represents the neighborhood node, v ti represents the convolution center point, v qi represents the neighborhood node in the spatial dimension, represents the distance between two joint nodes in the spatial dimension, L represents the threshold value for determining that two joint nodes are adjacent joint nodes, v ni represents the neighborhood node in the time dimension, represents the neighborhood node of the convolution center point of the n-th image in the time dimension and the corresponding position joint nodes of the n-1-th image and the n+1-th image;

[0075] A multi-dimensional adjacency matrix with time dimension and space dimension is constructed by using the spatio-temporal feature information and the second model;

[0076] The expression of the second model is:

[0077]

[0078] In the formula, U ti represents the multi-dimensional adjacency matrix, U qi represents the adjacency matrix in the space dimension, U ni represents the adjacency matrix in the time dimension, Ltq represents the distance between the convolution center point and the neighborhood node in the space domain, ltq represents the distance between the convolution center point and the neighborhood node in the time domain.

[0079] In the embodiment, in order to create the multi-dimensional adjacency matrix with time dimension and space dimension, first, the spatio-temporal feature information of the target joint node information is extracted by using the first model, and the neighborhood nodes in the graph convolution network are defined.

[0080] The expression of the first model is:

[0081]

[0082] In the formula, A(v ti ) represents the neighborhood node, v ti represents the convolution center point, v qi represents the neighborhood node in the space dimension, represents the distance between two joint nodes in the space dimension, L represents the threshold value for determining whether two joint nodes are adjacent joint nodes, in the embodiment, L is set to 2, that is, all the nodes that reach the convolution node by crossing one node with the shortest distance are the neighborhood nodes of the convolution node. ni represents the neighborhood node in the time dimension, represents the neighborhood node of the convolution center point of the nth image in the time dimension and the corresponding position joint nodes of the (n-1) th image and the (n+1) th image.

[0083] Obviously, all the neighborhood nodes of the convolution center point in the space domain and the time domain can be found by using the first model, so that the spatio-temporal feature information of all the joint nodes of the object to be recognized can be extracted. Since the neighborhood nodes in the traditional graph convolution network only include first-order neighborhood, the receptive field of each joint node is very limited, and only 0 and 1 are included in the matrix elements of the adjacency matrix in the traditional graph convolution network, so it is difficult to reflect the mutual relationship between the joint nodes. Therefore, in the embodiment, the multi-dimensional adjacency matrix with time dimension and space dimension is created by using the extracted spatio-temporal feature information.

[0084] In creating the multi-dimensional adjacency matrix with time dimension and space dimension, first, the convolution center point v ti and the vector between the joint node v qi ∪v ni , the direction information θ and the length information l between the convolution center point and the neighbor node are obtained; then, the multi-dimensional adjacency matrix with time dimension and space dimension is constructed by using the space-time feature information and the second model.

[0085] The expression of the second model is:

[0086]

[0087] In the formula, U ti represents the multi-dimensional adjacency matrix, U qi represents the adjacency matrix in the space dimension, and U ni represents the adjacency matrix in the time dimension, Ltq represents the distance between the convolution center point and the neighbor node in the space domain, ltq represents the distance between the convolution center point and the neighbor node in the time domain.

[0088] Through the above process, the multi-dimensional adjacency matrix with distance weight information and with time dimension and space dimension can be obtained. That is, through the above process, each joint node in the multi-dimensional adjacency matrix has its own weight, and the purpose of this operation is to enable the graph convolution network to treat each joint node information more differently, so as to pay more attention to those joint nodes with stronger recognition ability.

[0089] It should be noted that in the traditional graph convolution network, the graph convolution network is composed of nodes and edges, the neighbor node represents the adjacent joint node of a certain joint node, and the adjacency matrix represents the association information between the certain joint node and the neighbor node in the form of a matrix, that is, the adjacency matrix represents the weight information between the certain joint node and the neighbor node in the form of a matrix. For the traditional graph convolution network, only the neighbor nodes in the space dimension are found to construct the adjacency matrix, so as to obtain the feature information in the space dimension. In the embodiment, in addition to finding the neighbor nodes in the space dimension, the neighbor nodes in the time dimension are also found to construct the adjacency matrix, so as to obtain the feature information in the time dimension.

[0090] In other words, in the embodiment, the neighbor node v qi found by the graph convolution in the same frame image is the neighbor node in the space dimension; the neighbor node v niThe two different types of neighborhood nodes represent adjacent node information in the spatial domain and adjacent node information in the time domain, respectively. Then, a multi-dimensional adjacency matrix with time and space dimensions is constructed according to the two types of node information. Through the multi-dimensional adjacency matrix, the association relationship of each key node of the object to be identified in the image sequence in the spatial domain and the association relationship in the time domain can be represented, thereby covering the space-time feature information of the object to be identified in the image sequence.

[0091] As a preferred embodiment, the step of dividing the target key node information into different regions and determining the association information between the key nodes in each region and the association information between the key nodes in different regions to obtain the target association information comprises:

[0092] The target key node information is divided into a head region, a left upper limb region, a right upper limb region, a left lower limb region and a right lower limb region.

[0093] The association information between the key nodes in the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region is determined according to a third model, and the association information between the key nodes in different regions in the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region is determined according to a fourth model to obtain the target association information.

[0094] The expression of the third model is:

[0095]

[0096] In the formula, cos(x represents the relationship between the neighborhood nodes in the region where v qi is located, b∈x 1 and b≠a, represents a reference vector, represents a vector between the key node a and the key node b in the region x 1 , x 1 represents the head region, x 2 represents the left upper limb region, x 3 represents the right upper limb region, x 4 represents the left lower limb region, x 5 represents the right lower limb region, i∈{1,2,3,4,5};

[0097] The expression of the fourth model is:

[0098]

[0099] In the formula, cos(x i ,x jrepresents the region x i and the region x j the association information between the nodes of interest x represents the region x i and the region x j the vector between the center node x i ∈{x 1 ,x 2 ,x 3 ,x 4 ,x 5}, x j ∈{x 1 ,x 2 ,x 3 ,x 4 ,x 5} and x i ≠x j , represents the reference vector.

[0100] In the embodiment, the target node information of interest is divided into five regions, i.e., the target node information of interest is divided into a head region, a left upper limb region, a right upper limb region, a left lower limb region and a right lower limb region, and the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region are marked by x 1 , x 2 , x 3 , x 4 and x 5 respectively.

[0101] It can be understood that, since the motion of the nodes in the same limb segment or the same region is highly similar, the vector between a fixed node and other nodes in the same region can be obtained in the same region, and here the vector between the node a and the node b in the region is represented by . Specifically, in the embodiment, the unit direction along the neck node and the nose node is taken as the reference vector

[0102] Then, the cosine value between the vector between the convolution center node in the region x 1 and other nodes in the region and the reference vector is calculated, and is denoted by The greater the value is, the greater the amplitude of the motion of the limb in the region is.

[0103] According to the third model , the relationship between the neighbor nodes in the region where the v qi is located can be calculated, The greater the value is, the greater the amplitude of the motion of the object to be recognized is. By calculating the above information of all the neighbor nodes in the region x 1 , the following information can be obtained wherein b e x 1 and b≠a; then the relative direction and distance information between each region can be extracted to obtain the association information between the joints in the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region.

[0104] To obtain the association information between the joints in different regions, first, the center nodes in each region are obtained, and the vectors between the center nodes in each region are calculated, i.e. the relative direction and relative position information between each region are obtained, wherein the relative direction and relative position information between each region represent the association information between the joints in different regions. Here, the association information between the joints in the regions x 1 and x 2 is taken as an example for specific description. When the regions x 1 and x 2 are determined, the association information between the regions x 1 and x 2 can be determined by calculating the vectors between the center nodes of the regions x 1 and x 2 .

[0105] As a preferred embodiment, the above behavior recognition method further comprises:

[0106] normalizing the association information between the joints in different regions in the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region by using the fifth model;

[0107]

[0108] wherein, represents a reference vector, represents the vector between the center nodes of the regions x i and x j , x i e {x 1 , x 2 , x 3 , x 4 , x 5}, x j e {x 1 , x 2 , x 3 , x 4 , x 5} and x i ≠x j .

[0109] In this embodiment, in order to facilitate subsequent classification and recognition of the action of the to-be-recognized object, the association information between different regions can also be normalized. In the process of normalizing the association information between different regions, the cosine of the angle between the reference vector and the vector of the region x can be calculated first The greater the |cos(x i ,x j | is, the more significant change of the to-be-recognized object in the region x i and the region x j occurs; then, the association information between the nodes of different regions can be normalized by using

[0110] Here, the association information between the nodes of the region x 1 and the region x 2 is taken as an example for specific description. In the process of normalizing the association information between the nodes of the region x 1 and the region x 2 , the cosine of the angle between the reference vector and the vector of the region x can be calculated first After that, the association information between the nodes of the region x 1 and the region x 2 is normalized. Obviously, for the region x 1 , the following can be obtained

[0111] In the process of behavior recognition of the to-be-recognized object, the finally obtained multi-dimensional adjacency matrix and the target association information are input into the graph convolution network for graph convolution operation, so that the action of the to-be-recognized object can be classified and recognized according to the target convolution information. It should be noted that in the traditional graph convolution network, the propagation mode between layers is as follows:

[0112]

[0113] In the formula, U represents the addition of the adjacency matrix U and the unit matrix I, D is the degree matrix of U, H c+1 is the feature of the layer, and sigma is a nonlinear activation function.

[0114] In this application, since the adjacency matrix in the graph convolution network is replaced by the multi-dimensional adjacency matrix with time dimension and space dimension, and the association information of each region and different regions is more carefully divided, the propagation mode between layers in the graph convolution network is as follows:

[0115] ​​​​​​​

[0116] In the formula, U represents the addition of the adjacent matrix U and the unit matrix I, D is the degree matrix of U, H c+1 is the feature of the first layer, σ is a nonlinear activation function, U t ” i = e -α ·U t ' i , U t ' i = U ti + I.

[0117] Based on the above embodiment, the technical solutions are further described and optimized. As a preferred embodiment, the process of classifying and identifying the action of the to-be-identified object by using the target convolution information comprises the following steps:

[0118] The target convolution information is classified by using the CNN to obtain a target classification result, and the action of the to-be-identified object is classified and identified by using the target classification result.

[0119] Specifically, in this embodiment, the CNN (Convolutional Neural Network) is used to classify the target convolution information to obtain a target classification result, and the action of the to-be-identified object is classified and identified by using the target classification result. Because the CNN has a unique advantage in image processing due to its special structure of local weight sharing, its layout is closer to the actual biological neural network, and the weight sharing reduces the complexity of the network, therefore, when the CNN is used to identify the behavior of the to-be-identified object, not only the accuracy and reliability of the behavior recognition result of the to-be-identified object can be improved, but also the algorithm complexity in the behavior recognition of the to-be-identified object can be reduced.

[0120] Specifically, the CNN can be set as a two-layer structure, the convolution kernel in the CNN is set as a 3*3 convolution kernel, and the parameters are set to recognize the action of the to-be-identified object.

[0121] As a preferred embodiment, the above behavior recognition method further comprises:

[0122] The target classification result is displayed in the form of probability by using the softmax.

[0123] In this embodiment, in order to further improve the user experience of the user when viewing the behavior recognition result of the to-be-identified object, after the action of the to-be-identified object is classified and identified by using the CNN, the target classification result can be displayed in the form of probability by using the softmax.

[0124] It is conceivable that through such a setting mode, the staff can clearly view the probability of the to-be-identified object belonging to various behavior states. Obviously, through the technical solution provided by the embodiment, the staff can better identify the behavior of the to-be-identified object, and the user experience of the staff in identifying the behavior of the to-be-identified object can be further improved.

[0125] Please refer to Figure 2 , Figure 2 A structural diagram of a behavior identification device provided by an embodiment of the application is shown in FIG. 3. The device comprises:

[0126] A joint point extraction module 21 is configured to acquire an image sequence of a to-be-identified object in continuous time, and extract joint points of the image sequence by using a pose estimation algorithm to obtain target joint point information.

[0127] A matrix construction module 22 is configured to extract spatiotemporal feature information of the target joint point information, and construct a multi-dimensional adjacency matrix with time dimension and space dimension by using the spatiotemporal feature information. Each joint point in the multi-dimensional adjacency matrix has a weight.

[0128] An information determination module 23 is configured to divide the target joint point information into different regions, and determine the association information between joint points in each region and the association information between joint points in different regions to obtain target association information.

[0129] A behavior identification module 24 is configured to perform graph convolution operation on the multi-dimensional adjacency matrix and the target association information to obtain target convolution information, and classify and identify the action of the to-be-identified object by using the target convolution information.

[0130] The behavior identification device provided by the embodiment of the application has the beneficial effects of the behavior identification method disclosed above.

[0131] Please refer to Figure 3 , Figure 3 A structural diagram of a behavior identification device provided by an embodiment of the application is shown in FIG. 3. The device comprises:

[0132] A memory 31 is configured to store a computer program.

[0133] A processor 32 is configured to execute the computer program to realize the steps of the behavior identification method disclosed above.

[0134] The behavior identification device provided by the embodiment of the application has the beneficial effects of the behavior identification method disclosed above.

[0135] Correspondingly, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the behavior recognition method disclosed above.

[0136] The computer readable storage medium provided by the embodiment of the present application has the beneficial effects of the behavior recognition method disclosed above.

[0137] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0138] Finally, it should be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the sentence "including a" does not exclude the existence of other identical elements in the process, method, article or equipment including the element.

[0139] The above describes in detail the behavior recognition method, device, equipment and medium provided by the present application. The principle and implementation mode of the present application are described by applying specific examples in this paper. The above embodiment is only used to help understand the method of the present application and its core idea; at the same time, for the general technical personnel in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; according to the above, the content of the specification should not be understood as the limitation of the present application.

Claims

1. A behavior recognition method, characterized in that, include: The image sequence of the object to be identified over a continuous time period is obtained, and the key points of the image sequence are extracted using a pose estimation algorithm to obtain the target key point information; The spatiotemporal feature information of the target key point information is extracted, and a multidimensional adjacency matrix with time and space dimensions is constructed using the spatiotemporal feature information; wherein each key point in the multidimensional adjacency matrix has a weight; The target joint information is divided into different regions, and the association information between joints in each region and between joints in different regions is determined to obtain the target association information. A graph convolution operation is performed on the multidimensional adjacency matrix and the target association information to obtain target convolution information, and the target convolution information is used to classify and identify the actions of the object to be identified. The process of extracting the spatiotemporal feature information of the target joint information and constructing a multidimensional adjacency matrix with time and space dimensions using the spatiotemporal feature information includes: The spatiotemporal feature information of the target joint information is extracted using the first model; The expression for the first model is: ； In the formula, Represents neighboring nodes. Represents the center point of the convolution. Represents the neighborhood nodes in the spatial dimension. This represents the distance between two key points in a spatial dimension. This represents the threshold value for determining whether two joints are adjacent joints. Represents the neighborhood nodes in the time dimension. Indicates the first in the time dimension The convolution center point of the nth image and the nth Zhang image and the first Neighboring nodes of corresponding key points on the image; The spatiotemporal feature information and the second model are used to construct the multidimensional adjacency matrix with time and spatial dimensions; The expression for the second model is: ； In the formula, Represents a multidimensional adjacency matrix. Represents the adjacency matrix in spatial dimensions. Represents the adjacency matrix in the time dimension. This represents the distance between the convolution center point and its neighboring nodes within the spatial domain. This represents the distance between the convolution center point and its neighboring nodes in the time domain. The process of dividing the target joint information into different regions, determining the association information between joints within each region and between joints in different regions to obtain target association information includes: The target joint information is divided into the head region, left upper limb region, right upper limb region, left lower limb region, and right lower limb region; The association information between joints in the head region, the left upper limb region, the right upper limb region, the left lower limb region, and the right lower limb region is determined according to the third model, and the association information between joints in different regions of the head region, the left upper limb region, the right upper limb region, the left lower limb region, and the right lower limb region is determined according to the fourth model, so as to obtain the target association information; The expression for the third model is: ； In the formula, express The relationship between neighboring nodes within the same region and , , This represents a reference vector, which is a vector defined along a unit direction at the cervical and nasal joints. Indicates the region Internal joints With key points Vectors between This refers to the head region. This refers to the area of ​​the left upper limb. This refers to the area of ​​the right upper limb. Indicates the left lower limb area. Indicates the right lower limb area. ; The expression for the fourth model is: ； In the formula, Indicates the region With the region The vector between the central nodes, , and ; The behavior recognition method further includes: The fifth model is used to normalize the association information between the joints in different regions of the head region, the left upper limb region, the right upper limb region, the left lower limb region, and the right lower limb region; ； In the formula, , This represents a reference vector, which is a vector defined along a unit direction at the cervical and nasal joints. Indicates the region With the region The vector between the central nodes, , and .

2. The behavior recognition method according to claim 1, characterized in that, The process of extracting the joints of the image sequence using a pose estimation algorithm to obtain target joint information includes: The key points of the image sequence are extracted using the Openpose algorithm to obtain the target key point information.

3. The behavior recognition method according to claim 1, characterized in that, The process of classifying and recognizing the actions of the object to be identified using the target convolutional information includes: The target convolutional information is classified using CNN to obtain the target classification result, and the action of the object to be identified is classified and identified using the target classification result.

4. The behavior recognition method according to claim 3, characterized in that, Also includes: The classification results of the target are displayed in the form of probabilities using softmax.

5. A behavior recognition device, characterized in that, include: The joint extraction module is used to acquire the image sequence of the object to be identified over a continuous time period, and to extract the joints of the image sequence using a pose estimation algorithm to obtain the target joint information. The matrix construction module is used to extract the spatiotemporal feature information of the target joint information and use the spatiotemporal feature information to construct a multidimensional adjacency matrix with time and space dimensions; wherein, each joint in the multidimensional adjacency matrix has a weight; The information determination module is used to divide the target joint information into different regions, and determine the association information between joints in each region and the association information between joints in different regions to obtain target association information; The behavior recognition module is used to perform graph convolution operation on the multidimensional adjacency matrix and the target association information to obtain target convolution information, and to use the target convolution information to classify and recognize the actions of the object to be identified. Specifically, the matrix construction module is used to extract the spatiotemporal feature information of the target key point information using a first model; wherein the expression of the first model is: ； In the formula, Represents neighboring nodes. Represents the center point of the convolution. Represents the neighborhood nodes in the spatial dimension. This represents the distance between two key points in a spatial dimension. This represents the threshold value for determining whether two joints are adjacent joints. Represents the neighborhood nodes in the time dimension. Indicates the first in the time dimension The convolution center point of the nth image and the nth Zhang image and the first Neighboring nodes of corresponding key points on the image; The multidimensional adjacency matrix with temporal and spatial dimensions is constructed using the aforementioned spatiotemporal feature information and the second model; wherein, the expression of the second model is: ； In the formula, Represents a multidimensional adjacency matrix. Represents the adjacency matrix in spatial dimensions. Represents the adjacency matrix in the time dimension. This represents the distance between the convolution center point and its neighboring nodes within the spatial domain. This represents the distance between the convolution center point and its neighboring nodes in the time domain. The information determination module is specifically used to divide the target joint information into a head region, a left upper limb region, a right upper limb region, a left lower limb region, and a right lower limb region; determine the association information between joints in the head region, the left upper limb region, the right upper limb region, the left lower limb region, and the right lower limb region according to a third model, and determine the association information between joints in different regions of the head region, the left upper limb region, the right upper limb region, the left lower limb region, and the right lower limb region according to a fourth model, thereby obtaining the target association information; wherein, the expression of the third model is: ； In the formula, express The relationship between neighboring nodes within the same region and , , This represents a reference vector, which is a vector defined along a unit direction at the cervical and nasal joints. Indicates the region Internal joints With key points Vectors between This refers to the head region. This refers to the area of ​​the left upper limb. This refers to the area of ​​the right upper limb. Indicates the left lower limb area. Indicates the right lower limb area. ; The expression for the fourth model is: ； In the formula, Indicates the region With the region The vector between the central nodes, , and ; The behavior recognition device is also used to normalize the association information between the joints of different regions in the head region, the left upper limb region, the right upper limb region, the left lower limb region and the right lower limb region using the fifth model; ； In the formula, , This represents a reference vector, which is a vector defined along a unit direction at the cervical and nasal joints. Indicates the area With the region The vector between the central nodes, , and .

6. A behavior recognition device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of a behavior recognition method as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a behavior recognition method as described in any one of claims 1 to 4.

Images