Skeleton behavior recognition method based on time multi-scale adaptive feature aggregation

Abstract

The application discloses a skeleton behavior recognition method based on time multi-scale adaptive feature aggregation, comprising: inputting acquired skeleton sequence data into a pre-trained space-time graph convolution network; wherein the space-time graph convolution network comprises multiple levels of space-time graph convolution layers connected in sequence, each level of space-time graph convolution layer comprises a spatial convolution module, a time convolution module and a multi-scale adaptive feature aggregation module; according to a time feature map output by the time convolution module and a first output feature map output by the multi-scale adaptive feature aggregation module, an output feature map of each level of space-time graph convolution layer is determined, and the output feature map is taken as input data of a next level of space-time graph convolution layer; and according to the output feature map of the last level of space-time graph convolution layer, a behavior recognition result of the skeleton sequence data is output. The application can improve the time sequence feature extraction capability for the skeleton sequence data, thereby improving the accuracy of the space-time graph convolution network behavior recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and particularly relates to a skeleton behavior recognition method based on time multi-scale adaptive feature aggregation. BACKGROUND

[0002] At present, in the field of deep learning, human behavior recognition methods can be divided into behavior recognition methods based on RGB videos and behavior recognition methods based on skeletons; among them, the behavior recognition method based on RGB videos usually takes an RGB image sequence, a video or a specific fusion of these modalities such as an RGB image and an optical flow as the input of a model, compared with human skeleton data, the aforementioned modalities will produce more computational consumption, and are not robust enough when facing complex backgrounds and human scale changes, view changes and motion speed changes. Therefore, the behavior recognition method based on skeleton gradually becomes the mainstream algorithm today.

[0003] In the behavior recognition method based on skeleton, a spatio-temporal graph convolution network takes skeleton sequence data as the input of a recognition model, uses graph convolution to extract features in space, and then uses two-dimensional convolution to extract features in time. However, the duration of human behavior actions is long or short, there is a short-term time sequence relationship and a long-term dependence relationship, for example, the behavior of throwing something, the short-term view is only the hand action, and only the long-term picture of the falling object can judge the action. For the spatio-temporal graph convolution network based on skeleton sequence data, the extraction of time sequence features adopts a simple two-dimensional convolution operation, which is good for short-time feature extraction of skeleton sequence, but cannot effectively solve the long-time dependence problem of skeleton sequence. SUMMARY

[0004] In order to solve the above problems existing in the prior art, the application provides a skeleton behavior recognition method and device based on time multi-scale adaptive feature aggregation. The technical problems to be solved by the application are realized through the following technical solutions:

[0005] In the first aspect, the application provides a skeleton behavior recognition method based on time multi-scale adaptive feature aggregation, comprising:

[0006] Obtaining skeleton sequence data of a human body from a to-be-processed video;

[0007] Inputting the skeleton sequence data into a pre-trained spatio-temporal graph convolution network; wherein the spatio-temporal graph convolution network comprises multiple levels of spatio-temporal graph convolution layers connected in sequence, each level of spatio-temporal graph convolution layer comprises a spatial convolution module, a time convolution module and a multi-scale adaptive feature aggregation module, and the output ends of the spatial convolution module are connected with the input ends of the time convolution module and the multi-scale adaptive feature aggregation module, respectively;

[0008] determine an output feature map of each level of the spatio-temporal graph convolutional layer according to the time feature map output by the time convolution module and the first output feature map output by the multi-scale adaptive feature aggregation module, and take the output feature map as input data of a next level of the spatio-temporal graph convolutional layer;

[0009] output the behavior recognition result of the skeleton sequence data according to the output feature map of the last level of the spatio-temporal graph convolutional layer.

[0010] In an embodiment of the present application, the multi-scale adaptive feature aggregation module comprises a pooling layer, a local branch and a global branch.

[0011] Each level of the spatio-temporal graph convolutional layer processes the input data according to the following steps:

[0012] extract spatial features of the input data by using the spatial convolution module to obtain a spatial feature map;

[0013] input the spatial feature map into the time convolution module and the multi-scale adaptive feature aggregation module respectively, so that the time convolution module extracts time features of the input data to obtain a time feature map, the pooling layer compresses the spatial feature map in space, and the local branch generates a time series sensitivity map based on the compressed spatial feature map, and then generates a second output feature map according to the time series sensitivity map, and the global branch generates an adaptive convolution kernel based on the compressed spatial feature map;

[0014] convolve the second output feature map by using the adaptive convolution kernel to obtain a first output feature map of the multi-scale adaptive feature aggregation module;

[0015] add the first output feature map and the time feature map to obtain an output feature map of each level of the spatio-temporal graph convolutional layer.

[0016] In an embodiment of the present application, the local branch comprises a first convolutional layer, a first Relu layer, a second convolutional layer, a Sigmod layer and a scaling layer connected in sequence, and the global branch comprises a first fully connected layer, a second Relu layer, a second fully connected layer and a first Softmax layer connected in sequence.

[0017] In an embodiment of the present application, the steps of inputting the spatial feature map into the time convolution module and the multi-scale adaptive feature aggregation module respectively, so that the time convolution module extracts time features of the input data to obtain a time feature map, the pooling layer compresses the spatial feature map in space, and the local branch generates a time series sensitivity map based on the compressed spatial feature map, and then generates a second output feature map according to the time series sensitivity map, and the global branch generates an adaptive convolution kernel based on the compressed spatial feature map, comprise:

[0018] input the spatial feature map into a time convolution module and a multi-scale adaptive feature aggregation module respectively;

[0019] The time convolution module extracts the time features of the input data to obtain a time feature map.

[0020] The pooling layer compresses the spatial feature map in space to obtain a compressed spatial feature map.

[0021] The first convolution layer in the local branch performs one-dimensional convolution on the compressed spatial feature map to obtain a first feature map with a channel number of C / 4, and after being processed by the excitation function Relu of the first Relu layer, the first feature map is input to the second convolution layer, C being the channel number of the compressed spatial feature map.

[0022] The second convolution layer restores the channel number of the processed first feature map to C to generate a time sensitivity map.

[0023] After the time sensitivity map is processed by the Sigmod layer, the scaling layer replicates the processed time sensitivity map in the spatial dimension and performs dot product operation with the spatial feature map to obtain a second output feature map of the local branch.

[0024] The first fully connected layer in the global branch expands the compressed spatial feature map into a second feature map with a dimension of 2T, and after being processed by the second Relu layer, the second feature map is input to the second fully connected layer, T being the time dimension of the compressed spatial feature map.

[0025] The second fully connected layer converts the second feature map into a third feature map with a preset size CxK, and after the third feature map is input into the first Softmax layer, an adaptive convolution kernel is obtained.

[0026] In an embodiment of the present application, the spatio-temporal graph convolution network further comprises a global average pooling layer, a third fully connected layer and a second Softmax layer connected in sequence with the multi-level spatio-temporal graph convolution layer.

[0027] The step of outputting the behavior recognition result of the skeleton sequence data according to the output feature map of the last level spatio-temporal graph convolution layer comprises:

[0028] The global average pooling layer is used to perform global average pooling processing on the output feature map of the last level spatio-temporal graph convolution layer to obtain an output feature.

[0029] The third fully connected layer is used to calculate the confidence of each behavior class based on the output feature.

[0030] The second Softmax layer is used to classify according to the confidence, and a behavior recognition result of the skeleton sequence data is obtained.

[0031] In a second aspect, the application provides a skeleton behavior recognition device based on time multi-scale adaptive feature aggregation, comprising:

[0032] An acquisition module is configured to acquire skeleton sequence data of a human body from a video to be processed;

[0033] An input module is configured to input the skeleton sequence data into a pre-trained spatio-temporal graph convolution network; wherein the spatio-temporal graph convolution network comprises multiple levels of spatio-temporal graph convolution layers connected in sequence, each level of spatio-temporal graph convolution layer comprises a spatial convolution module, a time convolution module and a multi-scale adaptive feature aggregation module, and the output end of the spatial convolution module is connected to the input end of the time convolution module and the multi-scale adaptive feature aggregation module, respectively;

[0034] A determination module is configured to determine an output feature map of each level of spatio-temporal graph convolution layer according to a time feature map output by the time convolution module and a first output feature map output by the multi-scale adaptive feature aggregation module, and take the output feature map as input data of a next level of spatio-temporal graph convolution layer;

[0035] An output module is configured to output a behavior recognition result of the skeleton sequence data according to an output feature map of a last level of spatio-temporal graph convolution layer.

[0036] In a third aspect, the application further provides an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus;

[0037] The memory is configured to store a computer program;

[0038] The processor is configured to execute the program stored on the memory to implement the method steps of the first aspect.

[0039] Compared with the prior art, the application has the following beneficial effects:

[0040] The application provides a skeleton behavior recognition method and device based on time multi-scale adaptive feature aggregation, wherein the time convolution module can effectively extract short-time features of skeleton sequence data, and in the multi-scale adaptive feature aggregation module, a local branch uses a time sequence sensitivity map to further enhance the short-time features, and a global branch can focus on long-time features of the skeleton sequence data, so that the combination of the two can improve the time sequence feature extraction capability of the model for the skeleton sequence data, thereby improving the accuracy of behavior recognition.

[0041] The application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of a skeleton behavior recognition method based on time multi-scale adaptive feature aggregation provided by an embodiment of the present application;

[0043] Figure 2 is a structural schematic diagram of a spatio-temporal graph convolution network provided by an embodiment of the present application;

[0044] Figure 3 is a structural schematic diagram of a multi-scale adaptive feature aggregation module provided by an embodiment of the present application;

[0045] Figure 4 is a structural schematic diagram of a skeleton behavior recognition device based on time multi-scale adaptive feature aggregation provided by an embodiment of the present application;

[0046] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0048] Figure 1 is a flowchart of a skeleton behavior recognition method based on time multi-scale adaptive feature aggregation provided by an embodiment of the present application, Figure 2 is a structural schematic diagram of a spatio-temporal graph convolution network provided by an embodiment of the present application, Figure 3 is a structural schematic diagram of a multi-scale adaptive feature aggregation module provided by an embodiment of the present application. As shown in Figures 1-3 a skeleton behavior recognition method based on time multi-scale adaptive feature aggregation is provided by an embodiment of the present application, comprising:

[0049] S1, obtaining skeleton sequence data of a human body from a video to be processed;

[0050] S2, inputting the skeleton sequence data into a pre-trained spatio-temporal graph convolution network; wherein the spatio-temporal graph convolution network comprises multi-stage spatio-temporal graph convolution layers connected in sequence, each stage of the spatio-temporal graph convolution layers comprises a spatial convolution module (GCN), a time convolution module (TCN) and a multi-scale adaptive feature aggregation module (MS-AA), and the output ends of the spatial convolution module are connected with the input ends of the time convolution module and the multi-scale adaptive feature aggregation module, respectively;

[0051] S3, determining an output feature map of each stage of the spatio-temporal graph convolution layers according to a time feature map output by the time convolution module and a first output feature map output by the multi-scale adaptive feature aggregation module, and taking the output feature map as input data of a next stage of the spatio-temporal graph convolution layers;

[0052] S4, output the behavior recognition result of the skeleton sequence data according to the output feature map of the last spatio-temporal graph convolution layer.

[0053] Specifically, the embodiment utilizes a pre-trained spatio-temporal graph convolution network to perform behavior recognition on the skeleton sequence data of the human body, the spatio-temporal graph convolution network comprises a plurality of spatio-temporal graph convolution layers, a global average pooling layer and a fully connected layer connected in sequence, and optionally, the spatio-temporal graph convolution network comprises nine spatio-temporal graph convolution layers, i.e., L1-L9, each spatio-temporal graph convolution layer has the same structure and comprises a spatial convolution module, a temporal convolution module and a multi-scale adaptive feature aggregation module.

[0054] In the above steps S2-S3, after the skeleton sequence data of the human body is input into the pre-trained spatio-temporal graph convolution network, the skeleton sequence data is first input into the nine spatio-temporal graph convolution layers L1-L9, and it should be understood that the input data of the first spatio-temporal graph convolution layer L1 is the skeleton sequence data of the human body, and the input data of the second spatio-temporal graph convolution layer L2 to the ninth spatio-temporal graph convolution layer L9 is the output feature map of the previous spatio-temporal graph convolution layer. Further, after the processing of the above nine spatio-temporal graph convolution layers L1-L9, the output feature map of the ninth spatio-temporal graph convolution layer L9 is input into the global average pooling layer, the global average pooling layer averages each channel in the output feature map of L9, and each average value is used as a neuron for a classification task, and then the probability belonging to each category is obtained by the fully connected layer and the Softmax function, and finally the behavior recognition result of the skeleton sequence data is determined.

[0055] As shown in FIG. 1, in the embodiment, the multi-scale adaptive feature aggregation module comprises a pooling layer, a local branch and a global branch. Figures 2-3

[0056] Optionally, each spatio-temporal graph convolution layer processes the input data according to the following steps:

[0057] The spatial feature of the input data is extracted by the spatial convolution module to obtain a spatial feature map;

[0058] The spatial feature map is input into the temporal convolution module and the multi-scale adaptive feature aggregation module respectively, so that the temporal convolution module extracts the temporal feature of the input data to obtain a temporal feature map, the pooling layer compresses the spatial feature map in space, the local branch generates a time series sensitivity map based on the compressed spatial feature map, the second output feature map is generated according to the time series sensitivity map, and the global branch generates an adaptive convolution kernel based on the compressed spatial feature map;

[0059] The adaptive convolution kernel is used to convolve the second output feature map to obtain the first output feature map of the multi-scale adaptive feature aggregation module;

[0060] ​The first output feature map is added to the time feature map to obtain an output feature map of each spatio-temporal graph convolution layer.

[0061] Specifically, the local branch in the multi-scale adaptive feature aggregation module includes a first convolution layer (Conv1D), a first Relu layer, a second convolution layer (Conv1D), a Sigmod layer and a rescale layer connected in sequence, wherein the convolution kernel sizes of the first convolution layer are set to 9, 9, 9, 9, 5, 5, 5, 3, 3 in sequence, the convolution steps are all 1, and the number of convolution kernels is 16, 16, 16, 32, 32, 32, 64, 64, 64 in sequence; the convolution kernel sizes of the second convolution layer are all 1, the convolution steps are all set to 1, and the number of convolution kernels is 64, 64, 64, 128, 128, 128, 256, 256, 256 in sequence; further, the global branch includes a first full connection layer (FC), a second Relu layer, a second full connection layer (FC) and a first Softmax layer connected in sequence, the output neuron numbers of the first full connection layer are set to 600, 600, 600, 600, 600, 300, 300, 300, 150, 150 in sequence, the output neuron number of the second full connection layer is the adaptive convolution kernel size K generated by the global branch, which is set to 9, 9, 9, 9, 5, 5, 5, 3, 3 in sequence, and when the generated adaptive convolution kernel is used for convolution operation on the second output feature map of the local branch, the convolution steps are set to 1, 1, 1, 2, 1, 1, 2, 1, 1 in sequence.

[0062] In the embodiment, the spatial feature map is input into the time convolution module and the multi-scale adaptive feature aggregation module respectively, so that the time convolution module extracts the time feature of the input data to obtain a time feature map, the pooling layer compresses the spatial feature map in space, and after the local branch generates a time sequence sensitivity map based on the compressed spatial feature map, the second output feature map is generated according to the time sequence sensitivity map, and the adaptive convolution kernel is generated based on the compressed spatial feature map by using the global branch.

[0063] The spatial feature map is input into the time convolution module and the multi-scale adaptive feature aggregation module respectively.

[0064] The time convolution module extracts the time feature of the input data to obtain a time feature map.

[0065] The pooling layer compresses the spatial feature map in space to obtain a compressed spatial feature map.

[0066] The first convolution layer in the local branch performs one-dimensional convolution on the compressed spatial feature map to obtain a first feature map with a channel number of C / 4, and after being processed by the excitation function Relu of the first Relu layer, the first feature map is input into the second convolution layer, and C is the channel number of the compressed spatial feature map.

[0067] The second convolutional layer restores the channel number of the processed first feature map to C to generate a time sensitivity map;

[0068] After the time sensitivity map is processed by the Sigmod layer, the scaling layer replicates the processed time sensitivity map in the spatial dimension and performs dot product operation with the spatial feature map to obtain the output feature map of the local branch;

[0069] The first fully connected layer in the global branch expands the compressed spatial feature map to a second feature map with a dimension of 2T, and after being processed by the second Relu layer, inputs the second feature map to the second fully connected layer, where T is the time dimension of the compressed spatial feature map;

[0070] The second fully connected layer converts the second feature map into a third feature map with a preset size of CxK, and after inputting the third feature map into the first Softmax layer, an adaptive convolution kernel is obtained.

[0071] Specifically, for the first spatio-temporal graph convolutional layer, the spatial convolutional module extracts the spatial features of the skeleton sequence data, and after generating the spatial feature map, on the one hand, the time convolutional module extracts the time features of the input data, i.e., the skeleton sequence data, to obtain a time feature map; on the other hand, for the spatial feature map XeR C×T×V , C, T, and V respectively represent the channel number, time dimension, and joint node number of the spatial feature map, since the time features need to be focused on, the pooling layer in the multi-scale adaptive feature aggregation module first compresses the spatial feature map in space to X'eR C×T , then the first convolutional layer in the local branch performs one-dimensional convolution on the compressed spatial feature map to reduce the channel number to C / 4, and then sends it to the second convolutional layer through the Relu activation function, which restores the channel number to C to generate a time sensitivity map, which has a discriminative role for local time features, then after the Sigmod activation function, the time sensitivity map is replicated in the spatial dimension and dot product operation is performed with the spatial feature map X to serve as the second output feature map of the local branch; at the same time, the first fully connected layer in the global branch expands the T dimension of the compressed spatial feature map to 2T to obtain a second feature map; then the second feature map is input into the second Relu layer, which is processed by the Relu activation function and then sent to the second fully connected layer, which converts the second feature map with a T dimension to a third feature map with a preset size of CxK, and finally generates an adaptive convolution kernel after processing by the Softmax activation function.

[0072] It should be noted that the processing process of the input data by the local branch and the global branch in each spatio-temporal graph convolutional layer is the same, so the processing process of the second spatio-temporal graph convolutional layer L2 to the ninth spatio-temporal graph convolutional layer L9 will not be repeated here.

[0073] Further, the spatio-temporal graph convolution network further comprises a global average pooling layer, a third fully connected layer and a second Softmax layer connected with the multi-level spatio-temporal graph convolution layer in sequence.

[0074] According to the output feature map of the last level spatio-temporal graph convolution layer, the step of outputting the behavior recognition result of the skeleton sequence data comprises:

[0075] The global average pooling layer is used to perform global average pooling processing on the output feature map of the last level spatio-temporal graph convolution layer to obtain an output feature.

[0076] The third fully connected layer is used to calculate the confidence of each behavior category based on the output feature.

[0077] The second Softmax layer is used to classify according to the confidence to obtain the behavior recognition result of the skeleton sequence data.

[0078] It should be noted that when the spatio-temporal graph convolution network is trained, the training set is composed of at least 70,000 training samples, each of which contains human skeleton data and a behavior category label value corresponding to each human skeleton data. During the training process, the training samples are input into the spatio-temporal graph convolution network, the gradient descent method is used to update the parameters of each layer of the neural network, the updated parameters are assigned to the parameters of each layer of the spatio-temporal graph convolution network, and the trained spatio-temporal graph convolution network is obtained through continuous iteration.

[0079] Further, the step of updating the parameters of each layer of the spatio-temporal graph convolution network by using the gradient descent method comprises the following steps:

[0080] Step 1, set the learning rate of the spatio-temporal graph convolution network to 0.01, the batch training size to 64, and the iteration number to 80, and reduce the learning rate to 0.001 at the 10th iteration and to 0.0001 at the 50th iteration.

[0081] Step 2, the output value of the spatio-temporal graph convolution network and the behavior category label value are used as the gradient value.

[0082] Step 3, the parameters of each layer of the spatio-temporal graph convolution network are updated by using the following formula:

[0083]

[0084] wherein, represents the parameter value of the updated spatio-temporal graph convolution network, represents the assignment operation, and represents the parameter value of the spatio-temporal graph convolution network before updating, represents the gradient value of the spatio-temporal graph convolution network.

[0085] Figure 4It is a structural schematic diagram of a skeleton behavior recognition device based on time multi-scale adaptive feature aggregation provided by an embodiment of the present application. Figure 4 As shown in the figure, the present application also provides a skeleton behavior recognition device based on time multi-scale adaptive feature aggregation, comprising:

[0086] The acquisition module 410 is configured to acquire skeleton sequence data of a human body from a video to be processed.

[0087] The input module 420 is configured to input the skeleton sequence data into a pre-trained spatio-temporal graph convolution network.

[0088] The determination module 430 is configured to determine an output feature map of each level of the spatio-temporal graph convolution layer according to a time feature map output by the time convolution module and a first output feature map output by the multi-scale adaptive feature aggregation module, and take the output feature map as input data of a next level of the spatio-temporal graph convolution layer.

[0089] The output module 440 is configured to output a behavior recognition result of the skeleton sequence data according to an output feature map of the last level of the spatio-temporal graph convolution layer.

[0090] As can be seen from the above embodiments, the present application has the following advantages:

[0091] The present application provides a skeleton behavior recognition method and device based on time multi-scale adaptive feature aggregation.

[0092] The present application also provides an electronic device, as shown in the figure, comprising a processor 501, a communication interface 502, a memory 503 and a communication bus 504. Figure 5 The processor 501, the communication interface 502 and the memory 503 complete mutual communication through the communication bus 504.

[0093] The memory 503 is configured to store a computer program.

[0094] The processor 501 is configured to execute the program stored in the memory 503 to implement the following steps:

[0095] obtain skeleton sequence data of a human body from a video to be processed;

[0096] input the skeleton sequence data into a pre-trained spatio-temporal graph convolution network; wherein the spatio-temporal graph convolution network comprises multiple levels of spatio-temporal graph convolution layers connected in sequence, each level of spatio-temporal graph convolution layer comprises a spatial convolution module, a temporal convolution module and a multi-scale adaptive feature aggregation module, the output end of the spatial convolution module is connected with the input end of the temporal convolution module and the multi-scale adaptive feature aggregation module respectively;

[0097] determine the output feature map of each level of spatio-temporal graph convolution layer according to the temporal feature map output by the temporal convolution module and the first output feature map output by the multi-scale adaptive feature aggregation module, and take the output feature map as the input data of the next level of spatio-temporal graph convolution layer;

[0098] output the behavior recognition result of the skeleton sequence data according to the output feature map of the last level of spatio-temporal graph convolution layer.

[0099] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0100] The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0101] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0102] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0103] The method provided by the embodiments of the present application can be applied to an electronic device. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. Herein, no limitation is made, and any electronic device that can implement the present application belongs to the protection scope of the present application.

[0104] For the device / electronic device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the related parts refer to the part of the method embodiments.

[0105] It should be noted that the device, the electronic device and the storage medium of the embodiments of the present application are respectively the device, the electronic device and the storage medium applying the skeleton behavior recognition method based on time multi-scale adaptive feature aggregation, and all the embodiments of the skeleton behavior recognition method based on time multi-scale adaptive feature aggregation are applicable to the device, the electronic device and the storage medium, and can achieve the same or similar beneficial effects.

[0106] In the description of the present application, the terms "first", "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0107] Reference to terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc., mean that a particular feature, structure, material or characteristic being described is included in at least one embodiment or example of the present application. The illustrative appearances of the above-mentioned terms in various places in the specification are not necessarily referred to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Also, the terminology used has sp ecial meanings as commonly understood by those who are skilled in the art to which the present application pertains.

[0108] Although the present application has been described herein in relation to particular embodiments thereof, many other changes, modifications and

[0109] The above description is further to make further detailed description of the present application in combination with specific preferred embodiments, and cannot be deemed to limit the specific implementation of the present application to these descriptions. For those skilled in the art of the present application, a number of simple deductions or replacements can be made without departing from the concept of the present application, and all of them shall be deemed to fall within the protection scope of the present application.

Claims

1. A skeleton behavior recognition method based on temporal multi-scale adaptive feature aggregation, characterized in that: include: Obtain human skeleton sequence data from the video to be processed; Inputting the skeleton sequence data into a pre-trained spatiotemporal graph convolutional network; wherein the spatiotemporal graph convolutional network includes multiple levels of spatiotemporal graph convolutional layers connected in sequence, each level of the spatiotemporal graph convolutional layer includes: a spatial convolution module, a temporal convolution module and a multi-scale adaptive feature aggregation module, and the output end of the spatial convolution module is connected to the input end of the temporal convolution module and the multi-scale adaptive feature aggregation module respectively; Determine the output feature map of each level of spatiotemporal graph convolution layer based on the temporal feature map output by the temporal convolution module and the first output feature map output by the multi-scale adaptive feature aggregation module, and use the output feature map as the input data of the next level of spatiotemporal graph convolution layer; Outputting the behavior recognition result of the skeleton sequence data according to the output feature map of the last level spatiotemporal graph convolutional layer; The multi-scale adaptive feature aggregation module includes a pooling layer, a local branch and a global branch; Each level of spatiotemporal graph convolutional layer processes the input data as follows: Use the spatial convolution module to extract the spatial features of the input data and obtain the spatial feature map; Inputting the spatial feature map into a temporal convolution module and a multi-scale adaptive feature aggregation module respectively, so that the temporal convolution module extracts the temporal features of the input data to obtain a temporal feature map, causing the pooling layer to spatially compress the spatial feature map, and using the local branch to generate a temporal sensitivity map based on the compressed spatial feature map, generating a second output feature map according to the temporal sensitivity map, and using the global branch to generate an adaptive convolution kernel based on the compressed spatial feature map; Convolving the second output feature map with the adaptive convolution kernel to obtain a first output feature map of a multi-scale adaptive feature aggregation module; Adding the first output feature map to the temporal feature map to obtain an output feature map of each level of the spatiotemporal graph convolution layer; The local branch includes a first convolutional layer, a first Relu layer, a second convolutional layer, a Sigmod layer and a scaling layer connected in sequence, and the global branch includes a first fully connected layer, a second Relu layer, a second fully connected layer and a first Softmax layer connected in sequence.

2. The skeleton behavior recognition method based on temporal multi-scale adaptive feature aggregation according to claim 1 is characterized in that: The spatial feature map is input into a temporal convolution module and a multi-scale adaptive feature aggregation module respectively, so that the temporal convolution module extracts the temporal features of the input data to obtain a temporal feature map, the pooling layer compresses the spatial feature map in space, and uses the local branch to generate a temporal sensitivity map based on the compressed spatial feature map, generates a second output feature map according to the temporal sensitivity map, and uses the global branch to generate an adaptive convolution kernel based on the compressed spatial feature map, including: Inputting the spatial feature map into the temporal convolution module and the multi-scale adaptive feature aggregation module respectively; The temporal convolution module extracts the temporal features of the input data to obtain a temporal feature map; The pooling layer compresses the spatial feature map in space to obtain a compressed spatial feature map; The first convolution layer in the local branch performs one-dimensional convolution on the compressed spatial feature map to obtain a channel number of C / 4, and is input to the second convolutional layer after being processed by the excitation function Relu of the first Relu layer. C is the number of channels of the compressed spatial feature map; The second convolutional layer restores the number of channels of the processed first feature map to C , generate timing sensitivity graph; After the Sigmod layer processes the temporal sensitivity map, the scaling layer copies the processed temporal sensitivity map in the spatial dimension and performs a dot product operation with the spatial feature map to obtain a second output feature map of the local branch; The first fully connected layer in the global branch expands the compressed spatial feature map to a dimension of 2 T The second feature map is processed by the second Relu layer and input to the second fully connected layer. T is the time dimension of the compressed spatial feature map; The second fully connected layer converts the second feature map into a preset size C × K The third feature map is input into the first Softmax layer to obtain an adaptive convolution kernel.

3. The skeleton behavior recognition method based on temporal multi-scale adaptive feature aggregation according to claim 1 is characterized in that: The spatiotemporal graph convolutional network further includes a global average pooling layer, a third fully connected layer and a second Softmax layer sequentially connected to the multi-level spatiotemporal graph convolutional layer; The step of outputting the behavior recognition result of the skeleton sequence data according to the output feature map of the last level spatiotemporal graph convolution layer includes: Use the global average pooling layer to perform global average pooling on the output feature map of the last level of spatiotemporal graph convolution layer to obtain the output features; Calculating the confidence of each behavior category based on the output features using the third fully connected layer; The second Softmax layer is used to perform classification according to the confidence level to obtain an action recognition result of the skeleton sequence data.

4. A skeleton behavior recognition device based on temporal multi-scale adaptive feature aggregation, characterized in that: include: An acquisition module is used to obtain human skeleton sequence data from the video to be processed; An input module, configured to input the skeleton sequence data into a pre-trained spatiotemporal graph convolutional network; wherein the spatiotemporal graph convolutional network comprises a plurality of sequentially connected spatiotemporal graph convolutional layers, each of which comprises a spatial convolution module, a temporal convolution module, and a multi-scale adaptive feature aggregation module, wherein the output of the spatial convolution module is connected to the input of the temporal convolution module and the multi-scale adaptive feature aggregation module, respectively; A determination module is configured to determine the output feature map of each level of spatiotemporal graph convolution layer based on the temporal feature map output by the temporal convolution module and the first output feature map output by the multi-scale adaptive feature aggregation module, and use the output feature map as input data for the next level of spatiotemporal graph convolution layer; An output module, configured to output an action recognition result of the skeleton sequence data based on an output feature map of the last level of spatiotemporal graph convolutional layer; The multi-scale adaptive feature aggregation module includes a pooling layer, a local branch and a global branch; Each level of spatiotemporal graph convolutional layer processes the input data as follows: Use the spatial convolution module to extract the spatial features of the input data and obtain the spatial feature map; Inputting the spatial feature map into a temporal convolution module and a multi-scale adaptive feature aggregation module respectively, so that the temporal convolution module extracts the temporal features of the input data to obtain a temporal feature map, causing the pooling layer to spatially compress the spatial feature map, and using the local branch to generate a temporal sensitivity map based on the compressed spatial feature map, generating a second output feature map according to the temporal sensitivity map, and using the global branch to generate an adaptive convolution kernel based on the compressed spatial feature map; Convolving the second output feature map with the adaptive convolution kernel to obtain a first output feature map of a multi-scale adaptive feature aggregation module; Adding the first output feature map to the temporal feature map to obtain an output feature map of each level of the spatiotemporal graph convolution layer; The local branch includes a first convolutional layer, a first Relu layer, a second convolutional layer, a Sigmod layer and a scaling layer connected in sequence, and the global branch includes a first fully connected layer, a second Relu layer, a second fully connected layer and a first Softmax layer connected in sequence.

5. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 3 when executing a program stored in a memory.

Images