Video scene understanding method and system based on attention fusion

Abstract

The invention discloses a video scene understanding method and system based on attention fusion, and the method comprises the steps: adding a multi-dimensional attention fusion module into a backbone network to construct a recognition network under a pedestrian video scene, such as a pedestrian action recognition network, independently calculating respective attention in three dimensions of the space, and then performing fusion; obtaining pedestrian data of each frame of image in the video data set, and training an identification network in a pedestrian video scene by using the pedestrian data of each frame of image in the video data set; and finally, performing video understanding on a to-be-detected video by using the trained recognition network in the pedestrian video scene, such as outputting a pedestrian action type. According to the technical scheme, the attention information is independently used, so that all kinds of attention information are utilized most effectively, the attention information is integrated in the time dimension and the space dimension, more comprehensive information is obtained, time sequence information utilization is enhanced, and the video scene understanding ability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian image / video recognition, and in particular relates to a video scene understanding method and system based on attention fusion. Background Art

[0002] Video scene understanding is an important topic in the field of computer vision and artificial intelligence. It involves extracting and interpreting visual information from videos to achieve understanding and analysis of video content, including technologies and tasks at multiple levels such as target detection, target tracking, behavior recognition, and scene segmentation.

[0003] PoseC3D is a commonly used network for video scene understanding. PoseC3D uses an architecture based on a three-dimensional convolutional neural network (3DCNN) to process spatiotemporal information in video data, thereby understanding video scenes and analyzing human postures. PoseC3D is more efficient and accurate in processing dynamic human postures in videos, and is suitable for application scenarios that require long-term motion recognition and complex motion capture. It first obtains a sequence of marked boxes containing only the human body through a target detection network, extracts and normalizes the video sequence, and uses a three-dimensional convolutional layer to extract features from the video frames to generate the key point positions of the human body; then the human key points output by the network are smoothed and corrected to improve the accuracy and robustness of posture estimation. Finally, based on posture estimation, the changes in human posture are further analyzed to identify specific human movements or behavior patterns.

[0004] When processing key points, PoseC3D uses a 3D-CNN network as its backbone network. 3D-CNN is usually designed with the assumption that the time interval between video frames is fixed. However, this may not always be true in practical applications. In addition, 3D-CNN usually treats time as another spatial dimension and is not specially optimized to understand complex changes in time. Therefore, it is not enough to capture more complex temporal dynamics in nonlinear motion and rapidly changing scenes. At the same time, since action recognition requires full use of information in the time dimension, there is no information processing module specifically designed for the time dimension, which means that PoseC3D still has room for improvement.

[0005] Adding several specially designed temporal attention modules to the backbone network can enhance the model's ability to process time series information in the video while adding a few parameters, thereby optimizing the accuracy and efficiency of posture recognition. CBAM (Convolutional Block Attention Module) is an attention mechanism module for convolutional neural networks, which improves the network's representation ability by jointly using channel attention and spatial attention. The CBAM module first weights the different channels of the input feature map through the channel attention module to highlight the important feature channels, and then weights the different spatial positions of the feature map through the spatial attention module to emphasize the important spatial areas.

[0006] However, the CBAM module may not be able to fully capture the global temporal dependency when processing long time series data, because its design mainly focuses on the channel and spatial dimensions, lacking the modeling and utilization of temporal information. At the same time, the convolution in CBAM is extended to 3D 3D-CBAM. Due to the use of 3D convolution, the number of parameters increases significantly. At the same time, the information of the time dimension is mixed with the channel or spatial dimension and then processed in the same way, which also leads to inefficient utilization of temporal information.

[0007] The above-mentioned technical obstacles have resulted in the need for further improvement in video scene understanding and pedestrian behavior technology. Summary of the invention

[0008] The purpose of the present invention is to overcome the problem that the processing capability of the time series information of the video in the prior art needs to be improved, enhance the utilization of time series information, and then improve the video scene understanding capability, and then provide a video scene understanding method and system based on attention fusion. Among them, the technical solution of the present invention proposes a new multi-dimensional attention fusion module, and integrates it into the backbone network to construct a recognition network under the pedestrian video scene. The multi-dimensional attention fusion module independently calculates the attention of the three dimensions of channel, time, and space, and then fuses the channel attention, time attention and space attention information. It not only uses the attention information separately to make the most effective use of various types of attention information, but also integrates the attention information in the two dimensions of time and space to obtain more comprehensive information. The above network is particularly outstanding in the application of pedestrian action recognition.

[0009] To this end, the present invention provides the following technical solutions:

[0010] On the one hand, the present invention provides a video scene understanding method based on attention fusion, comprising the following steps:

[0011] A multi-dimensional attention fusion module is added to the backbone network to construct a recognition network in the pedestrian video scene. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information;

[0012] Obtain pedestrian data for each frame of the video data set, and then use the pedestrian data for each frame of the video data set to train a recognition network in the pedestrian video scene to obtain video understanding content;

[0013] Using the trained recognition network in the pedestrian video scene to perform video understanding on the video to be detected;

[0014] Among them, the input and output data of the recognition network in the pedestrian video scene are set according to the goal of video understanding, the input data corresponds to the pedestrian data, and the output data is the content of video understanding.

[0015] Further optionally, the video is understood as pedestrian action recognition, and the recognition network corresponding to the pedestrian video scene is a pedestrian action recognition network;

[0016] The pedestrian data is a 3D volume heat map based on the integration of pedestrian key point posture data. The pedestrian key points represent the joint positions of the human body. The pedestrian key point posture data includes the joint positions and the limbs represented by the joint lines. The output data of the recognition network in the pedestrian video scene is the pedestrian action type. The specific content of the pedestrian action category is manually set and set according to the recognition requirements, such as walking, running, etc.

[0017] The technical solution of the present invention processes the information of the three dimensions of channel, time and space independently, and calculates the attention information separately. The attention information is then used uniformly through feature fusion. Specifically, the input first passes through the channel attention module, and the output after attention calculation passes through the spatial attention module and the temporal attention module at the same time. The spatial attention module and the temporal attention module each perform attention calculation, and finally the attention information of the three modules is fused and calculated in the feature fusion module to obtain the final output of the module.

[0018] Further optionally, the multi-dimensional attention fusion module includes a channel preprocessing module, a spatial shrink sampling module, a time expansion focusing module and a multi-dimensional feature calculation module, wherein the channel preprocessing module, the spatial shrink sampling module and the time expansion focusing module are used to obtain channel attention, spatial attention and time attention respectively, and the output X obtained by the channel preprocessing module c_out As inputs of the spatial contraction sampling module and the temporal expansion focusing module respectively;

[0019] Among them, the multi-dimensional attention fusion module is used to fuse channel attention, time attention and spatial attention information, and the processing process is:

[0020] The output X of the spatial shrink sampling module sa With the time expansion focusing module output X ta Multiply the output X c_out Get the activation output X of spatial attention information s_out , the activation output of temporal attention X t_out ; and output X sa With output X ta Multiply to get the spatiotemporal attention information X sta , then multiply the output X c_out Get the output X st_out ; Finally, X s_out , X t_out , X st_out The output of the multidimensional feature calculation module is weighted according to a preset weight ratio, that is, the output of the multidimensional attention fusion module.

[0021] The technical solution of the present invention uses a multi-dimensional attention fusion module to integrate channel attention with time attention and space attention. At the same time, it also integrates time attention with space attention to obtain spatiotemporal attention, and finally integrates X s_out , X t_out , X st_out By weighting according to a preset weight ratio, the technical solution of the present invention not only uses the attention information separately to make the most effective use of the attention information, but also integrates the attention information in the two dimensions of time and space to obtain more comprehensive information.

[0022] Further optionally, the processing process of the time expansion focusing module is as follows:

[0023] Enter X c_out The spatial information is expanded frame by frame to the time channel to obtain X t Among them, a submodule in the time expansion focusing module, namely the time information fusion expansion module, inputs X c_our The spatial information is expanded frame by frame to the time channel to obtain X t The specific implementation process is: "Divide each frame, that is, the information of the spatial dimension, into several 2x2 small blocks, and splice the pixels in the same position of the small blocks together to obtain four spliced ​​sub-frames, and then splice the four spliced ​​sub-frames in sequence to complete the information fusion in a single frame. Then splice the results of the information fusion of each single frame in the order of the time dimension;

[0024] Then, 3D average pooling and 3D maximum pooling are used to obtain X t_avg , X t_max ;

[0025] Next, X is passed through a shared-weight MLP layer. t_avg , X t_max Add each element one by one, and then use the sigmoid activation function to get the output X ta , i.e., temporal attention information; wherein the MLP layer consists of two 1D convolutions and a ReLU function.

[0026] The time expansion focusing module provided by the technical solution of the present invention concentrates the spatial information into the time dimension to avoid the loss of effective information. In addition, the MLP with shared weights in the CBAM module is improved from 2D convolution to 1D convolution, which reduces the amount of calculation while isolating information in other dimensions, so that the attention information is more concentrated in the time dimension.

[0027] Further optionally, the processing process of the spatial contraction sampling module is as follows:

[0028] Enter X c_out Get X by average pooling and maximum pooling s_avg , X s_max ;

[0029] Then X s_avg , X s_max Splice to get X s ;

[0030] Finally, through the dilated convolution And sigmoid activation function to get the output X sa , that is, spatial attention information.

[0031] Further optionally, after the input of the multi-dimensional attention fusion module enters the channel preprocessing module, the processing process is as follows:

[0032] First, 3D average pooling and 3D maximum pooling are used to obtain X c_avg , X c_max ∈R B×C×1×1×1 ;

[0033] Then, X is transformed into c_avg , X c_max Add element by element, and then use the sigmoid activation function to get the channel attention information X ca ;

[0034] Finally, the channel attention information X ca Multiply input X to get output X c_out .

[0035] The channel preprocessing module provided by the technical solution of the present invention improves the MLP of shared weights in the CBAM module from 2D convolution to 1D convolution, which reduces the amount of calculation while isolating information in other dimensions, so that the attention information is more concentrated in the channel dimension.

[0036] In addition, in some implementations, a video scene understanding method based on attention fusion includes the following steps:

[0037] Step 1: Establish a video dataset about pedestrian behavior and a pedestrian image dataset;

[0038] Step 2: Using the pedestrian image dataset to train a pedestrian key point detection network to obtain a pedestrian detection model, wherein the input of the pedestrian detection model is a pedestrian image and the output is a pedestrian boundary box;

[0039] Step 3: Use the pedestrian detection model to perform frame-by-frame recognition on the video data set to obtain pedestrian recognition results for each frame of the image, and identify the posture of pedestrian key points based on the pedestrian recognition results. The pedestrian key points represent the positions of human joints, including joint positions and limbs represented by joint lines;

[0040] That is, pedestrian detection is performed through the constructed pedestrian detection model;

[0041] Step 4: Add the multi-dimensional attention fusion module to the backbone network to construct a pedestrian action recognition network, and then use the 3D volume heat map integrated with the pedestrian key point posture data of each frame of the video data to train the pedestrian action recognition network;

[0042] The input of the pedestrian action recognition network is a 3D volume heat map based on the integration of pedestrian key point posture data; the output is the pedestrian action category, which is the content of video understanding;

[0043] Step 5: Input the video to be detected into the pedestrian detection model and perform pedestrian key point recognition, and then input the recognition result of the pedestrian key point into the pedestrian action recognition network to obtain the action category.

[0044] Further optionally, the backbone network of the pedestrian action recognition network is the SlowPath of Slow Fast-RCNN, and the multi-dimensional attention fusion module is added to each res-laye residual layer of ResNet-3D.

[0045] In another aspect, the present invention provides a system based on the video scene understanding method, comprising at least:

[0046] A recognition network construction module in a pedestrian video scene is used to add a multi-dimensional attention fusion module to the backbone network to construct a recognition network in a pedestrian video scene. The multi-dimensional attention fusion module independently calculates the attention of the three dimensions of channel, time, and space, and then fuses the channel attention, time attention, and space attention information;

[0047] A pedestrian data acquisition module is used to acquire pedestrian data of each frame of image in the video data set;

[0048] A training module, used to train a recognition network in the pedestrian video scene using pedestrian data in each frame of the video data set to obtain video understanding content;

[0049] A detection module, used to perform video understanding on the video to be detected using the trained recognition network in the pedestrian video scene;

[0050] Among them, the input and output data of the recognition network in the pedestrian video scene are set according to the goal of video understanding, the input data corresponds to the pedestrian data, and the output data is the content of video understanding.

[0051] In a third aspect, the present invention provides an electronic terminal, comprising:

[0052] one or more processors;

[0053] a memory storing one or more computer programs;

[0054] Wherein, the processor calls the computer program to implement: the video scene understanding method.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is called by a processor to implement: the video scene understanding method.

[0056] Beneficial Effects

[0057] Compared with the existing method, the advantages of the present invention are:

[0058] A video scene understanding method based on attention fusion provided by the present invention proposes a new multi-dimensional attention fusion module, and adds it to the backbone network to construct a new pedestrian action recognition network. The multi-dimensional attention fusion module independently processes the information of three dimensions: channel, time, and space. That is, three different dimensions use their own modules to calculate attention separately. The attention information of each dimension has different emphases, and separate processing can make the attention information more effective. At the same time, the time and space attention information are finally combined into spatiotemporal attention information to avoid the single-dimensional attention information from accounting for too large a proportion in the output, thereby improving the processing capability of the time series information of the video, making full and effective use of the time series information, and ultimately improving the video scene understanding capability.

[0059] Compared with the existing CBAM, the MDAF proposed in the present invention expands the data from 2D to 3D, and can realize the processing of video data. At the same time, compared with CBAM, the present invention adds a time expansion focusing module, which is specially used to process timing information. CBAM is used to process 2D information and does not have the ability to utilize timing information.

[0060] Moreover, the technical solution of the present invention not only performs new processing in three dimensions, but also adopts different structures to process the information of each dimension according to the characteristics of the information of each dimension. At the same time, there is a multi-dimensional feature calculation module, which balances the effective information between each dimension through self-learning weights, thereby ensuring the reliability of the recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the network detection process provided by the present invention;

[0062] Figure 2 is a schematic diagram of the architecture of the MDAF module provided by the present invention;

[0063] Figure 3 It is a processing flow diagram of the channel preprocessing module;

[0064] Figure 4 is a schematic diagram of the processing flow of the time expansion focusing module;

[0065] Figure 5 It is a schematic diagram of the processing flow of the spatial shrink sampling module. DETAILED DESCRIPTION

[0066] The present invention will be further described below with reference to the embodiments.

[0067] The technical solution of the present invention provides a video scene understanding method based on attention fusion, the core improvement of which is to propose a new multi-dimensional attention fusion module and integrate it into the backbone network. To this end, the technical ideas of the technical solution of the present invention are as follows:

[0068] The multi-dimensional attention fusion module is added to the backbone network to construct a recognition network for pedestrian video scenes. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information;

[0069] Obtain pedestrian data for each frame of the video data set, and then use the pedestrian data for each frame of the video data set to train a recognition network in a pedestrian video scene to obtain video understanding content;

[0070] Use the trained recognition network in pedestrian video scenes to understand the video to be detected;

[0071] Among them, the input and output data of the recognition network in the pedestrian video scene are set according to the goal of video understanding. The input data corresponds to the pedestrian data, and the output data is the content of video understanding. It should be understood that video understanding is often divided into pedestrian action recognition, target recognition, target tracking, scene segmentation, etc. Therefore, the selection of input and output is closely related to the goal of video understanding and corresponds one to one, and can be set according to the needs.

[0072] Since the technical solution of the present invention has outstanding performance in the application of pedestrian action recognition, the following will be explained using this application as an example, that is, the video is understood as pedestrian action recognition, and the recognition network corresponding to the pedestrian video scene is the pedestrian action recognition network; the pedestrian data is a 3D volume heat map integrated based on the pedestrian key point posture data, the pedestrian key point represents the human body joint position, the pedestrian key point posture data includes the joint position and the limbs represented by the joint connection line; the output data of the recognition network in the pedestrian video scene is the pedestrian action type.

[0073] The details are as follows:

[0074] S1: Add the multi-dimensional attention fusion module to the backbone network to build a pedestrian action recognition network. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information. Specifically, the multi-dimensional attention fusion module includes a channel preprocessing module, a spatial shrink sampling module, a temporal expansion focusing module, and a multi-dimensional feature calculation module. The output X obtained by the channel preprocessing module c_out They are used as inputs of the spatial shrink sampling module and the temporal expansion focusing module respectively; the multi-dimensional attention fusion module is used to fuse channel attention, temporal attention and spatial attention information. The processing process is as follows:

[0075] The output X of the spatial shrink sampling module sa With the time expansion focusing module output X ta Multiply the output Xc_out Get the activation output X of spatial attention information s_out , the activation output of temporal attention X t_out ; and output X sa With output X ta Multiply to get the spatiotemporal attention information X sta , then multiply the output X c_out Get the output X st_out ; Finally, X s_out , X t_out , X st_out The preset weight ratio is weighted as the output of the multidimensional feature calculation module, that is, the output of the multidimensional attention fusion module. As can be seen from the above, the present invention realizes multidimensional feature fusion by using the multidimensional attention fusion module.

[0076] S2: Obtain pedestrian key point posture data for each frame of the video data set related to pedestrian behavior, wherein the pedestrian key point represents the position of the human body joint. The purpose of this step is to obtain pedestrian key point posture data. The following embodiment 1 is a preferred feasible method of the present invention. Other feasible embodiments do not restrict the method of obtaining pedestrian key point posture data.

[0077] S3: Train the pedestrian action recognition network using the 3D volume heat map integrated with the pedestrian key point posture data of each frame image in the video dataset, wherein the input of the pedestrian action recognition network is the 3D volume heat map integrated based on the pedestrian key point posture data, and the output is the recognized pedestrian action type.

[0078] S4: Using the trained pedestrian action recognition network to understand the video to be detected. It should be understood that according to the method of S2, the pedestrian key point posture data of the video to be detected is obtained, and then integrated into a 3D volume heat map and then input into the trained pedestrian action recognition network to identify the pedestrian action type.

[0079] The following embodiment 1 is a preferred embodiment of the present invention, and is specifically as follows:

[0080] Example 1

[0081] The embodiment of the present invention provides a video scene understanding method based on attention fusion, comprising the following steps:

[0082] Step 1: Establish a video dataset about pedestrian behavior and establish a pedestrian image dataset. The video dataset of this embodiment is used to train the subsequent pedestrian action recognition network, and the pedestrian image dataset is used to train the pedestrian detection model.

[0083] In this embodiment, some images of the pedestrian image dataset are preferably obtained by sampling the video dataset, that is, intercepting a few frames; some images are obtained from other collected pedestrian images, and the pedestrian image dataset is jointly established. The pedestrian image dataset established by this method can be expanded, and pedestrian images can be more fully and comprehensively collected.

[0084] Step 2: Use the pedestrian image dataset to train a pedestrian key point detection network to obtain a pedestrian detection model, wherein the input of the pedestrian detection model is a pedestrian image and the output is a pedestrian boundary box.

[0085] In this embodiment, the Faster-RCNN network is selected as the pedestrian key point detection network. Since the Faster-RCNN network is an existing network architecture, the present invention does not elaborate on it in detail; in other feasible embodiments, the network for implementing pedestrian detection can also be applicable to the requirements of the technical solution of the present invention.

[0086] It should be understood that the Faster-RCNN network is an existing technology. On the premise of clarifying its input and output, those skilled in the art can understand that the network is trained using the pedestrian image dataset as a sample set. Therefore, its training process and network architecture are not described in detail.

[0087] Step 3: Use the pedestrian detection model to perform frame-by-frame recognition on the video dataset to obtain the pedestrian recognition results for each frame, and identify the pedestrian key point posture based on the pedestrian recognition results. Pedestrian key points represent the position of human joints, usually including the main joints of the human body, such as the head, shoulders, elbows, wrists, hips, knees and ankles. The connection between key points represents the structure of the human skeleton, such as the connection from the shoulder to the elbow and then to the wrist.

[0088] It should be understood that after obtaining the bounding box of the pedestrian, the posture of the pedestrian's key points is estimated within the bounding box according to the coordinates of the bounding box, that is, the pedestrian's joints and limbs are extracted. Among them, identifying the posture of the pedestrian's key points is achievable by existing technology, so it is not described in detail.

[0089] Step 4: Add the multi-dimensional attention fusion module to the backbone network to build a pedestrian action recognition network, and then use the 3D volume heat map integrated with the pedestrian key point posture data of each frame of the video data to train the pedestrian action recognition network. Among them, the input data of the pedestrian action recognition network is the 3D volume heat map integrated based on the pedestrian key point posture data, and the output is the recognized pedestrian action category. Scene understanding includes behavior recognition, and the detection result of behavior recognition is the pedestrian action category.

[0090] In this embodiment, the Slow Path network of Slow Fast-RCNN is preferably used as the backbone network, and a multi-dimensional attention fusion module is used to enhance the utilization of temporal feature information. In other feasible embodiments, the Slow Fast-RCNN backbone network is not the only option. For example, using ResNet-3D as the backbone network and adding the multi-dimensional attention fusion module MDAF to the residual layer of ResNet-3D also falls within the protection scope of the present invention.

[0091] like Figure 1 As shown, this embodiment adds a multi-dimensional attention fusion module to each res-layer residual layer of ReSNet-3D of Slow Fast-RCNN. Each res-layer in ResNet 3D has a different size. Adding modules to each residual layer can process and model information of different sizes, paying attention to information of different sizes in the data. It should be noted that, in theory, the multi-dimensional attention fusion module MDAF can be added to any position of the backbone network, because the input and output sizes of the MDAF module are the same.

[0092] like Figure 2 As shown in Figure 2, the implementation process of the channel preprocessing module, spatial shrink sampling module, temporal expansion focusing module, and multi-dimensional feature calculation module in the MDAF module is as follows: Channel preprocessing module: Assume that the input is X ∈ R B×C×TxH×W , B is the batch size, which indicates the number of data samples input at a time; C is the number of channels, which indicates the number of channels of the input data; T is time, which indicates the length of the time dimension or sequence; H is the height, which indicates the height of the input data in the spatial dimension; W is the width, which indicates the width of the input data in the spatial dimension. Figure 3 As shown, the channel preprocessing module obtains X through 3D average pooling and 3D maximum pooling for the input X. c_avg , X c_max ∈R B×C×1×1×1 , then the two output results are added element by element through a shared weight MLP layer, and then the channel attention information X is obtained through the sigmoid activation function ca ∈R B×C×1×1×1 , X ca ∈R B×C×1×1×1 Multiply the input X to get the output X of the channel preprocessing module c_out ∈R B×C×T×H×W ,The MLP layer of the channel preprocessing module consists of two 1D convolutions and a ReLU function.

[0093] Among them, the technical solution of the present invention improves the shared weight MLP in the CBAM module from 2D convolution to 1D convolution, which reduces the amount of calculation while isolating information in other dimensions, so that the attention information is more concentrated in the channel dimension.

[0094] Spatial shrink sampling module: such as Figure 5 As shown, the output X of the channel preprocessing module c_out As input into the spatial shrink sampling module, we first obtain X by average pooling and maximum pooling. s_avg , X s_max ∈R B×1×T×H×W , and then concatenate the two to get X s ∈R B×2×T×H×W , followed by dilated convolution And sigmoid activation function to get spatial attention information X sa ∈R B ×1×1×H×W .

[0095] From X s To X sa Involving X s ∈R B×2×T×H×W T in becomes 1. In order to process the information of each dimension separately, if ordinary 3D convolution is used here, the information of the time dimension will be used while calculating the attention information of the spatial dimension. Therefore, it is preferred to use 3D convolution that is only expanded in the time dimension for calculation, reduce the utilization of the spatial dimension attention information on the time dimension information, shrink the time dimension information, and also calculate the attention information of the spatial dimension, thereby expanding the receptive field and reducing the loss of information.

[0096] Time extension focus module: such as Figure 4 As shown, the output X of the channel preprocessing module c_out As input, it enters the time expansion focusing module. First, the input X is converted into c_out ∈R B×C×T×H×W The spatial information in is expanded frame by frame to the time channel to obtain X t ∈R B×C×4T×H / 2×W / 2 , and then obtain X through 3D average pooling and 3D maximum pooling respectively. t_avg , X t_max ∈R B×1×T×1×1 ; Then, the two output results are added element by element through a shared weight MLP layer, and then X is obtained through the sigmoid activation function. ta ∈R B×1×T×1×1 The temporal attention information of the temporal expansion focusing module is composed of two 1D convolutions and a ReLU function.

[0097] Among them, the time expansion focusing module divides each frame, that is, the information of the spatial dimension, into several 2x2 small blocks, and splices the pixels in the same position of the small blocks together to obtain four spliced ​​subframes, and then splices the four spliced ​​subframes in sequence to complete the information fusion in a single frame; then splices the results of the information fusion of each single frame in the order of the time dimension to obtain the output of the time information fusion expansion module, avoiding the loss of effective information. It should be understood that the shared weight MLP in the CBAM module is improved from 2D convolution to 1D convolution, which reduces the amount of calculation while isolating information in other dimensions, so that the attention information is more concentrated in the time dimension.

[0098] Multi-dimensional feature calculation module: Multiple inputs enter the multi-dimensional feature calculation module, and the spatial attention information X sa Multiply by X c_out Get the activation output X of spatial attention information s_out ; The time attention information X ta Multiply by X c_out Get the activation output X of the temporal attention t_out ; Multiply the spatial attention and temporal attention to get the spatiotemporal attention information X sta ∈R B×1×T×H×W , the spatiotemporal attention information X sta Multiply by X c_out Get the output X st_out . X s_out , X t_out , X st_out The three are added in different weight ratios to get the output X of the MDAF module. out The weight ratio is obtained by training the network according to different data and application scenarios, and is an empirical value. The present invention does not restrict its specific value.

[0099] In summary, the MDAF module provided by the technical solution of the present invention will pass the output after attention calculation through the spatial attention module and the temporal attention module at the same time. The spatial attention module and the temporal attention module will perform attention calculation respectively. Finally, the attention information of the three modules will be fused and calculated in the feature fusion module to obtain the final output of the module. This process not only uses the attention information separately, making the most effective use of the attention information, but also integrates the attention information in the two dimensions of time and space to obtain more comprehensive information.

[0100] Step 5: Input the video to be detected into the pedestrian detection model and perform pedestrian key point recognition, and then input the recognition results of the pedestrian key points into the trained pedestrian action recognition network to obtain the action category. That is, in this embodiment, the video to be detected is input into the network, and the posture recognition of the key points is first performed through the Faster-RCNN network and the posture estimation of the key points, and then the key points are recognized through the Slow Fast-RCNN network with the spatiotemporal attention module to obtain the detection result.

[0101] In order to fully compare the technical solution provided by this embodiment with the prior art, the experimental results of Table 1 (comparison of the accuracy index of MDAF in three data sets with other existing methods) are obtained as follows:

[0102] Table 1

[0103]

[0104] The MDAF module proposed in this paper shows excellent performance in three datasets over the Posec3d baseline.

[0105] Example 2

[0106] This embodiment provides a system based on a video scene understanding method, which at least includes: a recognition network construction module in a pedestrian video scene, a pedestrian data acquisition module, a training module, and a detection module.

[0107] The recognition network construction module in the pedestrian video scene is used to add the multi-dimensional attention fusion module to the backbone network to build a pedestrian action recognition network. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information;

[0108] The pedestrian data acquisition module is used to obtain the pedestrian key point posture data of each frame image in the video data set about pedestrian behavior and integrate it into a 3D volume heat map. The pedestrian key point represents the position of the human body joint;

[0109] The training module is used to train the pedestrian action recognition network using the 3D volume heat map integrated with the pedestrian key point posture data of each frame image in the video dataset;

[0110] The detection module is used to use the trained pedestrian action recognition network to perform video understanding on the video to be detected; the input of the pedestrian action recognition network is a 3D volume heat map integrated based on the pedestrian key point posture data; the output is the pedestrian action category, which is the content of the video understanding.

[0111] In some embodiments, the pedestrian data acquisition module can be divided into a data set construction module, a pedestrian detection model construction module, and a key point posture recognition module, wherein the data set construction module is used to establish a video data set about pedestrian behavior and a pedestrian image data set; the pedestrian detection model construction module is used to use the pedestrian image data set to train a pedestrian key point detection network to obtain a pedestrian detection model, wherein the input of the pedestrian detection model is a pedestrian image and the output is a pedestrian's bounding box; the key point posture recognition module is used to use the pedestrian detection model to perform frame-by-frame recognition on the video data set to obtain a pedestrian recognition result for each frame of the image, and to identify the posture of the pedestrian key points based on the pedestrian recognition result.

[0112] It should be understood that the specific implementation process of each module please refer to the above method content, the present invention will not be repeated here, and the division of the above functional modules is only for example illustration. In some embodiments, some functional modules can be merged, and some functional modules can be split. Each functional module can be implemented in software or hardware or a combination of software and hardware. Among them, the hardware and software equipment includes but is not limited to general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0113] Embodiment 3:

[0114] An embodiment of the present invention provides an electronic terminal, comprising: one or more processors, and a memory storing one or more computer programs;

[0115] Among them, the processor calls the computer program to implement: a video scene understanding method.

[0116] For example, perform / implement the following steps:

[0117] The multi-dimensional attention fusion module is added to the backbone network to construct a recognition network for pedestrian video scenes. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information;

[0118] Obtain pedestrian data for each frame of the video data set, and then use the pedestrian data for each frame of the video data set to train a recognition network in a pedestrian video scene to obtain video understanding content;

[0119] Use the trained recognition network in pedestrian video scenes to understand the video to be detected;

[0120] Among them, the input and output data of the recognition network in the pedestrian video scene are set according to the goal of video understanding, the input data corresponds to the pedestrian data, and the output data is the content of video understanding.

[0121] Taking pedestrian action recognition as an example, execute:

[0122] The multi-dimensional attention fusion module is added to the backbone network to construct a pedestrian action recognition network. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information;

[0123] Obtain the pedestrian key point posture data of each frame image in the video dataset about pedestrian behavior and integrate it into a 3D volume heat map. The pedestrian key point represents the position of the human body joint;

[0124] The pedestrian action recognition network is trained using the 3D volume heat map integrated with the pedestrian key point posture data of each frame image in the video dataset;

[0125] Use the trained pedestrian action recognition network to perform video understanding on the video to be detected.

[0126] Or perform / implement the following steps:

[0127] Step 1: Establish a video dataset about pedestrian behavior and a pedestrian image dataset;

[0128] Step 2: Use the pedestrian image dataset to train the pedestrian key point detection network to obtain a pedestrian detection model, where the input of the pedestrian detection model is the pedestrian image and the output is the pedestrian's bounding box;

[0129] Step 3: Use the pedestrian detection model to perform frame-by-frame recognition on the video dataset to obtain the pedestrian recognition results of each frame image, and identify the posture of the pedestrian key points based on the pedestrian recognition results. The pedestrian key points represent the positions of the human body joints.

[0130] Step 4: Add the multi-dimensional attention fusion module to the backbone network to build a pedestrian action recognition network, and then use the 3D volume heat map integrated with the pedestrian key point posture data of each frame of the video data to train the pedestrian action recognition network;

[0131] The input of the pedestrian action recognition network is a 3D volume heat map based on the integration of pedestrian key point posture data; the output is the pedestrian action category, which is the content of video understanding;

[0132] Step 5: Input the video to be detected into the pedestrian detection model and perform pedestrian key point recognition, and then input the recognition results of the pedestrian key points into the pedestrian action recognition network to obtain the action category.

[0133] For the specific implementation process of each step, please refer to the description of the above method.

[0134] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.

[0135] Embodiment 4:

[0136] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which is called by a processor to implement a video scene understanding method.

[0137] For example, perform / implement the following steps:

[0138] The multi-dimensional attention fusion module is added to the backbone network to construct a recognition network for pedestrian video scenes. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information;

[0139] Obtain pedestrian data for each frame of the video data set, and then use the pedestrian data for each frame of the video data set to train a recognition network in a pedestrian video scene to obtain video understanding content;

[0140] Use the trained recognition network in pedestrian video scenes to understand the video to be detected;

[0141] Among them, the input and output data of the recognition network in the pedestrian video scene are set according to the goal of video understanding, the input data corresponds to the pedestrian data, and the output data is the content of video understanding.

[0142] Taking pedestrian action recognition as an example, execute:

[0143] The multi-dimensional attention fusion module is added to the backbone network to construct a pedestrian action recognition network. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information;

[0144] Obtain the pedestrian key point posture data of each frame image in the video dataset about pedestrian behavior and integrate it into a 3D volume heat map. The pedestrian key point represents the position of the human body joint;

[0145] The pedestrian action recognition network is trained using the 3D volume heat map integrated with the pedestrian key point posture data of each frame image in the video dataset;

[0146] Use the trained pedestrian action recognition network to perform video understanding on the video to be detected.

[0147] Or perform / implement the following steps:

[0148] Step 1: Establish a video dataset about pedestrian behavior and a pedestrian image dataset;

[0149] Step 2: Use the pedestrian image dataset to train the pedestrian key point detection network to obtain a pedestrian detection model, where the input of the pedestrian detection model is the pedestrian image and the output is the pedestrian's bounding box;

[0150] Step 3: Use the pedestrian detection model to perform frame-by-frame recognition on the video dataset to obtain the pedestrian recognition results of each frame image, and identify the posture of the pedestrian key points based on the pedestrian recognition results. The pedestrian key points represent the positions of the human body joints.

[0151] Step 4: Add the multi-dimensional attention fusion module to the backbone network to build a pedestrian action recognition network, and then use the 3D volume heat map integrated with the pedestrian key point posture data of each frame of the video data to train the pedestrian action recognition network;

[0152] The input of the pedestrian action recognition network is a 3D volume heat map based on the integration of pedestrian key point posture data; the output is the pedestrian action category, which is the content of video understanding;

[0153] Step 5: Input the video to be detected into the pedestrian detection model and perform pedestrian key point recognition, and then input the recognition results of the pedestrian key points into the pedestrian action recognition network to obtain the action category.

[0154] For the specific implementation process of each step, please refer to the description of the above method.

[0155] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the software and hardware device described in any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0156] Based on this understanding, the technical solution of the present invention, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.

[0157] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is a device for realizing the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram with reference to the instructions executed by the processor according to the method, device (system) and computer program product of the embodiment of the present application. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which realizes the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0158] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A video scene understanding method based on attention fusion, characterized by: The following steps are involved: A multi-dimensional attention fusion module is added to the backbone network to construct a recognition network in the pedestrian video scene. The multi-dimensional attention fusion module calculates the attention of the three dimensions of channel, time, and space independently, and then fuses the channel attention, time attention, and space attention information; Obtain pedestrian data for each frame of the video data set, and then use the pedestrian data for each frame of the video data set to train a recognition network in the pedestrian video scene to obtain video understanding content; Using the trained recognition network in the pedestrian video scene to perform video understanding on the video to be detected; Among them, the input and output data of the recognition network in the pedestrian video scene are set according to the goal of video understanding, the input data corresponds to the pedestrian data, and the output data is the content of video understanding.

2. The method according to claim 1, characterized in that: The video is understood as pedestrian action recognition, and the recognition network corresponding to the pedestrian video scene is a pedestrian action recognition network; Among them, the pedestrian data is a 3D volume heat map integrated based on the pedestrian key point posture data, the pedestrian key points represent the human body joint positions, and the pedestrian key point posture data includes the joint positions and the limbs represented by the joint lines; the output data of the recognition network in the pedestrian video scene is the pedestrian action type.

3. The method according to claim 1, characterized in that: The multi-dimensional attention fusion module includes a channel preprocessing module, a spatial shrinkage sampling module, a time expansion focusing module and a multi-dimensional feature calculation module. The channel preprocessing module, the spatial shrinkage sampling module and the time expansion focusing module are used to obtain channel attention, spatial attention and time attention respectively. The output X obtained by the channel preprocessing module c_out As inputs of the spatial contraction sampling module and the temporal expansion focusing module respectively; Among them, the multi-dimensional attention fusion module is used to fuse channel attention, time attention and spatial attention information, and the processing process is: The output X of the spatial shrink sampling module sa With the time expansion focusing module output X ta Multiply the output X c_out Get the activation output X of spatial attention information s_out , the activation output of temporal attention X t_out ; and output X sa With output X ta Multiply to get the spatiotemporal attention information X sta , then multiply the output X c_out Get the output X st_out ; Finally, X s_out , X t_out , X st_out The output of the multidimensional feature calculation module is weighted according to a preset weight ratio, that is, the output of the multidimensional attention fusion module.

4. The method according to claim 3, characterized in that: The processing process of the time expansion focusing module is as follows: Enter X c_out The spatial information is expanded frame by frame to the time channel to obtain X t ; Then, 3D average pooling and 3D maximum pooling are used to obtain X t_avg , X t_max ; Next, X is passed through a shared-weight MLP layer. t_avg , X t_max Add each element one by one, and then use the sigmoid activation function to get the output X ta , i.e., temporal attention information; wherein the MLP layer consists of two 1D convolutions and a ReLU function.

5. The method according to claim 3, characterized in that: The processing process of the spatial shrink sampling module is as follows: Enter X c_out Get X by average pooling and maximum pooling s_avg , X s_max ; Then X s_avg , X s_max Splice to get X s ; Finally, through the dilated convolution And sigmoid activation function to get the output X sa , that is, spatial attention information.

6. The method according to claim 3, characterized in that: After the input of the multi-dimensional attention fusion module enters the channel preprocessing module, the processing process is as follows: First, 3D average pooling and 3D maximum pooling are used to obtain X c_avg , X c_max ; Then, X is transformed into c_avg , X c_max Add element by element, and then use the sigmoid activation function to get the channel attention information X ca ; Finally, the channel attention information X ca Multiply input X to get output X c_out .

7. The method according to claim 2, characterized in that: The backbone network of the pedestrian action recognition network is the Slow Path of SlowFast-RCNN, and the multi-dimensional attention fusion module is added to each res-laye residual layer of ResNet-3D.

8. A system based on the video scene understanding method according to any one of claims 1 to 7, characterized in that: At least: A recognition network construction module in a pedestrian video scene is used to add a multi-dimensional attention fusion module to the backbone network to construct a recognition network in a pedestrian video scene. The multi-dimensional attention fusion module independently calculates the attention of the three dimensions of channel, time, and space, and then fuses the channel attention, time attention, and space attention information; A pedestrian data acquisition module is used to acquire pedestrian data of each frame of image in the video data set; A training module, used to train a recognition network in the pedestrian video scene using pedestrian data in each frame of the video data set to obtain video understanding content; A detection module, used to perform video understanding on the video to be detected using the trained recognition network in the pedestrian video scene; Among them, the input and output data of the recognition network in the pedestrian video scene are set according to the goal of video understanding, the input data corresponds to the pedestrian data, and the output data is the content of video understanding.

9. An electronic terminal, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to implement: The video scene understanding method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The video scene understanding method according to any one of claims 1 to 7.

Images