A video behavior recognition method and device based on multi-time scale convolution

Abstract

The present invention provides a method and device for video behavior recognition based on multi-time-scale convolution, the method comprising: constructing at least one multi-time-scale convolution module; embedding the at least one multi-time-scale convolution module into a preset skeleton network to form a target feature extraction model; extracting behavior features in the video through the target feature extraction model; constructing a behavior recognition model, and recognizing the behavior features through the behavior recognition model. The present invention extracts behavior features in the video through the target feature extraction model. Since the convolution kernel parameters in the multi-time-scale convolution module can be learned and adjusted, the spatiotemporal features between different frames of the video can be effectively extracted, thereby improving the accuracy of behavior feature extraction. In addition, there will be no information loss problem, further improving the accuracy of behavior feature extraction, thereby improving the recognition accuracy of video behavior recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of video behavior recognition, and in particular to a video behavior recognition method and device based on multi-time-scale convolution. Background Art

[0002] Behavior recognition based on video information is a hot topic in the field of computer vision. It primarily utilizes image processing, image analysis, and computer vision technologies to detect, classify, and track objects in video information, and to understand and describe behaviors within the video information. Based on different skeleton networks, behavior recognition networks are generally divided into 2D behavior recognition networks and 3D behavior recognition networks. Behavior recognition networks based on 2D convolutional neural networks typically fuse high-dimensional features extracted by the skeleton network to identify behaviors, but lack the extraction of spatiotemporal features during the convolution process. Behavior recognition networks based on 3D convolutional neural networks can extract spatiotemporal features during the convolution process, but their parameter and computational overhead limit their practical application deployment.

[0003] To address these issues, the Temporal Shift Module (TSM) has been proposed. The TSM model is a network structure that maintains the complexity of a 2D convolutional neural network while achieving the effects of a 3D convolutional neural network. Specifically, the TSM model uses the same number of parameters and computational complexity as a 2D convolutional neural network, but can extract spatiotemporal features during the convolution process.

[0004] There are the following shortcomings in using the TSM model to identify video behaviors: 1. The convolution kernel parameters in the TSM model are fixed, which makes it difficult to effectively extract the spatiotemporal features between different frames, resulting in low recognition accuracy; 2. The TSM model extracts spatiotemporal features by moving some feature channels. After moving some feature channels, the features of the first and last frames will have information missing, further resulting in low recognition accuracy. Summary of the Invention

[0005] In view of this, it is necessary to provide a video behavior recognition method and device based on multi-time-scale convolution to solve the technical problem in the prior art that the recognition accuracy of behavior recognition through the TSM model is low.

[0006] In order to solve the above technical problems, the present invention provides a video action recognition method based on multi-time-scale convolution, comprising:

[0007] Build at least one multi-timescale convolutional module;

[0008] Embedding the at least one multi-timescale convolutional module into a preset skeleton network to form a target feature extraction model;

[0009] Extracting behavioral features in the video using the target feature extraction model;

[0010] A behavior recognition model is constructed, and the behavior characteristics are recognized by the behavior recognition model.

[0011] In some possible implementations, the skeleton network is ResNet50.

[0012] In some possible implementations, embedding the at least one multi-time-scale convolutional module into a preset skeleton network to form a target feature extraction model includes:

[0013] determining a plurality of candidate embedding schemes for the at least one multi-timescale convolutional module in the skeleton network;

[0014] determining a target embedding scheme among the plurality of candidate embedding schemes;

[0015] According to the target embedding scheme, the at least one multi-time-scale convolution module is embedded into the skeleton network to form the target feature extraction model.

[0016] In some possible implementations, determining a target embedding scheme among the multiple candidate embedding schemes includes:

[0017] Constructing multiple candidate feature extraction models corresponding one-to-one to the multiple candidate embedding schemes;

[0018] Determine a training sample set, train the multiple candidate feature extraction models respectively according to the training sample set, and obtain multiple model performance indicators corresponding to the multiple candidate feature extraction models;

[0019] The alternative embedding scheme corresponding to the optimal model performance indicator among the multiple model performance indicators is used as the target embedding scheme.

[0020] In some possible implementations, the model performance indicator includes Top 1 accuracy and / or Top 5 accuracy.

[0021] In some possible implementations, extracting behavioral features from a video using a target feature extraction model includes:

[0022] Extracting spatial behavior features in the video through the skeleton network;

[0023] extracting temporal behavior features in the video through the at least one multi-timescale convolution module;

[0024] The spatial behavior feature is fused with the temporal behavior feature to obtain the behavior feature in the video.

[0025] In some possible implementations, each multi-time-scale convolution module in the at least one multi-time-scale convolution module includes at least two parallel one-dimensional convolution layers.

[0026] In some possible implementations, extracting temporal behavior features in the video by using the at least one multi-time-scale convolution module includes:

[0027] Determining a truncation parameter, and determining the number of the one-dimensional convolution layers in the multi-time-scale convolution module according to the truncation parameter and the total channels of the spatial behavior feature;

[0028] Dividing the total channels into intercepted channels and retained channels along the channel order according to the number of the one-dimensional convolutional layers; the intercepted channels correspond one-to-one to the one-dimensional convolutional layers;

[0029] Inputting the intercepted features corresponding to the intercepted channels into the corresponding one-dimensional convolutional layer to obtain at least two temporal behavior sub-features of different scales;

[0030] The time behavior sub-features are fused to obtain a fused time behavior sub-feature, and the fused time behavior sub-feature is spliced ​​with the retained feature to obtain the time behavior feature.

[0031] In some possible implementations, before extracting the behavioral features in the video using the target feature extraction model, the method further includes:

[0032] Sparsely sampling the video to obtain multiple video frames;

[0033] Then the behavior features extracted from the video by the target feature extraction model are specifically:

[0034] The behavior features in the multiple video frames are extracted by the target feature extraction model.

[0035] On the other hand, the present invention also provides a video behavior recognition device based on multi-time-scale convolution, comprising:

[0036] A multi-time-scale convolution module construction unit, used to construct at least one multi-time-scale convolution module;

[0037] a target feature extraction model construction unit, configured to embed the at least one multi-time-scale convolutional module into a preset skeleton network to form a target feature extraction model;

[0038] A behavior feature extraction unit, configured to extract behavior features from a video using the target feature extraction model;

[0039] The behavior feature recognition unit is used to build a behavior recognition model and recognize the behavior feature through the recognition model.

[0040] The beneficial effect of adopting the above embodiment is that the video behavior recognition method based on multi-time-scale convolution provided by the present invention constructs at least one multi-time-scale convolution module, embeds at least one multi-time-scale convolution module into a preset skeleton network, forms a target feature extraction model, and extracts the behavior features in the video through the target feature extraction model. Compared with the prior art of extracting behavior features through the TSM module, since the convolution kernel parameters in the multi-time-scale convolution module can be learned and adjusted, it can effectively extract the spatiotemporal features between different frames of the video, thereby improving the accuracy of behavior feature extraction. Moreover, the target feature extraction model formed by the multi-scale convolution module and the skeleton network will not cause the information loss problem existing in the TSM module, further improving the accuracy of behavior feature extraction, thereby improving the recognition accuracy of video behavior recognition.

[0041] Furthermore, by obtaining behavioral features through a multi-time-scale convolution module, the spatiotemporal features at different time scales can be integrated, and each frame will also obtain spatiotemporal information over a larger time range, enhancing the robustness of the target feature extraction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A schematic flow chart of an embodiment of the video behavior recognition method based on multi-time-scale convolution provided by the present invention;

[0044] Figure 2 A schematic structural diagram of an embodiment of the bottleneck structure of the skeleton network provided by the present invention;

[0045] Figure 3 A schematic diagram of the structure of an embodiment of the target feature extraction model provided by the present invention;

[0046] Figure 4 For the present invention Figure 1 A schematic flow chart of an embodiment of S102;

[0047] Figure 5 For the present invention Figure 4 A schematic flow chart of an embodiment of S402;

[0048] Figure 6 For the present invention Figure 1 A schematic flow chart of an embodiment of S103;

[0049] Figure 7 For the present invention Figure 6 A schematic flow chart of an embodiment of S602;

[0050] Figure 8 A schematic diagram of the structure of an embodiment of a video behavior recognition device based on multi-time-scale convolution provided by the present invention;

[0051] Figure 9 This is a schematic structural diagram of an embodiment of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more.

[0054] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] The present invention provides a video behavior recognition method and device based on multi-time-scale convolution, which are described below.

[0056] Figure 1 A flow chart of an embodiment of the video behavior recognition method based on multi-time scale convolution provided by the present invention is shown as follows: Figure 1 As shown in Figure 2, the video action recognition method based on multi-time-scale convolution includes:

[0057] S101, construct at least one multi-time-scale convolution module;

[0058] S102, embedding at least one multi-time-scale convolutional module into a preset skeleton network to form a target feature extraction model;

[0059] S103, extracting behavioral features in the video through a target feature extraction model;

[0060] S104: Build a behavior recognition model and identify behavior features through the behavior recognition model.

[0061] Compared with the prior art, the video behavior recognition method based on multi-time-scale convolution provided by the embodiment of the present invention constructs at least one multi-time-scale convolution module, embeds at least one multi-time-scale convolution module into a preset skeleton network, forms a target feature extraction model, and extracts the behavior features in the video through the target feature extraction model. Compared with the prior art of extracting behavior features through the TSM module, since the convolution kernel parameters in the multi-time-scale convolution module can be learned and adjusted, it can effectively extract the spatiotemporal features between different frames of the video, thereby improving the accuracy of behavior feature extraction. Moreover, the target feature extraction model formed by the multi-scale convolution module and the skeleton network will not cause the information loss problem existing in the TSM module, further improving the accuracy of behavior feature extraction, thereby improving the recognition accuracy of video behavior recognition.

[0062] Furthermore, by obtaining behavioral features through a multi-time-scale convolution module, the spatiotemporal features at different time scales can be integrated, and each frame will also obtain spatiotemporal information over a larger time range, enhancing the robustness of the target feature extraction model.

[0063] In a preferred embodiment of the present invention, the backbone network is ResNet50.

[0064] This is because residual neural networks can be divided into ResNet18, ResNet34, ResNet50, ResNet101, ResNet152, and other networks according to the number of network layers. Table 1 shows the image classification accuracy of ResNet networks with different layers on the open source dataset (ImageNet).

[0065] Table 1 Accuracy of ResNet on ImageNet dataset

[0066]

[0067] As shown in Table 1, ResNet152 achieves an accuracy improvement of over 2% compared to ResNet50, but the number of parameters has more than doubled. ResNet50, on the other hand, achieves a performance improvement of over 3% compared to ResNet34, while only increasing the number of parameters by 4M. Therefore, ResNet50 offers a better balance between parameter count and accuracy among residual networks. Based on this consideration, the backbone network of the embodiment of the present invention is ResNet50.

[0068] Specifically, ResNet50 consists of 50 network layers. Except for the first convolutional layer and the last fully connected layer, the other convolutional layers are organized in the form of a bottleneck structure. The bottleneck structure is divided into the convolutional layers between ResNet50 except the first convolutional layer and the last fully connected layer according to the input features of different sizes.

[0069] like Figure 2 As shown, each bottleneck structure contains three convolution kernels with parameters of 1×1, 3×3, and 1×1. The two 1×1 convolution kernels are placed at the top and bottom of the structure, and the 3×3 convolution kernel is placed in the middle. The 1×1 convolution kernel compresses and restores the channels of the input features, while the 3×3 convolution kernel maintains the same number of channels during calculation. This design significantly reduces the number of network parameters while accelerating network training.

[0070] A short-circuit mechanism is also designed between the input and output of the bottleneck structure, called residual connection identify. Specifically, the relationship between the input features and output features of the convolutional layer with a bottleneck structure is:

[0071]

[0072] Where x in is the input feature, x out is the output feature, is the mapping from input to output represented by a convolutional layer, where ω is the convolution operation.

[0073] By setting the residual connection, the mapping When the value of approaches 0, it is considered that the network layer has not learned any additional features, that is, the network layer is an identity mapping from input to output. This allows the network layer to be deepened while maintaining network performance without degradation.

[0074] And, as Figure 2 As shown in Figure 1, ResNet50 uses the activation function ReLu and a normalization layer BN after the convolution kernel. The BN layer normalizes the network output, speeds up the convergence of network training, reduces the occurrence of gradient explosion or gradient diffusion, and alleviates the overfitting phenomenon of the network. Therefore, the BN layer is widely used in various network structures. The calculation of the BN layer is as shown in formula (5):

[0075]

[0076] Where, is the output data, x is the input data, μ is the mean of the input data, σ 2 is the variance of the input data; γ and β are trainable parameters.

[0077] The activation function ReLu is:

[0078] ReLu(x)=max(0,x)

[0079] The activation function ReLu sets the values ​​in the input data that are less than 0 to 0, and the values ​​that are greater than 0 remain unchanged to increase the nonlinearity of the convolutional neural network.

[0080] In a specific embodiment of the present invention, Figure 3 As shown in Figure 2, the specific form of embedding the multi-time-scale convolution module into the skeleton network is: inserting the multi-scale convolution module before the bottleneck structure of different convolution layers.

[0081] Since the number of multi-scale convolution modules and the number of bottleneck structures are not necessarily the same, and embedding the multi-scale convolution modules in different positions in the skeleton network will affect the accuracy of video behavior recognition. Therefore, in some embodiments of the present invention, Figure 4 As shown, step S102 includes:

[0082] S401, determining multiple candidate embedding schemes for at least one multi-time-scale convolutional module in a skeleton network;

[0083] S402, determining a target embedding scheme among multiple candidate embedding schemes;

[0084] S403: According to the target embedding scheme, embed at least one multi-time-scale convolutional module into the skeleton network to form a target feature extraction model.

[0085] By determining a target embedding scheme, the embodiments of the present invention can further improve the reliability of the behavior features extracted by the target feature extraction model, thereby improving the accuracy of behavior recognition.

[0086] In a specific embodiment of the present invention, Figure 5 As shown, step S402 includes:

[0087] S501, constructing multiple candidate feature extraction models corresponding to multiple candidate embedding schemes;

[0088] S502: Determine a training sample set, train multiple candidate feature extraction models according to the training sample set, and obtain multiple model performance indicators corresponding to the multiple candidate feature extraction models;

[0089] S503: Using the alternative embedding scheme corresponding to the optimal model performance indicator among the multiple model performance indicators as the target embedding scheme.

[0090] By taking the alternative embedding scheme corresponding to the optimal model performance indicator as the target embedding scheme, the accuracy of the behavioral features extracted by the target feature extraction model can be improved.

[0091] In a specific embodiment of the present invention, the training sample set is the open source Something-Something v2 dataset.

[0092] In a specific embodiment of the present invention, the model performance indicator in step S502 includes Top 1 accuracy and / or Top 5 accuracy.

[0093] The Top1 accuracy rate refers to the proportion of the most probable categories in the model output that are consistent with the actual categories of the video. The Top5 accuracy rate refers to the proportion of the top five most probable categories in the model output that contain the actual categories of the video.

[0094] In a specific embodiment of the present invention, Figure 3 As shown, ResNet50 includes, from top to bottom, the first convolutional layer conv1, the second convolutional layer layer1, the third convolutional layer layer2, the fourth convolutional layer layer3, the fifth convolutional layer layer4 and the fully connected layer fc. The second convolutional layer layer1, the third convolutional layer layer2, the fourth convolutional layer layer3 and the fifth convolutional layer layer4 all include a bottleneck structure. Step S402 includes three alternative embedding schemes: [1,2,3,4], [2,3,4] and [3,4]. Specifically, the alternative embedding scheme [1,2,3,4] represents inserting a multi-time scale convolution before each bottleneck structure of layer1, layer2, layer3 and layer4 of ResNet50, and the rest of the alternative embedding schemes are similar. After experiments, the multiple model performance indicators corresponding to multiple alternative feature extraction models are shown in Table 2:

[0095] Table 2 Model performance indicators of alternative embedding schemes

[0096]

[0097] Table 3 shows that model performance decreases with decreasing number of inserted layers, indicating that increasing the number of inserted convolutions can enhance the model's spatiotemporal modeling capabilities. However, increasing the number of multi-scale convolutional modules also increases model parameters and computational overhead. Therefore, for some computationally sensitive applications, it is advisable to insert fewer layers, such as in alternative embedding solutions [2, 3, 4]. Compared to alternative embedding solutions [1, 2, 3, 4], their accuracy only decreases by 0.4%. For some accuracy-sensitive applications, alternative embedding solutions [1, 2, 3, 4] are recommended.

[0098] In some embodiments of the present invention, Figure 6 As shown, step S103 includes:

[0099] S601, extracting spatial behavior features in the video through a skeleton network;

[0100] S602, extracting temporal behavior features in the video through at least one multi-time-scale convolution module;

[0101] S603: Fusing spatial behavior features with temporal behavior features to obtain behavior features in the video.

[0102] That is, the behavior features in the embodiment of the present invention include spatial behavior features and temporal behavior features, which improves the accuracy of behavior recognition.

[0103] In order to further improve the acquisition of temporal behavior features at different scales, in a preferred embodiment of the present invention, each multi-time-scale convolution module in at least one multi-time-scale convolution module includes at least two parallel one-dimensional convolution layers.

[0104] By setting up a multi-time-scale convolution module including at least two one-dimensional convolution layers, while increasing the scale of temporal behavior features, the one-dimensional convolution layer has a simple structure and fewer parameters compared to other high-dimensional convolution layers such as two-dimensional convolution layers, which can improve the efficiency of obtaining temporal behavior features.

[0105] In some embodiments of the present invention, Figure 7 As shown, step S602 includes:

[0106] S701, determining a truncation parameter, and determining the number of one-dimensional convolution layers in a multi-time-scale convolution module according to the truncation parameter and the total channels of the spatial behavior characteristics;

[0107] S702, dividing the total channels into intercepted channels and retained channels along the channel order according to the number of one-dimensional convolutional layers; the intercepted channels correspond one-to-one to the one-dimensional convolutional layers;

[0108] S703: Input the clipped features corresponding to the clipped channels into the corresponding one-dimensional convolutional layer to obtain at least two temporal behavior sub-features of different scales;

[0109] S704: Fuse the time behavior sub-features to obtain a fused time behavior sub-feature, and concatenate the fused time behavior sub-feature with the retained feature to obtain a time behavior feature.

[0110] In some embodiments of the present invention, the corresponding relationship between the interception parameters, the total channels and the interception channels is:

[0111]

[0112] Where α is the interception parameter; C in is the total number of channels; C conv is the number of intercepted channels.

[0113] In a specific embodiment of the present invention, the process of determining the truncation parameters in step S701 is as follows: determining multiple candidate truncation parameters, and selecting the candidate truncation parameters corresponding to the optimal model performance indicator from the multiple candidate truncation parameters as the truncation parameters. When the truncation parameters are too few, the features of each frame in the spatial behavior feature only retain half of the original features, which causes a serious loss of information, thereby damaging the spatial modeling capability of the model and causing a decrease in network performance. When the truncation parameters are too many, although most of the features of the spatial behavior feature are retained, the temporal information fusion is less, so the network performance still has room for improvement.

[0114] After experiments, in a preferred embodiment of the present invention, the interception parameter is 4.

[0115] In a specific embodiment of the present invention, the total number of channels is 8, the truncation parameter is 4, the truncation channels are 2, the retention channels are 6, and the multi-time convolution module includes two parallel one-dimensional convolution layers.

[0116] Since a video often includes a large number of video frames, in order to reduce the amount of calculation, in some embodiments of the present invention, before step S103, the following steps are further included:

[0117] Sparsely sample the video to obtain multiple video frames;

[0118] Then step S103 is specifically as follows:

[0119] The behavioral features in multiple video frames are extracted through the target feature extraction model.

[0120] The embodiment of the present invention can reduce the video processing amount and improve the behavior recognition speed by sparsely sampling the video before extracting the behavior features of the video.

[0121] In order to further verify the superiority of the video behavior recognition method based on multi-time-scale convolution eliminated in the embodiment of the present invention, in some embodiments of the present invention, the accuracy of the embodiment of the present invention and the TSN model, TRN model, TRN-2Stream model, and TSM model in the prior art are compared. The comparison results are shown in Table 3:

[0122] Table 3 Performance comparison of different models

[0123]

[0124] Table 3 shows that, with the exception of the TSN model, which uses 16 frames as input, all other models use 8 frames as input and achieve higher accuracy. The video action recognition method based on multi-timescale convolution proposed in this embodiment of the present invention achieves an accuracy 11.33% higher than that of the TRN model, 3.63% higher than that of the TRN-2Stream model, and 0.77% higher than that of the TSM model with the same settings. This means that the video action recognition method based on multi-timescale convolution proposed in this embodiment of the present invention outperforms existing action recognition methods.

[0125] It should be noted that the experimental environment for conducting various experiments in the embodiments of the present invention is: experiments are carried out in the system environment of Ubuntu 16.04, an NVIDIA RTX 2080ti graphics card is used for model training and testing, the version of the Pytorch framework is 1.4, and the version of Cuda is 10.0. The model of the embodiment of the present invention uses the ImageNet pre-training parameters, and the dropout of the fully connected layer is set to 0.5 to reduce overfitting. SGD is selected as the optimizer in the training process, and the initial learning rate is set to 0.01. At the 20th epoch and the 40th epoch, the learning rate drops to 1 / 10 of the current learning rate. The momentum of the optimizer is 0.8, and the number of rounds of model training is 50 epochs.

[0126] In order to better implement the video behavior recognition method based on multi-time scale convolution in the embodiment of the present invention, based on the video behavior recognition method based on multi-time scale convolution, correspondingly, Figure 8 As shown, the embodiment of the present invention further provides a video behavior recognition device 800 based on multi-time-scale convolution, comprising:

[0127] A multi-time-scale convolution module construction unit 801 is used to construct at least one multi-time-scale convolution module;

[0128] A target feature extraction model construction unit 802 is used to embed at least one multi-time scale convolution module into a preset skeleton network to form a target feature extraction model;

[0129] A behavior feature extraction unit 803 is used to extract behavior features in the video using a target feature extraction model;

[0130] The behavior feature recognition unit 804 is used to build a behavior recognition model and recognize the behavior features through the recognition model.

[0131] The video behavior recognition device 800 based on multi-time-scale convolution provided in the above embodiment can implement the technical solution described in the above embodiment of the video behavior recognition method based on multi-time-scale convolution. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of the video behavior recognition method based on multi-time-scale convolution, which will not be repeated here.

[0132] like Figure 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902 and a display 903. Figure 9 Only some of the components of the electronic device 900 are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0133] In some embodiments, the memory 902 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. In other embodiments, the memory 902 may also be an external storage device of the electronic device 900, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 900.

[0134] Furthermore, the memory 902 may include both an internal storage unit of the electronic device 900 and an external storage device. The memory 902 is used to store application software installed in the electronic device 900 and various data.

[0135] In some embodiments, the processor 901 can be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run the program code or process data stored in the memory 902, such as the video behavior recognition method based on multi-time-scale convolution in the present invention.

[0136] In some embodiments, the display 903 can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 903 is used to display information on the electronic device 900 and to display a visual user interface. Components 901-903 of the electronic device 900 communicate with each other via a system bus.

[0137] In one embodiment, when the processor 901 executes the multi-time-scale convolution-based video behavior recognition program in the memory 902, the following steps may be implemented:

[0138] Build at least one multi-timescale convolutional module;

[0139] Embedding at least one multi-timescale convolutional module into a preset skeleton network to form a target feature extraction model;

[0140] Extract behavioral features from the video through the target feature extraction model;

[0141] Build a behavior recognition model and identify behavioral features through the behavior recognition model.

[0142] It should be understood that, when the processor 901 executes the multi-time-scale convolution-based video behavior recognition program in the memory 902 , in addition to the above functions, it can also implement other functions. For details, please refer to the description of the corresponding method embodiment above.

[0143] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 900 mentioned. The electronic device 900 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, or a laptop computer. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with iOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0144] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the method steps or functions provided by the above-mentioned method embodiments can be implemented.

[0145] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0146] The above is a detailed introduction to the video behavior recognition method and device based on multi-time-scale convolution provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A video action recognition method based on multi-time-scale convolution, characterized by: include: Build at least one multi-timescale convolutional module; Embedding the at least one multi-timescale convolutional module into a preset skeleton network to form a target feature extraction model; Extracting spatial behavior features in the video through the skeleton network; extracting temporal behavior features in the video through the at least one multi-timescale convolution module; fusing the spatial behavior features with the temporal behavior features to obtain behavior features in the video; Building a behavior recognition model, and identifying the behavior characteristics through the behavior recognition model; Each multi-time-scale convolution module in the at least one multi-time-scale convolution module includes at least two one-dimensional convolution layers in parallel; The extracting temporal behavior features in the video by the at least one multi-time-scale convolution module includes: Determining a truncation parameter, and determining the number of the one-dimensional convolution layers in the multi-time-scale convolution module according to the truncation parameter and the total channels of the spatial behavior feature; Dividing the total channels into intercepted channels and retained channels along the channel order according to the number of the one-dimensional convolutional layers; the intercepted channels correspond one-to-one to the one-dimensional convolutional layers; Inputting the intercepted features corresponding to the intercepted channels into the corresponding one-dimensional convolutional layer to obtain at least two temporal behavior sub-features of different scales; The time behavior sub-features are fused to obtain a fused time behavior sub-feature, and the fused time behavior sub-feature is spliced ​​with the retained feature corresponding to the retained channel to obtain the time behavior feature.

2. The video behavior recognition method based on multi-time scale convolution according to claim 1 is characterized in that: The skeleton network is ResNet50.

3. The video behavior recognition method based on multi-time scale convolution according to claim 1 is characterized in that: The at least one multi-time-scale convolution module is embedded into a preset skeleton network to form a target feature extraction model, comprising: determining a plurality of candidate embedding schemes for the at least one multi-timescale convolutional module in the skeleton network; determining a target embedding scheme among the plurality of candidate embedding schemes; According to the target embedding scheme, the at least one multi-time-scale convolution module is embedded into the skeleton network to form the target feature extraction model.

4. The video behavior recognition method based on multi-time scale convolution according to claim 3 is characterized in that: Determining a target embedding scheme among the multiple candidate embedding schemes includes: Constructing multiple candidate feature extraction models corresponding one-to-one to the multiple candidate embedding schemes; Determine a training sample set, train the multiple candidate feature extraction models respectively according to the training sample set, and obtain multiple model performance indicators corresponding to the multiple candidate feature extraction models; The alternative embedding scheme corresponding to the optimal model performance indicator among the multiple model performance indicators is used as the target embedding scheme.

5. The video behavior recognition method based on multi-time-scale convolution according to claim 4 is characterized in that: The model performance indicators include Top 1 accuracy and / or Top 5 accuracy.

6. The video behavior recognition method based on multi-time-scale convolution according to claim 1 is characterized in that: Before extracting the behavioral features in the video by the target feature extraction model, the method further includes: Sparsely sampling the video to obtain multiple video frames; Then the behavior features extracted from the video by the target feature extraction model are specifically: The behavior features in the multiple video frames are extracted by the target feature extraction model.

7. A video behavior recognition device based on multi-time scale convolution, characterized in that: include: A multi-time-scale convolution module construction unit, used to construct at least one multi-time-scale convolution module; a target feature extraction model construction unit, configured to embed the at least one multi-time-scale convolutional module into a preset skeleton network to form a target feature extraction model; A behavior feature extraction unit, configured to extract behavior features from a video using the target feature extraction model; A behavior feature recognition unit, configured to construct a behavior recognition model and recognize the behavior feature using the recognition model; The behavior feature extraction unit is specifically configured to: extract spatial behavior features in the video through the skeleton network; and extract temporal behavior features in the video through the at least one multi-time-scale convolution module; fusing the spatial behavior features with the temporal behavior features to obtain behavior features in the video; Each multi-time-scale convolution module in the at least one multi-time-scale convolution module includes at least two one-dimensional convolution layers in parallel; The extracting of temporal behavior features in the video through the at least one multi-time-scale convolution module includes: determining a clipping parameter, and determining the number of one-dimensional convolution layers in the multi-time-scale convolution module according to the clipping parameter and the total channel of the spatial behavior feature; dividing the total channel into clipping channels and retained channels along the channel sequence according to the number of the one-dimensional convolution layers; the clipping channels correspond one-to-one to the one-dimensional convolution layers; inputting the clipping features corresponding to the clipping channels into the corresponding one-dimensional convolution layers to obtain at least two temporal behavior sub-features of different scales; fusing the temporal behavior sub-features to obtain a fused temporal behavior sub-feature, and splicing the fused temporal behavior sub-feature with the retained features corresponding to the retained channels to obtain the temporal behavior features.

Images