Action recognition method of C3D convolutional neural network

Abstract

The invention provides an action recognition method of a C3D convolutional neural network, which comprises the following steps: firstly, adding a residual block into a C3D network model for improvement, improving the network recognition accuracy and accelerating the training speed, and then optimizing two hyper-parameters, namely a learning rate and momentum in a network optimizer by using a PSO algorithm, so as to improve the recognition accuracy of the improved C3D network and reduce an overfitting phenomenon. Compared with a basic C3D network model, the method has the advantages of being high in prediction accuracy, low in network over-fitting degree and the like, can achieve a more accurate prediction result under the condition of short training time, and has potential application prospects in video data processing and computer vision development.

Description

Technical Field

[0001] The present invention relates to the field of computer technology and application technology, and more specifically, to an action recognition method of a C3D convolutional neural network based on particle swarm algorithm (PSO) optimization. Background Art

[0002] With the development of computer technology and the Internet, electronic surveillance equipment has penetrated into every aspect of life, generating a large amount of video data. With the development of artificial intelligence and computer vision, using computer technology to process images and recognize actions to improve information processing efficiency has become a research hotspot.

[0003] The C3D network model was proposed by Facebook in 2015. It is a deep learning network structure based on 3D CNN. It has achieved good results in different video analysis tasks and has been widely used. However, the accuracy of the C3D network model still needs to be improved.

[0004] In view of this, the present invention proposes a motion recognition method of a C3D convolutional neural network. Particle swarm optimization (PSO) is a classic intelligent optimization algorithm with the advantages of few adjustment parameters, stable optimization, strong convergence ability, etc. It is applied to the training of neural networks to find a suitable hyperparameter for the network and improve the recognition accuracy of the network. Summary of the invention

[0005] The purpose of the present invention is to propose a motion recognition method of a C3D convolutional neural network. By connecting the residual block structure in series with the C3D network model, features are further extracted from the data to improve the accuracy of the C3D network. This network is called a RESC3D network model. The PSO algorithm is used to optimize the learning rate and momentum in the RESC3D network model. Finally, the optimized RESC3D network has a prediction accuracy of 5.2% compared with an unoptimized RESC3D network when using the same number of training rounds, and the overfitting phenomenon is improved. Compared with the original C3D model, the accuracy is improved by 13.64%, and the training time is also reduced by 57 minutes.

[0006] A C3D convolutional neural network action recognition method, characterized by comprising the following steps:

[0007] (1) The UCF50 dataset was selected as the basic database, and the images in the dataset were processed to ensure that the number of extracted video frames was at least 16. All processed data were divided into a training set and a test set, with a ratio of 8.5:1.

[0008] (2) The specific steps for optimizing the C3D network model are as follows:

[0009] (2a) Instantiate the C3D network model; define the loss function of the C3D network model as the cross entropy loss function, define the optimizer algorithm as the SGD stochastic gradient descent algorithm, and set the two hyperparameters of the optimizer: learning rate and momentum;

[0010] (2b) Define the learning rate update strategy and output the optimal learning rate and momentum hyperparameters to update the C3D network model;

[0011] (2c) Input the training set into the C3D network model with optimized hyperparameters, perform forward propagation to obtain the C3D network model output, and calculate the accuracy of the training set;

[0012] (2d) Calculate the loss using the training set accuracy and true label output by the C3D network model, and then perform back propagation based on the loss value to update the C3D network model parameters;

[0013] (2e) Load the test set to obtain the test set accuracy;

[0014] (2f) Determine whether the maximum number of training rounds has been reached. If not, return to step (2c) to continue training. If the maximum number of training rounds has been reached, save the trained model parameters and accuracy values.

[0015] (3) The test set is input into the optimized C3D network model for forward propagation to obtain the classification results of each image.

[0016] Furthermore, the maximum number of training rounds is 70 rounds, and the learning rate is reduced to 0.1 times the previous value in the 50th round. Too large a learning rate may cause the model to oscillate near the optimal solution, or cause the model to diverge in extreme cases. By reducing the learning rate, the model can converge quickly in the early stage of training and converge to the optimal value more stably in the later stage of training.

[0017] Furthermore, the cross entropy loss function is expressed as follows:

[0018]

[0019] where y i is the actual label value, is the probability value predicted by the model.

[0020] Furthermore, the learning rate of the optimizer mainly controls the speed of gradient descent during subsequent back propagation. If the learning rate is too small, it will lead to slow convergence. If the learning rate is too large, it will hinder convergence. Momentum controls the amount of change in each round of updates. Its main function is to increase the model convergence speed and help the network escape from the local optimal solution. Since these two parameters have a great influence on the model training results, they are subsequently selected as the optimization targets of the PSO algorithm.

[0021] Furthermore, backpropagation is a process that uses the chain rule to calculate the loss gradient layer by layer according to the output of the output layer to the input layer, and then updates the weights and bias parameters of the network to minimize the loss function.

[0022] Furthermore, in step (2b), a learning rate update strategy is defined, and the optimal learning rate and momentum hyperparameters are output to update the hyperparameters of the C3D network model. The PSO algorithm is used for parameter iteration. The specific steps are as follows:

[0023] (2b1) Create the initial PSO algorithm population based on the training set according to the learning rate and momentum parameters; the number of population particles is 3;

[0024] (2b2) ​​Calculate the fitness, use the position of each particle as the hyperparameter of the C3D network model for 60 rounds of training, and after the training, use the test set test accuracy to take the inverse number as the fitness value to obtain the optimal fitness value of the initial population, and store it as the global optimal fitness value and the corresponding particle position;

[0025] (2b3) Compare the obtained optimal fitness value with the value of the optimal fitness of the previous generation. If the value is less than the optimal fitness of the previous generation, the optimal fitness value of this generation is used as the value of the global optimal fitness and the corresponding position information is stored. Continue to complete the iteration until the maximum number of iterations is reached, and output the optimal learning rate and momentum parameters; if it is not satisfied, iteratively update the particle position and repeat step (2b2) ​​to obtain the optimal fitness and position information of this generation;

[0026] (2b4) The optimal learning rate and momentum parameters output from step (2b3) are used as hyperparameters of the RESC3D network model for 70 rounds of training and testing to obtain the optimized accuracy. The original parameters in the RESC3D network model are replaced with the optimized hyperparameters to complete the optimization of the RESC3D network model for action recognition.

[0027] Furthermore, in step (2b1), the population of the PSO algorithm is a 3×2 matrix, the first column is used as the momentum of RESC3D network training, ranging from [0.6, 0.95], the second column is the learning rate of model training, ranging from [0.0001, 0.001], and the three rows represent the positions of each particle.

[0028] Furthermore, in step (2b3), the update formula for iteratively updating the particle position is:

[0029]

[0030] Where: is the position of the i-th particle in the k-th generation, k is the current iteration number, d=1,2 is the search dimension, i=1,2,···,5 is the individual number of the population; Indicates the update speed of the i-th particle in the k-th generation; c1=c2=2 is the learning factor of the PSO algorithm; r1, r2 are random numbers between [0,1]; It represents the optimal value of the individual searched by the i-th individual when iterating to the k-th generation; Represents the global optimal value of the entire population when iterating to the kth generation; during the iteration process, Continuously updated; the value range of weight ω is [ω1,ω2].

[0031] Furthermore, in order to avoid the problem that the PSO algorithm is prone to premature maturity in the early stage and easy to oscillate in the later stage, the weight update formula is used to update the weight so that the weight decreases with the number of iterations. The weight update formula is as follows:

[0032]

[0033] In some embodiments, before step (2a), building a RESC3D network model based on the C3D network model includes the following steps:

[0034] Step A: Build the 3D The deep learning network structure model constructed by CNN includes: multiple convolutional layers, five pooling layers of MaxPool1, MaxPool2, MaxPool3, MaxPool4, and MaxPool5, and two fully connected layers. The kernel size of all convolutional layers is 3×3×3, the padding is 1×1×1, and the stride is 1×1×1; the receptive field of the MaxPool1 maximum pooling layer is 1×2×2, the stride is 1×2×2, and the padding is 0; the receptive field of the MaxPool2, MaxPool3, and MaxPool4 maximum pooling layers is 2×2×2, the stride is 2×2×2, and the padding is 0; the receptive field of the MaxPool5 maximum pooling layer is 2×2×2, the stride is 2×2×2, and the padding is 0×1×1; the number of neurons in the two fully connected layers is 4096; the fully connected layer and all convolutional layers are activated by the ReLU activation function. In order to limit the overfitting effect caused by the huge amount of parameters, a Dropout of 0.5 is added to the two fully connected layers;

[0035] Step B: Connect two layers of residual blocks in series to multiple convolutional layer input ends of the deep learning network structure model constructed by 3D CNN. Each layer of the residual block includes two convolutional layers of the first channel and one convolutional layer of the second channel, namely Conv1, Conv2 and Conv3. The convolution kernel sizes of Conv1 and Conv2 are both 3×3×3, the padding is 1×1×1, and the stride is 1×1×1. After the convolution, Conv1 and Conv2 are connected to a BN layer for batch normalization. The second channel has only one convolution Conv3, and the convolution kernel size of Conv3 is 1×1×1. The stride is 1×2×2. Conv3 adjusts the input data size so that it can be added to the data of the first channel after being processed by the second BN layer. The BN layer is not connected after convolution to restore the characteristics of the input data as much as possible. Input and Output are the input and output of the residual block respectively. By adding the convolution data of the two channels, the data is reduced in dimension to achieve the purpose of reducing the loss of data features. BN1 and BN2 are batch normalization layers, which can make the distribution of input data in each layer of the network relatively stable and speed up the training and convergence of the model.

[0036] Step C: Adjust the padding of the third and fifth pooling layers to adapt to the change in data size, add 0×1×1 padding to the pooling layer MaxPool3, adjust the stride of MaxPool4 to 1×1×1, adjust the stride of MaxPool5 to 1×1×1 and remove its padding to adapt to the change in data size, so that the receptive field of the pooling layer MaxPool3 of the final RESC3D network module is 2×2×2, the stride is 2×2×2, and the padding is 0×1×1, the receptive fields of MaxPool4 and MaxPool5 are both 2×2×2, the stride is 1×1×1, and the padding is 0.

[0037] Step D: Change the number of neurons in the fully connected layer from 4096 to 256 to adapt to the reduction of data. At the same time, due to the reduction of the fully connected layer, the 0.5 dropout rate of the two Dropouts is changed to 0.25.

[0038] Furthermore, in step (1), in the RESC3D network model, 50 categories in the UCF50 dataset are taken as the dataset. The UCF50 dataset was released by researchers from the University of Central Florida and aims to provide high-quality video data for research in the field of action recognition. The dataset consists of real videos from YouTube and contains more than 6,000 videos of 50 different action categories, covering sports, daily activities, imitation actions, etc. Each action category has enough video samples to ensure the accuracy and generalization ability of model training. Some common action categories include baseball pitching, basketball shooting, bench press, cycling, billiards, breaststroke, clean and jerk, diving, and drumming. These action categories are not only representative, but also can reflect the characteristics and difficulty of different types of sports.

[0039] Furthermore, each video data set is processed with an extraction frequency of 4 frames. The extracted frame size is adjusted to 128 pixels high and 171 pixels wide. If the number of video frames after extraction at this frequency is less than 16, the extraction frequency is gradually reduced to ensure that the number of extracted video frames is at least 16.

[0040] Furthermore, the input of the RESC3D network model is 3×16×112×112 data read from 16 frames of images sampled from the training set. After being processed by the first layer of residual blocks, the data dimension becomes 16×16×56×56, and then after being processed by the second layer of residual blocks, the data becomes 32×16×28×28. After a series of convolution and pooling operations, the data dimension becomes 512×1×1×1. At this time, the data has 4 dimensions. The data is flattened into one-dimensional data of size 512 and sent to the fully connected layer. The output of the RESC3D network model is the label value calculated by SoftMax of the fully connected layer of 50 neurons; finally, the parameters of the RESC3D network model are reduced from 78409573 parameters before the improvement to 27979701, and the parameters are reduced by about 5000w. The reduction in weight parameters will greatly shorten the training time of the model.

[0041] Furthermore, the input of the RESC3D network model is 3×16×112×112 data read from 16 frames of images sampled from the training set, and the output is the label value calculated by SoftMax through a fully connected layer of 50 neurons.

[0042] Beneficial effects of the present invention:

[0043] The present invention improves the prediction accuracy of the C3D network by combining the residual block with the C3D network model, and optimizes the learning rate and momentum and other hyperparameters of the neural network through the PSO algorithm, further improving the accuracy of the RESC3D network and reducing the overfitting degree of the network. It has potential application prospects in video data processing and computer vision development. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the network structure diagram of the basic C3D network model.

[0045] Figure 2 It is a schematic diagram of the introduced residual block structure.

[0046] Figure 3 It is the network structure diagram of RESC3D network model.

[0047] Figure 4 It is a flowchart of network training.

[0048] Figure 5 It is a specific flow chart of the action recognition method of the present invention based on particle swarm algorithm optimization and improvement of C3D network model.

[0049] Figure 6 It is the accuracy and loss value curve of the basic C3D network model training and testing before adding the residual block.

[0050] Figure 7It is the accuracy and loss value curve of the unoptimized RESC3D network model training and testing.

[0051] Figure 8 It is an iterative curve diagram of the particle swarm algorithm in the example of the present invention.

[0052] Fig. 9 The present invention adopts the PSO algorithm to optimize the accuracy and loss value curve of RESC3D network model training and testing. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0055] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and examples:

[0056] The present invention proposes a method for action recognition based on a C3D convolutional neural network. By introducing a residual block, the data is further extracted to improve the accuracy of the C3D network model. Then, the PSO algorithm is used to optimize the learning rate and momentum of the RESC3D network model training. Compared with the traditional C3D network model, the accuracy of the optimized RESC3D network model is improved by 13.64%, the training time is reduced by 57 minutes, and the overfitting of the model is improved. It has potential application prospects in action recognition and video data processing.

[0057] The dataset of this example uses a 50-category dataset UCF50. The UCF50 dataset was released by researchers at the University of Central Florida and aims to provide high-quality video data for research in the field of action recognition. The dataset consists of real videos from YouTube and contains more than 6,000 videos of 50 different action categories, covering sports, daily activities, imitation actions and many other aspects. Each action category has enough video samples to ensure the accuracy and generalization ability of model training. Some common action categories include baseball pitching, basketball shooting, bench press, cycling, billiards, breaststroke, clean and jerk, diving and drumming. These action categories are not only representative, but also reflect the characteristics and difficulty of different types of sports. The dataset contains training sets and test sets, divided into 8.5:1 ratios, and all training datasets are divided into 16 batches.

[0058] Figure 1 The C3D model shown is a schematic diagram of the structure before improvement. The C3D network model was proposed by Facebook in 2015 and is a deep learning network structure based on 3D CNN. In this model, all convolution kernels are 3×3×3 in size, with 1×1×1 padding and 1×1×1 stride. The receptive field of the MaxPool1 maximum pooling layer is 1×2×2, with a stride of 1×2×2 and a padding of 0. The receptive field of the MaxPool2, MaxPool3, and MaxPool4 maximum pooling layers is 2×2×2, with a stride of 2×2×2 and a padding of 0. The receptive field of the MaxPool5 maximum pooling layer is 2×2×2, with a stride of 2×2×2 and a padding of 0×1×1. The number of neurons in the two fully connected layers is 4096. The fully connected layer and all convolutional layers are activated with the ReLU activation function. In order to limit the overfitting effect caused by the huge number of parameters, a Dropout of 0.5 is added to the two fully connected layers. The input of the model is 3×16×112×112 data read from 16 frames of images sampled from the training set, and the output is the label value calculated by SoftMax through a fully connected layer of 50 neurons.

[0059] Figure 2The figure is a schematic diagram of the structure of the added residual block. The residual block is used to reduce the dimension of the data. By adding the convolution data of the two channels, the purpose of reducing the loss of data features is achieved. BN1 and BN2 are batch normalization layers, which can make the input data distribution of each layer in the network relatively stable and accelerate the training and convergence speed of the model. The convolution kernel size of the two convolution layers Conv1 and Conv2 of the first channel is 3×3×3, the padding is 1×1×1, and the stride is 1×1×1. After each convolution, a BN layer is connected for batch normalization. The second channel has only one convolution layer with a convolution kernel size of 1×1×1 and a stride of 1×2×2 to adjust the input data size so that it can be added to the one-channel data. After the convolution, the BN layer is not connected to restore the characteristics of the input data as much as possible. The Input and Output in the figure are the input and output of the residual block respectively.

[0060] Figure 3 is the model structure of the RESC3D network model. Compared with the basic C3D network, two residual blocks, the first layer residual block and the second layer residual block, are added to extract data features, and 0×1×1 padding is added to the pooling layer MaxPool3, the stride of MaxPool4 is adjusted to 1×1×1, and the stride of MaxPool5 is adjusted to 1×1×1 and its padding is removed to adapt to the change in data size. Finally, the receptive field of the pooling layer MaxPool3 of the RESC3D network is 2×2×2, the stride is 2×2×2, and the padding is 0×1×1. The receptive fields of MaxPool4 and MaxPool5 are both 2×2×2, the stride is 1×1×1, and the padding is 0. The input of the model is 3×16×112×112 data read from 16 frames of images sampled from the training set. After being processed by residual block 1, the data dimension becomes 16×16×56×56, and then after being processed by residual block 2, the data dimension becomes 32×16×28×28. After a series of convolution and pooling operations, the data dimension becomes 512×1×1×1. At this time, the data has 4 dimensions. The data is flattened into one-dimensional data of size 512 and sent to the fully connected layer. Both fully connected layers are changed from 4096 to 256 to adapt to the reduction of data. At the same time, due to the reduction of the fully connected layer, the 0.5 dropout rate of the two Dropouts is changed to 0.25. The output of the model is the label value calculated by the SoftMax of the fully connected layer of 50 neurons. Finally, the overall model parameters are reduced from 78409573 parameters before improvement to 27979701, and the parameters are reduced by about 50 million. The reduction of weight parameters will greatly shorten the training time of the model.

[0061] Figure 4 It is the process of training the neural network model, which is divided into the following steps:

[0062] Step (2a): Instantiate the C3D network model, define the model's loss function as the cross entropy loss function, define the optimizer algorithm as the SGD (stochastic gradient descent) algorithm, and set the optimizer's learning rate and momentum (i.e., the optimization target of the subsequent PSO algorithm).

[0063] (2b): Define the learning rate update strategy, output the optimal learning rate and momentum hyperparameters to update the C3D network model hyperparameters, and adjust the learning rate to 0.1 times the previous value after 50 rounds of training.

[0064] (2c): Input the training set into the C3D network model with optimized hyperparameters, perform forward propagation to obtain the C3D network model output, and calculate the accuracy of the training set.

[0065] (2d): The loss is calculated using the C3D network model output and the true label, and then backpropagation is performed based on the loss value to update the C3D network model parameters.

[0066] (2e): Load the test set to obtain the test set accuracy.

[0067] (2f): Determine whether the maximum number of training rounds has been reached. If not, continue this process to update the model parameters. If the maximum number of training rounds has been reached, save the trained model parameters and accuracy values.

[0068] Among them, the cross entropy loss function expression is as follows:

[0069]

[0070] where y i is the actual label value, is the probability value predicted by the model.

[0071] The learning rate of the optimizer mainly controls the speed of gradient descent during subsequent back propagation. If the learning rate is too small, it will lead to slow convergence. If the learning rate is too large, it will hinder convergence. Momentum controls the amount of change in each round of updates. Its main function is to increase the convergence speed of the model and help the network escape the local optimal solution. Since these two parameters have a great influence on the model training results, they are also selected as the optimization targets of the PSO algorithm in the future.

[0072] Backpropagation is the process of using the chain rule to calculate the loss gradient layer by layer according to the output of the output layer to the input layer, and then updating the weights and bias parameters of the network to minimize the loss function.

[0073] Figure 5 This is the flowchart of the entire improvement and PSO optimization algorithm. First, the residual block is introduced into the C3D network model and the structure is adjusted to improve the RESC3D network. The specific improvement process is as follows: Figure 3 Then the PSO algorithm is used to optimize the learning rate and momentum of the RESC3D network, and finally the optimized RESC3D network model is obtained by training with the optimal learning rate and momentum. Each training is carried out according to the above Figure 4 The training process is used for training.

[0074] The algorithm specifically includes the following steps:

[0075] Step 1: First, introduce the residual block into the C3D network model, the structure is as follows Figure 3 As shown, two residual blocks are connected in series with the C3D network model, and the padding of the third and fifth pooling layers is adjusted to adapt to the change of data size;

[0076] Step 2: Initialize the population position of the PSO algorithm, set the initial population to a 3×2 matrix, the first column is used as the momentum during RESC3D network model training, the second column is used as the learning rate during training, and the three rows represent the positions of each particle;

[0077] Step 3: Take the position of each particle in the initial population as the hyperparameter of the RESC3D network model for training, and train 60 rounds each time. After the training, get the accuracy of the test set, take the inverse of the accuracy as the fitness value of the PSO optimization algorithm, save the optimal fitness value of the initial population as the global optimal fitness value, and save the optimal fitness position;

[0078] Step 4: Update the initial particle position, use the updated particle position information as the hyperparameter of the RESC3D network model for training, repeat step 3 to obtain the optimal fitness value of this generation, and save the optimal fitness and optimal position;

[0079] Step 5: Determine the relationship between the optimal fitness obtained in step 4 and the global optimal fitness. If the current optimal fitness is less than the global optimal fitness, replace the global optimal fitness with the current optimal fitness and update the optimal position, otherwise do not update;

[0080] Step 6: Repeat steps 4 and 5 until 10 generations are completed. Use the position information corresponding to the optimal fitness value as the hyperparameter of the C3D network model for training. After 70 rounds of training, load the test set to obtain the optimized accuracy.

[0081] The population of the PSO algorithm is a 3×2 matrix. The first column is the momentum used for RESC3D network model training, which is limited to [0.6, 0.95]. The second column is the learning rate of network training, which is limited to [0.0001, 0.001]. The particles are initialized to random numbers within this range. After each position update, the particle position is checked. If it exceeds the constraint range, the particle is reset to a random number within this range and the iteration continues.

[0082] The particle position update formula in step 4 is:

[0083]

[0084] Where: is the position of the i-th particle in the k-th generation, k is the current iteration number, d=1,2 is the search dimension, i=1,2,3 is the population individual number, c1=c2=2 is the learning factor of the PSO algorithm, r1,r2 are random numbers between [0,1], Indicates the update speed of the i-th particle in the k-th generation, the initial speed Set to 0, the maximum value of the speed in column 1 Used to constrain the update of the momentum parameters in the first column and the maximum value of the velocity in the second column Used to constrain the update of the learning rate parameters in the second column. If the speed exceeds the maximum value during the iteration, the speed will be set to the respective maximum value. It represents the optimal value of the individual searched by the i-th individual when iterating to the k-th generation. represents the global optimal value of the entire population when iterating to the kth generation. The value range of weight ω is [ω1, ω2], where ω1 = 0.6 and ω2 = 1.2. In order to avoid the problem of premature maturity in the early stage of PSO algorithm and easy oscillation in the later stage, the weight is updated using the weight update formula so that the weight decreases with the number of iterations. The weight update formula is as follows:

[0085]

[0086] Figure 6 The training and test accuracy and loss curves of the C3D network model before improvement. The number of training rounds was 70, the learning rate was set to 0.0001, the momentum was set to 0.9, the training time was 90.44 minutes, the test time was 15.90 minutes, and the final training set accuracy was 85.88%, and the test set accuracy was 85.69%.

[0087] Figure 7The training and test accuracy and loss curves of the RESC3D network model after the introduction of the residual block, the number of training rounds is still 70 rounds, the learning rate is set to 0.001, the momentum is set to 0.9, the training time is 36.20 minutes, the test time is 12.58 minutes, and the final training set accuracy is 98.11%, and the test set accuracy is 88.30%. Compared with the unimproved C3D network, the training set accuracy is improved by 12.23%, the test set accuracy is improved by 2.61%, and the training time is shortened by 54.24 minutes when the number of training rounds is the same. While shortening a lot of training time, the accuracy is improved, and a lot of time is saved for the subsequent PSO algorithm calculation, but there is still a little overfitting.

[0088] Considering that unreasonable settings of learning rate and momentum may affect the accuracy of the model, in order to improve the accuracy and reduce the overfitting phenomenon, the PSO algorithm is used to optimize the two hyperparameters of learning rate and momentum during network training.

[0089] according to Figure 7 From the training curve, we can see that the model basically converges after about 60 rounds, and the accuracy and loss values ​​remain unchanged. Therefore, 60 rounds are selected as the number of training rounds for each particle of the PSO algorithm. The fitness value of the PSO algorithm is obtained by taking the inverse of the accuracy of the loaded test set after 60 rounds of training of the RESC3D network model. After 10 rounds of optimization by the PSO algorithm, the best learning rate is 0.0009424, the best momentum is 0.81744443, and the total time is 1300.29 minutes. The optimal solution is then used as the hyperparameter of the RESC3D network for another 70 rounds of training and compared with the previous unoptimized results.

[0090] Figure 8 This is the fitness value change curve of the PSO algorithm during the optimization period of 10 iterations. The fitness value of the initial population is -0.89306. The global optimal fitness value drops to -0.90751 from the second iteration to the end of the eighth iteration. From the eighth iteration, the global optimal fitness value converges to -0.92197. It can be judged that the PSO algorithm has basically converged.

[0091] Fig. 9The figure shows the training and testing accuracy and loss curves of the RESC3D network after the PSO algorithm is optimized. The training set accuracy of the final model is 99.52%, and the test set accuracy is 93.50%. The 70 rounds of training took a total of 33.4 minutes, which is 57.04 minutes less than the C3D network. With the 1300.29 minutes of the PSO algorithm, the entire training process took 1333.69 minutes, and the test process took 3.74 minutes. Compared with the model before optimization, the training set accuracy increased by 1.41%, the test set accuracy increased by 5.2%, and the overfitting phenomenon was significantly improved. Compared with the C3D network, the training set accuracy increased by 13.64%, and the test set accuracy increased by 7.81%, and the performance was significantly improved.

[0092] Table 1 shows the comparison of the results of the basic C3D network model, the RESC3D network model and the RESC3D network model optimized by PSO (PSO-RESC3D), which are superior to the C3D network in terms of accuracy and training time.

[0093] This example is written in Python 3.9 and uses the PyTorch deep learning framework. The CPU of the computer used is Intel Core i7-14700KF and the GPU is NVIDIA GeForce RTX 4070TI SUPER.

[0094] Table 1 Comparison of C3D, RESC3D, and PSO-RESC3D model results

[0095] C3D RESC3D PSO-RESC3D Training set accuracy 85.88％ 98.11％ 99.52％ Test set accuracy 85.69％ 88.30％ 93.50％ Training time / min 90.44 36.20 1333.69 Test time / min 15.90 12.58 12.25

[0096] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A C3D convolutional neural network action recognition method, characterized in that: The steps include: (1) The UCF50 dataset was selected as the basic database, and the images in the dataset were processed to ensure that the number of extracted video frames was at least 16. All processed data were divided into a training set and a test set, with a ratio of 8.5:

1. (2) The specific steps for optimizing the C3D network model are as follows: (2a) Instantiate the C3D network model; define the loss function of the C3D network model as the cross entropy loss function, define the optimizer algorithm as the SGD stochastic gradient descent algorithm, and set the two hyperparameters of the optimizer: learning rate and momentum; (2b) Define the learning rate update strategy and output the optimal learning rate and momentum hyperparameters to update the C3D network model; (2c) Input the training set into the C3D network model with optimized hyperparameters, perform forward propagation to obtain the C3D network model output, and calculate the accuracy of the training set; (2d) Calculate the loss using the training set accuracy and true label output by the C3D network model, and then perform back propagation based on the loss value to update the C3D network model parameters; (2e) Load the test set to obtain the test set accuracy; (2f) Determine whether the maximum number of training rounds has been reached. If not, return to step (2c) to continue training. If the maximum number of training rounds has been reached, save the trained model parameters and accuracy values. (3) The test set is input into the optimized C3D network model for forward propagation to obtain the classification results of each image.

2. The C3D convolutional neural network action recognition method according to claim 1, characterized in that: The maximum number of training rounds is 70. In the 50th round, the learning rate is reduced to 0.1 times the previous one. Too large a learning rate may cause the model to oscillate near the optimal solution, or in extreme cases, cause the model to diverge. By reducing the learning rate, the model can converge quickly in the early stage of training and converge to the optimal value more stably in the later stage of training.

3. The C3D convolutional neural network action recognition method according to claim 1, characterized in that: in, The cross entropy loss function expression is as follows: where y i is the actual label value, is the probability value predicted by the model.

4. The C3D convolutional neural network action recognition method according to claim 1, characterized in that: In step (2b), the learning rate update strategy is defined, and the optimal learning rate and momentum hyperparameters are output to update the hyperparameters of the C3D network model. The PSO algorithm is used for parameter iteration. The specific steps are as follows: (2b1) Create the initial PSO algorithm population based on the training set according to the learning rate and momentum parameters; the number of population particles is 3; (2b2) ​​Calculate the fitness, use the position of each particle as the hyperparameter of the C3D network model for 60 rounds of training, and after the training, use the test set test accuracy to take the inverse number as the fitness value to obtain the optimal fitness value of the initial population, and store it as the global optimal fitness value and the corresponding particle position; (2b3) Compare the obtained optimal fitness value with the value of the optimal fitness of the previous generation. If it is less than the value of the optimal fitness of the previous generation, then use the value of the optimal fitness of this generation as the value of the global optimal fitness and store the corresponding position information. Continue to complete the iteration until the maximum number of iterations is reached, and output the optimal learning rate and momentum parameters. If it is not satisfied, iteratively update the particle position and repeat step (2b2) ​​to obtain the optimal fitness and position information of this generation. (2b4) Use the optimal learning rate and momentum parameters output by step (2b3) as the hyperparameters of the RESC3D network model for 70 rounds of training and testing to obtain the optimized accuracy. Use the optimized hyperparameters to replace the original parameters in the RESC3D network model to complete the optimization of the RESC3D network model and perform action recognition.

5. The C3D convolutional neural network action recognition method according to claim 4, characterized in that: In step (2b1), the population of the PSO algorithm is a 3×2 matrix, the first column is the momentum of RESC3D network training, ranging from [0.6, 0.95], the second column is the learning rate of model training, ranging from [0.0001, 0.001], and the three rows represent the positions of each particle.

6. The C3D convolutional neural network action recognition method according to claim 4, characterized in that: In step (2b3), the update formula for iteratively updating the particle position is: Where: is the position of the i-th particle in the k-th generation, k is the current iteration number, d=1,2 is the search dimension, i=1,2,···,5 is the individual number of the population; Indicates the update speed of the i-th particle in the k-th generation; c1=c2=2 is the learning factor of the PSO algorithm; r1, r2 are random numbers between [0,1]; It represents the optimal value of the individual searched by the i-th individual when iterating to the k-th generation; Represents the global optimal value of the entire population when iterating to the kth generation; during the iteration process, Continuously updated; the value range of weight ω is [ω1,ω2].

7. The C3D convolutional neural network action recognition method according to claim 6, characterized in that: Use the weight update formula to update the weight so that the weight decreases with the number of iterations. The weight update formula is as follows:

8. The C3D convolutional neural network action recognition method according to claim 1, characterized in that: Before step (2a), a RESC3D network model based on the C3D network model is constructed, including the following steps: Step A: Build the 3D The deep learning network structure model constructed by CNN includes: multiple convolutional layers, five pooling layers of MaxPool1, MaxPool2, MaxPool3, MaxPool4, and MaxPool5, and two fully connected layers. The kernel size of all convolutional layers is 3×3×3, the padding is 1×1×1, and the stride is 1×1×1; the receptive field of the MaxPool1 maximum pooling layer is 1×2×2, the stride is 1×2×2, and the padding is 0; the receptive field of the MaxPool2, MaxPool3, and MaxPool4 maximum pooling layers is 2×2×2, the stride is 2×2×2, and the padding is 0; the receptive field of the MaxPool5 maximum pooling layer is 2×2×2, the stride is 2×2×2, and the padding is 0×1×1; the number of neurons in the two fully connected layers is 4096; the fully connected layer and all convolutional layers are activated by the ReLU activation function. In order to limit the overfitting effect caused by the huge amount of parameters, a Dropout of 0.5 is added to the two fully connected layers; Step B: Connect two layers of residual blocks in series to multiple convolutional layer input ends of the deep learning network structure model constructed by 3D CNN. Each layer of the residual block includes two convolutional layers of the first channel and one convolutional layer of the second channel, namely Conv1, Conv2 and Conv3. The convolution kernel sizes of Conv1 and Conv2 are both 3×3×3, the padding is 1×1×1, and the stride is 1×1×1. After the convolution, Conv1 and Conv2 are connected to a BN layer for batch normalization. The second channel has only one convolution Conv3, and the convolution kernel size of Conv3 is 1×1×1. The stride is 1×2×2. Conv3 adjusts the input data size so that it can be added to the data of the first channel after being processed by the second BN layer. The BN layer is not connected after convolution to restore the characteristics of the input data as much as possible. Input and Output are the input and output of the residual block respectively. By adding the convolution data of the two channels, the data is reduced in dimension to achieve the purpose of reducing the loss of data features. BN1 and BN2 are batch normalization layers, which can make the distribution of input data in each layer of the network relatively stable and speed up the training and convergence of the model. Step C: Adjust the padding of the third and fifth pooling layers to adapt to the change in data size, add 0×1×1 padding to the pooling layer MaxPool3, adjust the stride of MaxPool4 to 1×1×1, adjust the stride of MaxPool5 to 1×1×1 and remove its padding to adapt to the change in data size, so that the receptive field of the pooling layer MaxPool3 of the final RESC3D network module is 2×2×2, the stride is 2×2×2, and the padding is 0×1×1, the receptive fields of MaxPool4 and MaxPool5 are both 2×2×2, the stride is 1×1×1, and the padding is 0. Step D: Change the number of neurons in the fully connected layer from 4096 to 256 to adapt to the reduction of data. At the same time, due to the reduction of the fully connected layer, the 0.5 dropout rate of the two Dropouts is changed to 0.

25.

9. The C3D convolutional neural network action recognition method according to claim 8, characterized in that: In step (1), in the RESC3D network model, 50 categories in the UCF50 dataset are taken as the dataset. The UCF50 dataset was released by researchers from the University of Central Florida and aims to provide high-quality video data for research in the field of action recognition. The dataset consists of real videos from YouTube and contains more than 6,000 videos of 50 different action categories, covering sports, daily activities, imitation actions, etc. Each action category has enough video samples to ensure the accuracy and generalization ability of model training. Some common action categories include baseball pitching, basketball shooting, bench press, cycling, billiards, breaststroke, clean and jerk, diving, and drumming. These action categories are not only representative, but also can reflect the characteristics and difficulty of different types of sports.

10. The C3D convolutional neural network action recognition method according to claim 8, characterized in that: The input of the RESC3D network model is 3×16×112×112 data read from 16 frames of images sampled from the training set. After being processed by the first layer of residual blocks, the data dimension becomes 16×16×56×56, and then after being processed by the second layer of residual blocks, the data dimension becomes 32×16×28×28. After a series of convolution and pooling operations, the data dimension becomes 512×1×1×1. At this time, the data has 4 dimensions. The data is flattened into one-dimensional data of size 512 and sent to the fully connected layer. The output of the RESC3D network model is the label value calculated by SoftMax through the fully connected layer of 50 neurons; finally, the parameters of the RESC3D network model are reduced from 78409573 parameters before improvement to 27979701, and the parameters are reduced by about 5000w. The reduction in weight parameters will greatly shorten the training time of the model.

Images