Video retrieval method based on three-dimensional convolutional neural network of fusion feature gate

Abstract

A video retrieval method based on a three-dimensional convolutional neural network of a fusion feature gate, which is composed of the steps of dataset preprocessing, dataset division, three-dimensional convolutional neural network construction, three-dimensional convolutional neural network training and three-dimensional convolutional neural network testing. The three-dimensional convolutional neural network is optimized and improved, the technical problem of low retrieval accuracy in the prior art is solved, and a different conception scheme is provided for solving similar problems. Three feature gates are adopted, the feature gate is composed of a reset gate and an update gate, the technical problem of video information redundancy in the prior art is solved, the video feature information can be more accurately extracted, and the foundation for further retrieval is laid. The present application has the advantages of high retrieval accuracy, fast retrieval speed and good retrieval effect, and can be used for video image retrieval.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of communication, and particularly relates to a video retrieval method. BACKGROUND

[0002] Video is a comprehensive medium integrating image, sound and text. With the continuous development of Internet technology, video data grows rapidly. Structured analysis of unstructured video data and extraction of video content features still have great technical difficulties. With the development of deep learning, this problem has gradually been alleviated. However, since video contains not only spatial information but also temporal information, the large amount of data is still a problem currently faced.

[0003] Convolutional neural networks are used to process multi-dimensional data such as images. The basic structure of a traditional convolutional neural network includes an input layer, a pooling layer and an output layer, in addition to an activation layer, a fully connected layer and a standardization layer. Traditional machine learning methods cannot effectively extract requirements in the time dimension. Convolutional neural networks have been widely used and have made breakthroughs in image classification. Convolutional neural networks are suitable for processing parallel data and have unique advantages in image processing. Weight sharing reduces the complexity of the network, and the feature that the image can be directly input into the network avoids the complexity of data reconstruction in the feature extraction and classification process.

[0004] The two-dimensional convolution method is the simplest and most convenient method to solve the problem of static images. It is easy to capture. Two-dimensional convolution can expand the network width by increasing the training parameters, increasing the number of convolution kernels and the spatial dimension to improve the performance of the system. Therefore, two-dimensional convolution neural networks have achieved good results in image classification, target recognition and target detection. However, dynamic recognition and retrieval require analysis of spatial and temporal information, and the network structure cannot well solve this technical problem.

[0005] Various network structure models have been proposed for video analysis using three-dimensional convolution. So far, the application of three-dimensional convolution has become increasingly sophisticated. Tran. D proposed a three-dimensional convolution method C3D network structure. This network structure is applied to video understanding for the first time, and the structure is relatively simple, but the retrieval accuracy is poor.

[0006] In addition, there is an S3D network that separately extracts temporal and spatial domain information, reducing the number of training parameters. This network has relatively low complexity and can improve the system retrieval accuracy, but it cannot handle large-scale video data. SUMMARY

[0007] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art and provide a three-dimensional convolutional neural network video retrieval method based on fusion feature gating, which has low estimation error, strong network stability and complete information extraction.

[0008] The technical scheme adopted to solve the above technical problems comprises the following steps:

[0009] (1) Data set preprocessing

[0010] Take 13320 videos of 5-10 seconds in UCF-101 database as a video data set, divide them into different categories, each category has 25 groups, each group has 4-7 videos with different lengths, label the data set as 0-101 according to the category, as the identification label, extract the optical flow image and color image of the data set, the optical flow image and color image contain x, y, i three-dimensional features, cut the image to 224 pixels, and pre-process the data set.

[0011] (2) Divide the data set

[0012] Divide the video data set into a training set and a test set according to a ratio of 5:1.

[0013] (3) Construct a three-dimensional convolutional neural network

[0014] The three-dimensional convolutional neural network is composed of a three-dimensional convolution module and a feature gating module in series.

[0015] The three-dimensional convolution module is composed of a first basic convolution layer in series with a first maximum pooling layer, a second basic convolution layer, a third basic convolution layer, a second maximum pooling layer, a first 3D convolution module, a first feature gating module, a third maximum pooling layer, a second 3D convolution module, a second feature gating module, a fourth maximum pooling layer, a third 3D convolution module, a third feature gating module, an average pooling layer, and a convolution layer.

[0016] The first feature gating module is composed of a reset gate and an update gate in series, and the reset gate is determined according to the following formula :

[0017] (1)

[0018] wherein, is a nonlinear activation function, is a current input sample, is a hidden state at the last moment, is a coefficient of a reset gate output sample , is a parameter of a reset gate hidden state in training, is a bias of the reset gate, , is an intermediate parameter.

[0019] The update gate is determined according to the following formula :

[0020] (2)

[0021] in, For the current input sample, To hide the previous state To update the gate input sample coefficients, To update the hidden state of the door coefficient, To update the door deviation, These are intermediate parameters.

[0022] The structures of the second and third feature gating are the same as those of the first feature gating.

[0023] (4) Training a three-dimensional convolutional neural network

[0024] The cross-loss function L is determined by the following formula:

[0025] (3)

[0026] in, For the true value of the data, The value can be [0, 1]. For the predicted value of the data, The value can be (0, 1];

[0027] The 3D convolutional neural network was trained in 8 groups, with 32 images per group and 256 images per batch. The training continued until the 3D convolutional neural network converged, and the trained model was used to detect on the test set.

[0028] The evaluation metric is the ratio of the number of correct results for all test data to the total number of test data.

[0029] The accuracy P is determined by the following formula:

[0030] (4)

[0031] Where TP represents the number of correctly retrieved data items by the model, and FP represents the number of incorrectly retrieved data items by the model; the training set is fed into the 3D convolutional neural network for training, and during the training process, the learning rate of the 3D convolutional neural network is... The optimizer used is the SGD optimizer, with a momentum value ranging from 0.1 to 1, iterating until the crossover loss function is reached. convergence.

[0032] (5) Testing the three-dimensional convolutional neural network

[0033] The test set is input into a 3D convolutional neural network with fused feature gating for testing.

[0034] In step (3) of this invention, the first 3D convolutional module is composed of two cascaded 3D convolutional layers, the second 3D convolutional module is composed of five cascaded 3D convolutional layers, and the third 3D convolutional module is composed of two cascaded 3D convolutional layers.

[0035] In the step of constructing a three-dimensional convolutional neural network in (3) of the present invention, the stride of the 3D convolutional layer is 2 and the kernel size is [missing information]. .

[0036] In step (3) of the present invention, the stride of the first max pooling layer is 2 and the kernel size is [missing value]. The structures of the second, third, and fourth max-pooling layers are the same as those of the first max-pooling layer.

[0037] In step (3) of the present invention, the stride of the first feature gate, the second feature gate, and the third feature gate is 1, and the kernel size is [missing value]. The average pooling layer stride is 2, and the convolution kernel size is [missing value]. The stride of the convolutional layer is 1, and the kernel size is [missing value]. .

[0038] This invention optimizes and enhances the three-dimensional convolutional neural network, solving the technical problem of low retrieval accuracy in existing technologies and providing a different conceptual solution for similar problems. It employs three feature gates, consisting of reset and update gates, which resolves the problem of redundant video information in existing technologies, enabling more accurate extraction of video feature information and laying the foundation for further retrieval. This invention boasts advantages such as high retrieval accuracy, fast retrieval speed, and good retrieval results, and can be used for video and image retrieval. Attached Figure Description

[0039] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0040] Figure 2 This is a schematic diagram of the three-dimensional convolutional neural network structure in Example 1. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0042] Example 1

[0043] exist Figure 1In this embodiment, the video retrieval method based on a 3D convolutional neural network with fusion feature gating consists of the following steps:

[0044] (1) Dataset preprocessing

[0045] A video dataset of 13,320 videos (5-10 seconds each) from the UCF-101 database was used. These were divided into different categories, with 25 groups per category and 4-7 videos of varying lengths per group. The datasets were labeled 0-101 according to their categories as identification tags. Optical flow and color images were extracted from the datasets. These images contain features in three dimensions: x, y, and i. 224 images were then cropped from each image. 224 pixels, preprocessing the dataset.

[0046] (2) Split the dataset

[0047] The video dataset was divided into training and testing sets in a 5:1 ratio.

[0048] (3) Constructing a three-dimensional convolutional neural network

[0049] A three-dimensional convolutional neural network is composed of a three-dimensional convolutional module and a feature gating module connected in series.

[0050] The three-dimensional convolutional module in this embodiment is composed of a first basic convolutional layer connected in series with a first max pooling layer, a second basic convolutional layer, a third basic convolutional layer, a second max pooling layer, a first 3D convolutional module, a first feature gating module, a third max pooling layer, a second 3D convolutional module, a second feature gating module, a fourth max pooling layer, a third 3D convolutional module, a third feature gating module, an average pooling layer, and a convolutional layer.

[0051] In this embodiment, the first 3D convolutional module consists of two cascaded 3D convolutional layers, the second 3D convolutional module consists of five cascaded 3D convolutional layers, and the third 3D convolutional module consists of two cascaded 3D convolutional layers.

[0052] In this embodiment, the stride of the 3D convolutional layer is 2, and the kernel size is [missing value]. .

[0053] In this embodiment, the stride of the first max pooling layer is 2, and the kernel size is [missing value]. The structures of the second, third, and fourth max-pooling layers are the same as those of the first max-pooling layer.

[0054] In this embodiment, the stride of the first feature gate, the second feature gate, and the third feature gate is 1, and the convolution kernel size is [missing value]. The average pooling layer stride is 2, and the convolution kernel size is [missing value]. The stride of the convolutional layer is 1, and the kernel size is [missing value]. .

[0055] The first feature gate control module in this embodiment consists of a reset gate and an update gate connected in series. The reset gate is determined by the following formula. :

[0056] (1)

[0057] in, It is a non-linear activation function. For the current input sample, This is the hidden state from the previous moment. To reset the gate output sample coefficient, To reset the parameters of the hidden state of the door during training, To reset the door's deviation, , For intermediate parameters;

[0058] Determine the update gate using the following formula :

[0059] (2)

[0060] in, For the current input sample, To hide the previous state To update the gate input sample coefficients, To update the hidden state of the door coefficient, To update the door deviation, These are intermediate parameters.

[0061] The structures of the second and third feature gating are the same as those of the first feature gating.

[0062] (4) Training a three-dimensional convolutional neural network

[0063] The cross-loss function L is determined by the following formula:

[0064] (3)

[0065] in, For the true value of the data, The value can be [0, 1]. For the predicted value of the data, The value is (0, 1], in this embodiment The value is 0.5. The value is 0.5.

[0066] The 3D convolutional neural network was trained in 8 groups, with 32 images per group and 256 images per batch. The training continued until the 3D convolutional neural network converged, and the trained model was used to detect on the test set.

[0067] The evaluation metric is the ratio of the number of correct results to the total number of test data.

[0068] The accuracy P is determined by the following formula:

[0069] (4)

[0070] Where TP represents the number of correctly retrieved data items by the model, and FP represents the number of incorrectly retrieved data items by the model; the training set is fed into the 3D convolutional neural network for training, and during the training process, the learning rate of the 3D convolutional neural network is... In this embodiment The value is 10 -3 The optimizer uses the SGD optimizer, with a momentum value ranging from 0.1 to 1. In this embodiment, the momentum value of the SGD optimizer is 0.5, and the iteration proceeds to the crossover loss function. convergence.

[0071] (5) Testing a 3D convolutional neural network with fused feature gating

[0072] The test set is input into a 3D convolutional neural network with fused feature gating for testing.

[0073] A video retrieval method based on a 3D convolutional neural network with fusion feature gating was developed.

[0074] Example 2

[0075] The video retrieval method based on a 3D convolutional neural network with fusion feature gating in this embodiment consists of the following steps:

[0076] (1) Dataset preprocessing

[0077] The steps are the same as in Example 1.

[0078] (2) Split the dataset

[0079] The steps are the same as in Example 1.

[0080] (3) Constructing a three-dimensional convolutional neural network

[0081] The steps are the same as in Example 1.

[0082] (4) Training a three-dimensional convolutional neural network

[0083] The cross-loss function L is determined by the following formula:

[0084] (3)

[0085] in, For the true value of the data, The value can be [0, 1]. For the predicted value of the data, The value is (0, 1], in this embodiment The value is 0. The value is 0.1.

[0086] The 3D convolutional neural network was trained in 8 groups, with 32 images per group and 256 images per batch. The training continued until the 3D convolutional neural network converged, and the trained model was used to detect on the test set.

[0087] The evaluation metric is the ratio of the number of correct results to the total number of test data.

[0088] The accuracy P is determined by the following formula:

[0089] (4)

[0090] Where TP represents the number of correctly retrieved data items by the model, and FP represents the number of incorrectly retrieved data items by the model; the training set is fed into the 3D convolutional neural network for training, and during the training process, the learning rate of the 3D convolutional neural network is... In this embodiment The value is 10 -4 The optimizer uses the SGD optimizer, whose momentum value is between 0.1 and 1. In this embodiment, the momentum value of the SGD optimizer is 0.1. Iteration is performed until the crossover loss function is reached. convergence.

[0091] The other steps are the same as in Example 1.

[0092] A video retrieval method based on a 3D convolutional neural network with fusion feature gating was developed.

[0093] Example 3

[0094] The video retrieval method based on a 3D convolutional neural network with fusion feature gating in this embodiment consists of the following steps:

[0095] (1) Dataset preprocessing

[0096] The steps are the same as in Example 1.

[0097] (2) Split the dataset

[0098] The steps are the same as in Example 1.

[0099] (3) Constructing a three-dimensional convolutional neural network

[0100] The steps are the same as in Example 1.

[0101] (4) Training a three-dimensional convolutional neural network

[0102] The cross-loss function L is determined by the following formula:

[0103] (3)

[0104] in, For the true value of the data, The value can be [0, 1]. For the predicted value of the data, The value is (0, 1], in this embodiment The value is 1. The value is 1.

[0105] The 3D convolutional neural network was trained in 8 groups, with 32 images per group and 256 images per batch. The training continued until the 3D convolutional neural network converged, and the trained model was used to detect on the test set.

[0106] The evaluation metric is the ratio of the number of correct results to the total number of test data.

[0107] The accuracy P is determined by the following formula:

[0108] (4)

[0109] Where TP represents the number of correctly retrieved data items by the model, and FP represents the number of incorrectly retrieved data items by the model; the training set is fed into the 3D convolutional neural network for training, and during the training process, the learning rate of the 3D convolutional neural network is... In this embodiment The value is 10 -2 The optimizer uses the SGD optimizer, whose momentum value ranges from 0.1 to 1. In this embodiment, the momentum value of the SGD optimizer is 1. Iteration is performed until the crossover loss function is reached. convergence.

[0110] The other steps are the same as in Example 1.

[0111] A video retrieval method based on a 3D convolutional neural network with fusion feature gating was developed.

[0112] Example 4

[0113] In the above embodiments 1, 2, and 3, the video retrieval method based on a 3D convolutional neural network with fused feature gating consists of the following steps:

[0114] (1) Dataset preprocessing

[0115] The steps are the same as in Example 1.

[0116] (2) Split the dataset

[0117] The steps are the same as in Example 1.

[0118] (3) Constructing a three-dimensional convolutional neural network

[0119] The steps are the same as in Example 1.

[0120] (4) Training a three-dimensional convolutional neural network

[0121] The cross-loss function L is determined by the following formula:

[0122] (3)

[0123] in, For the true value of the data, The value can be [0, 1]. For the predicted value of the data, The value is (0, 1], in this embodiment The value is 0. The value is 1.

[0124] The 3D convolutional neural network was trained in 8 groups, with 32 images per group and 256 images per batch. The training continued until the 3D convolutional neural network converged, and the trained model was used to detect on the test set.

[0125] The other steps in this procedure are the same as in the corresponding embodiments.

[0126] The other steps are the same as in Example 1. This completes the video retrieval method based on a 3D convolutional neural network with fused feature gating.

[0127] Example 5

[0128] In the above embodiments 1, 2, and 3, the video retrieval method based on a 3D convolutional neural network with fused feature gating consists of the following steps:

[0129] (1) Dataset preprocessing

[0130] The steps are the same as in Example 1.

[0131] (2) Split the dataset

[0132] The steps are the same as in Example 1.

[0133] (3) Constructing a three-dimensional convolutional neural network

[0134] The steps are the same as in Example 1.

[0135] (4) Training a three-dimensional convolutional neural network

[0136] The cross-loss function L is determined by the following formula:

[0137] (3)

[0138] in, For the true value of the data, The value can be [0, 1]. For the predicted value of the data, The value is (0, 1], in this embodiment The value is 1. The value is 0.

[0139] The 3D convolutional neural network was trained in 8 groups, with 32 images per group and 256 images per batch. The training continued until the 3D convolutional neural network converged, and the trained model was used to detect on the test set.

[0140] The other steps in this procedure are the same as in the corresponding embodiments.

[0141] The other steps are the same as in Example 1. This completes the video retrieval method based on a 3D convolutional neural network with fused feature gating.

[0142] To verify the beneficial effects of the present invention, the inventors conducted comparative experiments using the method of Embodiment 1 of the present invention with the Convolutional3D method (abbreviated as C3D in the experiment), the Sperational 3D method (abbreviated as S3D in the experiment), and the Inflated 3D method (abbreviated as I3D in the experiment). The method was trained on sports, action, and image / video datasets (abbreviated as Sport-1m, Kinetics-400, Kinetics / ImageNet in the experiment), and the parameters were fine-tuned. Color images and optical flow images (abbreviated as rgb and flow in the experiment) were input respectively and experiments were conducted on the UCF-101 dataset. The retrieval accuracy was calculated according to formula (4). The experimental and calculation results are shown in Table 1.

[0143] Table 1. Experimental results of Example 1 of the present invention and the comparative method.

[0144]

[0145] As shown in Table 1, when the input is a color image, the retrieval accuracy of the method of the present invention is 0.94, which is 0.23 higher than the C3D method, 0.21 higher than the S3D method, and 0.08 higher than the I3D method. When the input is an optical flow image, the retrieval accuracy of the method of the present invention is 0.96, which is 0.05 higher than the I3D method. The method of Embodiment 1 of the present invention has high retrieval accuracy, fast retrieval speed, and good retrieval effect.

Claims

1. A video retrieval method based on a three-dimensional convolutional neural network with fusion feature gating, characterized in that... It consists of the following steps: (1) Dataset preprocessing A video dataset of 13,320 videos (5-10 seconds each) from the UCF-101 database was used. These videos were divided into different categories, with 25 groups per category and 4-7 videos per group. The dataset was labeled 0-101 according to the category, serving as identification labels. Optical flow and color images were extracted from the dataset. These images contain features in three dimensions: x, y, and i. 224-bit images were then extracted. 224 pixels, preprocess the dataset; (2) Split the dataset The video dataset was divided into training and testing sets in a 5:1 ratio. (3) Constructing a three-dimensional convolutional neural network A three-dimensional convolutional neural network is composed of a three-dimensional convolutional module and a feature gating module connected in series. The aforementioned 3D convolutional module is composed of a first basic convolutional layer connected in series with a first max pooling layer, a second basic convolutional layer, a third basic convolutional layer, a second max pooling layer, a first 3D convolutional module, a first feature gating module, a third max pooling layer, a second 3D convolutional module, a second feature gating module, a fourth max pooling layer, a third 3D convolutional module, a third feature gating module, an average pooling layer, and a convolutional layer. The first feature gate control module consists of a reset gate and an update gate connected in series. The reset gate is determined by the following formula. : （1） in, It is a non-linear activation function. For the current input sample, This is the hidden state from the previous moment. To reset the gate output sample coefficient, To reset the parameters of the hidden state of the door during training, To reset the door's deviation, , For intermediate parameters; Determine the update gate using the following formula : （2） in, For the current input sample, To hide the state from the previous moment To update the gate input sample coefficients, To update the hidden state of the door coefficient, To update the door deviation, For intermediate parameters; The structures of the second and third feature gating are the same as those of the first feature gating. (4) Training a three-dimensional convolutional neural network The cross-loss function L is determined by the following formula: （3） in, For the true value of the data, The value can be [0, 1]. For the predicted value of the data, The value can be (0, 1]; The 3D convolutional neural network was trained in 8 groups, with 32 frames per group and 256 frames per batch. The training continued until the 3D convolutional neural network converged. The trained model was then used to detect on the test set. The evaluation metric is the ratio of the number of correct results to the total number of test data. The accuracy P is determined by the following formula: （4） Where TP represents the number of correctly retrieved data items by the model, and FP represents the number of incorrectly retrieved data items by the model; the training set is fed into the 3D convolutional neural network for training, and during the training process, the learning rate of the 3D convolutional neural network is... The optimizer used is the SGD optimizer, with a momentum value ranging from 0.1 to 1, iterating until the crossover loss function is reached. convergence; (5) Testing the three-dimensional convolutional neural network The test set is input into a 3D convolutional neural network with fused feature gating for testing.

2. The video retrieval method based on a three-dimensional convolutional neural network with fusion feature gating according to claim 1, characterized in that: In step (3), the first 3D convolutional module consists of two cascaded 3D convolutional layers, the second 3D convolutional module consists of five cascaded 3D convolutional layers, and the third 3D convolutional module consists of two cascaded 3D convolutional layers.

3. The video retrieval method based on a three-dimensional convolutional neural network with fusion feature gating according to claim 2, characterized in that: In step (3) of constructing the three-dimensional convolutional neural network, the stride of the 3D convolutional layer is 2 and the kernel size is [missing information]. .

4. The video retrieval method based on a three-dimensional convolutional neural network with fusion feature gating according to claim 1, characterized in that: In step (3), the stride of the first max pooling layer is 2, and the kernel size is [missing value]. The structures of the second, third, and fourth max-pooling layers are the same as those of the first max-pooling layer.

5. The video retrieval method based on a three-dimensional convolutional neural network with fusion feature gating according to claim 1, characterized in that: In step (3), the stride of the first feature gate, the second feature gate, and the third feature gate is 1, and the kernel size is [missing value]. The average pooling layer stride is 2, and the convolution kernel size is [missing value]. The stride of the convolutional layer is 1, and the kernel size is [missing value]. .

Images