Video feature extraction method, device, readable storage medium and computer equipment
By adopting a combination method of one-dimensional, two-dimensional and three-dimensional convolution processing in video feature extraction, the problem of large amount of calculation in traditional three-dimensional convolution kernel is solved, and efficient video feature extraction is achieved.
Patent Information
- Application Number
- CN201910987655.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-10-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-10-17
AI Technical Summary
Traditional video processing methods are based on three-dimensional convolution kernels, which leads to huge computational volume and makes it difficult to efficiently extract video features.
The convolution layer of the feature extraction model performs one-dimensional convolution processing in the time domain, followed by two-dimensional separation convolution processing in the spatial domain, and then performs three-dimensional convolution processing to reduce the calculation amount and improve feature extraction efficiency.
It effectively reduces the amount of calculation in the video feature extraction process, improves the efficiency of feature extraction, and reduces the consumption of computing resources.
Smart Images

Figure CN110705513B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a video feature extraction method, device, computer-readable storage medium and computer equipment. Background Art
[0002] With the development of computer technology, video capture and video processing technologies have emerged. Video can be seen as a combination of multiple image data along time, and a picture in a video is called a frame. Video processing technology can be used to perform a series of processing on the captured video so that the video can meet various user needs. For example, it can achieve different types of video tasks such as video classification, video segmentation, and video description.
[0003] Traditional video processing is generally based on a 3D convolution network with a 3D convolution kernel, which obtains feature abstraction of video content by stacking convolution layers. It may take trillions of operations to process a video, so the video convolution method based on the 3D convolution kernel will bring a huge amount of calculation. Summary of the invention
[0004] Based on this, it is necessary to provide a video feature extraction method, apparatus, computer-readable storage medium and computer equipment to address the technical problem of large computational complexity of traditional feature extraction.
[0005] A video feature extraction method, comprising:
[0006] Get the video to be processed;
[0007] Performing one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain a first video feature;
[0008] Performing a two-dimensional separable convolution process on the first video feature in a spatial domain to obtain a second video feature;
[0009] Perform three-dimensional convolution processing on the second video feature to obtain a target video feature.
[0010] A video feature extraction device, comprising:
[0011] An acquisition module is used to acquire the video to be processed;
[0012] A one-dimensional convolution module, used for performing one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain a first video feature;
[0013] A two-dimensional convolution module, used for performing a two-dimensional separable convolution process on the first video feature in a spatial domain to obtain a second video feature;
[0014] The three-dimensional convolution module is used to perform three-dimensional convolution processing on the second video feature to obtain a target video feature.
[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the following steps:
[0016] Get the video to be processed;
[0017] Performing one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain a first video feature;
[0018] Performing a two-dimensional separable convolution process on the first video feature in a spatial domain to obtain a second video feature;
[0019] Perform three-dimensional convolution processing on the second video feature to obtain a target video feature.
[0020] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps:
[0021] Get the video to be processed;
[0022] Performing one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain a first video feature;
[0023] Performing a two-dimensional separable convolution process on the first video feature in a spatial domain to obtain a second video feature;
[0024] Perform three-dimensional convolution processing on the second video feature to obtain a target video feature.
[0025] The above-mentioned video feature extraction method, device, computer-readable storage medium and computer equipment obtain a video to be processed, perform one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain a first video feature, perform two-dimensional separable convolution processing on the first video feature in the spatial domain to obtain a second video feature, and perform three-dimensional convolution processing on the second video feature to obtain a target video feature, which can reduce the amount of calculation in the video feature extraction process and improve the efficiency of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 A diagram showing an application environment of a video feature extraction method in an embodiment;
[0027] Figure 2 Schematic diagram of a process of extracting video features in one embodiment;
[0028] Figure 3 A schematic diagram of a two-dimensional convolution kernel performing convolution processing in one embodiment;
[0029] Figure 4 A schematic diagram of performing one-dimensional convolution, two-dimensional convolution, and three-dimensional convolution processing in one embodiment;
[0030] Figure 5 A schematic diagram of a flow chart of steps for detecting a target object category in one embodiment;
[0031] Figure 6 A flowchart of steps for generating a video description in one embodiment;
[0032] Figure 7 A flowchart of steps for generating a video description in another embodiment;
[0033] Figure 8 A schematic flow chart of video segmentation steps in one embodiment;
[0034] Fig. 9 A schematic diagram of a flow chart of the steps of training a feature extraction model in one embodiment;
[0035] Fig.10 A schematic diagram of a process of extracting video features in another embodiment;
[0036] Fig.11 is a structural block diagram of a video feature extraction device in another embodiment;
[0037] Fig.12 FIG. 4 is a structural block diagram of a computer device in one embodiment. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0039] Figure 1 FIG. 1 is an application environment diagram of a video feature extraction method in an embodiment. Figure 1, the video feature extraction method is applied to a feature extraction system. The feature extraction system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected via a network. The terminal 110 may be a desktop terminal or a mobile terminal, and the mobile terminal may be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 may be implemented by an independent server or a server cluster consisting of multiple servers. In this embodiment, the feature extraction model may be stored on the server 120, and the terminal 110 may obtain a video to be processed and send the video to be processed to the server 120. The server 120 receives the video to be processed, performs a one-dimensional convolution process on the video to be processed in the time domain through the convolution layer of the feature extraction model, obtains a first video feature, performs a two-dimensional separable convolution process on the first video feature in the spatial domain, obtains a second video feature, and performs a three-dimensional convolution process on the second video feature to obtain a target video feature. Then, the server 120 returns the target video feature to the terminal 110.
[0040] In this embodiment, the feature extraction model may exist in the terminal 110. The terminal 110 may obtain a video to be processed, perform a one-dimensional convolution process on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain a first video feature, perform a two-dimensional separable convolution process on the first video feature in the spatial domain to obtain a second video feature, and perform a three-dimensional convolution process on the second video feature to obtain a target video feature. This solution can reduce the amount of calculation in the feature extraction process and improve the efficiency of video feature extraction.
[0041] like Figure 2 As shown, in one embodiment, a video feature extraction method is provided. This embodiment mainly applies the method to the above Figure 1 The terminal 110 in FIG. Figure 2 , the video feature extraction method specifically includes the following steps:
[0042] Step 202: Obtain the video to be processed.
[0043] The video to be processed refers to the video for which features need to be extracted. The video to be processed may be a video stored locally on the terminal, a video received from other devices, a video obtained from the network, or a video captured in real time by the terminal.
[0044] Specifically, the terminal may obtain the video to be processed from a local device or other device or network, or may obtain the video to be processed by real-time shooting.
[0045] Step 204: Perform one-dimensional convolution processing in the time domain on the video to be processed through the convolution layer of the feature extraction model to obtain a first video feature.
[0046] Among them, the time domain refers to the time dimension, and the independent variable is time, that is, the horizontal axis is time and the vertical axis is the change of the signal. The feature extraction model is trained based on the training video samples and the labels of the training video samples. The first video feature refers to the feature information on the time dimension extracted from the video to be processed. One-dimensional convolution processing refers to convolution calculation of the channel features through a one-dimensional convolution kernel.
[0047] Specifically, the terminal can input the video to be processed into the feature extraction model, and perform one-dimensional convolution processing on the video to be processed in the time dimension through the convolution layer of the feature extraction model to extract feature information of the video to be processed in the time dimension. After the one-dimensional convolution processing, the first video feature of each output channel is obtained.
[0048] In this embodiment, the terminal may perform feature extraction on the processed video through the current convolution layer of the feature extraction model to obtain features corresponding to each input channel of the next layer of the current convolution layer. When the current convolution layer is the target convolution layer, the features corresponding to each input channel are subjected to one-dimensional convolution processing in the time domain through the target convolution layer to obtain the first video feature.
[0049] Furthermore, when the current convolution layer is not the target convolution layer, the feature maps of each input channel of the convolution layer can be convolved with the convolution kernels of each input channel of the convolution layer to obtain the first video features corresponding to each input channel of the next layer of the convolution layer, wherein one convolution kernel corresponds to one input channel of the next layer.
[0050] For example, when all the convolution layers from the input layer to the third layer are non-target convolution layers, each convolution layer from the first layer to the third layer takes the convolution kernel of each input channel and the features of each input channel for convolution calculation, and obtains the features of the corresponding input channel of the next layer. An output channel in one layer is an input channel of the next layer. An input channel in one layer can be convolved with multiple convolution kernels of the input channel. Therefore, the convolution calculation of a convolution kernel and an input channel can correspond to the features of an input channel of the next layer. It can be seen that the relationship between the current layer channel and the next layer channel can be one-to-one or one-to-many. Therefore, by calculating layer by layer, more and more channel subsets, that is, more and more features, will be obtained. The more features in each layer, the finer the feature description, so that it can better represent the characteristics of the video to be processed, which is conducive to the subdivision of individual differences.
[0051] Step 206: Perform a two-dimensional separable convolution process on the first video feature in the spatial domain to obtain a second video feature.
[0052] The spatial domain is also called image space, which is a space composed of image pixels. Directly processing pixel values in the image space with height and width as independent variables is called spatial domain processing. The second video feature refers to the feature information in the spatial domain extracted from the first video feature. Two-dimensional separable convolution processing refers to convolution calculation of channel features through a two-dimensional convolution kernel.
[0053] Specifically, the feature information of the video to be processed in the time dimension is extracted through one-dimensional convolution processing, that is, the first video feature. Then, the terminal can input the first video feature into the next layer of the convolution layer for two-dimensional convolution processing to extract the feature information of the video to be processed in the spatial domain. After the two-dimensional convolution processing of the first video feature, the second video feature can be obtained, which carries the feature information in the independent time dimension and the information in the independent space dimension.
[0054] Step 208: Perform three-dimensional convolution processing on the second video feature to obtain the target video feature.
[0055] The target video feature refers to the video feature information obtained by fusing the feature information in the time domain and the spatial domain. The three-dimensional convolution processing refers to performing convolution calculation on the video features of the channel through a simple three-dimensional convolution kernel. For example, the three-dimensional convolution kernel can be a simple 1*1*1 convolution kernel.
[0056] Specifically, the terminal performs three-dimensional convolution processing on the second video feature that carries feature information in an independent time dimension and information in an independent space dimension, so as to fuse the feature information in the time domain and the space domain to obtain the target video feature.
[0057] The above-mentioned video feature extraction method obtains a video to be processed, performs one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain a first video feature, performs two-dimensional separable convolution processing on the first video feature in the spatial domain to obtain a second video feature, and performs three-dimensional convolution processing on the second video feature to obtain a target video feature, which can reduce the amount of calculation in the feature extraction process and improve the efficiency of feature extraction.
[0058] In one embodiment, the convolution layer of the feature extraction model performs one-dimensional convolution processing on the video to be processed in the time domain to obtain the first video feature, including:
[0059] The video to be processed is input into a feature extraction model to obtain a one-dimensional convolution kernel of a convolution layer of the feature extraction model; the features of each channel of the video to be processed are convolved in the time domain through each one-dimensional convolution kernel to obtain first video features corresponding to each input channel of the next layer of the convolution layer.
[0060] Among them, a one-dimensional convolution kernel refers to a convolution kernel whose width (height) in the spatial dimension is 1, and whose width in the time dimension may not be 1. For example, a one-dimensional convolution kernel of M*1*1, where M is the width of the convolution kernel in the time dimension.
[0061] Specifically, the terminal inputs the video to be processed into the convolution layer of the feature extraction model, and the feature extraction model obtains the one-dimensional convolution kernel of the convolution layer, and performs convolution calculation on the features of each channel of the video to be processed through a one-dimensional convolution kernel to obtain the video features of each output channel, and fuses the video features of each output channel to obtain a first video feature. The first video feature is the feature output by a one-dimensional convolution kernel. Then, the same processing is performed on each one-dimensional convolution kernel in the convolution layer to obtain the first video feature corresponding to the output of each one-dimensional convolution kernel. The first video feature corresponding to the output of each one-dimensional convolution kernel is used as the first video feature corresponding to each input channel of the next layer of the convolution layer. It can be understood that the first video features corresponding to each output channel of the previous convolution layer are the first video features corresponding to each input channel of the next layer.
[0062] For example, a color video has 3 channels, and there are 10 one-dimensional convolution kernels in the convolution layer. One one-dimensional convolution kernel performs convolution calculation on the video features of each of the three channels, and obtains the features of the three channels after convolution calculation respectively. The three features are fused to obtain a video feature and output to obtain the first video feature. The first video feature is the video feature output by the one-dimensional convolution kernel. The same process is performed on the remaining 9 one-dimensional convolution kernels to obtain 9 first video features output by each of the 9 one-dimensional convolution kernels. Then, the total of 10 first video features are used as the features of the input channel of the next layer of the convolution layer.
[0063] In this embodiment, a mapping relationship between each group of one-dimensional convolution kernels and each group of input channels is obtained, and a group of one-dimensional convolution kernels corresponds to a group of input channels; according to the mapping relationship between each group of one-dimensional convolution kernels and each group of input channels, each group of one-dimensional convolution kernels is respectively convolved with the features of the video to be processed of the corresponding group of input channels in the time domain to obtain the first video features of each group of output channels.
[0064] Specifically, the one-dimensional convolution kernels of the convolution layer in the feature extraction model are divided into multiple groups, and the input channels of the same convolution layer are also divided into multiple groups. The number of groups into which the one-dimensional convolution kernels of the same layer are divided is the same as the number of groups into which the input channels of the same layer are divided. In addition, the mapping relationship between each group of one-dimensional convolution kernels and each group of input channels is pre-constructed in the feature extraction model, and a group of one-dimensional convolution kernels corresponds to a group of input channels. The terminal can obtain the mapping relationship between each group of one-dimensional convolution kernels and each group of input channels in a layer. According to the mapping relationship between each group of one-dimensional convolution kernels and each group of input channels, each group of convolution kernels is respectively subjected to group convolution processing with the video features of the corresponding groups of input channels, and the first video features corresponding to each output channel in a group of output channels of the layer can be obtained. It can be understood that the number of a group of convolution kernels in the same convolution layer is equal to the number of a group of output channels. The first video features of each group of output channels are used as the first video features of each input channel of the next layer.
[0065] In this embodiment, the size of the one-dimensional convolution kernel may be M*1*1, where M is the width of the convolution kernel in the time dimension. For example, the size of the video to be processed is T*W*H*K, T is the time length of the video to be processed, W and H are the width and height of each image frame in the video to be processed, and K is the number of channels (for example, the number of channels for color video is 3). The video to be processed or the feature of the video to be processed of size T*W*H*K is first passed through X one-dimensional convolution kernels of size M*1*1 with K channels, and the output size after the one-dimensional convolution is T*W*H*X. Therefore, the multiplication and addition calculation amount of the one-dimensional convolution is 2*T*W*H*K*M*X, the parameter amount is K*M*X, and the output video feature size is T*W*H*X. The one-dimensional convolution performs convolution calculation on the time channel of the video or video feature, thereby completing the extraction of the time dimension information of the video or video feature by the current convolution layer. Therefore, the one-dimensional convolution uses a convolution kernel with a window width of M*1*1 to perform feature calculation and information extraction on a signal with a length of T.
[0066] The video feature extraction method in this embodiment, by inputting the video to be processed into a feature extraction model, obtains the one-dimensional convolution kernel of the convolution layer of the feature extraction model, and performs convolution processing on the features of each channel of the video to be processed in the time domain through each one-dimensional convolution kernel, to obtain the first video features corresponding to each input channel of the next layer of the convolution layer, so as to separately extract the feature information of the video to be processed in the time dimension and reduce the number of parameters for feature extraction.
[0067] In one embodiment, the first video feature is subjected to a two-dimensional separable convolution process in a spatial domain to obtain a second video feature, including:
[0068] A first mapping relationship between each two-dimensional separable convolution kernel and each input channel is obtained, where one two-dimensional separable convolution kernel corresponds to one input channel; and according to the first mapping relationship, each two-dimensional separable convolution kernel is convolved with the first video feature of the corresponding input channel in the spatial domain to obtain the second video features corresponding to the output channels of each two-dimensional separable convolution kernel.
[0069] The first mapping relationship refers to the correspondence between a two-dimensional separable convolution and an input channel. The number of channels of a two-dimensional separable convolution kernel is 1, and the width of the convolution kernel in the time dimension is 1. For example, a convolution kernel with a channel number of 1 and a size of 1*N*N is a two-dimensional separable convolution kernel, where N is the width (height) of the spatial dimension of the convolution kernel.
[0070] Specifically, the terminal can pre-construct a mapping relationship between a two-dimensional separable convolution kernel and an input channel, so that one two-dimensional separable convolution kernel corresponds to only one input channel, and the two-dimensional separable convolution kernel only performs convolution processing on the features of the input channel with the mapping relationship, that is, one-to-one convolution processing. After obtaining the first video features of each output channel of the previous layer, the first video features of each output channel are used as the second video features of each input channel of the next layer. The terminal obtains the first mapping relationship between each two-dimensional separable convolution kernel and each input channel, and according to the first mapping relationship, performs convolution calculations in the spatial domain on each two-dimensional separable convolution kernel and the first video features of the corresponding input channel, respectively, to obtain the second video features corresponding to the output channels of each two-dimensional separable convolution kernel.
[0071] For example, after one-dimensional convolution, a video feature of size T*W*H*X will be convolved by X two-dimensional separable convolution kernels of size 1*N*N, where N is the width (height) of the spatial dimension of the convolution kernel. The traditional two-dimensional convolution kernel is a convolution kernel of size 1*N*N with X channels of k. When each convolution kernel of size k convolves the input feature, it will calculate the input features of K channels. Therefore, the multiplication and addition calculation amount for the above-mentioned input features is 2*T*W*H*X*N*N*k, the parameter amount is X*N*N*k, and the output feature size is T*W*H*k. Therefore, the amount of calculation is large and consumes a lot of computing resources. The two-dimensional separable convolution kernel in this embodiment is a convolution kernel of size 1*N*N with X channels of size 1*N*N, which performs independent convolution operations on the channels of the input features, and each convolution kernel of size 1*N*N with a channel number of 1 only corresponds to one dimension of the input feature and performs calculations. For example, the 1*N*N 2D separable convolution kernel with the first channel number of 1 corresponds to the video features of the first channel, and the output size is T*W*H*1; the 1*N*N 2D separable convolution kernel with the second channel number of 1 corresponds to the video features of the second channel, and the output size is T*W*H*1, and so on. Therefore, the amount of multiplication and addition calculations for the above input features is only 2*T*W*H*X*N*N, the number of parameters is X*N*N, and the output feature size is T*W*H*X. Therefore, the amount of calculation and the number of parameters of the 2D separable convolution are both 1 / X of the traditional 2D convolution kernel, but have the same output feature size. Although the amount of calculation and the number of parameters of the 2D separable convolution kernel are less than those of the traditional 2D convolution kernel, the 2D separable convolution kernel can still extract the spatial features of the video and obtain independent spatial structure information of each dimension of the input video or video features.
[0072] like Figure 3 As shown, Figure 3 (d) in the figure is a schematic diagram of the traditional two-dimensional convolution kernel for convolution processing. Figure 3 As can be seen from (d) in the figure, each convolution kernel has 3 channels. When each convolution kernel performs convolution on the input features, it will calculate the input features of 3 channels. Figure 3 (e) in FIG. 1 is a schematic diagram of a two-dimensional separable convolution kernel performing convolution processing in this embodiment. Figure 3 As can be seen from (e) in the figure, each convolution kernel has only one channel. When each convolution kernel performs convolution on the input features, it only needs to calculate the input features of one channel, which significantly reduces the number of parameters and calculations.
[0073] The above-mentioned video feature extraction method obtains a first mapping relationship between each two-dimensional separable convolution kernel and each input channel, and one two-dimensional separable convolution kernel corresponds to one input channel. According to the first mapping relationship, each two-dimensional separable convolution kernel is convolved with the first video feature of the corresponding input channel in the spatial domain to obtain the second video features corresponding to the output channels of each two-dimensional separable convolution kernel. Through the one-to-one convolution processing of the two-dimensional separable convolution kernel and the input channel, the independent feature information of the video to be processed in the spatial domain can be extracted. A two-dimensional separable convolution kernel is convolved with the feature of only one input channel, which can reduce the number of parameters and the amount of calculation of the convolution processing.
[0074] In one embodiment, the first video feature is subjected to a two-dimensional separable convolution process in a spatial domain to obtain a second video feature, including:
[0075] Obtain a second mapping relationship between each group of two-dimensional separable convolution kernels and each group of input channels; perform convolution processing on each group of two-dimensional separable convolution kernels with the first video features of each corresponding group of input channels in the spatial domain according to the second mapping relationship to obtain second video features of each group of output channels, wherein a group of two-dimensional separable convolution kernels corresponds to a group of input channels.
[0076] Among them, the first mapping relationship refers to the correspondence between a set of two-dimensional separable convolution kernels and a set of input channels.
[0077] Specifically, the two-dimensional separable convolution kernels of the convolution layer in the feature extraction model are divided into multiple groups, and the input channels of the same convolution layer are also divided into multiple groups. The number of groups into which the two-dimensional separable convolution kernels of the same layer are divided is the same as the number of groups into which the input channels of the same layer are divided. In addition, the mapping relationship between each group of two-dimensional separable convolution kernels and each group of input channels is pre-constructed in the feature extraction model, and a group of two-dimensional separable convolution kernels corresponds to a group of input channels. After obtaining the first video features corresponding to each output channel of the previous layer, the convolution processing of the next layer is entered, and the terminal can obtain the second mapping relationship between each group of two-dimensional separable convolution kernels and each group of input channels in the next layer. According to the second mapping relationship, each group of convolution kernels is respectively subjected to group convolution processing with the second video features of each corresponding group of input channels, that is, a group of convolution kernels is subjected to convolution processing on a group of input channels having a mapping relationship with the group of convolution kernels, and the second video features corresponding to each output channel in a group of output channels of the layer can be obtained. It can be understood that the number of a group of convolution kernels in the same convolution layer is equal to the number of a group of output channels. According to the same processing method, the second video features output by each group of output channels of the target convolution layer can be obtained. The second video features of each group of output channels are used as the second video features of each input channel of the next layer.
[0078] The above-mentioned video feature extraction method obtains the second mapping relationship between each group of two-dimensional separable convolution kernels and each group of input channels, and performs convolution processing on each group of two-dimensional separable convolution kernels with the first video features of each corresponding group of input channels in the spatial domain according to the second mapping relationship to obtain the second video features of each group of output channels, wherein a group of two-dimensional separable convolution kernels corresponds to a group of input channels, so that a group of convolution kernels only performs convolution processing on one group of input channels, and does not need to perform convolution processing on the input channels of other groups, thereby reducing the amount of convolution processing calculation and improving the speed of operation. In addition, a group of two-dimensional separable convolution kernels can perform convolution processing on a group of input channels, and the features of each input channel in a group can be fused to obtain a second video feature with more feature details.
[0079] In one embodiment, the second video feature is subjected to three-dimensional convolution processing to obtain a target video feature, including:
[0080] The second video feature is convolved using a three-dimensional convolution kernel to obtain a target video feature.
[0081] The target video feature refers to a feature that can represent key information of the video to be processed. The target video feature can be a video feature obtained after multiple one-dimensional convolutions, two-dimensional convolutions, and three-dimensional convolutions.
[0082] Specifically, after the terminal obtains the second video features corresponding to each output channel of the previous layer, it performs convolution processing of the next layer, that is, three-dimensional convolution processing. The terminal can obtain the three-dimensional convolution kernel of this layer, and perform convolution calculations on a three-dimensional convolution kernel and the second video features of each input channel respectively to obtain multiple video features, and fuse the multiple video features into one video feature. The fused video feature is the video feature of the output channel corresponding to the three-dimensional convolution kernel. Corresponding processing is performed on each three-dimensional convolution kernel of this layer to obtain the video features of the output channel corresponding to each three-dimensional convolution kernel. After multiple one-dimensional convolutions, two-dimensional convolutions and three-dimensional convolutions of the output video features, the target video features can be obtained. The video features in the time domain and the video features in the spatial domain of the video to be processed are fused through three-dimensional convolution processing.
[0083] For example, the output features of the last two-dimensional separable convolution are input to a 1*1*1 three-dimensional convolution kernel with D channels of X. The multiplication and addition calculation amount of the convolution kernel for the input video feature of T*W*H*X is 2*T*W*H*X*D, the parameter amount is D*X, and the size of the output video feature is T*W*H*D. The convolution kernel fuses the X channels of the input video feature by multiplication and addition, thereby completing the fusion of the video's temporal information and spatial information, and finally outputs features with the video's temporal and spatial information, completing the three-dimensional video convolution operation based on separable convolution.
[0084] In summary, for the three-dimensional video convolution module based on separable convolution, for an input video or video feature of size T*W*H*K, where W is the width of the video frame, H is the height of the video frame, and K is the number of channels. The total amount of calculation using the video feature extraction method in this embodiment is: 2*T*W*H*X*(K*M+N*N+D), and the amount of parameters is: X*(K*M+N*N+D). The total amount of calculation for traditional three-dimensional convolution is: 2*T*W*H*K*M*N*N*D, and the amount of parameters is K*M*N*N*D. Assuming X=tK, then, compared with the traditional three-dimensional convolution, the amount of calculation and parameter of video feature extraction of the three-dimensional video convolution module based on separable convolution are both the original t(K*M+N*N+D) / M*N*N*D. Assuming that M and N are 3, D=K=1024, and t=3, the multiplication and addition calculation amount and parameter amount of the three-dimensional video convolution module based on separable convolution are approximately 1 / 3 of those of the traditional three-dimensional convolution kernel.
[0085] In one embodiment, the three-dimensional convolution kernel is a 1*1*1 convolution kernel. Specifically, multiple 1*1*1 convolution kernels can be used to quickly and easily fuse features in the time dimension and the space dimension with minimal computation.
[0086] In one embodiment, the one-dimensional convolution kernel is an M*1*1 convolution kernel, the two-dimensional separable convolution kernel is a 1*N*N convolution kernel, and the three-dimensional convolution kernel is a 1*1*1 convolution kernel.
[0087] Specifically, the size of the one-dimensional convolution kernel is M*1*1, and the convolution kernel of M*1*1 is a one-dimensional convolution kernel with a width of M in the time dimension and a width (height) of 1 in the space dimension. The size of the two-dimensional separable convolution kernel is 1*N*N, and the convolution kernel of 1*N*N is a convolution kernel with a width of 1 in the time dimension, a width (height) of M in the space dimension, and a channel number of 1. The size of the three-dimensional convolution kernel is 1*1*1, that is, a convolution kernel with a width of 1 in the time dimension and a width (height) of 1 in the space dimension. Among them, M refers to the width of the convolution kernel in the time dimension, and N is the width (height) of the convolution kernel in the space dimension. The terminal inputs the video to be processed into the feature extraction model, and convolutions the video to be processed through the M*1*1 convolution kernels with X channels and K in the convolution layer, and obtains the first video feature of the video to be processed in the time dimension. Among them, X is the number of convolution kernels in the convolution layer, and K is the number of channels of the convolution kernel. Then, the first video feature of each input channel is input into X two-dimensional separable convolution kernels with a size of 1*N*N and a channel number of 1 for one-to-one convolution processing to obtain the second video feature of each output channel. Next, the second video feature of each input channel is input into X three-dimensional convolution kernels with a size of 1*1*1 and a channel number of 1 for convolution processing, thereby fusing the video features of the video to be processed in the time dimension and the space dimension.
[0088] like Figure 4 As shown, Figure 4 (a) is a schematic diagram of performing convolution processing on the features of the video to be processed in the time dimension (i.e., the T dimension in the coordinate axis) through a one-dimensional convolution kernel of 3*1*1. Through the convolution processing in Figure (a), the first video features of the video to be processed in the T dimension can be obtained. Figure 4 (b) is a schematic diagram of performing convolution processing on the features of the first video features in the spatial dimension (i.e., the H and W dimensions in the coordinate axis) through a 1*3*3 two-dimensional convolution kernel. Through the convolution processing in Figure (b), the second video features of the video to be processed in the H and W dimensions can be obtained. Figure 4 (c) in FIG. 1 is a schematic diagram of performing convolution processing on the second video feature through a 1*1*1 three-dimensional convolution kernel. Through the convolution processing in (c), independent features in the time dimension and the space dimension can be fused.
[0089] In one embodiment, the convolution layer of the feature extraction model performs one-dimensional convolution processing on the video to be processed in the time domain to obtain the first video feature, including:
[0090] Video images are extracted from the video to be processed at preset time intervals to obtain multiple frames of video images; the multiple frames of video images are input into a feature extraction model, and one-dimensional convolution processing is performed on the multiple frames of video images in the time domain through the convolution layer of the feature extraction model to obtain a first video feature.
[0091] Specifically, the terminal can extract video images from the video to be processed at preset time intervals to obtain multiple frames of video images, input the multiple frames of video images into the feature extraction model, and perform one-dimensional convolution processing on the multiple frames of video images in the time domain through the convolution layer of the feature extraction model to obtain the first video feature. By extracting video images from the video to be processed at preset time intervals, some image frames in the video to be processed can be extracted, and one-dimensional convolution processing on some image frames in the time domain is performed through the convolution layer of the feature extraction model to obtain the first video feature, which can reduce the amount of computation for feature extraction and increase the speed of feature extraction.
[0092] In one embodiment, Figure 5 As shown, the video feature extraction method also includes:
[0093] Step 502: Divide the target video feature into at least two areas.
[0094] Specifically, the terminal may divide the target video into at least two areas. Further, the target video feature may be divided into a preset number of areas. The preset number may be set as needed, such as 8, 10, etc.
[0095] Step 504: Detect the probability of the target object existing in each area.
[0096] Specifically, a plurality of convolutional layers may be used to perform convolution processing on each region of the target video feature, and the probability of the existence of the target object in each region may be output.
[0097] Step 506: When the probability that the target object exists in the region exceeds a threshold, determine the category of the target object in the region.
[0098] The threshold refers to the critical value of whether there is a target object in the region. If the threshold is exceeded, it is determined that there is a target object in the region, and if the threshold is not exceeded, it is determined that there is no target object in the region.
[0099] Specifically, when the probability that a target object exists in a region exceeds a threshold, it is determined that the target object exists in the region. When the probability that a target object exists in multiple regions exceeds a threshold, it is determined that the target object exists in multiple regions. Further, when it is determined that a target object exists in a region, the category of the target object in each region can be suppressed by non-maximum suppression to obtain the category corresponding to the target object in each region.
[0100] In this embodiment, when the probability that the target object exists in the area exceeds a threshold, the position and category of the target object in the area are determined.
[0101] Specifically, when the probability of a target object existing in a region exceeds a threshold, it is determined that the target object exists in the region. When the probability of a target object existing in multiple regions exceeds a threshold, it is determined that the target object exists in multiple regions. Further, when it is determined that the target object exists in the region, the category of the target object in each region and the position of the target object in the target image can be suppressed by non-maximum suppression to obtain the category corresponding to the target object in each region and the accurate position of the target object in the target image.
[0102] In the above-mentioned video feature extraction method, the video feature extraction method is applied to the scene of target object detection in the video. By dividing the target video feature into at least two areas, the probability of the target object existing in each area is detected, and when the probability of the target object existing in the area exceeds a threshold, the category of the target object in the area is determined. The video feature extraction method is used for target object detection, which can improve the speed of target detection.
[0103] In one embodiment, Figure 6 As shown, the video feature extraction method also includes:
[0104] Step 602: Generate a video feature sequence according to the target video features of each frame.
[0105] The video feature sequence refers to a sequence obtained by extracting features at the encoding time in the video to be processed. The encoding time is the time when the image from the 1st frame to the tth frame in the video to be processed is obtained. For example, if there are a total of T frames of images in the video to be processed, then at the encoding time t, the image frames from the 1st moment to the tth moment can be obtained.
[0106] Specifically, the video to be processed can be regarded as a combination of multiple images along the time axis, and an image in the video to be processed is called an image frame. Therefore, the video to be processed has an additional time dimension for the image data. In other words, the video to be processed corresponds to a video image sequence, and this video image sequence is formed by each frame of the video image in the corresponding time sequence. By sequentially extracting video features from the video to be processed along the time axis sequence of the video to be processed using the above-mentioned video feature extraction method, the target video features corresponding to each frame of the image can be obtained, thereby obtaining a video feature sequence at the video encoding moment.
[0107] Step 604: Encode the video feature sequence through a recursive neural network to obtain an implicit state sequence corresponding to each moment.
[0108] Specifically, the terminal can input the video feature sequence into a recursive neural network, and encode the target video features of each frame in the video feature sequence through the recursive neural network. The implicit state is a vector description of the target video features at the corresponding video encoding moment, and the implicit state carries the video information at the corresponding video encoding moment. By encoding the video feature sequence through a recursive neural network, the implicit state sequence corresponding to each moment can be obtained.
[0109] Step 606: Generate description information of the video to be processed according to the implicit state sequence.
[0110] The description information of the video refers to summary information generated based on the video features and describing the content displayed in the video.
[0111] Specifically, the corresponding words are generated by the recursive neural network according to the implicit state sequence corresponding to each moment, so that the words corresponding to each moment can be obtained. Thus, the words corresponding to each moment generate the description information of the video to be processed.
[0112] In the above embodiment, the video feature extraction method is applied to the scenario of generating video description information. Through the above video feature extraction method, the number of parameters and the amount of calculation in the feature extraction process can be reduced, and the target video features can be quickly obtained. A video feature sequence is generated according to the target video features of each frame, and the video feature sequence is encoded by a recursive neural network to obtain an implicit state sequence corresponding to each moment. The description information of the video to be processed is generated according to the implicit state sequence, which can improve the speed of generating video description information.
[0113] In one embodiment, Figure 7 As shown, the implicit state sequence includes a forward implicit state sequence and a backward implicit state sequence, and the description information of the video to be processed is generated according to the implicit state sequence, including:
[0114] Step 702: Determine the event corresponding to each moment and the interval corresponding to the event at the moment according to the forward implicit state sequence and the backward implicit state sequence.
[0115] Among them, the forward implicit state sequence refers to the sequence obtained by encoding the video feature sequence from front to back by the forward recurrent neural network according to the corresponding video encoding time. The backward implicit state sequence refers to the sequence obtained by encoding the video feature sequence from back to front by the backward recurrent neural network according to the corresponding video encoding time. The event refers to the content information displayed by the video frame image corresponding to the encoding time.
[0116] Specifically, the terminal can encode the video feature sequence from front to back according to the corresponding video encoding moment through a forward recursive neural network to obtain a forward implicit state sequence corresponding to each moment. The forward implicit state in the forward implicit state sequence carries the context information of the corresponding video encoding moment. Then, the terminal can encode the video feature sequence from back to front according to the corresponding video encoding moment through a backward recursive neural network to obtain a backward implicit state sequence corresponding to each moment. The backward implicit state in the backward implicit state sequence carries the context information of the corresponding video encoding moment.
[0117] Next, the video feature description and context relevance corresponding to each video encoding moment can be obtained based on the forward implicit state sequence and the backward implicit state sequence, and then the event at this video encoding moment can be accurately predicted. A video encoding moment often corresponds to one or more events, but these events may overlap in time at the video encoding moment. By identifying the temporally overlapping events through the forward implicit state sequence and the backward implicit state sequence, the event corresponding to each video encoding moment can be determined. The video to be processed can be regarded as composed of events occurring in each time period, so the interval corresponding to each event is the corresponding time period on the timeline.
[0118] Step 704: for each moment, predict the description information of the event according to the interval corresponding to the event at the moment, and obtain the word corresponding to the event at the moment.
[0119] The event description information refers to the summary information of the content displayed in a certain interval in the video.
[0120] Specifically, the terminal can predict the description information of the event according to the interval corresponding to the event at the moment through a recursive neural network, and filter the description information of the event to generate the description word corresponding to the event at the corresponding moment. Furthermore, the event corresponding to each moment is different, so the interval corresponding to each event is also different. By predicting the interval corresponding to the event corresponding to each moment, the description information corresponding to the event can be obtained, and thus the word corresponding to each moment can be obtained.
[0121] Step 706: Generate description information of the video to be processed according to the words corresponding to each moment.
[0122] Specifically, multiple description information can be generated according to the words corresponding to each moment. The multiple description information are all summary information describing the displayed content of the video to be processed. Further, the multiple description information can be sorted according to credibility, and the description information with the highest credibility is output, so as to obtain the description information of the video to be processed.
[0123] In the above embodiment, the video feature extraction method is applied to the scene of generating video description information. Through the above video feature extraction method, the parameter amount and calculation amount of the feature extraction process can be reduced, and the target video features can be quickly obtained. According to the forward implicit state sequence and the backward implicit state sequence, the event corresponding to each moment and the interval corresponding to the event at the moment are determined. For each moment, the description information of the event is predicted according to the interval corresponding to the event at the moment, and the word corresponding to the event at the moment is obtained. According to the word corresponding to each moment, the description information of the video to be processed is generated. Through the forward implicit state sequence and the backward implicit state sequence, the relevance of the context can be linked, which can ensure the accuracy and comprehensiveness of the description information of the generated video to be processed. The application of the video feature extraction method to the generation of video description information can further improve the speed of generating video description information.
[0124] In one embodiment, Figure 8 As shown, the method also includes:
[0125] Step 802: Detect the target subject in the video to be processed based on the target video feature.
[0126] Among them, the target subject refers to various objects, such as people, flowers, cats, dogs, cows, blue sky, white clouds, background, etc. The target subject refers to the required subject and can be selected according to needs.
[0127] Specifically, the terminal may input the target video features output by the feature extraction model into the detection model, and detect the target subject in the video to be processed through the detection model.
[0128] Furthermore, when there are multiple subjects, the priority of the category to which each subject belongs is obtained, and the subject with the highest or second highest priority is selected as the target subject.
[0129] When there are multiple subjects, the area occupied by each subject in the video frame is obtained, and the subject with the largest or second largest area in the video frame is selected as the target subject.
[0130] Step 804, obtaining the time point at which the target subject appears in the video to be processed.
[0131] Specifically, the video to be processed can be regarded as a combination of multiple images along the time axis, and each image frame corresponds to a time point. The terminal can determine each image frame where the target subject appears, obtain the time point corresponding to the image frame, and use the time point corresponding to each image frame as each time point when the target subject appears in the video to be processed.
[0132] Step 806, determine the time difference between two adjacent time points. When the time difference is greater than the time threshold, extract the current video segment containing the target subject from the video to be processed; the starting time point of the current video segment is the time point after the ending time point of the previous video segment, and the ending time point of the current video segment is the previous time point among the two adjacent time points.
[0133] Specifically, the terminal can sort each time point according to the size of the time point, and calculate the time difference between each two adjacent time points, which can be a difference or a ratio. The terminal can obtain a time threshold, compare each time difference with the time threshold, and when the time difference is less than or equal to the time threshold, the video to be processed is not segmented. When the time difference is greater than the time threshold, it indicates that the target subject has not appeared in the video for more than a preset time, and the terminal obtains two time points corresponding to the time difference. The previous time point of the two time points is used as a node for video segmentation, and the video to be processed is segmented with the node, and the node is also used as the end time point of the current video segment. Then, the terminal obtains the end time point of the previous video segment, and determines the next time point after the end time point, and uses the next time point as another node of the current video segment, that is, the start time point of the current video segment, and segments the video to be processed according to the node, so that the current video segment containing the target subject can be extracted from the video to be processed.
[0134] For example, the time points at which the target subject appears in the video to be processed are A, B, C, D, E and F, respectively. If the time difference between the two time points B and C is greater than the time threshold, the video to be processed is segmented with B as the node, and the current video segment with A as the starting time point and B as the ending time point is obtained. Then, when the time difference between C and D, and D and E is less than or equal to the time threshold, the time difference between E and F is continued to be calculated. When the time difference between E and F is greater than the time threshold, the video to be processed is cut with E as the node, and the point E is the ending time point of the current video segment. Then, the previous video segment is the video segment with A as the starting time point and B as the ending time point. Determine the next time point after point B, that is, C, and then cut the video segment with C as the node, and the current video segment with C as the starting time point and E as the ending time point is obtained.
[0135] In the above embodiment, the video feature extraction method is applied to video segment extraction, and the target video feature is quickly obtained based on the video feature extraction method. The target subject in the video to be processed is detected based on the target video feature, and the target subject in the video to be processed can be accurately detected. The time point at which the target subject appears in the video to be processed is obtained, and the time difference between the two adjacent time points is determined. When the time difference is greater than the time threshold, the current video segment containing the target subject is extracted from the video to be processed. The starting time point of the current video segment is the time point after the ending time point of the previous video segment, and the ending time point of the current video segment is the previous time point among the two adjacent time points. The various video segments containing the target subject in the video to be processed can be quickly and accurately extracted, and the speed of video segmentation is accelerated.
[0136] In one embodiment, the method further comprises:
[0137] Obtain a target video feature corresponding to the video segment; classify the video segment based on the target video feature to obtain a classification result corresponding to the video segment.
[0138] The classification result is to determine the category of the video segment, for example, the video segment belongs to a food video, a fighting video, etc.
[0139] Specifically, after extracting each video segment from the video to be processed, the terminal can obtain the target video features corresponding to each video segment, classify the target video features corresponding to each video segment, and obtain the target video feature classification result. The target video feature classification result is the classification result of the corresponding video segment.
[0140] The above-mentioned video feature extraction method can obtain the target video feature corresponding to the video segment, classify the video segment based on the target video feature, and obtain the classification result corresponding to the video segment, thereby quickly realizing video classification.
[0141] In one embodiment, the terminal can directly classify the video to be processed based on the target video feature. The target subject in the video to be processed can also be classified based on the target video feature to determine the category of the target subject. Category refers to the category of the subject, such as portrait, flower, animal, landscape, etc. The category and position of the target subject can also be determined at the same time. Position refers to the position of the target subject in the video to be processed, which can be expressed in coordinates.
[0142] In one embodiment, Fig. 9 As shown, before obtaining the video to be processed, the method further includes:
[0143] Step 902: Obtain training video samples and labels corresponding to the training video samples.
[0144] The label corresponding to the training video sample is the category to which the training video sample belongs.
[0145] Specifically, the terminal may obtain a training video sample and a label corresponding to the training video sample from a local device or other device or network.
[0146] Step 904: Perform one-dimensional convolution processing on the training video sample in the time domain through the convolution layer of the feature extraction model to be trained to obtain the first video features corresponding to each input channel of the next layer of the convolution layer.
[0147] Specifically, the terminal inputs the training video sample into the convolution layer of the feature extraction model, and the feature extraction model obtains the one-dimensional convolution kernel of the convolution layer, and performs convolution calculation on the features of each channel of the training video sample through a one-dimensional convolution kernel to obtain the video features of each output channel, and fuses the video features of each output channel to obtain a first video feature. The first video feature is the feature output by a one-dimensional convolution kernel. Then, the same process is performed on each one-dimensional convolution kernel in the convolution layer to obtain the first video feature corresponding to the output of each one-dimensional convolution kernel. The first video feature corresponding to the output of each one-dimensional convolution kernel is used as the first video feature corresponding to each input channel of the next layer of the convolution layer. It can be understood that the first video features corresponding to each output channel of the previous convolution layer are the first video features corresponding to each input channel of the next layer.
[0148] Step 906, obtaining a first mapping relationship between each two-dimensional separable convolution kernel and each input channel, where one two-dimensional separable convolution kernel corresponds to one input channel.
[0149] Step 908: Convolution processing is performed in the spatial domain on each of the two-dimensional separable convolution kernels with the first video features of the corresponding input channels according to the first mapping relationship to obtain second video features corresponding to the output channels of each of the two-dimensional separable convolution kernels.
[0150] Specifically, the terminal obtains a first mapping relationship between each two-dimensional separable convolution kernel and each input channel, and performs convolution calculation in the spatial domain on each two-dimensional separable convolution kernel with the first video feature of the corresponding input channel according to the first mapping relationship to obtain the second video features corresponding to the output channels of each two-dimensional separable convolution kernel.
[0151] Step 910: Perform three-dimensional convolution processing on the second video feature to obtain a target video feature.
[0152] Specifically, after obtaining the second video features corresponding to each output channel of the previous layer, the convolution processing of the next layer, that is, the three-dimensional convolution processing, can be performed. The terminal can obtain the three-dimensional convolution kernel of this layer, and perform convolution calculations on a three-dimensional convolution kernel and the second video features of each input channel respectively to obtain multiple video features, and fuse the multiple video features into one video feature. The fused video feature is the video feature of the output channel corresponding to the three-dimensional convolution kernel. Corresponding processing is performed on each three-dimensional convolution kernel of this layer to obtain the video features of the output channel corresponding to each three-dimensional convolution kernel. After multiple one-dimensional convolutions, two-dimensional convolutions and three-dimensional convolutions of the output video features, the target video features can be obtained.
[0153] Step 912: predict and classify the training video sample based on the target video feature to obtain a classification result of the training video sample.
[0154] Step 914, according to the difference between the classification result and the label, adjust the parameters of the feature extraction model and continue training until the training stop condition is met.
[0155] Specifically, the terminal predicts and classifies the training video sample based on the target video feature, and obtains the classification result of the training video sample. The terminal compares the label corresponding to the training video sample with the predicted classification result to determine the difference between the detection result and the label. According to the difference between the classification result and the label, the parameters of the feature extraction model are adjusted and training is continued until the training stop condition is met, and the training is terminated to obtain a trained feature extraction model.
[0156] In the above-mentioned video feature extraction method, by obtaining a training video sample and a label corresponding to the training video sample, the video feature of the training video sample in the time domain is extracted by one-dimensional convolution processing, the video feature of the training video sample in the space domain is extracted by two-dimensional convolution processing, and the video features in the time domain and the space domain are fused by three-dimensional convolution to obtain the target video feature. The classification result of the training video sample is obtained based on the target video feature, and according to the difference between the classification result and the label, the parameters of the feature extraction model are continuously adjusted and the training is continued, so as to obtain a trained feature extraction model. The trained feature extraction model reduces the computational amount of the convolution processing, improves the speed of the operation, and thus speeds up the efficiency of feature extraction.
[0157] like Fig.10As shown, it is a schematic diagram of feature extraction in one embodiment. The three-dimensional video convolution processing includes 3*1*1 convolution processing, 1*3*3 separable convolution processing and 1*1*1 convolution processing. First, input the video to be processed and multiple three-dimensional video convolution processing to extract the features of the video to be processed. For each three-dimensional video convolution processing, the first video feature of the video to be processed in the time dimension is first extracted by 3*1*1 convolution processing, and then the second video feature of the video to be processed in the spatial dimension is extracted by 1*3*3 separable convolution processing, that is, the first video feature is convoluted by a 1*3*3 convolution kernel with a channel number of 1 to obtain the second video feature. Then, the video features in the time domain and the space domain are fused by 1*1*1 convolution processing, that is, the second video feature is convoluted by 1*1*1 convolution and. After the last three-dimensional video convolution processing, the target video feature is obtained. The target video feature can be used for video classification, video segmentation, video beautification, target detection, etc., but is not limited to this. It can be understood that the target video features extracted by the video feature extraction method according to this embodiment can be used in any application scenario that requires further processing based on the features.
[0158] In one embodiment, a video feature extraction method is provided, comprising:
[0159] The terminal obtains a training video sample and a label corresponding to the training video sample.
[0160] The terminal performs one-dimensional convolution processing on the training video sample in the time domain through the convolution layer of the feature extraction model to be trained, and obtains the first video features corresponding to each input channel of the next layer of the convolution layer.
[0161] The terminal obtains a first mapping relationship between each two-dimensional separable convolution kernel and each input channel, and one two-dimensional separable convolution kernel corresponds to one input channel.
[0162] The terminal performs convolution processing in the spatial domain on each of the two-dimensional separable convolution kernels and the first video features of the corresponding input channels according to the first mapping relationship to obtain second video features corresponding to the output channels of each of the two-dimensional separable convolution kernels.
[0163] The terminal performs three-dimensional convolution processing on the second video feature to obtain a target video feature.
[0164] The terminal predicts and classifies the training video sample based on the target video feature to obtain a classification result of the training video sample.
[0165] The terminal adjusts the parameters of the feature extraction model according to the difference between the classification result and the label and continues training until the training stop condition is met.
[0166] The terminal obtains the video to be processed; extracts video images from the video to be processed at preset time intervals to obtain multiple frames of video images.
[0167] The terminal inputs the multiple frames of video images into a feature extraction model, and performs one-dimensional convolution processing in the time domain on the multiple frames of video images through the convolution layer of the feature extraction model to obtain a first video feature.
[0168] The terminal performs convolution processing in the time domain on the features of each channel of the video to be processed through each one-dimensional convolution kernel to obtain first video features corresponding to each input channel of the next layer of the convolution layer.
[0169] The terminal obtains a first mapping relationship between each two-dimensional separable convolution kernel and each input channel, and one two-dimensional separable convolution kernel corresponds to one input channel.
[0170] The terminal performs convolution processing in the spatial domain on each of the two-dimensional separable convolution kernels and the first video features of the corresponding input channels according to the first mapping relationship to obtain second video features corresponding to the output channels of each of the two-dimensional separable convolution kernels.
[0171] The terminal performs convolution processing on the second video feature through a three-dimensional convolution kernel to obtain a target video feature.
[0172] The terminal divides the target video feature into at least two areas.
[0173] The terminal detects the probability of the target object existing in each area; when the probability of the target object existing in the area exceeds a threshold, the category of the target object in the area is determined.
[0174] In the above-mentioned video feature extraction method, by obtaining a training video sample and a label corresponding to the training video sample, the video feature of the training video sample in the time domain is extracted by one-dimensional convolution processing, the video feature of the training video sample in the space domain is extracted by two-dimensional convolution processing, and the video features in the time domain and the space domain are fused by three-dimensional convolution to obtain the target video feature. Based on the target video feature, the classification result of the training video sample is obtained, and according to the difference between the classification result and the label, the parameters of the feature extraction model are continuously adjusted and the training is continued, so as to obtain a trained feature extraction model.
[0175] By inputting the video to be processed into a trained feature extraction model, a one-dimensional convolution kernel of the convolution layer of the feature extraction model is obtained, and the features of each channel of the video to be processed are convolved in the time domain through each one-dimensional convolution kernel to obtain the first video features corresponding to each input channel of the next layer of the convolution layer. The feature information of the video to be processed in the time dimension can be extracted separately, thereby reducing the number of parameters for feature extraction.
[0176] Through the one-to-one convolution processing of the two-dimensional separable convolution kernel and the input channel, the independent feature information of the video to be processed in the spatial domain can be extracted. A two-dimensional separable convolution kernel only performs convolution calculation with the features of one input channel, which can reduce the number of parameters and calculations of the convolution processing. Then, the video features of the video to be processed in the time domain and the video features in the spatial domain are fused through three-dimensional convolution processing to obtain the target video features. By dividing the target video features into at least two regions, the probability of the target object existing in each region is detected. When the probability of the target object existing in the region exceeds a threshold, the category of the target object in the region is determined. The video feature extraction method is used for target object detection, which can improve the speed of target detection.
[0177] Figure 2-Figure 9 FIG. 1 is a flow chart of a method for extracting video features in one embodiment. It should be understood that although Figure 2-Figure 9 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 9 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0178] In one embodiment, Fig.11 As shown, a video feature extraction device is provided, which includes: an acquisition module 1102, a one-dimensional convolution module 1104, a two-dimensional convolution module 1106 and a three-dimensional convolution module 1108. Among them,
[0179] The acquisition module 1102 is used to acquire the video to be processed.
[0180] The one-dimensional convolution module 1104 is used to perform one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain the first video feature.
[0181] The two-dimensional convolution module 1106 is used to perform a two-dimensional separable convolution process on the first video feature in the spatial domain to obtain a second video feature.
[0182] The three-dimensional convolution module 1108 is used to perform three-dimensional convolution processing on the second video feature to obtain a target video feature.
[0183] The above-mentioned video feature extraction device obtains a video to be processed, performs one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model to obtain a first video feature, performs two-dimensional separable convolution processing on the first video feature in the spatial domain to obtain a second video feature, and performs three-dimensional convolution processing on the second video feature to obtain a target video feature, which can reduce the amount of calculation in the feature extraction process and improve the efficiency of feature extraction.
[0184] In one embodiment, the one-dimensional convolution module 1104 is also used to: input the video to be processed into a feature extraction model to obtain a one-dimensional convolution kernel of the convolution layer of the feature extraction model; perform convolution processing on the features of each channel of the video to be processed in the time domain through each one-dimensional convolution kernel to obtain the first video features corresponding to each input channel of the next layer of the convolution layer.
[0185] The video feature extraction device in this embodiment obtains the one-dimensional convolution kernel of the convolution layer of the feature extraction model by inputting the video to be processed into the feature extraction model, and performs convolution processing on the features of each channel of the video to be processed in the time domain through each one-dimensional convolution kernel to obtain the first video features corresponding to each input channel of the next layer of the convolution layer. The feature information of the video to be processed in the time dimension can be extracted separately, thereby reducing the number of parameters for feature extraction.
[0186] In one embodiment, the two-dimensional convolution module 1106 is further used to: obtain a first mapping relationship between each two-dimensional separable convolution kernel and each input channel, where one two-dimensional separable convolution kernel corresponds to one input channel;
[0187] According to the first mapping relationship, each two-dimensional separable convolution kernel is convolved with the first video feature of the corresponding input channel in the spatial domain to obtain the second video features corresponding to the output channels of each two-dimensional separable convolution kernel.
[0188] The above-mentioned video feature extraction device obtains a first mapping relationship between each two-dimensional separable convolution kernel and each input channel, one two-dimensional separable convolution kernel corresponds to one input channel, and performs convolution processing on each two-dimensional separable convolution kernel with the first video feature of the corresponding input channel in the spatial domain according to the first mapping relationship to obtain the second video features corresponding to the output channels of each two-dimensional separable convolution kernel. Through the one-to-one convolution processing of the two-dimensional separable convolution kernel and the input channel, the independent feature information of the video to be processed in the spatial domain can be extracted. A two-dimensional separable convolution kernel only performs convolution calculation with the feature of one input channel, which can reduce the number of parameters and the amount of calculation of the convolution processing.
[0189] In one embodiment, the two-dimensional convolution module 1106 is also used to: obtain a second mapping relationship between each group of two-dimensional separable convolution kernels and each group of input channels; perform convolution processing in the spatial domain on each group of two-dimensional separable convolution kernels and the first video features of each corresponding group of input channels according to the second mapping relationship to obtain second video features of each group of output channels, wherein a group of two-dimensional separable convolution kernels corresponds to a group of input channels.
[0190] The above-mentioned video feature extraction device obtains the second mapping relationship between each group of two-dimensional separable convolution kernels and each group of input channels, and performs convolution processing on each group of two-dimensional separable convolution kernels with the first video features of each corresponding group of input channels in the spatial domain according to the second mapping relationship, so as to obtain the second video features of each group of output channels, wherein a group of two-dimensional separable convolution kernels corresponds to a group of input channels, so that a group of convolution kernels only performs convolution processing on a group of input channels, and does not need to perform convolution processing on the input channels of other groups, thereby reducing the amount of convolution processing calculation and improving the speed of operation. In addition, a group of two-dimensional separable convolution kernels can perform convolution processing on a group of input channels, and the features of each input channel in a group can be fused to obtain a second video feature with more feature details.
[0191] In one embodiment, the three-dimensional convolution module 1108 is further used to: perform convolution processing on the second video feature through a three-dimensional convolution kernel to obtain a target video feature, and the three-dimensional convolution kernel is a 1*1*1 convolution kernel. Through the three-dimensional convolution processing, the video features in the time domain and the video features in the space domain of the video to be processed are fused.
[0192] In one embodiment, the one-dimensional convolution module 1104 is also used to: extract video images from the video to be processed at preset time intervals to obtain multiple frames of video images; input the multiple frames of video images into a feature extraction model, and perform one-dimensional convolution processing on the multiple frames of video images in the time domain through the convolution layer of the feature extraction model to obtain a first video feature. By extracting video images from the video to be processed at preset time intervals, partial image frames in the video to be processed can be extracted, and the convolution layer of the feature extraction model performs one-dimensional convolution processing on the partial image frames in the time domain to obtain the first video feature, which can reduce the amount of computation for feature extraction and improve the speed of feature extraction.
[0193] In one embodiment, the device further includes: a detection module. The detection module is used to: divide the target video feature into at least two areas; detect the probability of the target object existing in each area; and determine the category of the target object in the area when the probability of the target object existing in the area exceeds a threshold. In the above-mentioned video feature extraction device, the video feature extraction device is applied to the scene of target object detection in the video. By dividing the target video feature into at least two areas, detecting the probability of the target object existing in each area, and determining the category of the target object in the area when the probability of the target object existing in the area exceeds a threshold, the video feature extraction device is used for target object detection, which can improve the speed of target detection.
[0194] In one embodiment, the device further includes: a generation module. The generation module is used to: generate a video feature sequence according to the target video features of each frame; encode the video feature sequence through a recursive neural network to obtain an implicit state sequence corresponding to each moment; and generate description information of the video to be processed according to the implicit state sequence. In the above embodiment, the video feature extraction device is applied to the scenario of generating video description information. Through the above video feature extraction device, the parameter amount and calculation amount of the feature extraction process can be reduced, and the target video features can be quickly obtained. A video feature sequence is generated according to the target video features of each frame, and the video feature sequence is encoded through a recursive neural network to obtain an implicit state sequence corresponding to each moment. The description information of the video to be processed is generated according to the implicit state sequence, which can increase the speed of generating video description information.
[0195] In one embodiment, the implicit state sequence includes a forward implicit state sequence and a backward implicit state sequence, and the generation module is further used to: determine the event corresponding to each moment and the interval corresponding to the event at the moment according to the forward implicit state sequence and the backward implicit state sequence; for each moment, predict the description information of the event according to the interval corresponding to the event at the moment, and obtain the word corresponding to the event at the moment; generate the description information of the video to be processed according to the word corresponding to each moment. In the above embodiment, the video feature extraction device is applied to the scene of generating video description information. Through the above video feature extraction device, the parameter amount and calculation amount of the feature extraction process can be reduced, and the target video feature can be quickly obtained. According to the forward implicit state sequence and the backward implicit state sequence, determine the event corresponding to each moment and the interval corresponding to the event at the moment, for each moment, predict the description information of the event according to the interval corresponding to the event at the moment, and obtain the word corresponding to the event at the moment, and generate the description information of the video to be processed according to the word corresponding to each moment. The backward implicit state sequence can be linked to the context relevance through the forward implicit state sequence, which can ensure the accuracy and comprehensiveness of the description information of the generated video to be processed. The video feature extraction device is applied to generate video description information, which can further improve the speed of generating video description information.
[0196] In one embodiment, the device further includes: a video segment extraction module. The video segment extraction module is used to: detect the target subject in the video to be processed based on the target video feature; obtain the time point when the target subject appears in the video to be processed; determine the time difference between two adjacent time points, and when the time difference is greater than a time threshold, extract the current video segment containing the target subject from the video to be processed; the starting time point of the current video segment is the time point after the ending time point of the previous video segment, and the ending time point of the current video segment is the previous time point among the two adjacent time points.
[0197] In the above embodiment, a video feature extraction device is applied to video segment extraction, and the target video feature is quickly obtained based on the video feature extraction device. The target subject in the video to be processed is detected based on the target video feature, and the target subject in the video to be processed can be accurately detected. The time point at which the target subject appears in the video to be processed is obtained, and the time difference between two adjacent time points is determined. When the time difference is greater than the time threshold, the current video segment containing the target subject is extracted from the video to be processed. The starting time point of the current video segment is the time point after the ending time point of the previous video segment, and the ending time point of the current video segment is the previous time point among the two adjacent time points. The various video segments containing the target subject in the video to be processed can be quickly and accurately extracted, and the speed of video segmentation is accelerated.
[0198] In one embodiment, the device further includes: a classification module. The classification module is used to: obtain a target video feature corresponding to the video segment; and classify the video segment based on the target video feature to obtain a classification result corresponding to the video segment.
[0199] The above-mentioned video feature extraction device can obtain the target video feature corresponding to the video segment, classify the video segment based on the target video feature, and obtain the classification result corresponding to the video segment, thereby quickly realizing video classification.
[0200] In one embodiment, before obtaining the video to be processed, the device further includes: a training module. The training module is used to: obtain a training video sample and a label corresponding to the training video sample; perform one-dimensional convolution processing on the training video sample in the time domain through the convolution layer of the feature extraction model to be trained, and obtain the first video features corresponding to each input channel of the next layer of the convolution layer; obtain the first mapping relationship between each two-dimensional separable convolution kernel and each input channel, and one two-dimensional separable convolution kernel corresponds to one input channel; perform convolution processing on each two-dimensional separable convolution kernel with the first video feature of the corresponding input channel in the spatial domain according to the first mapping relationship, and obtain the second video features corresponding to the output channels of each two-dimensional separable convolution kernel; perform three-dimensional convolution processing on the second video feature to obtain the target video feature; perform predictive classification on the training video sample based on the target video feature to obtain the classification result of the training video sample; according to the difference between the classification result and the label, adjust the parameters of the feature extraction model and continue training until the training stop condition is met and the training is terminated.
[0201] In the above-mentioned video feature extraction device, by obtaining a training video sample and a label corresponding to the training video sample, the video feature of the training video sample in the time domain is extracted by one-dimensional convolution processing, the video feature of the training video sample in the space domain is extracted by two-dimensional convolution processing, and the video features in the time domain and the space domain are fused by three-dimensional convolution to obtain the target video feature. Based on the target video feature, the classification result of the training video sample is obtained, and according to the difference between the classification result and the label, the parameters of the feature extraction model are continuously adjusted and the training is continued, so as to obtain a trained feature extraction model. The trained feature extraction model reduces the computational amount of the convolution processing, improves the speed of the operation, and thus speeds up the efficiency of feature extraction.
[0202] Fig.12 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The terminal 110 (or server 120) in FIG. Fig.12As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the video feature extraction method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can execute the video feature extraction method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0203] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0204] In one embodiment, the video feature extraction device provided in the present application can be implemented in the form of a computer program. The computer program can be Fig.12 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the video feature extraction device, for example, Fig.11 The acquisition module 1102, the one-dimensional convolution module 1104, the two-dimensional convolution module 1106 and the three-dimensional convolution module 1108 are shown. The computer program composed of various program modules enables the processor to execute the steps of the video feature extraction method of each embodiment of the present application described in this specification.
[0205] For example, Fig.12 The computer device shown can be Fig.11 The acquisition module 1102 in the video feature extraction device shown in the figure performs the step of acquiring the video to be processed. The computer device can perform the step of performing one-dimensional convolution processing on the video to be processed in the time domain through the convolution layer of the feature extraction model through the one-dimensional convolution module 1104 to obtain the first video feature. The computer device can perform the step of performing two-dimensional separable convolution processing on the first video feature in the spatial domain through the two-dimensional convolution module 1106 to obtain the second video feature. The computer device can perform the step of performing three-dimensional convolution processing on the second video feature through the three-dimensional convolution module 1108 to obtain the target video feature.
[0206] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned video feature extraction method. The steps of the video feature extraction method here may be the steps of the video feature extraction method in the above-mentioned embodiments.
[0207] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the video feature extraction method described above. The steps of the video feature extraction method here can be the steps of the video feature extraction method in the above-mentioned embodiments.
[0208] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0209] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0210] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A video feature extraction method, characterized in that: include: Get the video to be processed; Inputting the video to be processed into a feature extraction model to obtain a one-dimensional convolution kernel of a convolution layer of the feature extraction model; Performing convolution processing on the features of each channel of the video to be processed in the time domain through each one-dimensional convolution kernel, to obtain first video features corresponding to each input channel of the next layer of the convolution layer, one one-dimensional convolution kernel corresponds to one input channel, and each one-dimensional convolution kernel has the same size; Obtain the mapping relationship between each group of two-dimensional separable convolution kernels and each group of input channels; The sizes of the two-dimensional separable convolution kernels in each group are the same, and the number of groups into which the two-dimensional separable convolution kernels of the same convolution layer are divided is the same as the number of groups into which the input channels of the same convolution layer are divided; According to the mapping relationship, each group of two-dimensional separable convolution kernels is respectively subjected to group convolution processing on the spatial domain with the first video features of each group of corresponding input channels to obtain the second video features of each group of output channels, wherein a group of two-dimensional separable convolution kernels corresponds to a group of input channels, and the number of a group of convolution kernels in the same convolution layer is equal to the number of a group of output channels; Perform three-dimensional convolution processing on the second video feature to obtain a target video feature.
2. The method according to claim 1, characterized in that The performing three-dimensional convolution processing on the second video feature to obtain a target video feature includes: The second video feature is convolved by a three-dimensional convolution kernel to obtain a target video feature, where the three-dimensional convolution kernel is a 1*1*1 convolution kernel.
3. The method according to claim 1, characterized in that The step of inputting the video to be processed into a feature extraction model to obtain a one-dimensional convolution kernel of a convolution layer of the feature extraction model; performing convolution processing on the features of each channel of the video to be processed in the time domain through each of the one-dimensional convolution kernels to obtain first video features corresponding to each input channel of a next layer of the convolution layer, including: Extracting video images from the video to be processed at preset time intervals to obtain multiple frames of video images; Input the multiple frames of video images into a feature extraction model to obtain a one-dimensional convolution kernel of a convolution layer of the feature extraction model; One-dimensional convolution processing is performed on the features of each channel of the multiple frames of video images in the time domain through each one-dimensional convolution kernel to obtain first video features corresponding to each input channel of the next layer of the convolution layer.
4. The method according to claim 1, characterized in that: The method further comprises: Dividing the target video feature into at least two regions; Detect the probability of the target object existing in each area; When the probability that the target object exists in the region exceeds a threshold, the category of the target object in the region is determined.
5. The method according to claim 1, characterized in that The method further comprises: Generate a video feature sequence according to the target video features of each frame; Encoding the video feature sequence through a recursive neural network to obtain an implicit state sequence corresponding to each moment; Generate description information of the video to be processed according to the implicit state sequence.
6. The method according to claim 5, characterized in that The implicit state sequence includes a forward implicit state sequence and a backward implicit state sequence, and the generating the description information of the video to be processed according to the implicit state sequence includes: Determine, according to the forward implicit state sequence and the backward implicit state sequence, the event corresponding to each moment and the interval corresponding to the event at the moment; For each moment, predict the description information of the event according to the interval corresponding to the event at the moment, and obtain the word corresponding to the event at the moment; Generate description information of the video to be processed according to the words corresponding to each moment.
7. The method according to claim 1, characterized in that The method further comprises: Detecting a target subject in the video to be processed based on the target video features; Obtaining the time point at which the target subject appears in the video to be processed; Determine the time difference between two adjacent time points. When the time difference is greater than a time threshold, extract a current video segment containing the target subject from the video to be processed; the starting time point of the current video segment is a time point after the ending time point of the previous video segment, and the ending time point of the current video segment is a previous time point among the two adjacent time points.
8. The method according to claim 7, characterized in that The method further comprises: Obtain target video features corresponding to the video segment; The video segment is classified based on the target video feature to obtain a classification result corresponding to the video segment.
9. The method according to claim 1, characterized in that: Before obtaining the video to be processed, the method further includes: Obtaining training video samples and labels corresponding to the training video samples; Performing one-dimensional convolution processing on the training video sample in the time domain through the convolution layer of the feature extraction model to be trained, so as to obtain first video features corresponding to each input channel of the next layer of the convolution layer; Obtain a first mapping relationship between each two-dimensional separable convolution kernel and each input channel, where one two-dimensional separable convolution kernel corresponds to one input channel; According to the first mapping relationship, convolution processing is performed on each of the two-dimensional separable convolution kernels with the first video features of the corresponding input channels in the spatial domain to obtain second video features corresponding to the output channels of each of the two-dimensional separable convolution kernels; Performing three-dimensional convolution processing on the second video feature to obtain a target video feature; Predicting and classifying the training video samples based on the target video features to obtain classification results of the training video samples; According to the difference between the classification result and the label, the parameters of the feature extraction model are adjusted and training is continued until the training stop condition is met.
10. A video feature extraction device, characterized in that: The device comprises: An acquisition module is used to acquire the video to be processed; A one-dimensional convolution module, used for inputting the video to be processed into a feature extraction model, obtaining a one-dimensional convolution kernel of a convolution layer of the feature extraction model; performing convolution processing on the features of each channel of the video to be processed in the time domain through each one-dimensional convolution kernel, obtaining first video features corresponding to each input channel of the next layer of the convolution layer, wherein one one-dimensional convolution kernel corresponds to one input channel, and each one-dimensional convolution kernel has the same size; A two-dimensional convolution module, used to obtain the mapping relationship between each group of two-dimensional separable convolution kernels and each group of input channels; the size of the two-dimensional separable convolution kernels in each group is the same, and the number of groups into which the two-dimensional separable convolution kernels of the same convolution layer are divided is the same as the number of groups into which the input channels of the same convolution layer are divided; according to the mapping relationship, each group of two-dimensional separable convolution kernels is respectively subjected to group convolution processing in the spatial domain with the first video features of the corresponding groups of input channels to obtain the second video features of each group of output channels, wherein a group of two-dimensional separable convolution kernels corresponds to a group of input channels, and the number of a group of convolution kernels in the same convolution layer is equal to the number of a group of output channels; The three-dimensional convolution module is used to perform three-dimensional convolution processing on the second video feature to obtain a target video feature.
11. The device according to claim 10, characterized in that The three-dimensional convolution module is also used to perform convolution processing on the second video feature through a three-dimensional convolution kernel to obtain a target video feature, and the three-dimensional convolution kernel is a 1*1*1 convolution kernel.
12. The device according to claim 10, characterized in that The one-dimensional convolution module is also used to extract video images from the video to be processed at preset time intervals to obtain multiple frames of video images; input the multiple frames of video images into a feature extraction model to obtain a one-dimensional convolution kernel of the convolution layer of the feature extraction model; and perform one-dimensional convolution processing in the time domain on the features of each channel of the multiple frames of video images through each one-dimensional convolution kernel to obtain the first video features corresponding to each input channel of the next layer of the convolution layer.
13. The device according to claim 10, characterized in that The device also includes: The detection module is used to divide the target video feature into at least two areas; detect the probability of the target object existing in each area; when the probability of the target object existing in the area exceeds a threshold, determine the category of the target object in the area.
14. The device according to claim 10, characterized in that The device also includes: A generation module is used to generate a video feature sequence according to the target video features of each frame; encode the video feature sequence through a recursive neural network to obtain an implicit state sequence corresponding to each moment; and generate description information of the video to be processed according to the implicit state sequence.
15. The device according to claim 14, characterized in that The implicit state sequence includes a forward implicit state sequence and a backward implicit state sequence, and the generating module is further used to determine the event corresponding to each moment and the interval corresponding to the event at the moment according to the forward implicit state sequence and the backward implicit state sequence; For each moment, the description information of the event is predicted according to the interval corresponding to the event at the moment, and the word corresponding to the event at the moment is obtained; and the description information of the video to be processed is generated according to the word corresponding to each moment.
16. The device according to claim 10, characterized in that The device also includes: A video segment extraction module is used to detect the target subject in the video to be processed based on the target video features; obtain the time point when the target subject appears in the video to be processed; determine the time difference between two adjacent time points, and when the time difference is greater than a time threshold, extract the current video segment containing the target subject from the video to be processed; the starting time point of the current video segment is the time point after the ending time point of the previous video segment, and the ending time point of the current video segment is the previous time point among the two adjacent time points.
17. The device according to claim 16, characterized in that The device also includes: The classification module is used to obtain target video features corresponding to the video segment; classify the video segment based on the target video features to obtain a classification result corresponding to the video segment.
18. The device according to claim 10, characterized in that The device also includes: A training module is used to obtain training video samples and labels corresponding to the training video samples; perform one-dimensional convolution processing on the training video samples in the time domain through the convolution layer of the feature extraction model to be trained, and obtain the first video features corresponding to each input channel of the next layer of the convolution layer; obtain the first mapping relationship between each two-dimensional separable convolution kernel and each input channel, and one two-dimensional separable convolution kernel corresponds to one input channel; according to the first mapping relationship, perform convolution processing on each two-dimensional separable convolution kernel with the first video feature of the corresponding input channel in the spatial domain, and obtain the second video features corresponding to the output channels of each two-dimensional separable convolution kernel; perform three-dimensional convolution processing on the second video feature to obtain the target video feature; predict and classify the training video samples based on the target video feature to obtain the classification result of the training video sample; according to the difference between the classification result and the label, adjust the parameters of the feature extraction model and continue training until the training stop condition is met to terminate the training.
19. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 9.
20. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Image related processing method and devices, intelligent terminal, server and memory medium
CN108259997A
Video quality enhancement method based on adaptive separable convolution
CN108900848A
A method and an apparatus for extracting video feature vectors
CN109165573A
Video feature extraction method and related equipment
CN110020639A
Video description generation method and device, video playing method and device and storage medium
CN110072142A
