Video emotion classification method combining multi-level branch convolution and dilated interaction sampling

Abstract

The application discloses a video emotion classification method combining multi-level branch convolution and dilated interactive sampling in the field of emotion classification, and comprises the following steps: converting a frame sequence into a tensor, using one layer of large-core convolution for covering down-sampling, and reducing the size of each frame image in the tensor; using a multi-level branch convolution algorithm for spatial feature extraction and further reducing the spatial dimension size; using a dilated interactive sampling algorithm for time sequence extraction, shortening the calculation time through parallel dilated convolution after sampling; and finally classifying video emotions. The application extracts information of different sizes and positions through three groups of strip-shaped large-core deep convolutions of different scales, arouses spatial attention through element multiplication, aggregates the connection between these local information through multi-level non-covering small-core down-sampling, down-samples time sequences in the structure of an odd-even binary tree, and combines non-causal dilated convolution and interactive learning between sequences.

Description

Technical Field

[0001] This invention belongs to the field of sentiment classification, specifically involving a video sentiment classification method that combines multi-level branch convolution with dilated interactive sampling. Background Technology

[0002] In daily life, we mostly express our emotions through facial expressions and body language. Dynamic emotion recognition based on computer vision uses specific algorithms to analyze video captured by a camera, allowing the computer to identify the emotions expressed based on the target's facial expressions and movements.

[0003] Current video sentiment classification models suffer from two main problems: First, emotion is expressed jointly by different parts of the body, while current video sentiment recognition models primarily focus on extracting inter-frame relationships in the temporal dimension, using only fixed-size mini-kernel convolutions in the spatial dimension. This approach fails to effectively perceive parts of different locations and sizes and cannot aggregate the connections between these parts. Second, most current methods use Transformers and their variants to model long-term inter-frame dependencies in the temporal dimension through multi-head self-attention. However, multi-head self-attention is computationally expensive and, due to the lack of inductive bias, requires a large amount of data, making training difficult and prone to overfitting. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a video sentiment classification method that combines multi-level branch convolution with dilated interactive sampling, in order to solve the problems mentioned in the background.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A video sentiment classification method combining multi-level branch convolution and dilated interactive sampling includes the following steps:

[0007] Step 1: Convert the frame sequence into a tensor and use a large kernel convolution layer for coverage downsampling to reduce the size of each frame image in the tensor.

[0008] Step 2: Use a multi-branch convolution algorithm to extract the spatial features output from Step 1, and further reduce the spatial dimension.

[0009] Step 3: Use the dilated interactive sampling algorithm to extract the time series output from Step 2, and shorten the computation time by using parallel dilated convolution after sampling.

[0010] Step 4: Finally, input the obtained features into the sentiment classification model to classify the sentiment of the video.

[0011] Preferably, the multi-branch convolution algorithm extraction process in step 2 is as follows:

[0012] Step 2.1: First, activate the output of Step 1 with the GELU nonlinear function and then feed it into a standard 3×3 convolution to extract local features. Then, after passing through LayerNorm, feed it into a 1×1 convolution to increase the number of channels. This effectively avoids information loss and can also capture the relationship between the channel dimensions of the feature map, thus enhancing the channel adaptability of the features.

[0013] Step 2.2: Then, the features after dimensionality increase from 1×1 convolution are fed into the parallel branch strip convolution module. Three pairs of depth-separable strip convolution kernels, namely 1×5, 5×1, 1×7, 7×1, 1×11, and 11×1, are used to extract regional association information of different sizes and positions based on small, medium, and large receptive fields.

[0014] Step 2.3: Next, the three outputs of the branched strip convolution are residually connected to the input, and then fed into the second 1×1 convolution to model the relationship between different channels. The output of the 1×1 convolution is directly used as the attention weight. The output of Step 1 is weighted by matrix multiplication, and the weighted result is first passed through the GELU activation function to introduce nonlinearity, and then reduced in dimension by the third 1×1 convolution.

[0015] Step 2.4: Finally, to fully extract features of different semantics and sizes, a multi-level architecture is introduced, consisting of four stages, each with a spatial resolution of [missing information]. Each stage includes a downsampling block and a stack of structure blocks as described above. The downsampling block uses a convolution with a stride of 2 and a kernel size of 2×2. LayerNorm is performed after each downsampling operation.

[0016] Preferably, the process in step 2.3 can be summarized by the following formula:

[0017]

[0018]

[0019] In the formula, Att represents the convolution weights, and Conv represents the weights. 1×1 For a 1×1 convolution, S i Let be the convolution result of the i-th branch, i∈{0,1,2,3}, DwConv be a set of depthwise separating strip convolutions, F be the input features, and Out be the module output. This refers to matrix multiplication.

[0020] Preferably, the extraction process of the dilatational interactive sampling algorithm in step 3 is as follows:

[0021] Step 3.1: First, decompose the time series output in Step 2 into two subsequences O. even and O odd Separate even-numbered elements from odd-numbered elements;

[0022] Step 3.2: Then use different dilated convolution modules EConv to convert O even and O odd The hidden states are mapped to two separate hidden states, which are then converted into exponential form and multiplied element-wise with the hidden states of another sequence. The result of the element-wise multiplication is then residually joined with the original sequence.

[0023] Preferably, the calculation formula in step 3.2 is as follows:

[0024]

[0025]

[0026] In the formula, O′ even To output an odd sequence, O even For the next odd sequence, For residual connections, EConv is the dilated convolution operation, ⊙ is the element-wise multiplication operation, exp is the pointer fetch operation, and O... odd For the lower-order pairing sequence, O′ odd To output an even sequence.

[0027] The beneficial effects of this invention are:

[0028] 1. The method of this invention proposes a multi-level branch convolution algorithm for spatial feature extraction. It uses three sets of strip-shaped large kernel depth convolutions of different scales to extract information of different sizes and positions, and uses element-wise multiplication to evoke spatial attention, extracting local key information while aggregating the connections between local parts.

[0029] 2. The method of this invention aggregates the relationships between these local information through multi-level non-covering small kernel downsampling, and as sampling deepens, semantic and spatial contextual information can be obtained in large kernel convolution.

[0030] 3. The method of this invention extracts temporal features through dilated interactive sampling algorithm. By downsampling, dilated convolution and information interaction, it captures the temporal dependencies of multiple time resolutions and realizes the learning of global temporal features. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0033] Figure 2 This is the spatiotemporal feature extraction structure in this invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Please see Figure 1 As shown, this invention provides a video sentiment classification method combining multi-level branch convolution and dilated interactive sampling, comprising the following steps:

[0036] Step 1: Convert the frame sequence into a tensor and use a large kernel convolution layer for coverage downsampling to reduce the size of each frame image in the tensor.

[0037] The process of using a large-kernel convolution layer for coverage downsampling is as follows:

[0038] Each frame of the frame sequence is input into a convolutional layer with 3 input channels, a stride of 4, a kernel size of 7×7, and 64 convolutions. The size of each frame is downsampled to [the specified value]. H and W are the height and width of each frame of input.

[0039] Step 2: Use the proposed multi-branch convolution algorithm to extract spatial features and further reduce the spatial dimension.

[0040] The multi-branch convolution algorithm process is as follows:

[0041] Step 2.1: After the output of Step 1 is activated by the GELU nonlinear function, it is fed into a standard 3×3 convolution to extract local features. Then, after passing through LayerNorm, it is fed into a 1×1 convolution to increase the number of channels, which effectively avoids information loss and can also capture the relationship between the channel dimensions of the feature map, thus enhancing the channel adaptability of the features.

[0042] Step 2.2: The features after dimensionality upscaling from 1×1 convolution are fed into a parallel branched strip convolution module. Three pairs of depth-separable strip convolution kernels, namely 1×5, 5×1, 1×7, 7×1, 1×11, and 11×1, are used to extract regional association information of different sizes and locations based on small, medium, and large receptive fields.

[0043] Step 2.3: The three outputs of the branched strip convolution are residually concatenated with the input, and then fed into a second 1×1 convolution to model the relationship between different channels. The output of the 1×1 convolution is directly used as the attention weight. The output of Step 1 is weighted by matrix multiplication, and the weighted result is first passed through the GELU activation function to introduce non-linearity, and then reduced in dimensionality by a third 1×1 convolution. The above process can be summarized by the following formula:

[0044]

[0045]

[0046] In the formula, Att represents the convolution weights, and Conv represents the weights. 1×1 For a 1×1 convolution, S i Let be the convolution result of the i-th branch, i∈{0,1,2,3}, DwConv be a set of depthwise separating strip convolutions, F be the input features, and Out be the module output. This is matrix multiplication;

[0047] Step 2.4: To fully extract features of different semantics and sizes, a multi-level architecture is introduced, consisting of four stages, each with a spatial resolution of [missing information]. Each stage consists of a downsampling block and a stack of structure blocks as described above. The downsampling block uses a convolution with a stride of 2 and a kernel size of 2×2, and LayerNorm is performed after each downsampling operation.

[0048] Step 3: Extract temporal features using the proposed dilated interactive sampling algorithm, and shorten the computation time by using parallel dilated convolution after sampling;

[0049] The process of the dilatational interactive sampling algorithm is as follows:

[0050] Step 3.1: Decompose the time series output from Step 2 into two subsequences O. even and O odd This separates even-numbered elements from odd-numbered elements. These two subsequences have a coarse temporal resolution but retain most of the information of the original sequence.

[0051] Step 3.2: Use different dilated convolution modules EConv to convert O even and O odd The hidden states are mapped to two separate hidden states, then converted to exponential form and multiplied element-wise with the hidden states of another sequence. The result of the element-wise multiplication is then residual-joined with the original sequence. The calculation formula is shown below:

[0052]

[0053]

[0054] In the formula, O′ even To output an odd sequence, O even For the next odd sequence, For residual connections, EConv is the dilated convolution operation, ⊙ is the element-wise multiplication operation, exp is the pointer fetch operation, and O... odd For the lower-order pairing sequence, O′ odd To output an even sequence;

[0055] The dilated convolution module EConv first reduces the boundary effect of convolution operations by boundary replication, then uses dilated convolution with an initial dilation factor of 3 to convolve the sampling time sequence, and then uses GELU and Dropout. The dilation factor decreases as the number of sampling layers increases, and remains fixed after it decreases to 1.

[0056] Figure 2 It extracts structure graphs for spatiotemporal features, extracts regional spatial information, aggregates multi-regional and multi-scale connections, and establishes global temporal correlations with minimal overhead.

[0057] Step 4: Finally, the obtained features are fed into a fully connected layer, and a softmax classifier is used for sentiment classification. This classifier uses three fully connected layers: the first fully connected layer reduces the dimension from 2048 to 1024, and uses dropout to prevent overfitting; the second fully connected layer reduces the dimension from 1024 to 200, and also uses dropout; the third fully connected layer reduces the dimension to the number of classes, does not use dropout, and outputs the sentiment classification result.

[0058] This invention extracts information of different sizes and locations through three sets of strip-shaped large-kernel depth convolutions of different scales, and evokes spatial attention through element-wise multiplication. At the same time, it aggregates the connections between these local information through multi-level non-covering small-kernel downsampling. In addition, this invention uses an odd-even binary tree structure to downsample time series, and combines non-causal dilated convolution and interaction learning between sequences. This enables global modeling, and the convolution kernels of different sequences in each layer can extract different but valuable information.

[0059] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0060] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A video sentiment classification method combining multi-level branch convolution and dilated interactive sampling, characterized in that, Includes the following steps: Step 1: Convert the frame sequence into a tensor and use a large kernel convolution layer for coverage downsampling to reduce the size of each frame image in the tensor. Step 2: Use a multi-branch convolution algorithm to extract the spatial features output from Step 1, and further reduce the spatial dimension. Step 3: Use the dilated interactive sampling algorithm to extract the time series output from Step 2, and shorten the computation time by using parallel dilated convolution after sampling. Step 4: Finally, input the obtained features into the sentiment classification model to classify the sentiment of the video. The extraction process of the multi-branch convolution algorithm in step 2 is as follows: Step 2.1: First, activate the output of Step 1 using the GELU nonlinear function and then send it to the standard input. Convolution is performed to extract local features, followed by LayerNorm processing before being fed into the system. Convolution increases the number of channels, effectively avoiding information loss, and can also capture the relationship between the channel dimensions of the feature map, thus enhancing the channel adaptability of the feature. Step 2.2, then... The features, after being increased in dimensionality through convolution, are fed into parallel branched strip convolution modules, which utilize... , , , , , Three pairs of depth-separable strip convolutional kernels extract region association information of different sizes and locations based on small, medium and large receptive fields; Step 2.3: Next, perform residual concatenation between the three outputs and inputs of the branched strip convolution, and then feed it into the second... Convolution models the relationship between different channels, The output of the convolution is directly used as the attention weight. The output of step 1 is weighted by matrix multiplication, and the weighted result is first passed through the GELU activation function to introduce non-linearity, and then through a third... Convolution is used for dimensionality reduction; Step 2.4: Finally, to fully extract features of different semantics and sizes, a multi-level architecture is introduced, consisting of four stages, each with a spatial resolution of [missing information]. , , , Each stage includes a stack of downsampling blocks and structure blocks. The downsampling blocks use a step size of 2 and a kernel size of [missing information]. The convolution is performed, and LayerNorm is applied after each downsampling operation; The extraction process of the dilation-interactive sampling algorithm in step 3 is as follows: Step 3.1: First, decompose the time series output in Step 2 into two subsequences. and Separate even-numbered elements from odd-numbered elements; Step 3.2: Then use different dilated convolution modules. Will and The hidden states are mapped to two separate hidden states, which are then converted into exponential form and multiplied element-wise with the hidden states of another sequence. The result of the element-wise multiplication is then residually joined with the original sequence.

2. The video sentiment classification method combining multi-level branch convolution and dilated interactive sampling according to claim 1, characterized in that, The process of using a large kernel convolution layer for coverage downsampling in step 1 is as follows: Each frame of the frame sequence is input into a convolutional network with 3 input channels, a stride of 4, and a kernel size of [size missing]. In a convolutional layer with 64 convolutions, the image size of each frame is downsampled to... In the formula, and It is the height and width of each frame input.

3. The video sentiment classification method combining multi-level branch convolution and dilated interactive sampling according to claim 1, characterized in that, The process in step 2.3 can be summarized by the following formula: In the formula, For convolution weights, for convolution, For the first The convolution results of each branch, , It is a set of depth-separated strip convolutions. As input features, For module output, This refers to matrix multiplication.

4. The video sentiment classification method combining multi-level branch convolution and dilated interactive sampling according to claim 1, characterized in that, The calculation formula in step 3.2 is as follows: In the formula, To output an odd sequence, For the next odd sequence, For residual links, This is a dilation convolution operation. For element-wise multiplication, exp is the pointer fetch operation. For the next even sequence, To output an even sequence.

5. The video sentiment classification method combining multi-level branch convolution and dilated interactive sampling according to claim 3, characterized in that, The dilated convolution module First, boundary effects of convolution operations are reduced by boundary replication. Then, dilated convolution with an initial dilation factor of 3 is used to convolve the sampling time sequence. GELU and Dropout are then used, and the dilation factor decreases as the number of sampling layers increases, and remains fixed after it decreases to 1.

6. The video sentiment classification method combining multi-level branch convolution and dilated interactive sampling according to claim 1, characterized in that, Step 4 specifically involves feeding the obtained features into a fully connected layer and using a softmax classifier for sentiment classification.

7. The video sentiment classification method combining multi-level branch convolution and dilated interactive sampling according to claim 6, characterized in that, The classifier uses three fully connected layers. The first fully connected layer reduces the dimension from 2048 to 1024 and uses dropout to prevent overfitting. The second fully connected layer reduces the dimension from 1024 to 200 and also uses dropout. The third fully connected layer reduces the dimension to the number of categories and does not use dropout, outputting the sentiment classification result.

Images