A video classification method, apparatus, storage medium, and terminal device.

Abstract

This invention discloses a video classification method, apparatus, storage medium, and terminal device. The method includes: acquiring a video segment to be classified, wherein the video segment to be classified includes several frames; obtaining deep appearance features and temporal features corresponding to the several frames; and determining the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features. By determining the category of the video segment to be classified based on its deep appearance features and temporal features, a more accurate video category can be obtained.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a video classification method, apparatus, storage medium, and terminal device. Background Technology

[0002] Currently, existing video classification methods all extract temporal and spatial semantic features from videos by training 3D convolutional networks or 2D+1D convolutional networks to obtain video classification models. Although convolution can extract temporal semantics, it cannot obtain long-term semantic information at shallow feature levels. As the input segment becomes longer, the information obtainable for frames with longer distances decreases. 3D or 2D+1D convolutional networks cannot extract long-term semantic information from videos, which leads to the inability to obtain accurate video classification based on 3D or 2D+1D convolutional networks. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a video classification method to address the shortcomings of the existing technology.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A video classification method, the method comprising:

[0006] Obtain the video segment to be classified, wherein the video segment to be classified includes several frames of images;

[0007] Based on the aforementioned frame images, deep appearance features and temporal features corresponding to the aforementioned frame images are obtained;

[0008] Based on the deep appearance features and the temporal features, the category corresponding to the video segment to be classified is determined.

[0009] The video classification method, wherein obtaining the video segment to be classified includes:

[0010] Get the videos to be categorized;

[0011] Based on the video to be classified, the video segment to be classified is obtained.

[0012] The video classification method, wherein obtaining the deep appearance features and temporal features corresponding to the plurality of frame images based on the plurality of frame images includes:

[0013] Based on the aforementioned frame images and the trained video classification model, the deep appearance features and temporal features corresponding to the aforementioned frame images are determined.

[0014] The video classification method, wherein the video classification model includes an appearance feature model, a semantic feature model, a feature fusion unit, and a fully connected layer, determines the deep appearance features and temporal features corresponding to the several frame images based on the trained video classification model, including:

[0015] For each of the plurality of frame images, the frame image is input into the appearance feature model, and the appearance feature model outputs a first feature map corresponding to the frame image, wherein the first feature map carries shallow appearance features;

[0016] The first feature map is input into the semantic feature model, and the semantic feature model outputs a sixth feature map and a fourth attention-bearing feature vector, wherein the sixth feature map carries deep appearance features and the fourth attention-bearing feature vector carries temporal features.

[0017] The video classification method, wherein the semantic feature model includes a pre-defined transformer network, convolutional modules, average pooling layers, feature fusion units, fully connected layers, a three-dimensional average pooling layer, a first convolutional block, a first normalization module, a second convolutional block, a second normalization module, a sigmoid function layer, and a ReLU function layer, wherein the convolutional module includes a first convolutional sub-module, a second convolutional sub-module, a third convolutional sub-module, and a fourth convolutional sub-module, and the transformer network includes a first transformer network layer, a second transformer network layer, a third transformer network layer, and a fourth transformer network layer; the step of inputting the first feature map into the semantic feature model and outputting a sixth feature map and a fourth attention-bearing feature vector through the semantic feature model includes:

[0018] The first feature map is input into the first convolutional submodule, and the second feature map of the frame image is output through the first convolutional submodule;

[0019] Based on the second feature map and the first transformer network layer, a first attention-bearing feature vector of the frame image is determined;

[0020] Based on the second feature map, the first attention-bearing feature vector, and the second convolutional submodule, a third feature map of the frame image is determined;

[0021] Based on the third feature map, the first attention-bearing feature vector, and the second transformer network layer, the second attention-bearing feature vector of the frame image is determined;

[0022] Based on the third feature map, the second attention-bearing feature vector, and the third convolutional submodule, the fourth feature map of the frame image is determined;

[0023] Based on the fourth feature map, the second attention-bearing feature vector, and the third transformer network layer, the third attention-bearing feature vector of the frame image is determined;

[0024] Based on the fourth feature map, the third attention-bearing feature vector, and the fourth convolutional submodule, the fifth feature map of the frame image is determined;

[0025] Based on the fifth feature map, the third attention-bearing feature vector, and the fourth transformer network layer, the fourth attention-bearing feature vector of the frame image is determined;

[0026] The fifth feature map is input into the average pooling layer, and the sixth feature map of the frame image is output.

[0027] The video classification method, wherein determining the first attention-bearing feature vector of the frame image based on the second feature map and the first transformer network layer includes:

[0028] The second feature map is input into the three-dimensional average pooling layer, and the first-level second feature vector is output through the three-dimensional average pooling layer.

[0029] The first-level second feature vector is input into the first convolutional block, and the second-level second feature vector is output through the first convolutional block;

[0030] The second-level second feature vector is input into the first normalization module, and the third-level second feature vector is output through the first normalization module.

[0031] The third-level second feature vector is input into the first transformer network layer, and the first attention-bearing feature vector of the frame image is output through the first transformer network layer.

[0032] The video classification method, wherein determining the third feature map of the frame image based on the second feature map, the first attention-bearing feature vector, and the second convolutional submodule includes:

[0033] The first attention-bearing feature vector is input into the second convolutional block, and the second convolutional block outputs the first-level first attention-bearing feature vector.

[0034] The first-level first attention feature vector is input into the second normalization module, and the second-level first attention feature vector is output through the second normalization module.

[0035] The second-level first attention feature vector is input into the Sigmoid function layer, and the third-level first attention feature vector is output through the Sigmoid function layer.

[0036] Based on the first attention-bearing feature vector, the second feature map, the third-level first attention-bearing feature vector, the ReLU function layer, and the second convolutional submodule, the third feature map of the frame image is determined.

[0037] The video classification method, wherein determining the second attention-bearing feature vector of the frame image based on the third feature map, the first attention-bearing feature vector, and the second transformer network layer includes:

[0038] The third feature map is input into the three-dimensional average pooling layer, and the first-level third feature vector is output through the three-dimensional average pooling layer.

[0039] Add the first-level third feature vector to the first attention-bearing feature vector to obtain the second-level third feature vector;

[0040] The second-level third feature vector is input into the first convolutional block, and the third-level third feature vector is output through the first convolutional block;

[0041] The third-level third feature vector is input into the first normalization module, and the fourth-level third feature vector is output through the first normalization module.

[0042] The fourth-level third feature vector is input into the second transformer network layer, and the second attention-bearing feature vector of the frame image is output through the second transformer network layer.

[0043] The video classification method, wherein determining the fourth feature map of the frame image based on the third feature map, the second attention-bearing feature vector, and the third convolutional submodule includes:

[0044] The second attention-bearing feature vector is input into the second convolutional block, and the second convolutional block outputs the first-level second attention-bearing feature vector.

[0045] The first-level second-band attention feature vector is input into the second normalization module, and the second-level second-band attention feature vector is output through the second normalization module;

[0046] The second-level second-band attention feature vector is input into the Sigmoid function layer, and the third-level second-band attention feature vector is output through the Sigmoid function layer.

[0047] Based on the second attention-bearing feature vector, the third-level second attention-bearing feature vector, the third feature map, the ReLU function layer, and the third convolutional submodule, the fourth feature map of the frame image is determined.

[0048] The video classification method, wherein determining the third attention-bearing feature vector of the frame image based on the fourth feature map, the second attention-bearing feature vector, and the third transformer network layer includes:

[0049] The fourth feature map is input into the three-dimensional average pooling layer, and the first-level fourth feature vector is output through the three-dimensional average pooling layer.

[0050] The first-level fourth feature vector and the second attention-bearing feature vector are added together to obtain the second-level fourth feature vector;

[0051] The second-level fourth feature vector is input into the first convolutional block, and the third-level fourth feature vector is output through the first convolutional block;

[0052] The third-level fourth feature vector is input into the first normalization module, and the fourth-level fourth feature vector is output through the first normalization module.

[0053] The fourth feature vector of the fourth level is input into the third transformer network layer, and the third attention-bearing feature vector of the frame image is output through the third transformer network layer.

[0054] The video classification method, wherein determining the fifth feature map of the frame image based on the fourth feature map, the third attention-bearing feature vector, and the fourth convolutional submodule includes:

[0055] The third attention-bearing feature vector is input into the second convolutional block, and the first-level third attention-bearing feature vector is output through the second convolutional block;

[0056] The first-level third-band attention feature vector is input into the second normalization module, and the second-level third-band attention feature vector is output through the second normalization module.

[0057] The second-level third-band attention feature vector is input into the Sigmoid function layer, and the third-level third-band attention feature vector is output through the Sigmoid function layer.

[0058] Based on the third-level third-band attention feature vector, the fourth feature map, the third-band attention feature vector, the ReLU function layer, and the fourth convolutional submodule, the fifth feature map of the frame image is determined.

[0059] The video classification method, wherein determining the fourth attention feature vector of the frame image based on the fifth feature map, the third attention feature vector, and the fourth transformer network layer includes:

[0060] The fifth feature map is input into the three-dimensional average pooling layer, and the first-level fifth feature vector is output through the three-dimensional average pooling layer.

[0061] The first-level fifth feature vector and the third attention-bearing feature vector are added together to obtain the second-level fifth feature vector;

[0062] The second-level fifth feature vector is input into the first convolutional block, and the third-level fifth feature vector is output through the first convolutional block;

[0063] The third-level fifth feature vector is input into the first normalization module, and the fourth-level fifth feature vector is output through the first normalization module.

[0064] The fourth-level fifth feature vector is input into the fourth transformer network layer, and the fourth attention-bearing feature vector of the frame image is output through the fourth transformer network layer.

[0065] The video classification method includes a first convolutional submodule, a second convolutional submodule, a third convolutional submodule, and a fourth convolutional submodule, each of which contains several residual modules.

[0066] The video classification method, wherein determining the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features includes:

[0067] The deep appearance features and the temporal features are input into the feature fusion unit for fusion to obtain fused semantic features;

[0068] The fused semantic features are input into a fully connected layer, and several video category probability sets are obtained through the fully connected layer.

[0069] Based on the probability sets of the aforementioned video categories, the video category corresponding to the video segment to be classified is determined.

[0070] The video classification method, wherein the training process of the video classification model includes:

[0071] Acquire several frame image samples and the predicted video category labels corresponding to the several frame image samples;

[0072] A number of frame image samples are input into a preset first network model, and the first network model outputs a predicted video category result corresponding to the number of frame image samples.

[0073] Based on the predicted video category results and the predicted video category labels, a loss function is obtained;

[0074] Based on the loss function, the first network model is trained to obtain a video classification model.

[0075] In the video classification method, each of the plurality of video category probability sets corresponds to a video category set.

[0076] Secondly, embodiments of the present invention provide a video classification device, wherein the device includes:

[0077] A module for acquiring video segments to be classified and several frames of images is used to acquire video segments to be classified and several frames of images corresponding to the video segments to be classified.

[0078] The deep appearance feature and temporal feature acquisition module is used to obtain deep appearance features and temporal features corresponding to the several frame images based on the several frame images;

[0079] The category acquisition module corresponding to the video segment to be classified is used to determine the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features.

[0080] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, performs the following steps:

[0081] Obtain the video segment to be classified, wherein the video segment to be classified includes several frames of images;

[0082] Based on the aforementioned frame images, deep appearance features and temporal features corresponding to the aforementioned frame images are obtained;

[0083] Based on the deep appearance features and the temporal features, the category corresponding to the video segment to be classified is determined.

[0084] Fourthly, embodiments of the present invention provide a terminal device, which includes: a processor, a memory, and a communication bus; the memory stores a computer-readable program that can be executed by the processor;

[0085] The communication bus enables communication between the processor and the memory;

[0086] When the processor executes the computer-readable program, it implements the steps in any of the video classification methods described above.

[0087] Beneficial effects: Compared with the prior art, the present invention provides a video classification method, apparatus, storage medium and terminal device, wherein the method includes acquiring a video segment to be classified, wherein the video segment to be classified includes several frames of images;

[0088] Based on the aforementioned frame images, deep appearance features and temporal features corresponding to the aforementioned frame images are obtained;

[0089] Based on the deep appearance features and the temporal features, the category corresponding to the video segment to be classified is determined. This invention determines the category of the video segment to be classified based on its deep appearance features and temporal features, resulting in more accurate video categories. Attached Figure Description

[0090] Figure 1 This is a schematic diagram illustrating an application scenario of the video classification method provided by the present invention.

[0091] Figure 2 A flowchart of the video classification method provided by the present invention.

[0092] Figure 3 This is a schematic diagram of the video classification model structure for the video classification method provided by the present invention.

[0093] Figure 4 This is a schematic diagram of the interaction between the transformer layer and the convolution module in the video classification method provided by the present invention.

[0094] Figure 5 This is a schematic diagram of the internal structure of the convolution module in the video classification method provided by the present invention.

[0095] Figure 6 This diagram illustrates the temporal and spatial attention allocation of the video classification model in the video classification method provided by this invention.

[0096] Figure 7 This is a schematic diagram illustrating the theoretical basis of the video classification model for the video classification method provided by this invention.

[0097] Figure 8 This is a schematic diagram illustrating the implementation of the video classification model of the video classification method provided by the present invention.

[0098] Figure 9 This is a schematic diagram of the video classification device provided by the present invention.

[0099] Figure 10The structural schematic diagram of the terminal device provided by the present invention. Detailed Implementation

[0100] This invention provides a video classification method, apparatus, storage medium, and terminal device. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0101] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0102] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0103] The inventors discovered through research that most current video semantic understanding technologies use 3D convolution or 2D+1D convolution to extract features in temporal and spatial semantics. Although convolution can extract temporal semantics, it cannot obtain long-term semantic information from shallow features. As the input segment becomes longer, less information can be obtained from frames with longer distances, resulting in poor performance in long-term video understanding.

[0104] To address the aforementioned problems, in this embodiment of the invention, a video segment to be classified is first acquired, wherein the video segment to be classified comprises several frames; then, based on the several frames, deep appearance features and temporal features corresponding to the several frames are obtained; finally, based on the deep appearance features and the temporal features, the category corresponding to the video segment to be classified is determined. This invention determines the category of the video segment to be classified based on its deep appearance features and temporal features, thus obtaining a more accurate video category.

[0105] For example, embodiments of the present invention can be applied to, for example... Figure 1 The scenario is illustrated below. In this scenario, firstly, terminal device 1 can collect video segments to be classified and upload them to server 2, so that server 2 can determine the probability set of video categories for the video segments to be classified. Server 2 can first obtain the deep appearance features and temporal features of the video segments to be classified; then, based on the obtained deep appearance features and temporal features, determine the category corresponding to the video segments to be classified.

[0106] It is understood that, in the above application scenarios, although the actions of the embodiments of the present invention are described as being partially performed by the terminal device 1 and partially performed by the server 2, these actions can also be performed entirely by the server 2 or entirely by the terminal device 1.

[0107] In the execution of this invention

[0108] The invention will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0109] In one implementation of this embodiment, such as Figure 2 As shown, the video classification method includes:

[0110] S100. Obtain the video segment to be classified, wherein the video segment to be classified includes several frames of images;

[0111] Specifically, the video segment to be classified can be a video of a preset duration extracted from a video to be classified, or it can be obtained by taking one frame every few frames from a video to be classified. The video segment to be classified includes several frames; in this embodiment, the video segment to be classified is a video clip of a preset duration that needs to be classified, and the video clip includes multiple frames, such as 32 frames. Accordingly, obtaining the video segment to be classified includes the following steps:

[0112] S101. Obtain the video to be categorized;

[0113] S102. Based on the video to be classified, obtain the video segment to be classified.

[0114] Specifically, video classification refers to classifying the content contained in a given video segment. After classification, the video can be categorized into classes, typically actions (e.g., making a cake), scenes (e.g., a beach), and objects (e.g., a table). Furthermore, two crucial features in video classification are appearance features and temporal features. With the prevalence of the internet, a large number of user-uploaded videos exist online. Therefore, video classification cannot be completed manually and requires deep learning algorithms. In practice, video classification can employ LSTM, 3D-ConvNet, or Two-Stream. To obtain the video classification results, the video to be classified must first be acquired. This video can be captured by a terminal device, sent to the terminal device by an external device, or downloaded by the terminal device from a server or cloud. Since a video consists of continuous multi-frame images, video processing deals with each frame. After acquiring the video to be classified, it can be downsampled at equal intervals to obtain the video segment to be classified, or it can be augmented to expand each segment of the original video into multiple video segments with a fixed number of k frames, ensuring that no two video segments have duplicate frames.

[0115] After obtaining the video segments to be classified, the following steps can be performed: Figure 2 The following steps are shown: S200, Based on the plurality of frame images, obtain the deep appearance features and temporal features corresponding to the plurality of frame images;

[0116] Specifically, classifying a video segment is based on extracted video features. The more accurate the feature extraction, the more accurate the video classification. Therefore, this embodiment of the invention obtains deep appearance features and temporal features corresponding to several frames of images in the video segment to be classified. Appearance features, also known as spatial features, refer to features containing multiple dimensions of information such as color, contour, angle, and position. Deep appearance features refer to the detailed features of the local image appearance, while temporal features refer to features in the video that contain the temporal sequence information of the frames.

[0117] In one implementation of the present invention, obtaining the deep appearance features and temporal features corresponding to the plurality of frame images includes the following steps:

[0118] S201. Based on the aforementioned frame images and the trained video classification model, determine the deep appearance features and temporal features corresponding to the aforementioned frame images.

[0119] In the feature extraction stage, existing technologies typically select global or local features of video frames to represent the video, such as HSV and LBP features. However, this representation method cannot comprehensively and accurately describe the video content and ignores the temporal features of the video. Video is a sequence of consecutive frames, thus possessing temporal and spatial (i.e., appearance features) correlations. An accurate description of a video should simultaneously capture both its spatial and temporal characteristics. This can be achieved through video classification models to determine the deep appearance and temporal features corresponding to the frames in the video.

[0120] In another implementation of the present invention, the video classification model includes an appearance feature model, a semantic feature model, a feature fusion unit, and a fully connected layer. Determining the deep appearance features and temporal features corresponding to the several frame images based on the trained video classification model includes the following steps:

[0121] S2011. For each of the plurality of frame images, the frame image is input into the appearance feature model, and the appearance feature model outputs a first feature map corresponding to the frame image, wherein the first feature map carries shallow appearance features.

[0122] S2012. Input the first feature map into the semantic feature model, and output a sixth feature map and a fourth attention-bearing feature vector through the semantic feature model, wherein the sixth feature map carries deep appearance features; and the fourth attention-bearing feature vector carries temporal features.

[0123] Specifically, the video classification model includes an appearance feature model, a semantic feature model, a feature fusion unit, and a fully connected layer. The appearance feature model is used to extract shallow appearance features from frame images. Shallow appearance features are the extracted global image features of the frame image, such as color information, contour information, corner information, position information, and other multi-dimensional information.

[0124] In this embodiment, the appearance feature model employs a 3D convolutional neural network, and the number of frames for a single sample input to the appearance feature model is 32 frames. The input dimensions of the frame images include the number of samples (B) inserted at one time, the number of channels (i.e., the number of features) (C), the length of the inserted samples (T) (i.e., the number of frame images), the height (H) of the feature map, and the width (W) of the feature map. Several frame images containing the input dimensions are simultaneously input into the convolutional layer to obtain an intermediate feature map. Then, the output intermediate feature map of the convolutional layer is input into the max pooling layer to obtain a first feature map of 32 frames. The shallow appearance features are carried in this first feature map of 32 frames.

[0125] For example, assuming the dimensions of several frame images are: B = 1, C = 3, T = 32, H = 256, W = 256, the output dimensions of the intermediate feature map obtained after the frame image input passes through the convolutional layer are: B = 1, C = 64, T = 32, H = 128, W = 128. It can be seen that after the frame image passes through the convolutional layer, the height and width of the feature map are halved, meaning the feature map becomes smaller. This smaller feature map reduces the computational load of subsequent networks, and the number of channels is increased from 3 to 64 according to the design. Then, the intermediate feature map is input to the max pooling layer. The dimensions of the first feature map of the frame image output by the max pooling layer are: B = 1, C = 64, T = 32, H = 64, W = 64. It can be seen that the height and width of the first feature map of the frame image output by the max pooling layer are further reduced, thus avoiding the introduction of excessive redundant information.

[0126] The semantic feature model is used to extract deep appearance features and temporal features from the input 32-frame feature maps. The output of the semantic feature model is a sixth feature map and a fourth attention-bearing feature vector. In this embodiment, the sixth feature map carries deep appearance features, and the fourth attention-bearing feature vector carries temporal features.

[0127] In another implementation of the present invention, the semantic feature model includes a preset transformer network, convolutional modules, average pooling layers, feature fusion units, fully connected layers, a three-dimensional average pooling layer, a first convolutional block, a first normalization module, a second convolutional block, a second normalization module, a sigmoid function layer, and a ReLU function layer. The convolutional modules include a first convolutional sub-module, a second convolutional sub-module, a third convolutional sub-module, and a fourth convolutional sub-module. The transformer network includes a first transformer network layer, a second transformer network layer, a third transformer network layer, and a fourth transformer network layer. The step of inputting the first feature map into the semantic feature model and outputting a sixth feature map and a fourth attention-bearing feature vector through the semantic feature model includes the following steps:

[0128] S20121. Input the first feature map into the first convolutional submodule, and output the second feature map of the frame image through the first convolutional submodule;

[0129] S20122. Based on the second feature map and the first transformer network layer, determine the first attention-enabled feature vector of the frame image;

[0130] S20123. Based on the second feature map, the first attention-bearing feature vector, and the second convolutional submodule, determine the third feature map of the frame image;

[0131] S20124. Based on the third feature map, the first attention-bearing feature vector, and the second transformer network layer, determine the second attention-bearing feature vector of the frame image;

[0132] S20125. Based on the third feature map, the second attention-bearing feature vector, and the third convolutional submodule, determine the fourth feature map of the frame image;

[0133] S20126. Based on the fourth feature map, the second attention-bearing feature vector, and the third transformer network layer, determine the third attention-bearing feature vector of the frame image;

[0134] S20127. Based on the fourth feature map, the third attention-bearing feature vector, and the fourth convolutional submodule, determine the fifth feature map of the frame image;

[0135] S20128. Based on the fifth feature map, the third attention-bearing feature vector, and the fourth transformer network layer, determine the fourth attention-bearing feature vector of the frame image;

[0136] S20129. Input the fifth feature map into the average pooling layer and output the sixth feature map of the frame image.

[0137] In existing technologies, 3D Convolutional Neural Networks (3D CNNs) can directly input video as 3D data. While the 3D input mode ensures the temporal continuity of the video, 3D CNNs cannot acquire long-term semantic information from shallow features. Therefore, this invention, after obtaining shallow appearance features through a 3D CNN, processes the feature maps output by the 3D CNN to obtain feature maps of different sizes, thus obtaining deep appearance features. These feature maps of different sizes are then input into a transformer network layer. Since self-attention and multi-head attention in the transformer network layer can provide complete attention enhancement in temporal semantics, deep appearance features and temporal features can be obtained after semantic modeling. In another implementation of this embodiment, the first, second, third, and fourth convolutional submodules are all composed of several layers of residual modules. Furthermore, the number of residual modules contained in the first, second, third, and fourth convolutional submodules can be the same or different.

[0138] For example, suppose the fourth convolutional submodule consists of 3 layers of residual modules, such as Figure 5 As shown, the three residual modules consist of a first residual module, a second residual module, and a third residual module. Each residual module contains a first convolutional layer and a second convolutional layer, and each convolutional layer contains a 3D convolutional module, a batch normalization module, and a ReLU function.

[0139] In S20121, the first feature map is input to the first convolutional submodule, and the second feature map of the frame image is output through the first convolutional submodule;

[0140] In one implementation of this invention, the first convolutional submodule consists of several residual modules. The output of each residual module is the input of the next residual module. By sequentially inputting the first feature map into each residual module, a second feature map of the frame image with a reduced feature map size can be output.

[0141] For example, suppose the size of the first feature map is: B = 1, C = 64, T = 32, H = 64, W = 64, and the size of the second feature map of the frame image is: B = 1, C = 128, T = 32, H = 32, W = 32.

[0142] In S20122, in this embodiment, the second feature map is output by the first convolutional submodule, and the size of the feature map is reduced compared to the size of the first feature map. Based on the second feature map with a smaller size and the first transformer network layer, a first attention-bearing feature vector of the frame image is obtained. For example, the first attention-bearing feature vector includes the temporal attention of the feature map whose height and width dimensions of the frame image are both reduced to half of their original size.

[0143] Specifically, step S20122 includes:

[0144] S201221. Input the second feature map into the three-dimensional average pooling layer, and output the first-level second feature vector through the three-dimensional average pooling layer;

[0145] In this embodiment of the invention, the second feature map contains the height and width information of the feature map. However, for the transformer network, the transformer network does not contain feature information in the two dimensions of height and width. Therefore, the second feature map needs to be input into the three-dimensional average pooling layer for dimensionality reduction, reducing the three-dimensional feature vector to a one-dimensional feature vector. That is, the height and width of the second feature map are both changed to 1 through the three-dimensional average pooling layer to obtain the first-level second feature vector.

[0146] For example, suppose the second feature map contains the following size information: the number of samples inserted at one time, B, is 1; the number of channels (i.e., the number of features), C, is 128; the length of the inserted samples, T (i.e., the number of frames), is 32; the height H of the feature map is 32; and the width W of the feature map is 32. Then the size information contained in the first-level second feature vector is: the number of samples inserted at one time, B, is 1; the number of deep apparent features, C, is 128; and the length of the inserted samples, T (i.e., the number of frames), is 32. s The value is 128, and the length T of the sample (i.e., the number of frames) is 32.

[0147] S201222: Input the first-level second feature vector into the first convolutional block, and output the second-level second feature vector through the first convolutional block;

[0148] In this embodiment of the invention, the first convolutional block is a one-dimensional convolutional block, and the first-level second feature vector is output as a second-level second feature vector through the one-dimensional convolutional block. This can reduce the amount of computation and avoid overfitting.

[0149] S201223. Input the second-level second feature vector into the first normalization module, and output the third-level second feature vector through the first normalization module;

[0150] In this embodiment of the invention, the first normalization module is a one-dimensional batch normalization module. The function of the one-dimensional batch normalization module is to normalize the values ​​to between 0 and 1. The second-level second feature map is input to the first normalization module, which can accelerate the convergence of the network. The third-level second feature vector is output through the one-dimensional batch normalization module.

[0151] S201224. Input the third-level second feature vector into the first transformer network layer, and output the first attention-bearing feature vector of the frame image through the first transformer network layer.

[0152] In this embodiment, as Figure 6As shown, the third-level second feature vector is the feature vector of the frame image after being reduced in size by the first convolutional submodule, then processed by three-dimensional average pooling, transforming it from a five-dimensional feature vector into a three-dimensional feature vector, and then processed by one-dimensional convolution and one-dimensional batch normalization. At this point, the feature vector conforms to the input vector representation of the transformer network. The third-level second feature vector is input to the first transformer network layer, and the first transformer network layer outputs the first attention-bearing feature vector of the frame image. The first transformer network layer is a transformer structure, which is a feedforward network using self-attention and multi-head attention mechanisms. Therefore, the first attention-bearing feature vector contains attention allocation between frames and attention allocation in subspaces. In addition, the number of features in the size information of the attention-bearing feature vector output from the transformer structure is the same as the number of deep apparent features in the size information of the input feature map. For example, if the number of deep apparent features in the size information of the second feature map is 128, then the number of features in the size information of the first attention-bearing feature vector is also 128.

[0153] In S20123, a third feature map of the frame image is determined based on the second feature map, the first attention-bearing feature vector, and the second convolutional submodule.

[0154] In this embodiment, the size of the feature map is further reduced based on the output of the previous convolutional submodule, the output of the previous transformer network layer, and the next convolutional submodule. That is, based on the second feature map, the first attention-bearing feature vector, and the second convolutional submodule, the third feature map of the frame image is determined. At this time, the third feature map carries deeper appearance features than the second feature map.

[0155] For example, the second feature map contains the following size information: the number of samples inserted at one time B is 1, the number of channels (i.e., the number of features) C is 128, the length of the inserted samples T (i.e., the number of frame images) is 32, the height H of the feature map is 32, and the width W of the feature map is 32. The third feature map of the frame image contains the following size information: the number of samples inserted at one time B is 1, the number of channels (i.e., the number of features) C is 256, the length of the inserted samples T (i.e., the number of frame images) is 32, the height H of the feature map is 16, and the width W of the feature map is 16.

[0156] Specifically, the steps in S20123 include:

[0157] S201231. Input the first attention-bearing feature vector into the second convolutional block, and output the first-level first attention-bearing feature vector through the second convolutional block;

[0158] In this embodiment, the second convolutional block is also a one-dimensional convolutional block. Inputting the first attention-bearing feature vector into the second convolutional block can reduce the amount of computation, avoid overfitting, and finally obtain the first-level first attention-bearing feature vector.

[0159] S201232, Input the first-level first attention feature vector into the second normalization module, and output the second-level first attention feature vector through the second normalization module;

[0160] In this embodiment, the second normalization module is also a one-dimensional batch normalization module. The function of the one-dimensional batch normalization module is to normalize the values ​​to between 0 and 1. Inputting the first-level first attention feature vector into the second normalization module can accelerate the convergence of the network and output the second-level first attention feature vector.

[0161] S201233: Input the second-level first attention feature vector into the Sigmoid function layer, and output the third-level first attention feature vector through the Sigmoid function layer;

[0162] In this embodiment, the Sigmoid function layer uses the Sigmoid function, which is a type of activation function. Its function is to add non-linear factors and is a non-linear regression that can increase the linear expression of the network. In other words, the Sigmoid function layer performs a non-linear mapping on the second-level first attention feature vector to obtain the third-level first attention feature vector. The third-level first attention feature vector carries the temporal attention corresponding to the second feature map.

[0163] S201234. Based on the first attention-bearing feature vector, the second feature map, the third-level first attention-bearing feature vector, the ReLU function layer, and the second convolutional submodule, determine the third feature map of the frame image.

[0164] In this embodiment, the third-level first attention-bearing feature vector is multiplied by the second feature map to obtain a first product feature map. Then, the first product feature map is added to the second feature map to obtain a first summed feature map. The first summed feature map is then input to the ReLU function layer, where a nonlinear factor is added to obtain a first nonlinear feature map. The first nonlinear feature map and the first attention-bearing feature vector are then input to the second convolution submodule to obtain the third feature map of the frame image. At this time, since both the third-level first attention-bearing feature vector and the first attention-bearing feature vector carry temporal attention, and the second feature map is post-processed by the second convolution submodule to obtain a third feature map with a changed feature map, the third feature map of the frame image not only carries spatial semantic information (deeper appearance features) that is smaller than the second feature map, but also carries the temporal semantic information corresponding to the third feature map.

[0165] S20124. Based on the third feature map, the first attention-bearing feature vector, and the second transformer network layer, determine the second attention-bearing feature vector of the frame image;

[0166] In this embodiment, the third feature map is output from the first convolutional submodule, and its size is further reduced compared to the second feature map. Based on the reduced-size third feature map, the first attention-bearing feature vector, and the second transformer network layer, the second attention-bearing feature vector of the frame image is obtained. For example, the second attention-bearing feature vector includes the temporal attention of the feature map whose height and width dimensions of the frame image are both reduced to one-quarter of their original size.

[0167] Specifically, step S20124 includes:

[0168] S201241. Input the third feature map into the three-dimensional average pooling layer, and output the first-level third feature vector through the three-dimensional average pooling layer;

[0169] In this embodiment, the third feature map contains the height and width information of the feature map. However, for the transformer network, the transformer network does not contain feature information in the two dimensions of height and width. Therefore, the third feature map needs to be input into the three-dimensional average pooling layer for dimensionality reduction, reducing the three-dimensional feature vector to a one-dimensional feature vector. That is, the height and width of the third feature map are both changed to 1 through the three-dimensional average pooling layer to obtain the first-level third feature vector.

[0170] For example, suppose the third feature map contains the following size information: the number of samples inserted at one time, B, is 1; the number of channels (i.e., the number of features), C, is 256; the length of the inserted samples, T (i.e., the number of frames), is 32; the height H of the feature map is 16; and the width W of the feature map is 16. Then the first-level third feature vector contains the following size information: the number of samples inserted at one time, B, is 1; the number of deep apparent features, C, is 256; and the length of the inserted samples, T (i.e., the number of frames), is 32. s The value is 256, and the length T of the sample (i.e., the number of frames) is 32.

[0171] S201242. Add the first-level third feature vector to the first attention-bearing feature vector to obtain the second-level third feature vector;

[0172] In this embodiment, since it is necessary to obtain not only the deep appearance features of the frame image but also the temporal features of the frame image for video, the first-level third feature vector contains the deep appearance features of the frame image, and the first attention-bearing feature vector contains the temporal features of the frame image. By adding the deep appearance features in the first-level third feature vector and the temporal features in the first attention-bearing feature vector together, the deep appearance features and temporal features of the second-level third feature vector can be obtained, that is, the deep appearance features and temporal features contained in the second-level third feature vector.

[0173] For example, the size information contained in the first-level third feature vector is: the number of samples inserted at one time, B, is 1, and the number of deep apparent features, C... s The length of the sample (T, i.e., the number of frames) is 32; the size information contained in the first attention-bearing feature vector is: the number of samples B inserted at one time is 1, and the number of deep appearance features C is 256. t The value is 128, and the length T of the sample (i.e., the number of frames) is 32.

[0174] S201243. Input the second-level third feature vector into the first convolutional block, and output the third-level third feature vector through the first convolutional block;

[0175] In this embodiment, since the computational cost of the neural network is very high, in order to reduce the computational cost and avoid overfitting, the second-level third feature map is convolved again in one dimension, that is, input into the first convolution block. The first convolution block is a one-dimensional convolution block, and finally the third-level third feature vector is obtained.

[0176] S201244. Input the third-level third feature vector into the first normalization module, and output the fourth-level third feature vector through the first normalization module;

[0177] In this embodiment, if the range of values ​​output by the neural network is very large, it will cause the network to converge very slowly. Therefore, it is necessary to normalize the output of the neural network, that is, to input the third-level third feature map into the first normalization module. The first normalization module is a one-dimensional batch normalization module. The range of values ​​of the fourth-level third feature map obtained by the first normalization module is between 0 and 1.

[0178] S201245. Input the fourth-level third feature vector into the second transformer network layer, and output the second attention-bearing feature vector of the frame image through the second transformer network layer.

[0179] In this embodiment, the value of the third feature vector of the fourth level is average pooled and contains only the three-dimensional features of the feature map. The three-dimensional features contain the deep appearance features of the feature map and also accumulate the temporal features of the first transformer network layer. Therefore, the second attention-bearing feature vector of the frame image is obtained by processing the reduced third feature map to obtain the temporal attention corresponding to the third feature map.

[0180] S20125. Based on the third feature map, the second attention-bearing feature vector, and the third convolutional submodule, determine the fourth feature map of the frame image;

[0181] In this embodiment, in order to obtain feature maps of different sizes and extract temporal semantic information from the feature maps of different sizes, it is necessary to further reduce the size of the feature maps. That is, based on the third feature map, the second attention-bearing feature vector and the third convolutional submodule, a fourth feature map of the frame image with a smaller size can be obtained.

[0182] For example, the third feature map contains the following size information: the number of samples placed at one time B is 1, the number of channels (i.e., the number of features) C is 256, the length of the placed samples T (i.e., the number of frames) is 32, the height H of the feature map is 16, and the width W of the feature map is 16. The fourth feature map contains the following size information: the number of samples placed at one time B is 1, the number of channels (i.e., the number of features) C is 512, the length of the placed samples T (i.e., the number of frames) is 32, the height H of the feature map is 8, and the width W of the feature map is 8.

[0183] Specifically, step S20125 includes:

[0184] S201251. Input the second attention-bearing feature vector into the second convolutional block, and output the first-level second attention-bearing feature vector through the second convolutional block;

[0185] In this embodiment, the second convolutional block is also a one-dimensional convolutional block. Inputting the second attention-bearing feature vector into the second convolutional block can reduce the amount of computation and avoid overfitting, ultimately obtaining the first-level second attention-bearing feature vector.

[0186] S201252, Input the first-level second-attention feature vector into the second normalization module, and output the second-level second-attention feature vector through the second normalization module;

[0187] In this embodiment, the second normalization module is also a one-dimensional batch normalization module. The function of the one-dimensional batch normalization module is to normalize the values ​​to between 0 and 1. Inputting the first-level second attention feature vector into the second normalization module can accelerate the convergence of the network and output the second-level second attention feature vector.

[0188] S201253. Input the second-level second-band attention feature vector into the Sigmoid function layer, and output the third-level second-band attention feature vector through the Sigmoid function layer;

[0189] In this embodiment, the Sigmoid function layer uses the Sigmoid function, which is a type of activation function. Its function is to add non-linear factors and is a non-linear regression that can increase the linear expression of the network. In other words, the Sigmoid function layer performs a non-linear mapping on the second-level second-band attention feature vector to obtain a third-level second-band attention feature vector. The third-level second-band attention feature vector carries the temporal attention corresponding to the third feature map.

[0190] S201254. Based on the second attention-bearing feature vector, the third-level second attention-bearing feature vector, the third feature map, the ReLU function layer, and the third convolutional submodule, a fourth feature map of the frame image is determined. At this time, the fourth feature map carries deeper appearance features than the third feature map.

[0191] In this embodiment, the third-level second attention-bearing feature vector is multiplied by the third feature map to obtain a second product feature map. Then, the second product feature map is added to the third feature map to obtain a second summed feature map. The second summed feature map is then input into the ReLU function layer to add a nonlinear factor, resulting in a second nonlinear feature map. The second nonlinear feature map and the second attention-bearing feature vector are then input into the third convolutional submodule to obtain the fourth feature map of the frame image. At this time, based on the same principle as the third feature map, the fourth feature map of the frame image not only carries less spatial semantic information than the third feature map, but also carries the temporal semantic information corresponding to the fourth feature map.

[0192] S20126. Based on the fourth feature map, the second attention-bearing feature vector, and the third transformer network layer, determine the third attention-bearing feature vector of the frame image;

[0193] In this embodiment, the fourth feature map is output from the third convolutional submodule, and its size is further reduced compared to the third feature map. Based on the reduced-size fourth feature map, the second attention-bearing feature vector, and the third transformer network layer, the third attention-bearing feature vector of the frame image is obtained. For example, the third attention-bearing feature vector includes the temporal attention of the feature map whose height and width dimensions of the frame image are both reduced to one-eighth of their original values.

[0194] Specifically, step S20126 includes:

[0195] S201261. Input the fourth feature map into the average pooling layer, and output the first-level fourth feature vector through the average pooling layer;

[0196] In this embodiment, the fourth feature map contains the height and width information of the feature map. However, for the transformer network, the transformer network does not contain feature information in the two dimensions of height and width. Therefore, the fourth feature map needs to be input into the three-dimensional average pooling layer for dimensionality reduction, reducing the three-dimensional feature vector to a one-dimensional feature vector. That is, the height and width of the fourth feature map are both changed to 1 through the three-dimensional average pooling layer to obtain the first-level fourth feature vector.

[0197] For example, suppose the fourth feature map contains the following size information: the number of samples inserted at one time, B, is 1; the number of channels (i.e., the number of features), C, is 512; the length of the inserted samples, T (i.e., the number of frames), is 32; the height H of the feature map is 8; and the width W of the feature map is 8. Then the size information contained in the first-level fourth feature vector is: the number of samples inserted at one time, B, is 1; the number of deep apparent features, C, is 512; and the length of the inserted samples, T (i.e., the number of frames), is 32. s The value is 512, and the length T of the sample (i.e., the number of frames) is 32.

[0198] S201262. Add the first-level fourth feature vector and the second attention-bearing feature vector to obtain the second-level fourth feature vector;

[0199] In this embodiment, since it is necessary to obtain not only the deep appearance features of the frame image but also the temporal features of the frame image for video, the first-level fourth feature vector contains the deep appearance features of the frame image, and the second attention-bearing feature vector contains the temporal features of the frame image. Adding the deep appearance features in the first-level fourth feature vector and the temporal features in the second attention-bearing feature vector together can obtain the deep appearance features and temporal features of the second-level fourth feature vector, that is, the deep appearance features and temporal features contained in the second-level fourth feature vector.

[0200] For example, the size information contained in the first-level fourth feature vector is as follows: the number of samples inserted at one time, B, is 1, and the number of deep apparent features, C... s The value is 512, and the length T of the sample (i.e., the number of frames) is 32; the size information contained in the second attention-bearing feature vector is: the number of samples B inserted at one time is 1, and the number of deep appearance features C is 1. t The value is 256, and the length T of the sample (i.e., the number of frames) is 32.

[0201] S201263. Input the second-level fourth feature vector into the first convolutional block, and output the third-level fourth feature vector through the first convolutional block;

[0202] In this embodiment, since the computational cost of the neural network is very high, in order to reduce the computational cost and avoid overfitting, the second-level fourth feature vector is subjected to one-dimensional convolution, that is, input into the first convolution block. The first convolution block is a one-dimensional convolution block, and finally the third-level fourth feature vector is obtained.

[0203] S201264. Input the third-level fourth feature vector into the first normalization module, and output the fourth-level fourth feature vector through the first normalization module;

[0204] In this embodiment, if the range of values ​​output by the neural network is very large, it will cause the network to converge very slowly. Therefore, it is necessary to normalize the output of the neural network, that is, to input the third-level fourth feature map into the first normalization module. The first normalization module is a one-dimensional batch normalization module. The range of values ​​of the fourth-level fourth feature map obtained by the first normalization module is between 0 and 1.

[0205] S201265. Input the fourth feature vector of the fourth level into the third transformer network layer, and output the third attention-bearing feature vector of the frame image through the third transformer network layer.

[0206] In this embodiment, the value of the fourth feature vector of the fourth level is average pooled and contains only the three-dimensional features of the feature map. The three-dimensional features contain the deep appearance features of the feature map and also accumulate the temporal features of the second transformer network layer. Therefore, the third attention feature vector of the frame image is obtained by processing the reduced fourth feature map to obtain the temporal attention corresponding to the fourth feature map.

[0207] S20127. Based on the fourth feature map, the third attention-bearing feature vector, and the fourth convolutional submodule, determine the fifth feature map of the frame image;

[0208] In this embodiment, in order to obtain feature maps of different sizes and extract temporal semantic information from the feature maps of different sizes, it is necessary to further reduce the size of the feature maps. That is, based on the fourth feature map, the third attention-bearing feature vector and the fourth convolutional submodule, a fifth feature map of the frame image with a smaller size can be obtained.

[0209] For example, the fourth feature map contains the following size information: the number of samples placed at one time B is 1, the number of channels (i.e., the number of features) C is 512, the length of the placed samples T (i.e., the number of frames) is 32, the height H of the feature map is 8, and the width W of the feature map is 8. The fifth feature map contains the following size information: the number of samples placed at one time B is 1, the number of channels (i.e., the number of features) C is 1024, the length of the placed samples T (i.e., the number of frames) is 32, the height H of the feature map is 4, and the width W of the feature map is 4.

[0210] Specifically, step S20127 includes:

[0211] S201271. Input the third attention-bearing feature vector into the second convolutional block, and output the first-level third attention-bearing feature vector through the second convolutional block;

[0212] In this embodiment, the second convolutional block is also a one-dimensional convolutional block. Inputting the third attention-bearing feature vector into the second convolutional block can reduce the amount of computation and avoid overfitting, ultimately obtaining the first-level third attention-bearing feature vector.

[0213] S201272, Input the first-level third-band attention feature vector into the second normalization module, and output the second-level third-band attention feature vector through the second normalization module;

[0214] In this embodiment, the second normalization module is also a one-dimensional batch normalization module. The function of the one-dimensional batch normalization module is to normalize the values ​​to between 0 and 1. Inputting the first-level third attention feature vector into the second normalization module can accelerate the convergence of the network and output the second-level third attention feature vector.

[0215] S201273. Input the second-level third-band attention feature vector into the Sigmoid function layer, and output the third-level third-band attention feature vector through the Sigmoid function layer;

[0216] In this embodiment, the Sigmoid function layer uses the Sigmoid function, which is a type of activation function. Its function is to add non-linear factors and is a non-linear regression that can increase the linear expression of the network. In other words, the Sigmoid function layer performs a non-linear mapping on the second-level third-band attention feature vector to obtain the third-level third-band attention feature vector. The third-level third-band attention feature vector carries the temporal attention corresponding to the fourth feature map.

[0217] S201274. Based on the third-level third-band attention feature vector, the fourth feature map, the third-band attention feature vector, the ReLU function layer, and the fourth convolutional submodule, determine the fifth feature map of the frame image.

[0218] In this embodiment, the third-level third attention-bearing feature vector is multiplied with the fourth feature map to obtain a third product feature map. Then, the third product feature map is added to the fourth feature map to obtain a third summed feature map. The third summed feature map is then input into the ReLU function layer to add a nonlinear factor, resulting in a third nonlinear feature map. The third nonlinear feature map and the third attention-bearing feature vector are then input into the fourth convolutional submodule to obtain the fifth feature map of the frame image. At this point, the fifth feature map carries deeper appearance features than the fourth feature map.

[0219] S20128. Based on the fifth feature map, the third attention-bearing feature vector, and the fourth transformer network layer, determine the fourth attention-bearing feature vector of the frame image;

[0220] In this embodiment, the fifth feature map is a feature map that is further scaled down from the fourth feature map. Based on the fifth feature map, the third attention-bearing feature vector, and the fourth transformer network layer, the fourth attention-bearing feature vector of the frame image is obtained. For example, the fourth attention-bearing feature vector includes the temporal attention of the feature map whose height and width dimensions of the frame image are both scaled down to one-sixteenth of their original values.

[0221] Specifically, step S20128 includes:

[0222] S201281. Input the fifth feature map into the three-dimensional average pooling layer, and output the first-level fifth feature vector through the three-dimensional average pooling layer;

[0223] In this embodiment, the fifth feature map contains the height and width information of the feature map. However, for the transformer network, the transformer network does not contain feature information in the two dimensions of height and width. Therefore, the fifth feature map needs to be input into the three-dimensional average pooling layer for dimensionality reduction, reducing the three-dimensional feature vector to a one-dimensional feature vector. That is, the height and width of the fifth feature map are both changed to 1 through the three-dimensional average pooling layer to obtain the first-level fifth feature vector.

[0224] For example, the fifth feature map contains the following size information: the number of samples inserted at one time, B, is 1; the number of channels (i.e., the number of features), C, is 1024; the length of the inserted samples, T (i.e., the number of frames), is 32; the height H of the feature map is 4; and the width W of the feature map is 4. Then, the size information contained in the first-level fifth feature vector is: the number of samples inserted at one time, B, is 1; the number of deep apparent features, C, is... s The value is 1024, and the length T of the sample (i.e., the number of frames) is 32.

[0225] S201282. Add the first-level fifth feature vector and the third attention-bearing feature vector to obtain the second-level fifth feature vector;

[0226] In this embodiment, since it is necessary to obtain not only the deep appearance features of the frame image but also the temporal features of the frame image for video, the first-level fifth feature map contains the deep appearance features of the frame image, and the third attention-bearing feature vector contains the temporal features of the frame image. Adding the deep appearance features of the first-level fifth feature vector and the temporal features of the third attention-bearing feature vector together can obtain the deep appearance features and temporal features of the second-level fifth feature vector, that is, the deep appearance features and temporal features contained in the second-level fifth feature vector.

[0227] For example, the size information contained in the fifth feature vector of the first level is as follows: the number of samples put in at one time, B, is 1, and the number of deep apparent features, C... s The value is 1024, and the length T of the sample (i.e., the number of frames) is 32; the size information contained in the third attention feature vector is: the number of samples B inserted at one time is 1, and the number of deep apparent features C is 1. tThe value is 512, and the length T of the sample (i.e., the number of frames) is 32.

[0228] S201283. Input the second-level fifth feature vector into the first convolutional block, and output the third-level fifth feature vector through the first convolutional block;

[0229] In this embodiment, since the computational cost of the neural network is very high, in order to reduce the computational cost and avoid overfitting, the second-level fifth feature vector is input into the first convolutional block, that is, input into the first convolutional block, which is a one-dimensional convolutional block, and finally the third-level fifth feature vector is obtained.

[0230] S201284. Input the third-level fifth feature vector into the first normalization module, and output the fourth-level fifth feature vector through the first normalization module;

[0231] In this embodiment, if the range of values ​​output by the neural network is very large, it will cause the network to converge very slowly. Therefore, it is necessary to normalize the output of the neural network, that is, to input the third-level fifth feature vector into the first normalization module. The first normalization module is a one-dimensional batch normalization module. The range of values ​​of the fourth-level fifth feature vector obtained by the first normalization module is between 0 and 1.

[0232] S201285. Input the fourth-level fifth feature vector into the fourth transformer network layer, and output the fourth attention-bearing feature vector of the frame image through the fourth transformer network layer.

[0233] In this embodiment, the value of the fifth feature map at the fourth level is averaged and pooled to include only the three-dimensional features of the feature map. These three-dimensional features include the deep appearance features of the feature map and also accumulate the temporal features of the third transformer network layer. Therefore, the fourth attention feature vector of the frame image is obtained by processing the reduced fifth feature map to obtain the temporal attention corresponding to the fifth feature map. In other words, the fourth attention feature vector carries temporal features.

[0234] S20129. Input the fifth feature map into the average pooling layer and output the sixth feature map of the frame image;

[0235] In this embodiment, as the depth of the neural network increases, it is necessary to decompose the network size. Therefore, the output signal of the neural network is subjected to average pooling to reduce the deviation of the estimated mean of the neural network model and improve the robustness of the model. Thus, the fifth feature map is passed through the average pooling layer to obtain the sixth feature map of the frame image. At this time, the sixth feature map carries deeper appearance features than the fifth feature map.

[0236] After obtaining the deep apparent features and temporal features, the following can be executed: Figure 2 The following steps are shown: S300, Determine the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features;

[0237] Specifically, after obtaining the deep appearance features and temporal features, the classifier can be used to determine the category corresponding to the video segment to be classified.

[0238] In one implementation of the present invention, determining the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features includes the following steps:

[0239] S301. Input the deep appearance features and the temporal features into the feature fusion unit for fusion to obtain fused semantic features;

[0240] S302. Input the fused semantic features into a fully connected layer to obtain several video category probability sets through the fully connected layer;

[0241] S303. Determine the video category corresponding to the video segment to be classified based on the probability set of the plurality of video categories.

[0242] In step S301, the sixth feature map is input into the average pooling layer to obtain the first-level sixth feature vector. The first-level sixth feature vector contains the deep appearance features of the feature map, and the fourth attention-bearing feature vector contains the temporal features of the feature map. The first-level sixth feature vector and the fourth attention-bearing feature vector are input into the feature fusion unit to obtain the sixth feature vector of the frame image containing the features and temporal features.

[0243] In step S302, the fully connected layer is used to classify the input data and extract and integrate useful information from the sixth feature vector to obtain several video category probability sets corresponding to the frame image.

[0244] In step S303, the video category corresponding to the video segment to be classified is determined according to the plurality of video category probability sets.

[0245] Specifically, each video category probability set in each video category probability set includes several video category probabilities, and the number of video category probabilities included in any two video category probability sets in each set is equal. For each video category probability set in the several video category probability sets, each video category probability in the set corresponds to a video category, and the video categories corresponding to any two video category probabilities in the set are different. Specifically, several frames of images corresponding to the video segment to be classified can obtain several video category probability sets for the video segment to be classified through the video classification model, and the category corresponding to the maximum value of the several video category probability sets is taken as the category of the video segment to be classified. For example, when several frames of images corresponding to the video segment to be classified are input into the video classification model, several video category probability sets for each category will be obtained, and the sum of the several video category probability sets is 1. For example, the probability of food is 0.1, the probability of sports is 0.2, the probability of news is 0.2, and the probability of animation is 0.5. Thus, the probability value of animation is the highest, and the category of the video segment to be classified is animation. Experiments have shown that, compared to the ordinary ResNet3D-50 network, which has a Top-1 accuracy of 42.4%, the video classification model in this embodiment has a Top-1 accuracy of 54.26%, representing an improvement of approximately 12%. This demonstrates that the video classification model in this embodiment has a significantly improved learning ability compared to ResNet3D.

[0246] In another implementation of the present invention, the training process of the video classification model includes the following steps: acquiring several frame image samples and predicted video category labels corresponding to the several frame image samples; inputting the several frame image samples into a preset first network model, and outputting predicted video category results corresponding to the several frame image samples through the first network model; obtaining a loss function based on the predicted video category results and the predicted video category labels; and training the first network model based on the loss function to obtain a video classification model.

[0247] In this embodiment, several frame image samples and corresponding predicted video category labels are first acquired. These frame image samples are then input into a preset first network model, which outputs the predicted video category results corresponding to the frame image samples. Based on the predicted video category results and the predicted video category labels, a loss function (cross-entropy loss function) is calculated. The first network model is then trained using this cross-entropy loss function, i.e., the cross-entropy loss function is backpropagated within the first network model, and the parameters in the first network model are updated. After calculating the loss function, it is determined whether the loss function is less than or equal to a preset threshold. If the loss function is less than or equal to the preset threshold, training ends. If the loss function is greater than the preset threshold, it is determined whether the number of training iterations of the first network model has reached a preset threshold. If the number of training iterations of the first network model has not reached the preset threshold, the network parameters of the first network model are corrected based on the loss function. If the number of training iterations of the first network model reaches the preset threshold, training ends. The preset threshold and the preset number of iterations threshold can be determined based on the model accuracy of the video classification model. In this embodiment, the preset training threshold can be the maximum number of training iterations for the first network model, such as 5000. Thus, by using the loss function and the number of training iterations to determine whether the training of the preset network model has ended, it is possible to avoid the first network model entering an infinite loop due to the loss function failing to reach a threshold value less than or equal to the preset threshold.

[0248] In another implementation of this embodiment, such as Figure 7-8 As shown, the first network model can be applied to temporal action localization. By obtaining a preset training set corresponding to temporal action localization, the first network model is trained to obtain a network model corresponding to temporal action localization.

[0249] In another implementation of this embodiment, such as Figure 7-8 As shown, the first network model can also be applied to anomaly detection. By obtaining a preset training set corresponding to anomaly detection, the first network model is trained to obtain a network model corresponding to anomaly detection.

[0250] In another implementation of this embodiment, such as Figure 7-8 As shown, the first network model can also be applied to video summarization. By obtaining a preset training set corresponding to the video summary, the first network model is trained to obtain a network model corresponding to the video summary.

[0251] In another implementation of this embodiment, such as Figure 7-8As shown, the first network model can also be applied to video description. By obtaining a preset training set corresponding to the video description, the first network model is trained to obtain a network model corresponding to the video description.

[0252] In another implementation of this embodiment, such as Figure 7-8 As shown, the first network model can also be applied to video generation. By obtaining a preset training set corresponding to video generation, the first network model can be trained to obtain a network model corresponding to video generation.

[0253] It should be noted that the above application scenarios are shown only for the purpose of understanding the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0254] Exemplary device

[0255] like Figure 9 As shown in the figure, this embodiment of the invention provides a video classification device, which includes a video segment to be classified and a plurality of frame image acquisition module 401, a deep appearance feature and temporal feature acquisition module 402, and a category acquisition module 403 corresponding to the video segment to be classified, wherein:

[0256] The video segment to be classified and several frame images acquisition module 401 is used to acquire the video segment to be classified and several frame images corresponding to the video segment to be classified.

[0257] The deep appearance feature and temporal feature acquisition module 402 is used to obtain deep appearance features and temporal features corresponding to the several frame images based on the several frame images;

[0258] The category acquisition module 403 for the video segment to be classified is used to determine the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features.

[0259] This embodiment of the invention acquires a video segment to be classified and several frames of images corresponding to the video segment by a video segment to be classified and a number of frames of images acquired by a module 401. Then, a deep appearance feature and temporal feature acquisition module 402 obtains deep appearance features and temporal features corresponding to the frames of images based on the frames of images. Finally, a category acquisition module 403 determines the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features. In this way, determining the category of the video segment to be classified based on its deep appearance features and temporal features can yield a more accurate video category.

[0260] In summary, this embodiment provides a video classification method, apparatus, storage medium, and terminal device. The method includes acquiring a video segment to be classified, wherein the video segment to be classified includes several frames; obtaining deep appearance features and temporal features corresponding to the several frames; and determining the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features. Thus, determining the category of the video segment to be classified based on its deep appearance features and temporal features allows for more accurate video category determination.

[0261] Based on the video classification method described above, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the video classification method described above.

[0262] Based on the above video classification method, the present invention also provides a terminal device, such as... Figure 10 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0263] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0264] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0265] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0266] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video classification method, characterized in that, The method includes: Obtain the video segment to be classified, wherein the video segment to be classified includes several frames of images; Based on the aforementioned frame images, deep appearance features and temporal features corresponding to the aforementioned frame images are obtained; The step of obtaining the deep appearance features and temporal features corresponding to the several frame images includes: Based on the aforementioned frame images and the trained video classification model, determine the deep appearance features and temporal features corresponding to the aforementioned frame images; The video classification model includes an appearance feature model, a semantic feature model, a feature fusion unit, and a fully connected layer. Based on the aforementioned frame images and the trained video classification model, the deep appearance features and temporal features corresponding to the aforementioned frame images are determined as follows: For each of the plurality of frame images, the frame image is input into the appearance feature model, and the appearance feature model outputs a first feature map corresponding to the frame image, wherein the first feature map carries shallow appearance features; The first feature map is input into the semantic feature model, and the semantic feature model outputs a sixth feature map and a fourth attention-bearing feature vector, wherein the sixth feature map carries deep appearance features; and the fourth attention-bearing feature vector carries temporal features. The semantic feature model includes a pre-defined transformer network, convolutional modules, average pooling layers, feature fusion units, fully connected layers, a 3D average pooling layer, a first convolutional block, a first normalization module, a second convolutional block, a second normalization module, a sigmoid function layer, and a ReLU function layer. The convolutional modules include a first convolutional sub-module, a second convolutional sub-module, a third convolutional sub-module, and a fourth convolutional sub-module. The transformer network includes a first transformer network layer, a second transformer network layer, a third transformer network layer, and a fourth transformer network layer. The step of inputting the first feature map into the semantic feature model and outputting a sixth feature map and a fourth attention-enabled feature vector through the semantic feature model includes: The first feature map is input into the first convolutional submodule, and the second feature map of the frame image is output through the first convolutional submodule; Based on the second feature map and the first transformer network layer, a first attention-bearing feature vector of the frame image is determined; Based on the second feature map, the first attention-bearing feature vector, and the second convolutional submodule, a third feature map of the frame image is determined; Based on the third feature map, the first attention-bearing feature vector, and the second transformer network layer, the second attention-bearing feature vector of the frame image is determined; Based on the third feature map, the second attention-bearing feature vector, and the third convolutional submodule, the fourth feature map of the frame image is determined; Based on the fourth feature map, the second attention-bearing feature vector, and the third transformer network layer, the third attention-bearing feature vector of the frame image is determined; Based on the fourth feature map, the third attention-bearing feature vector, and the fourth convolutional submodule, the fifth feature map of the frame image is determined; Based on the fifth feature map, the third attention-bearing feature vector, and the fourth transformer network layer, the fourth attention-bearing feature vector of the frame image is determined; The fifth feature map is input into the average pooling layer, and the sixth feature map of the frame image is output. Based on the deep appearance features and the temporal features, the category corresponding to the video segment to be classified is determined.

2. The video classification method according to claim 1, characterized in that, The acquisition of the video segment to be classified includes: Get the videos to be categorized; Based on the video to be classified, the video segment to be classified is obtained.

3. The video classification method according to claim 1, characterized in that, The step of determining the first attention-bearing feature vector of the frame image based on the second feature map and the first transformer network layer includes: The second feature map is input into the three-dimensional average pooling layer, and the first-level second feature vector is output through the three-dimensional average pooling layer. The first-level second feature vector is input into the first convolutional block, and the second-level second feature vector is output through the first convolutional block; The second-level second feature vector is input into the first normalization module, and the third-level second feature vector is output through the first normalization module. The third-level second feature vector is input into the first transformer network layer, and the first attention-bearing feature vector of the frame image is output through the first transformer network layer.

4. The video classification method according to claim 1, characterized in that, The step of determining the third feature map of the frame image based on the second feature map, the first attention-bearing feature vector, and the second convolutional submodule includes: The first attention-bearing feature vector is input into the second convolutional block, and the second convolutional block outputs the first-level first attention-bearing feature vector. The first-level first attention-bearing feature vector is input into the second normalization module, and the second-level first attention-bearing feature vector is output through the second normalization module. The second-level first attention-bearing feature vector is input into the Sigmoid function layer, and the third-level first attention-bearing feature vector is output through the Sigmoid function layer. Based on the first attention-bearing feature vector, the second feature map, the third-level first attention-bearing feature vector, the ReLU function layer, and the second convolutional submodule, the third feature map of the frame image is determined.

5. The video classification method according to claim 1, characterized in that, The step of determining the second attention-bearing feature vector of the frame image based on the third feature map, the first attention-bearing feature vector, and the second transformer network layer includes: The third feature map is input into the three-dimensional average pooling layer, and the first-level third feature vector is output through the three-dimensional average pooling layer. The first-level third feature vector is added to the first attention-bearing feature vector to obtain the second-level third feature vector; The second-level third feature vector is input into the first convolutional block, and the third-level third feature vector is output through the first convolutional block; The third-level third feature vector is input into the first normalization module, and the fourth-level third feature vector is output through the first normalization module. The fourth-level third feature vector is input into the second transformer network layer, and the second attention-bearing feature vector of the frame image is output through the second transformer network layer.

6. The video classification method according to claim 1, characterized in that, The process of determining the fourth feature map of the frame image based on the third feature map, the second attention-bearing feature vector, and the third convolutional submodule includes: The second attention-bearing feature vector is input into the second convolutional block, and the second convolutional block outputs the first-level second attention-bearing feature vector. The first-level second-band attention feature vector is input into the second normalization module, and the second-level second-band attention feature vector is output through the second normalization module; The second-level second-band attention feature vector is input into the Sigmoid function layer, and the third-level second-band attention feature vector is output through the Sigmoid function layer. Based on the second attention-bearing feature vector, the third-level second attention-bearing feature vector, the third feature map, the ReLU function layer, and the third convolutional submodule, the fourth feature map of the frame image is determined.

7. The video classification method according to claim 1, characterized in that, The step of determining the third attention-bearing feature vector of the frame image based on the fourth feature map, the second attention-bearing feature vector, and the third transformer network layer includes: The fourth feature map is input into the three-dimensional average pooling layer, and the first-level fourth feature vector is output through the three-dimensional average pooling layer. The first-level fourth feature vector and the second attention-bearing feature vector are added together to obtain the second-level fourth feature vector; The second-level fourth feature vector is input into the first convolutional block, and the third-level fourth feature vector is output through the first convolutional block; The third-level fourth feature vector is input into the first normalization module, and the fourth-level fourth feature vector is output through the first normalization module. The fourth feature vector of the fourth level is input into the third transformer network layer, and the third attention-bearing feature vector of the frame image is output through the third transformer network layer.

8. The video classification method according to claim 1, characterized in that, The process of determining the fifth feature map of the frame image based on the fourth feature map, the third attention-bearing feature vector, and the fourth convolutional submodule includes: The third attention-bearing feature vector is input into the second convolutional block, and the first-level third attention-bearing feature vector is output through the second convolutional block; The first-level third-band attention feature vector is input into the second normalization module, and the second-level third-band attention feature vector is output through the second normalization module. The second-level third-band attention feature vector is input into the Sigmoid function layer, and the third-level third-band attention feature vector is output through the Sigmoid function layer. Based on the third-level third-band attention feature vector, the fourth feature map, the third-band attention feature vector, the ReLU function layer, and the fourth convolutional submodule, the fifth feature map of the frame image is determined.

9. The video classification method according to claim 1, characterized in that, The determination of the fourth attention feature vector of the frame image based on the fifth feature map, the third attention feature vector, and the fourth transformer network layer includes: The fifth feature map is input into the three-dimensional average pooling layer, and the first-level fifth feature vector is output through the three-dimensional average pooling layer. The first-level fifth feature vector and the third attention-bearing feature vector are added together to obtain the second-level fifth feature vector; The second-level fifth feature vector is input into the first convolutional block, and the third-level fifth feature vector is output through the first convolutional block; The third-level fifth feature vector is input into the first normalization module, and the fourth-level fifth feature vector is output through the first normalization module. The fourth-level fifth feature vector is input into the fourth transformer network layer, and the fourth attention-bearing feature vector of the frame image is output through the fourth transformer network layer.

10. The video classification method according to claim 1, characterized in that, The first convolution submodule, the second convolution submodule, the third convolution submodule, and the fourth convolution submodule each contain several residual modules.

11. The video classification method according to claim 1, characterized in that, The step of determining the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features includes: The deep appearance features and the temporal features are input into the feature fusion unit for fusion to obtain fused semantic features; The fused semantic features are input into a fully connected layer, and several video category probability sets are obtained through the fully connected layer; based on the several video category probability sets, the video category corresponding to the video segment to be classified is determined.

12. The video classification method according to any one of claims 1-11, characterized in that, The training process of the video classification model includes: Acquire several frame image samples and the predicted video category labels corresponding to the several frame image samples; A number of frame image samples are input into a preset first network model, and the first network model outputs a predicted video category result corresponding to the number of frame image samples. Based on the predicted video category results and the predicted video category labels, a loss function is obtained; Based on the loss function, the first network model is trained to obtain a video classification model.

13. The video classification method according to claim 12, characterized in that, Each of the several video category probability sets Each video category probability set corresponds to a video category set.

14. A video classification device, characterized in that, The video classification device is used to implement the video classification method according to any one of claims 1-13, the device comprising: A module for acquiring video segments to be classified and several frames of images is used to acquire video segments to be classified and several frames of images corresponding to the video segments to be classified. The deep appearance feature and temporal feature acquisition module is used to obtain deep appearance features and temporal features corresponding to the several frame images based on the several frame images; The category acquisition module corresponding to the video segment to be classified is used to determine the category corresponding to the video segment to be classified based on the deep appearance features and the temporal features.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the video classification method as described in any one of claims 1-13.

16. A terminal device, characterized in that, include: Processor, memory, and communication bus; The memory stores a computer-readable program that can be executed by the processor; The communication bus enables communication between the processor and the memory; When the processor executes the computer-readable program, it implements the video classification method as described in any one of claims 1-13.

Images