Video feature extraction method, device, equipment and storage medium

Abstract

The application discloses a video feature extraction method, device and equipment and a storage medium, and belongs to the field of video processing. In the embodiment of the application, in the process of extracting video features, a first feature extraction model based on supervised training and a second feature extraction model based on self-supervised training are introduced. Since the first feature extraction model is trained based on sample videos and labels corresponding to the sample videos, the target of the output first visual feature is stronger, and it is more suitable to add video labels to target videos. The second feature extraction model is a model based on self-supervised training, and the target of the feature predicted by the self-supervised model is not as strong as that of the supervised model. Therefore, the second visual feature extracted by the second feature extraction model includes more abundant information, the first visual feature and the second visual feature are complementary to each other, and the obtained target visual feature can more completely reflect the features of the target video.

Description

Technical Field

[0001] This application relates to the field of video processing, and in particular to a video feature extraction method, apparatus, device, and storage medium. Background Technology

[0002] With the development of computer technology, more and more users are watching videos through various video platforms. In order to improve user engagement and increase daily active users, each video platform recommends videos to users based on their preferences.

[0003] In related technologies, various video platforms use artificial intelligence algorithms to recommend videos to users. For example, they may perform transfer learning on image recommendation models pre-trained using publicly available datasets to obtain video recommendation models. These models then extract video features of the videos to be recommended and recommend videos to users based on these features.

[0004] However, due to the significant differences between image and video data in publicly available datasets, video recommendation models obtained through transfer learning cannot accurately extract video features, resulting in poor performance in subsequent video recommendations. Summary of the Invention

[0005] This application provides a video feature extraction method, apparatus, device, and storage medium, which can improve the accuracy of video feature extraction. The technical solution is as follows:

[0006] On the one hand, a video feature extraction method is provided, the method comprising:

[0007] Obtain multiple video frames from the target video;

[0008] The first number of video frames are input into the first feature extraction model. The first feature extraction model is used to extract the first visual features of the target video. The first feature extraction model is a visual feature extraction model trained with multiple sample videos as training samples and multiple sample video labels corresponding to the multiple sample videos as supervised training.

[0009] The second number of video frames are input into the second feature extraction model. The second feature extraction model extracts the second number of video frame features of the second number of video frames. The second feature extraction model is a visual feature extraction model obtained by self-supervised training with multiple sample video frames as training samples.

[0010] The first visual feature of the target video and the second number of video frame features are fused together to obtain the target visual feature of the target video.

[0011] In one possible implementation, extracting the second number of video frame features from the second number of video frames using the second feature extraction model includes:

[0012] The second feature extraction model is used to perform convolution processing on the second number of video frames to obtain the second number of video frame features of the second number of video frames.

[0013] In one possible implementation, the training method for the first feature extraction model includes:

[0014] Multiple sample video frames are obtained from the multiple sample videos respectively;

[0015] The sample video frame is input into the first model, and the first sample visual features of the corresponding first sample video are extracted through the first model.

[0016] Based on the visual features of the first sample, the first model is used to predict the predicted video label of the first sample video.

[0017] Based on the difference information between the predicted video label and the sample video label corresponding to the first sample video, the model parameters of the first model are adjusted.

[0018] In response to the fact that the model parameters of the first model meet the first target condition, the first model is determined as the first feature extraction model.

[0019] In one possible implementation, obtaining the first sample video frame from the plurality of sample video frames includes:

[0020] Obtain a sample video frame queue from the plurality of sample video frames;

[0021] The first sample video frame is obtained from the sample video frame queue.

[0022] In one possible implementation, after adjusting the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the reference video frame, the method further includes:

[0023] Based on the model parameters of the second model, the model parameters of the reference model are adjusted.

[0024] On the one hand, a video feature extraction device is provided, the device comprising:

[0025] The video frame acquisition module is used to acquire multiple video frames from the target video;

[0026] The first input module is used to input a first number of video frames into a first feature extraction model, and extract the first visual features of the target video through the first feature extraction model. The first feature extraction model is a visual feature extraction model trained with multiple sample videos as training samples and multiple sample video labels corresponding to the multiple sample videos as supervised training.

[0027] The second input module is used to input a second number of video frames into the second feature extraction model, and extract the second number of video frame features of the second number of video frames through the second feature extraction model. The second feature extraction model is a visual feature extraction model obtained by self-supervised training with multiple sample video frames as training samples.

[0028] The feature fusion module is used to fuse the first visual features of the target video and the second number of video frame features to obtain the target visual features of the target video.

[0029] In one possible implementation, the first feature extraction model includes a first sub-model and a second sub-model. The first sub-model is used to extract static visual features of the target video, and the second sub-model is used to extract dynamic visual features of the target video. The first input module is used to uniformly sample a first number of video frames to obtain a third number of video frames, where the third number is less than the first number. The third number of video frames are then input into the first sub-model, and the first number of video frames are then input into the second sub-model.

[0030] In one possible implementation, the first input module is used to perform convolution processing on the third number of video frames using multiple convolution kernels of the first sub-model to obtain static visual features of the target video; to perform convolution processing on the first number of video frames using multiple convolution kernels of the second sub-model to obtain dynamic visual features of the target video; and to obtain a first visual feature of the target video based on the static visual features and the dynamic visual features; wherein the number of convolution kernels in the first sub-model is greater than the number of convolution kernels in the second sub-model.

[0031] In one possible implementation, the first input module is used to perform spatiotemporal attention processing on the static visual features to obtain a first static feature of the target video; perform spatiotemporal attention processing on the dynamic visual features to obtain a first dynamic feature of the target video; fuse the first static feature and the first dynamic feature to obtain a fused feature; perform global pooling processing on the fused feature and the first dynamic feature respectively to obtain a second static feature and a second dynamic feature of the target video; and fuse the second static feature and the second dynamic feature to obtain a first visual feature of the target video.

[0032] In one possible implementation, the first input module is used to perform channel attention processing on the static visual features to obtain static channel attention weights corresponding to the static visual features; multiply the static channel attention weights by the static visual features to obtain a third static feature; perform max pooling, mean pooling, and dilated convolution processing on the third static feature to obtain a first spatiotemporal attention weight corresponding to the third static feature; multiply the third static feature by the first spatiotemporal attention weight and then add it to the static visual features to obtain the first static feature.

[0033] In one possible implementation, the first input module is used to perform max pooling and mean pooling on the third static feature to obtain a fourth static feature; to perform convolution processing on the fourth static feature using convolution kernels with different dilatation rates to obtain multiple fifth static features; and to perform normalization processing after adding the multiple fifth static features to obtain the first spatiotemporal attention weight corresponding to the third static feature.

[0034] In one possible implementation, the first input module is used to perform channel attention processing on the dynamic visual features to obtain dynamic channel attention weights corresponding to the dynamic visual features; multiply the dynamic channel attention weights by the dynamic visual features to obtain a third dynamic feature; perform max pooling, mean pooling, and dilated convolution processing on the third dynamic feature to obtain a second spatiotemporal attention weight corresponding to the third dynamic feature; multiply the third dynamic feature by the second spatiotemporal attention weight and then add it to the dynamic visual features to obtain the first dynamic feature.

[0035] In one possible implementation, the first input module is used to perform max pooling and mean pooling on the third dynamic feature to obtain a fourth dynamic feature; to perform convolution processing on the fourth dynamic feature using convolution kernels with different dilatation rates to obtain multiple fifth dynamic features; and to perform normalization processing after adding the multiple fifth dynamic features to obtain the second spatiotemporal attention weight corresponding to the third dynamic feature.

[0036] In one possible implementation, the first input module is used to perform mean pooling on the second static feature and the second dynamic feature respectively, and then concatenate the mean pooled second static feature and the mean pooled second dynamic feature to obtain the first visual feature of the target video.

[0037] In one possible implementation, the second input module is used to perform convolution processing on the second number of video frames respectively through the second feature extraction model to obtain the second number of video frame features of the second number of video frames.

[0038] In one possible implementation, the feature fusion module is used to perform mean pooling on the second number of video frame features to obtain the second visual feature of the target video; and to concatenate the first visual feature and the second visual feature to obtain the target visual feature of the target video.

[0039] In one possible implementation, the training device for the first feature extraction model includes:

[0040] The first sample video frame acquisition module is used to acquire multiple sample video frames from the multiple sample videos respectively;

[0041] The first sample input module is used to input the sample video frame into the first model and extract the first sample visual features of the corresponding first sample video through the first model.

[0042] The label prediction module is used to predict the predicted video label of the first sample video based on the visual features of the first sample using the first model.

[0043] The first parameter adjustment module is used to adjust the model parameters of the first model based on the difference information between the predicted video label and the sample video label corresponding to the first sample video.

[0044] The first model acquisition module is used to determine the first model as the first feature extraction model in response to the first model parameters meeting the first target condition.

[0045] In one possible implementation, the training apparatus for the second feature extraction model includes:

[0046] The second sample video frame acquisition module is used to acquire the first sample video frame from the plurality of sample video frames;

[0047] The data augmentation module is used to perform two types of data augmentation on the first sample video frame to obtain a first augmented video frame and a second augmented video frame.

[0048] The second sample input module is used to input the first enhanced video frame into the second model and extract the first sample video frame features of the first enhanced video frame through the second model.

[0049] The third sample input module is used to input the second enhanced video frame into the reference model, and extract the reference video frame features of the second enhanced video frame through the reference model. The reference model and the second model have the same model structure.

[0050] The second model parameter adjustment module is used to adjust the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the reference video frame.

[0051] The second model acquisition module is used to determine the second model as the second feature extraction model in response to the fact that the model parameters of the second model meet the second target condition.

[0052] In one possible implementation, the second sample video frame acquisition module is configured to acquire a sample video frame queue from the plurality of sample video frames; and acquire the first sample video frame from the sample video frame queue.

[0053] In one possible implementation, the second sample video frame acquisition module is further configured to acquire a second sample video frame from the plurality of sample videos;

[0054] The third sample input module is further configured to input the second sample video frame into the reference model, and extract the second sample video frame features of the second sample video frame through the reference model;

[0055] The second model parameter adjustment module is further configured to adjust the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the second sample video frame.

[0056] In one possible implementation, the second model parameter adjustment module is further configured to adjust the model parameters of the reference model based on the model parameters of the second model.

[0057] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the video feature extraction method.

[0058] On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the video feature extraction method.

[0059] On one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the video feature extraction method described above.

[0060] In this embodiment, during the extraction of video features, a first feature extraction model based on supervised training and a second feature extraction model based on self-supervised training are introduced. Since the first feature extraction model is trained based on sample videos and their corresponding labels, the output first visual features are highly targeted and suitable for adding video tags to target videos. The second feature extraction model is based on self-supervised training. Since the self-supervised model's feature prediction is not as targeted as the supervised model, the second visual features extracted by the second feature extraction model contain richer information. The first and second visual features complement each other, and the resulting target visual features can more completely reflect the characteristics of the target video. Attached Figure Description

[0061] To illustrate the technical solutions in the embodiments of this application more clearly, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of the implementation environment of a video feature extraction method provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the structure of a first feature extraction model provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of the structure of a spatiotemporal attention module provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of a training method for a second feature extraction model provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of a training method for a second feature extraction model provided in an embodiment of this application;

[0067] Figure 6 This is a flowchart of a video feature extraction method provided in an embodiment of this application;

[0068] Figure 7 This is a flowchart of a video feature extraction method provided in an embodiment of this application;

[0069] Figure 8 This is a flowchart of a video feature extraction method provided in an embodiment of this application;

[0070] Figure 9 This is a schematic diagram illustrating the effect of adding video tags according to an embodiment of this application;

[0071] Figure 10 This is a schematic diagram of the structure of a video feature extraction device provided in an embodiment of this application;

[0072] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0073] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0075] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0076] In this application, the term "at least one" means one or more, and "multiple" means two or more, for example, multiple reference face images means two or more reference face images.

[0077] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0078] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0079] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0080] Attention weights represent the importance of a piece of data during training or prediction. Importance indicates the magnitude of the influence of input data on output data. Data with high importance corresponds to higher attention weights, while data with low importance corresponds to lower attention weights. The importance of data varies in different scenarios, and training the model to assign attention weights is essentially the process of determining data importance.

[0081] Figure 1 This is a schematic diagram illustrating the implementation environment of a video feature extraction method provided in this application embodiment. See also... Figure 1 The implementation environment may include terminal 110 and server 140.

[0082] Terminal 110 is connected to server 140 via a wireless or wired network. Optionally, terminal 110 may be a smartphone, tablet, laptop, desktop computer, smartwatch, etc., but is not limited to these. Terminal 110 has an application installed and running that supports video playback.

[0083] Optionally, the server is an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0084] Optionally, terminal 110 refers to one of a plurality of terminals. This application embodiment only uses terminal 110 as an example for illustration.

[0085] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more, in which case other terminals may also be included in the above implementation environment. This application does not limit the number of terminals or the type of device in its embodiments.

[0086] The video feature extraction method provided in this application can be applied to scenarios such as adding video tags and video recommendations. These two application scenarios will be introduced below.

[0087] In scenarios involving adding video tags to videos, after obtaining the target visual features of the target video using the technical solution provided in this application embodiment, the server queries the video tag library based on the target visual features to obtain the video tag corresponding to the target visual features and adds the video tag to the target video. In some embodiments, the video tag library includes multiple types of video tags, such as video type tags, video region tags, video highlight tags, video actor tags, and video title tags. The server can match different types of video tags to the target video based on the target visual features, thereby increasing the richness of the video tags for the target video, and subsequently, it can use the rich video tags to improve user profiles.

[0088] Furthermore, in scenarios involving adding video tags to videos, after obtaining the target visual features of the target video through the technical solution provided in this application embodiment, the server can also use the target visual features as input to a subsequent multimodal fusion model. The multimodal fusion model then fuses the visual, audio, and textual features of the target video to obtain multimodal features that more accurately represent the target video. Based on these multimodal features, the server adds video tags, enabling the addition of more appropriate video tags to the target video.

[0089] In video recommendation scenarios, after obtaining the target visual features of a target video using the technical solution provided in this application embodiment, the server can compare the target visual features with the visual features of other videos to obtain the similarity between the target visual features and the visual features of other videos. The server recommends the video corresponding to the visual features with the highest similarity to the user. Of course, the server can also sort the videos according to similarity and prioritize recommending videos with higher similarity to the user.

[0090] In this embodiment, the server can implement the video feature extraction method provided in this application using two models, namely a first feature extraction model and a second feature extraction model. The first feature extraction model is a visual feature extraction model obtained based on supervised learning, and the second feature extraction model is a visual feature extraction model obtained based on self-supervised learning. Both the first and second feature extraction models are used to extract visual features from the video.

[0091] To provide a clearer explanation of the technical solutions provided in the embodiments of this application, the model structures of the first feature extraction model and the second feature extraction model will be explained first.

[0092] See Figure 2 , Figure 2 This illustration shows a first feature extraction model provided in an embodiment of this application. The first feature extraction model includes a first sub-model 201 and a second sub-model 202. The first sub-model 201 is used to extract static visual features of the video, and the second sub-model 202 is used to extract dynamic visual features of the video. In some embodiments, the first sub-model is referred to as a "Slow" model, and the second sub-model is referred to as a "Fast" model. The first sub-model 201 includes convolutional kernels of multiple sizes, each size corresponding to multiple convolutional kernels. Figure 2 In this context, C represents the number of convolutional kernels in the first sub-model, and T represents the sampling frequency of the first sub-model. The second sub-model 202 includes convolutional kernels of multiple sizes, with each size corresponding to multiple convolutional kernels. Figure 2In this context, βC represents the number of convolutional kernels in the second sub-model, and αT represents the sampling frequency of the second sub-model. The value range of β is (0, 1), and the value range of α is (1, +∞). In some embodiments, α = 4, and β = 1 / 8. That is, in this embodiment, the number of convolutional kernels in the first sub-model is greater than the number of convolutional kernels in the second sub-model. The number of convolutional kernels determines both the depth of feature extraction and the computational speed of the feature extraction process. In some embodiments, the first feature extraction model also integrates two spatiotemporal attention modules 2031 and 2032, which are used to add spatiotemporal attention weights. In some embodiments, see... Figure 3 , Figure 3 The structure of the spatiotemporal attention module 2031 is shown. The spatiotemporal attention module 2031 includes a convolutional layer 301, a channel attention layer 302, a multi-scale spatiotemporal attention layer 303, and an activation layer 304. The convolutional layer 301 performs convolution processing on the visual features input to the spatiotemporal attention module 2031. The channel attention layer 302 performs channel attention processing on visual features from different channels. The multi-scale spatiotemporal attention layer 303 performs spatiotemporal attention processing on visual features. The activation layer 304 normalizes the spatiotemporal attention processed features to obtain the spatiotemporal attention weights of the visual features. In some embodiments, the spatiotemporal attention module 2031 and the spatiotemporal attention module 2032 have the same structure. In some embodiments, if the first feature extraction model integrates a spatiotemporal attention module, then the first feature extraction model can also be called "Slow Fast Net-MCBAM".

[0093] In this embodiment, the second feature extraction model includes multiple convolutional layers, and the first feature extraction model extracts video frame features from video frames through multiple convolutional layers.

[0094] After introducing the structure of the first feature extraction model and the second feature extraction model provided in the embodiments of this application, the training methods of the first feature extraction model and the second feature extraction model will be described below based on their structures. The training method of each model may include a data preparation process and a model training process.

[0095] 1. The training methods for the first feature extraction model include:

[0096] During data preparation, the server acquires multiple sample videos and their corresponding sample video tags. Optionally, the multiple sample videos may be of different types, such as sports videos, animation videos, game videos, or movie videos, etc., which is not limited in this embodiment. In some embodiments, the sample video tags corresponding to the sample videos are manually added by reviewers when reviewing the sample videos.

[0097] During model training, the server acquires multiple sample video frames from multiple sample videos. The server inputs the first sample video frame from these frames into a first model, which extracts the first sample visual features. Based on these first sample visual features, the server predicts the predicted video label for the first sample video. The server adjusts the model parameters of the first model based on the difference between the predicted video label and the corresponding sample video label for the first sample video. If the model parameters of the first model meet a first objective condition, the server designates the first model as the first feature extraction model.

[0098] For example, the server initializes the model parameters of the first model, inputs the first sample image into the first model, and the first model performs convolution processing on the first sample image through a convolutional layer to obtain the first sample visual features corresponding to the first sample image. Based on these first sample visual features, the first model determines the probability that the first sample image corresponds to multiple video tags. For example, the first model can output a vector to represent the probability, [0.2, 0.1, 0.1, 0.7...], where different numbers represent the probability that the first sample image corresponds to different video tags, the number of numbers represents the number of video tags, and the number represents the probability of the corresponding video tag. The server can determine the video tag corresponding to the largest number as the predicted video tag. Based on the difference between the predicted video tag and the corresponding sample video tag, the server adjusts the model parameters of the first model until the loss function value of the first model reaches the target function value or the number of iterations reaches the target number, at which point the training of the first model stops. The server uses the first model at this point as the first feature extraction model. The training process of the first feature extraction model is also the supervised training process. In some embodiments, the loss function of the first model is binary cross-entropy.

[0099] 2. The training methods for the second feature extraction model include:

[0100] During the data preparation process, the server acquires multiple sample videos. Optionally, the multiple sample videos are videos of different types, such as sports videos, animation videos, game videos, or movie videos, etc. This application embodiment does not limit this.

[0101] During model training, the server obtains a first sample video frame from multiple sample video frames. The server performs two data augmentations on the first sample video frame, resulting in a first augmented video frame and a second augmented video frame. The server inputs the first augmented video frame into a second model, which extracts the first sample video frame features. The server inputs the second augmented video frame into a reference model, which extracts the reference video frame features. The reference model and the second model have the same model structure. Based on the similarity between the features of the first sample video frame and the features of the reference video frame, the server adjusts the model parameters of the second model. When the model parameters of the second model meet the second objective condition, the server determines the second model as the second feature extraction model. The training process of the second feature extraction model is also a self-supervised training process.

[0102] When obtaining the first sample video frame from multiple sample video frames, sampling can be performed at preset time intervals, such as collecting one frame every 0.1 seconds, 1 second, 2 seconds, 5 seconds, etc.

[0103] Data augmentation methods include, but are not limited to, rotating video frames, mirroring, random cropping, color dithering, translation transformation, noise perturbation, and contrast transformation.

[0104] For example, see Figure 4The server can train the second feature extraction model using a reference model. The server can obtain a queue of sample video frames from multiple sample video frames. The first sample video frame is obtained from this queue. The server performs two different data augmentation methods on the first sample video frame: rotation, mirror symmetry, random cropping, color dithering, translation transformation, noise perturbation, and contrast transformation, resulting in a first augmented video frame and a second augmented video frame. Since the first and second augmented video frames are obtained by applying different data augmentation methods to the first sample video frame, and data augmentation does not change the content of the first sample video frame, they have the same content and thus constitute a positive sample pair. The difference between the first and second enhanced video frames is related to the data augmentation method. If different rotation angles are used to augment the first sample video frame, for example, the server rotates the first sample video frame by 90° to obtain the first enhanced video frame, and rotates it by 180° to obtain the second enhanced video frame, then the first and second enhanced video frames are two video frames with the same content but different rotation angles. The server inputs the first and second enhanced video frames into the second model and the reference model, respectively, where the second model and the reference model have the same model structure. In some embodiments, before performing the first iteration of training on the second model, the server can uniformly initialize the model parameters of the second model and the reference model, that is, initialize the model parameters of the second model and the reference model to the same model parameters. The server performs convolution processing on the first and second enhanced video frames through multiple convolutional layers of the second model and the reference model, respectively, to obtain the first sample video frame features of the first enhanced video frame and the reference video frame features of the second enhanced video frame. The server obtains the similarity between the features of the first sample video frame and the features of the reference video frame. Based on the similarity between the features of the first sample video frame and the features of the reference video frame, the server adjusts the model parameters of the second model. The goal of the adjustment is to make the similarity between the features of the first sample video frame and the features of the reference video frame as high as possible, that is, to enable the second feature extraction model to focus on the content of the video frame when extracting video frame features.

[0105] In one possible implementation, to further improve the feature extraction capability of the second feature extraction model, during model training, the server can adjust the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the reference video frame, and then obtain second sample video frames from multiple sample videos. The server inputs the second sample video frames into the reference model and extracts the second sample video frame features through the reference model. The server adjusts the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the second sample video frame.

[0106] For example, the server can obtain a second sample video frame from multiple sample video frames. This second sample video frame has different content from the first enhanced video frame and can serve as a negative sample of the first enhanced video frame. The server inputs this second sample video frame as a negative sample into a reference model, performs convolution processing on it through multiple convolutional layers of the reference model, and obtains the second sample video frame features. The server obtains the similarity between the features of the first and second sample video frames. Based on this similarity, the server adjusts the model parameters of the second model, aiming to minimize the similarity between the features of the first and second sample video frames. Combining the previous description, during the server's training of the second model, data augmentation is performed on the first sample video frame, resulting in the first and second enhanced video frames as a positive sample pair. In addition, the server also obtains the second sample video frame, which, together with the first sample video frame, constitutes a negative sample pair. The server can train the second model using both positive and negative sample pairs, thereby improving the feature extraction capability of the second model.

[0107] Furthermore, the server can not only obtain a single second sample video frame from multiple sample video frames to train the second model, but also obtain multiple second sample video frames from multiple sample video frames to train the second model. For example, the server can divide multiple sample video frames into multiple queues. During training, the server can determine a target queue from these queues; the target queue is the queue that does not contain the first sample video frame. The server can then sequentially input multiple second sample video frames from this target queue into the second model, training the second model based on the similarity between the first enhanced video frame and each of the multiple second sample video frames.

[0108] Furthermore, besides acquiring the second sample video frame features of the second sample video frames in real-time through the reference model during model training, the server can also generate a dictionary before model training, storing the second sample video frame features corresponding to multiple second sample images. During model training, the second sample video frame features can be directly retrieved from the dictionary as negative samples for the first enhanced video frame features. In this implementation, the second sample video frame features are acquired through a reference model, resulting in good consistency among multiple second sample video frame features. Training the second model using these second sample video frame features achieves better training results.

[0109] In one possible implementation, the server can also adjust the model parameters of the reference model based on the adjusted parameters of the second model. That is, during the training of the second model, the server can also train the reference model, thereby improving the feature extraction capability of the reference model. Since the training of the second model requires interaction with the reference model, improving the feature extraction capability of the reference model also improves the feature extraction capability of the second model.

[0110] For example, the server can use the momentum update method to adjust the model parameters of the reference model. That is, the server can use the method provided by the following formula (1) to update the model parameters of the reference model.

[0111] θ k ←mθ k +(1-m)θ q (1)

[0112] Where, θ k The model parameters for the reference model are θ. q Here are the model parameters for the second model, and m is the momentum coefficient.

[0113] To provide a clearer explanation of the training method for the second feature extraction model, an example will be used below to illustrate the training method for the second feature extraction model, taking into account the various possible implementation methods described above.

[0114] See Figure 5The server retrieves four sample video frames from each sample video, and these multiple sample video frames constitute an image training set. The server then obtains a queue of sample video frames from the training set, and finally retrieves the first sample video frame from this queue. The server performs two data augmentation techniques on the first sample video frame to obtain a first augmented video frame and a second augmented video frame. In some embodiments, the first augmented video frame is also referred to as the Query image X. q The second enhanced video frame is also known as the Key image X. k The server inputs the Query image into the second model and the Key image into the reference model. The Query image and Key image are then convolved through multiple convolutional layers in both the second and reference models to obtain the first sample video frame features of the Query image and the reference video frame features of the Key image. In some embodiments, the second model can also be called the Query encoder, and the reference model can also be called the Key encoder. Correspondingly, the first sample frame features are called Query Embedding, and the reference sample frame features are called Key Embedding. The server pre-generates a dictionary using the Key encoder, which includes multiple second sample video frame features. The server divides the multiple second sample video frame features in the dictionary into multiple batches, each batch containing multiple second sample video frame features. The server constructs a contrastive loss function (InfoNCE) to train the second model. The form of the contrastive loss function is shown in formula (2). The server updates the model parameters of the second model using the contrastive loss function, and updates the model parameters of the reference model using the model parameters of the second model.

[0115] (2)

[0116] Where L() is the contrastive loss function, W is the feature of the first sample video frame, X1 is the feature of the reference video frame, X2 is the feature of the second sample video frame, Y is the label used to indicate whether it is a positive or negative sample. In some embodiments, Y=1 represents a positive sample and Y=0 represents a negative sample. D w The cosine similarity is the feature between video frames, where N is the number of the first sample video frames and m is the threshold, which is set by technicians according to the actual situation.

[0117] It should be noted that the above descriptions all use the example of using a server to train the model. In other possible implementations, the two models can also be trained by the terminal, or the model can be trained through interaction between the terminal and the server. For example, the terminal can collect the training images of the two models and send the training images of the two models to the server, and the server can train the two models. This application embodiment does not limit this.

[0118] In addition, before training the two models mentioned above, the server can use open-source models with similar functions on the Internet as initial models and train on the basis of the initial models, which can improve the convergence speed of the models.

[0119] In the embodiments of this application, the technical solutions provided in the embodiments of this application can be implemented by a server or a terminal as the execution subject, or the technical methods provided in this application can be implemented through the interaction between the terminal and the server. The embodiments of this application do not limit this. The following description will take the server as the execution subject as an example.

[0120] Figure 6 This is a flowchart of a video feature extraction method provided in an embodiment of this application. See also... Figure 6 The methods include:

[0121] 601. The server retrieves multiple video frames from the target video.

[0122] The target video can be any type of video, such as TV series videos, game videos, instructional videos, and movie videos, etc. This application embodiment does not limit this.

[0123] 602. The server inputs a first number of video frames into the first feature extraction model. The first feature extraction model extracts the first visual features of the target video. The first feature extraction model is a visual feature extraction model trained with multiple sample videos as training samples and multiple sample video labels corresponding to the multiple sample videos as supervised training.

[0124] Since the first feature extraction model is trained under supervision using multiple sample video labels corresponding to multiple sample videos, the first visual feature extracted by the first feature extraction model is also the visual feature suitable for adding video labels. The training method of the first feature extraction model is described in the previous description and will not be repeated here.

[0125] 603. The server inputs a second number of video frames into the second feature extraction model. The second feature extraction model extracts the features of the second number of video frames. The second feature extraction model is a visual feature extraction model obtained by self-supervised training with multiple sample video frames as training samples.

[0126] In this model, the second feature extraction model uses multiple sample video frames as training samples and is trained using a self-supervised method. Therefore, the video frame features extracted by the second feature extraction model can contain more information related to the video frame content. Compared to the first feature extraction model, the video frame features extracted by the second feature extraction model carry richer information and have better scalability.

[0127] 604. The server fuses the first visual feature and the second number of video frame features of the target video to obtain the target visual feature of the target video.

[0128] Among them, since the target visual features are obtained by fusing the first visual features and the video frame features, they contain more content-related information than the first visual features. Therefore, the target video features have better scalability when used to add new video tags.

[0129] See Figure 7 The flowchart of steps 601-604 above is shown.

[0130] In this embodiment, during the extraction of video features, a first feature extraction model based on supervised training and a second feature extraction model based on self-supervised training are introduced. Since the first feature extraction model is trained based on sample videos and their corresponding labels, the output first visual features are highly targeted, making it suitable for adding video tags to target videos. The second feature extraction model is based on self-supervised training. Since the self-supervised model's feature prediction is less targeted than the supervised model, the second visual features extracted by the second feature extraction model contain richer information. The first and second visual features complement each other, resulting in target visual features that more completely reflect the characteristics of the target video.

[0131] In the embodiments of this application, the technical solutions provided in the embodiments of this application can be implemented by a server or a terminal as the execution subject, or the technical methods provided in this application can be implemented through the interaction between the terminal and the server. The embodiments of this application do not limit this. The following description will take the server as the execution subject as an example.

[0132] Figure 8 This is a flowchart of a video feature extraction method provided in an embodiment of this application. See also... Figure 8 The methods include:

[0133] 801. The server retrieves multiple video frames from the target video.

[0134] In one possible implementation, the server performs uniform sampling on the target video, extracting multiple video frames from it. This approach reduces the number of video frames the server needs to process while preserving the target video content, thereby improving server processing efficiency.

[0135] For example, the server divides the target video into multiple parts, each with the same number of video frames. One video frame is extracted from each part, and the video frames extracted from multiple parts constitute a set of video frames.

[0136] In one possible implementation, the server acquires video frames from the target video at fixed intervals, resulting in multiple video frames. This implementation maximizes the preservation of the target video's content while reducing the number of video frames, and improves the accuracy of subsequent feature extraction while reducing server computational load.

[0137] In one possible implementation, the server randomly selects multiple video frames from the target video. This implementation allows the server to select video frames from the target video in a more flexible manner.

[0138] In one possible implementation, the server acquires all video frames from the target video. In this implementation, multiple video frames constitute the complete content of the target video, improving the accuracy of subsequent video feature extraction.

[0139] It should be noted that the server can obtain video frames from the target video through any of the above methods, and this application embodiment does not limit this.

[0140] Optionally, after the server obtains multiple video frames from the target video, the server can resize the video frames to convert video frames of different sizes into video frames of the same size, wherein the same size is the size of the sample video frames used by the server when training the first feature extraction model and the second feature extraction model.

[0141] In this implementation, the server can preprocess multiple video frames in the target video, adjusting the size of the multiple video frames to be the same as the sample video frames used when training the feature extraction model, thereby improving the accuracy of extracting visual features of the target video using the feature extraction model in the subsequent process.

[0142] For example, different target videos may have different sizes. For instance, video A has a size of 800×600 and video B has a size of 1024×768. The server can resize multiple video frames obtained from video A and multiple video frames obtained from video B to obtain multiple video frames of the same size.

[0143] 802. The server inputs a first number of video frames into the first feature extraction model. The first feature extraction model extracts the first visual features of the target video. The first feature extraction model is a visual feature extraction model trained with multiple sample videos as training samples and multiple sample video labels corresponding to the multiple sample videos as supervised training.

[0144] In one possible implementation, the first feature extraction model includes a first sub-model and a second sub-model. The first sub-model is used to extract static visual features of the target video, and the second sub-model is used to extract dynamic visual features of the target video. The server uniformly samples a first number of video frames to obtain a third number of video frames, where the third number is less than the first number. The server inputs the third number of video frames into the first sub-model. The server inputs the first number of video frames into the second sub-model. The server performs convolution processing on the third number of video frames using multiple convolution kernels of the first sub-model to obtain the static visual features of the target video. The server performs convolution processing on the first number of video frames using multiple convolution kernels of the second sub-model to obtain the dynamic visual features of the target video. Based on the static and dynamic visual features, the server obtains the first visual feature of the target video. The number of convolution kernels in the first sub-model is greater than the number of convolution kernels in the second sub-model.

[0145] For example, see Figure 2 The first feature extraction model includes a first sub-model 201 and a second sub-model 202. The server obtains a first number of video frames from multiple video frames and inputs these first number of video frames into the second sub-model 202. In some embodiments, the first number is 24. The server obtains 24 video frames from multiple video frames through uniform sampling and inputs these 24 video frames into the second sub-model 202. It should be noted that when the server inputs the first number of video frames into the second sub-model 202, it does so simultaneously. That is, the two-dimensional pixel matrices of the first number of video frames are concatenated in chronological order to obtain a three-dimensional pixel matrix. Taking three video frames as an example, if there are video frames A, B, and C, for ease of understanding, the following description will use grayscale images (single-channel images) as examples. The two-dimensional pixel value matrix corresponding to video frame A is... The two-dimensional pixel value matrix corresponding to video frame B is: The two-dimensional pixel value matrix corresponding to video frame C is: The server can process the three two-dimensional pixel value matrices according to the order in which video frames A, B, and C appear in the target video. , as well as By superimposing the values, a three-dimensional pixel value matrix is ​​obtained. The server will store this three-dimensional pixel value matrix. Input the second sub-model 202. The server processes the 3D pixel value matrix through multiple 3D convolution kernels of the second sub-model 202. Convolution processing is performed to obtain the dynamic visual features of the target video. The server obtains a third number of video frames from a first number of video frames and inputs these third number of video frames into the first sub-model 201. In some embodiments, if the first number is 24, then the second number can be 6. The server can obtain 6 video frames from 24 video frames through uniform sampling and input these 6 video frames into the first sub-model 201. Similarly, as described previously, the server also assembles the two-dimensional pixel value matrices of the 6 video frames into a three-dimensional pixel value matrix and inputs this three-dimensional pixel value matrix into the first sub-model 201. The server performs convolution processing on the three-dimensional pixel value matrix using multiple three-dimensional convolution kernels of the first sub-model 201 to obtain the static visual features of the target video. The process of the first sub-model 201 performing convolution processing on 6 video frames and the process of the second sub-model 202 performing convolution processing on 24 video frames belong to the same inventive concept and will not be described further here. As described earlier when introducing the first feature extraction model, the number of convolutional kernels in the first sub-model 201 is greater than the number of convolutional kernels in the second sub-model 202. In some embodiments, the first sub-model 201 is a 3D ResNet-50 residual network, where 50 represents the number of model layers in the residual network. The second sub-model 202 is a lightweight 3D convolutional network, such as a 3D ResNet-18 residual network. The server obtains the first visual features of the target video based on static and dynamic visual features.

[0146] The following describes the method by which the server obtains the first visual feature of the target video based on static and dynamic visual features.

[0147] In one possible implementation, the server performs spatiotemporal attention processing on the static visual features to obtain a first static feature of the target video. The server then performs spatiotemporal attention processing on the dynamic visual features to obtain a first dynamic feature of the target video. The server fuses the first static feature and the first dynamic feature to obtain a fused feature. The server then performs global pooling processing on the fused feature and the first dynamic feature respectively to obtain a second static feature and a second dynamic feature of the target video. Finally, the server fuses the second static feature and the second dynamic feature to obtain a first visual feature of the target video.

[0148] In this implementation, the server can perform spatiotemporal attention processing on the static and dynamic visual features of the target video respectively. This spatiotemporal attention processing enables the server to focus on more discernible spatiotemporal positions when fusing the static and dynamic visual features of the target video. By comprehensively utilizing different video frames, the first visual feature obtained can more accurately reflect the features of the target video.

[0149] Because this implementation involves many processing details, the processing details will be introduced in several parts below for a clearer explanation.

[0150] The first part describes the method of using the server to perform spatiotemporal attention processing on dynamic visual features to obtain the first dynamic feature of the target video.

[0151] In one possible implementation, the server performs channel attention processing on the dynamic visual features to obtain dynamic channel attention weights corresponding to the dynamic visual features. The server multiplies the dynamic channel attention weights with the dynamic visual features to obtain a third dynamic feature. The server performs max pooling, mean pooling, and dilated convolution processing on the third dynamic feature to obtain a second spatiotemporal attention weight corresponding to the third dynamic feature. The server multiplies the third dynamic feature with the second spatiotemporal attention weight and then adds it to the dynamic visual features to obtain a first dynamic feature.

[0152] For example, see Figure 2The server inputs the dynamic visual features of the target video into the spatiotemporal attention module 2031 connected to the first sub-model 201. The server performs convolution processing on the dynamic visual features of the target video through the convolutional layer 301 of the spatiotemporal attention module 2031, and inputs the convolutionally processed dynamic visual features into the channel attention layer 302. The server performs one max pooling and one mean pooling on the convolutionally processed dynamic visual features through the channel attention layer 302 to obtain the dynamic channel pooling features corresponding to the dynamic visual features. The server performs multi-head channel attention processing on the dynamic channel pooling features through the channel attention layer 302, and normalizes the dynamic channel pooling features after multi-head channel attention processing to obtain the dynamic channel attention weights corresponding to the dynamic visual features. The server multiplies the dynamic channel attention weights with the dynamic visual features to obtain the third dynamic feature. The server performs max pooling and mean pooling on the third dynamic feature to obtain the fourth dynamic feature. The server uses multiple convolutional kernels with different dilation rates to perform convolution processing on the fourth dynamic feature to obtain multiple fifth dynamic features. The server sums multiple fifth dynamic features and then normalizes them to obtain the second spatiotemporal attention weight corresponding to the third dynamic feature. The server multiplies the third dynamic feature by the second spatiotemporal attention weight and then adds it to the dynamic visual feature to obtain the first dynamic feature.

[0153] The above implementation method will be illustrated below with a specific example.

[0154] If the dynamic visual features of the target video are The server will dynamically display visual features. The input spatiotemporal attention module 2031 processes dynamic visual features through its convolutional layer 301. Perform convolution processing to obtain dynamic visual features after convolution. The server uses channel attention layer 302 to process the dynamic visual features after convolution. Perform one max pooling and one mean pooling operation, then concatenate the results of the max pooling and mean pooling operations to obtain the dynamic channel pooling feature. The first row of numbers represents the result of max pooling, and the second row represents the result of mean pooling. The server uses a channel attention layer 302 to process the dynamic channel pooling features. Multi-channel attention processing is performed to obtain the vector (120, 190, 110). The server normalizes the vector (120, 190, 110) to obtain the dynamic channel attention weights (0.29, 0.45, 0.26) corresponding to the dynamic visual features. The server then convolves the dynamic channel attention weights (0.29, 0.45, 0.26) with the convolved dynamic visual features. Multiplying them together yields the third dynamic feature. The server's response to the third dynamic feature The fourth dynamic feature is obtained by performing max pooling and mean pooling respectively. The server uses three dilated convolution kernels with dilation rates of 1, 2, and 3, respectively. T , T as well as T Regarding the fourth dynamic feature Convolution processing is performed to obtain three fifth dynamic features. , as well as It should be noted that, for ease of understanding, this explanation uses a video frame as a grayscale image (single-channel image). If the video frame were a color image, such as an RGB three-channel image, then the corresponding dilated convolution kernel would be a three-dimensional dilated convolution kernel. The server will use three fifth dynamic features. , as well as Add them together to get a matrix. The server supports the matrix. After normalization, the second spatiotemporal attention weights corresponding to the third dynamic feature are obtained. The server will use the third dynamic feature. With the second spatiotemporal attention weight Multiply to obtain a matrix The server will store the matrix. With dynamic visual features Adding them together yields the first dynamic feature. The server will hold the matrix With dynamic visual features The process of addition can also be called the residual processing process.

[0155] The second part introduces the method of using the server to perform spatiotemporal attention processing on static visual features to obtain the first static features of the target video.

[0156] In one possible implementation, the server performs channel attention processing on the static visual features to obtain static channel attention weights corresponding to the static visual features. The server multiplies the static channel attention weights with the static visual features to obtain a third static feature. The server performs max pooling, mean pooling, and dilated convolution processing on the third static feature to obtain a first spatiotemporal attention weight corresponding to the third static feature. The server multiplies the third static feature with the first spatiotemporal attention weight and then adds it to the static visual features to obtain the first static feature.

[0157] For example, see Figure 2 The server inputs the static visual features of the target video into the spatiotemporal attention module 2032 connected to the second sub-model 202. The server performs convolution processing on the static visual features of the target video through the convolutional layer 301 of the spatiotemporal attention module 2032, and inputs the convolutionally processed static visual features into the channel attention layer 302. The server performs one max pooling and one mean pooling on the convolutionally processed static visual features through the channel attention layer 302 to obtain the static channel pooled features corresponding to the static visual features. The server performs multi-head channel attention processing on the static channel pooled features through the channel attention layer 302, and normalizes the static channel pooled features after multi-head channel attention processing to obtain the static channel attention weights corresponding to the static visual features. The server multiplies the static channel attention weights with the static visual features to obtain the third static feature. The server performs max pooling and mean pooling on the third static feature to obtain the fourth static feature. The server uses multiple convolutional kernels with different dilation rates to perform convolution processing on the fourth static feature to obtain multiple fifth static features. The server sums multiple fifth static features and then normalizes them to obtain the second spatiotemporal attention weight corresponding to the third static feature. The server multiplies the third static feature by the first spatiotemporal attention weight and then adds it to the static visual feature to obtain the first static feature.

[0158] The above implementation method will be illustrated below with a specific example.

[0159] If the static visual features of the target video are The server will display static visual features. Input to the spatiotemporal attention module 2032, and process static visual features through the convolutional layer 301 of the spatiotemporal attention module 2032. Perform convolution processing to obtain the static visual features after convolution. The server uses channel attention layer 302 to process the static visual features after convolution. Perform one max pooling and one mean pooling operation, then concatenate the results of the max pooling and mean pooling operations to obtain the static channel pooling feature. The first row of numbers represents the results of max pooling, and the second row represents the results of mean pooling. The server processes the static channel pooling features through channel attention layer 302. Multi-channel attention processing is performed to obtain the vector (10, 15, 25). The server normalizes the vector (10, 15, 25) to obtain the static channel attention weights (0.2, 0.3, 0.5) corresponding to the static visual features. The server then convolves the static channel attention weights (0.2, 0.3, 0.5) with the convolved static visual features. Multiplying them together yields the third static feature. The server's third static feature The fourth static feature is obtained by performing max pooling and mean pooling respectively. The server uses three dilated convolution kernels with dilation rates of 1, 2, and 3, respectively. T , T as well as T For the fourth static feature Convolution processing is performed to obtain three fifth static features. , as well as It should be noted that, for ease of understanding, this explanation uses a video frame as a grayscale image (single-channel image). If the video frame were a color image, such as an RGB three-channel image, then the corresponding dilated convolution kernel would be a three-dimensional dilated convolution kernel. The server uses three fifth static features... , as well as Add them together to get a matrix. The server supports the matrix. After normalization, the first spatiotemporal attention weights corresponding to the third static feature are obtained. The server will use the third static feature. With first-space attention weight Multiply to obtain a matrix The server will store the matrix. Compared with static visual features Adding them together yields the first static feature. The server will hold the matrix Compared with static visual features The process of addition can also be called the residual processing process.

[0160] The third part describes the method by which the server fuses the first static feature and the first dynamic feature to obtain the fused feature.

[0161] In one possible implementation, the server can add the first static feature to the first dynamic feature to obtain a fused feature.

[0162] With the first static feature First dynamic feature Taking the first static feature as an example, the server will... With the first dynamic feature Adding them together yields the fusion characteristics. In some embodiments, the method by which the server obtains the fusion feature may also be referred to as a bypass connection, see [link to documentation]. Figure 2The bypass connection is essentially an arrow pointing from the second sub-model 202 "Fast" to the first sub-model 201 "Slow". The number of bypass connections can be one or more, and this embodiment does not limit this. By performing bypass connections, the processing results of the second sub-model and the first sub-model can be combined during data processing, improving the accuracy of subsequent extraction of target visual features from the target video.

[0163] Part Four describes the method by which the server performs global pooling processing on the fused features and the first dynamic features to obtain the second static features and the second dynamic features of the target video.

[0164] In one possible implementation, the server can perform global mean pooling on the fused features and the first dynamic features respectively to obtain the second static features and the second dynamic features of the target video.

[0165] With fusion features First dynamic feature Taking this as an example, the server uses fusion features. Global mean pooling is performed to obtain the second static feature (232.24, 280.87, 298.44) of the target video. The server then processes the first dynamic feature. Global mean pooling is performed to obtain the second dynamic feature of the target video (142.75, 168.7, 162.45).

[0166] Part Five describes the method by which the server fuses the second static feature and the second dynamic feature to obtain the first visual feature of the target video.

[0167] In one possible implementation, the server performs mean pooling on the second static feature and the second dynamic feature respectively, and then concatenates the mean pooled second static feature and the mean pooled second dynamic feature to obtain the first visual feature of the target video.

[0168] Taking the second static feature (232.24, 280.87, 298.44) and the second dynamic feature (142.75, 168.7, 162.45) as examples, the server performs mean pooling on the second static feature (232.24, 280.87, 298.44) to obtain the mean pooled second static feature (270.5). The server also performs mean pooling on the second dynamic feature (142.75, 168.7, 162.45) to obtain the mean pooled second dynamic feature (157.9). The server then concatenates the mean pooled second static feature (270.5) and the mean pooled second dynamic feature (157.9) to obtain the first visual feature of the target video (270.5, 157.9).

[0169] It should be noted that the descriptions in Parts 1 to 5 above are based on the example of a video frame being a grayscale image (single-channel image). In other possible real-time methods, the video frame can also be a color image (multi-channel image). If the video frame is a color image, the dimension of the features will increase accordingly with the number of channels, but the processing of each channel belongs to the same inventive concept as that of a grayscale image, and will not be described in detail here.

[0170] 803. The server inputs a second number of video frames into the second feature extraction model. The second feature extraction model extracts the features of the second number of video frames. The second feature extraction model is a visual feature extraction model obtained by self-supervised training with multiple sample video frames as training samples.

[0171] The training method for the second feature extraction model is described in the previous model training section and will not be repeated here. The second quantity can be set by technicians according to the actual situation, or determined by the duration of the target video. In some embodiments, the second quantity is the same as the number of multiple video frames obtained from the target video in step 801. If the multiple video frames are obtained by the server from the target video every second, then the second quantity is the same as the duration (in seconds) of the target video.

[0172] In one possible implementation, the server uses a second feature extraction model to perform convolution processing on a second number of video frames to obtain the second number of video frame features.

[0173] For example, if the second number is 3, the server can input video frame A, video frame B and video frame C into the second feature extraction model respectively. The second feature extraction model performs convolution processing on video frame A, video frame B and video frame C respectively to obtain the video frame feature (1,1,1) corresponding to video frame A, the video frame feature (1,0,1) corresponding to video frame B and the video frame feature (1,0,0) corresponding to video frame C.

[0174] 804. The server performs mean pooling on the features of the second number of video frames to obtain the second visual features of the target video.

[0175] In one possible implementation, the server performs global mean pooling on the features of a second number of video frames, and then concatenates the mean pooled features of the second number of video frames to obtain the second visual features of the target video.

[0176] Taking the second quantity as 3, and the video frame features corresponding to video frame A (1, 1, 1), video frame B (1, 0, 1), and video frame C (1, 0, 0) as examples, the server performs global mean pooling on the video frame feature (1, 1, 1) corresponding to video frame A, resulting in mean pooled video frame feature (1). The server performs global mean pooling on the video frame feature (1, 0, 1) corresponding to video frame B, resulting in mean pooled video frame feature (0.66). The server performs global mean pooling on the video frame feature (1, 0, 0) corresponding to video frame C, resulting in mean pooled video frame feature (0.33). The server concatenates the three mean pooled video frame features to obtain the second visual feature of the target video (1, 0.66, 0.33).

[0177] 805. The server splices the first visual feature and the second visual feature to obtain the target visual feature of the target video.

[0178] For example, if the first visual feature of the target video is (270.5, 157.9) and the second visual feature is (1, 0.66, 0.33), the server will concatenate the first visual feature (270.5, 157.9) and the second visual feature (1, 0.66, 0.33) to obtain the target visual feature (270.5, 157.9, 1, 0.66, 0.33). The server can then use this target visual feature (270.5, 157.9, 1, 0.66, 0.33) to recommend videos or add video tags to the target video. For an example of adding video tags to a target video, see [link to example]. Figure 9901 represents the type of video tag, 902 represents the video tag, and 903 represents the probability corresponding to the video tag.

[0179] It should be noted that after step 803, in addition to executing steps 804 and 805 to obtain the target visual features of the target video, the server can also obtain the multimodal features of the target video by executing the following steps. The multimodal features include the visual features, audio features, and text features of the target video. The text features are the features corresponding to the subtitles of the target video.

[0180] In one possible implementation, the server can obtain audio and subtitle information from the target video, input the audio information into an audio feature extraction model to obtain the audio features of the target video, and input the subtitle information into a text feature extraction model to obtain the text features of the target video. In some embodiments, both the audio feature extraction model and the text feature extraction model are models obtained through supervised training, and have the ability to extract audio features and text features from the video, respectively. The server inputs a second number of video frame features, the audio features of the target video, and the text features of the target video into a multimodal model, and fuses the second number of video frame features, the audio features of the target video, and the text features of the target video through an aggregation network of the multimodal model to obtain the multimodal fused features of the target video. In some embodiments, the aggregation network is NeXtVLAD (Multimodal Video Classification) or TRN (Topology Representing Network), and this application embodiment does not limit this. The server fuses the multimodal fused features of the target video with the first visual features of the target video to obtain the multimodal features of the target video. Since the multimodal features of the target video integrate the visual, audio, and textual features of the target video, these multimodal features can more completely represent the target video. Subsequent video tagging and video recommendation using these multimodal features can achieve better results.

[0181] During the experiment, the video feature extraction method provided in this application embodiment was integrated into the video tag adding model, and the results are as follows:

[0182] Currently, the baseline metrics for common labeling models are accuracy = 79.3%, recall = 57.3%, and F1 = 66.5%, where F1 is the F1 measure.

[0183] 1) By simply adding the visual features output by the first and second sub-models of the first feature extraction model, the accuracy improved to 80.0%, the recall improved to 59.9%, and the F1 score improved to 68.5%. The first and second sub-models can be collectively referred to as "Slow Fast".

[0184] 2) After adding a spatiotemporal attention module to the first feature extraction model, the accuracy improved to 80.3%, the recall improved to 60.9%, and the F1 score improved to 69.3%. The first feature extraction model plus the spatiotemporal attention module can be called "Slow Fast Net-MCBAM".

[0185] 3) After adding a second feature extraction model on this basis, the accuracy improved to 80.9%, the recall improved to 62.0%, and the F1 score improved to 70.2%. The first feature extraction model plus the spatiotemporal attention module and the second feature extraction model can be collectively referred to as "Slow Fast Net-MCBAM + MoCo". MoCo is the second feature extraction model used in the experiment.

[0186] Specific data are shown in Table 1. It is evident that after adopting the technical solution provided in this application, the accuracy and recall of the video tagging model have been significantly improved, demonstrating the effectiveness of this application's solution.

[0187] Table 1

[0188]

[0189] In this embodiment, during the extraction of video features, a first feature extraction model based on supervised training and a second feature extraction model based on self-supervised training are introduced. Since the first feature extraction model is trained based on sample videos and their corresponding labels, the output first visual features are highly targeted, making it suitable for adding video tags to target videos. The second feature extraction model is based on self-supervised training. Since the self-supervised model's feature prediction is less targeted than the supervised model, the second visual features extracted by the second feature extraction model contain richer information. The first and second visual features complement each other, resulting in target visual features that more completely reflect the characteristics of the target video.

[0190] Figure 10 This is a schematic diagram of a video feature extraction device provided in an embodiment of this application. See also... Figure 10 The device includes: a video frame acquisition module 1001, a first input module 1002, a second input module 1003, and a feature fusion module 1004.

[0191] The video frame acquisition module 1001 is used to acquire multiple video frames from the target video.

[0192] The first input module 1002 is used to input a first number of video frames into the first feature extraction model, and extract the first visual features of the target video through the first feature extraction model. The first feature extraction model is a visual feature extraction model trained with multiple sample videos as training samples and multiple sample video labels corresponding to the multiple sample videos as supervised training.

[0193] The second input module 1003 is used to input a second number of video frames into the second feature extraction model, and to extract the features of the second number of video frames through the second feature extraction model. The second feature extraction model is a visual feature extraction model obtained by self-supervised training with multiple sample video frames as training samples.

[0194] The feature fusion module 1004 is used to fuse the first visual features and the features of a second number of video frames of the target video to obtain the target visual features of the target video.

[0195] In one possible implementation, the first feature extraction model includes a first sub-model and a second sub-model. The first sub-model is used to extract static visual features of the target video, and the second sub-model is used to extract dynamic visual features of the target video. A first input module is used to uniformly sample a first number of video frames to obtain a third number of video frames, where the third number is less than the first number. The third number of video frames are then input into the first sub-model. The first number of video frames are then input into the second sub-model.

[0196] In one possible implementation, the first input module is used to perform convolution processing on a third number of video frames using multiple convolution kernels of the first sub-model to obtain the static visual features of the target video. Then, it performs convolution processing on a first number of video frames using multiple convolution kernels of the second sub-model to obtain the dynamic visual features of the target video. Based on the static and dynamic visual features, the first visual feature of the target video is obtained. The number of convolution kernels in the first sub-model is greater than the number of convolution kernels in the second sub-model.

[0197] In one possible implementation, the first input module performs spatiotemporal attention processing on the static visual features to obtain a first static feature of the target video. It then performs spatiotemporal attention processing on the dynamic visual features to obtain a first dynamic feature of the target video. The first static feature and the first dynamic feature are fused to obtain a fused feature. Global pooling is then performed on the fused feature and the first dynamic feature respectively to obtain a second static feature and a second dynamic feature of the target video. Finally, the second static feature and the second dynamic feature are fused to obtain a first visual feature of the target video.

[0198] In one possible implementation, the first input module performs channel attention processing on the static visual features to obtain static channel attention weights corresponding to the static visual features. The static channel attention weights are multiplied by the static visual features to obtain a third static feature. The third static feature is then subjected to max pooling, mean pooling, and dilated convolution processing to obtain a first spatiotemporal attention weight corresponding to the third static feature. The third static feature is multiplied by the first spatiotemporal attention weight and then added to the static visual features to obtain the first static feature.

[0199] In one possible implementation, the first input module performs max pooling and mean pooling on the third static feature to obtain the fourth static feature. The fourth static feature is then convolved using multiple convolutional kernels with different dilation rates to obtain multiple fifth static features. These multiple fifth static features are then summed and normalized to obtain the first spatiotemporal attention weight corresponding to the third static feature.

[0200] In one possible implementation, the first input module performs channel attention processing on the dynamic visual features to obtain dynamic channel attention weights corresponding to the dynamic visual features. The dynamic channel attention weights are multiplied by the dynamic visual features to obtain a third dynamic feature. The third dynamic feature is then subjected to max pooling, mean pooling, and dilated convolution processing to obtain a second spatiotemporal attention weight corresponding to the third dynamic feature. The third dynamic feature is multiplied by the second spatiotemporal attention weight and then added to the dynamic visual features to obtain a first dynamic feature.

[0201] In one possible implementation, the first input module performs max pooling and mean pooling on the third dynamic feature to obtain the fourth dynamic feature. The fourth dynamic feature is then convolved using multiple convolutional kernels with different dilation rates to obtain multiple fifth dynamic features. These multiple fifth dynamic features are then summed and normalized to obtain the second spatiotemporal attention weight corresponding to the third dynamic feature.

[0202] In one possible implementation, the first input module is used to perform mean pooling on the second static feature and the second dynamic feature respectively, and then concatenate the mean pooled second static feature and the mean pooled second dynamic feature to obtain the first visual feature of the target video.

[0203] In one possible implementation, the second input module is used to perform convolution processing on a second number of video frames respectively through a second feature extraction model to obtain the second number of video frame features of the second number of video frames.

[0204] In one possible implementation, the feature fusion module performs mean pooling on the features of a second number of video frames to obtain the second visual features of the target video. The first visual features and the second visual features are then concatenated to obtain the target visual features of the target video.

[0205] In one possible implementation, the training apparatus for the first feature extraction model includes:

[0206] The first sample video frame acquisition module is used to acquire multiple sample video frames from multiple sample videos respectively.

[0207] The first sample input module is used to input sample video frames into the first model and extract the first sample visual features of the corresponding first sample video through the first model.

[0208] The label prediction module is used to predict the video label of the first sample video based on the visual features of the first sample using the first model.

[0209] The first parameter adjustment module is used to adjust the model parameters of the first model based on the difference information between the predicted video label and the sample video label corresponding to the first sample video.

[0210] The first model acquisition module is used to determine the first model as the first feature extraction model in response to the first target condition being met by the model parameters of the first model.

[0211] In one possible implementation, the training device for the second feature extraction model includes:

[0212] The second sample video frame acquisition module is used to acquire the first sample video frame from multiple sample video frames.

[0213] The data augmentation module is used to perform data augmentation on the first sample video frame in two ways to obtain the first augmented video frame and the second augmented video frame.

[0214] The second sample input module is used to input the first enhanced video frame into the second model and extract the first sample video frame features of the first enhanced video frame through the second model.

[0215] The third sample input module is used to input the second enhanced video frame into the reference model, and extract the reference video frame features of the second enhanced video frame through the reference model. The reference model and the second model have the same model structure.

[0216] The second model parameter adjustment module is used to adjust the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the reference video frame.

[0217] The second model acquisition module is used to determine the second model as the second feature extraction model in response to the fact that the model parameters of the second model meet the second target condition.

[0218] In one possible implementation, the second sample video frame acquisition module is used to acquire a sample video frame queue from a plurality of sample video frames. The first sample video frame is then acquired from the sample video frame queue.

[0219] In one possible implementation, the second sample video frame acquisition module is further configured to acquire a second sample video frame from multiple sample videos.

[0220] The third sample input module is also used to input the second sample video frame into the reference model and extract the second sample video frame features through the reference model.

[0221] The second model parameter adjustment module is also used to adjust the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the second sample video frame.

[0222] In one possible implementation, the second model parameter adjustment module is further configured to adjust the model parameters of the reference model based on the model parameters of the second model.

[0223] It should be noted that the video feature extraction device provided in the above embodiments is only illustrated by the division of the above functional modules when extracting video features. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video feature extraction device and the video feature extraction method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0224] In this embodiment, during the extraction of video features, a first feature extraction model based on supervised training and a second feature extraction model based on self-supervised training are introduced. Since the first feature extraction model is trained based on sample videos and their corresponding labels, the output first visual features are highly targeted, making it suitable for adding video tags to target videos. The second feature extraction model is based on self-supervised training. Since the self-supervised model's feature prediction is less targeted than the supervised model, the second visual features extracted by the second feature extraction model contain richer information. The first and second visual features complement each other, resulting in target visual features that more completely reflect the characteristics of the target video.

[0225] This application provides a computer device for performing the above-described method. This computer device can be implemented as a terminal or a server. The structure of the terminal will be described below:

[0226] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal 1100 can be a smartphone, tablet computer, laptop computer, or desktop computer. The terminal 1100 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0227] Typically, terminal 1100 includes one or more processors 1101 and one or more memories 1102.

[0228] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0229] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one computer program, which is executed by the processor 1101 to implement the video feature extraction method provided in the method embodiments of this application.

[0230] In some embodiments, the terminal 1100 may also optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, a positioning assembly 1108, and a power supply 1109.

[0231] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0232] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc.

[0233] Display screen 1105 is used to display a user interface (UI). This UI may include graphics, text, icons, video, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1101 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard.

[0234] The camera assembly 1106 is used to capture images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal.

[0235] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1101 for processing, or input to the radio frequency circuit 1104 to realize voice communication.

[0236] The positioning component 1108 is used to locate the current geographical location of the terminal 1100 in order to enable navigation or LBS (Location Based Service).

[0237] The power supply 1109 is used to supply power to the various components in the terminal 1100. The power supply 1109 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0238] In some embodiments, the terminal 1100 further includes one or more sensors 1110. The one or more sensors 1110 include, but are not limited to: an accelerometer 1111, a gyroscope 1112, a pressure sensor 1113, a fingerprint sensor 1114, an optical sensor 1115, and a proximity sensor 1116.

[0239] Accelerometer 1111 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established with terminal 1100.

[0240] The gyroscope sensor 1112 can detect the orientation and rotation angle of the terminal 1100. The gyroscope sensor 1112 can work in conjunction with the accelerometer sensor 1111 to collect the user's 3D movements on the terminal 1100.

[0241] The pressure sensor 1113 can be installed on the side bezel of the terminal 1100 and / or on the lower layer of the display screen 1105. When the pressure sensor 1113 is installed on the side bezel of the terminal 1100, it can detect the user's grip signal on the terminal 1100, and the processor 1101 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is installed on the lower layer of the display screen 1105, the processor 1101 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1105.

[0242] The fingerprint sensor 1114 is used to collect the user's fingerprint. The processor 1101 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1114, or the fingerprint sensor 1114 identifies the user's identity based on the collected fingerprint.

[0243] An optical sensor 1115 is used to collect ambient light intensity. In one embodiment, a processor 1101 can control the display brightness of a display screen 1105 based on the ambient light intensity collected by the optical sensor 1115.

[0244] The proximity sensor 1116 is used to detect the distance between the user and the front of the terminal 1100.

[0245] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0246] The aforementioned computer equipment can also be implemented as a server. The structure of a server is described below:

[0247] Figure 12 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1200 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1201 and one or more memories 1202. The one or more memories 1202 store at least one computer program, which is loaded and executed by the one or more processors 1201 to implement the methods provided in the various method embodiments described above. Of course, the server 1200 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1200 may also include other components for implementing device functions, which will not be elaborated upon here.

[0248] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the video feature extraction method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0249] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the video feature extraction method described above.

[0250] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0251] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A video feature extraction method, characterized in that, The method includes: Obtain multiple video frames from the target video; A first number of video frames are uniformly sampled to obtain a third number of video frames, wherein the third number is less than the first number. The third number of video frames are input into the first sub-model of the first feature extraction model, and the first number of video frames are input into the second sub-model of the first feature extraction model. The first visual feature of the target video is extracted through the first feature extraction model. The first feature extraction model is a visual feature extraction model trained with multiple sample videos as training samples and multiple sample video labels corresponding to the multiple sample videos as supervised training. The first sub-model is used to extract the static visual feature of the target video, and the second sub-model is used to extract the dynamic visual feature of the target video. The second number of video frames are input into the second feature extraction model. The second feature extraction model extracts the second number of video frame features of the second number of video frames. The second feature extraction model is a visual feature extraction model obtained by self-supervised training with multiple sample video frames as training samples. The first visual feature of the target video and the second number of video frame features are fused together to obtain the target visual feature of the target video.

2. The method according to claim 1, characterized in that, The step of extracting the first visual features of the target video using the first feature extraction model includes: The third number of video frames are convolved using multiple convolution kernels of the first sub-model to obtain the static visual features of the target video. By using multiple convolution kernels of the second sub-model, the first number of video frames are convolved to obtain the dynamic visual features of the target video. Based on the static visual features and the dynamic visual features, the first visual features of the target video are obtained; In the first sub-model, the number of convolutional kernels is greater than the number of convolutional kernels in the second sub-model.

3. The method according to claim 2, characterized in that, The step of obtaining the first visual feature of the target video based on the static visual features and the dynamic visual features includes: Spatiotemporal attention processing is performed on the static visual features to obtain the first static features of the target video; Spatiotemporal attention processing is performed on the dynamic visual features to obtain the first dynamic features of the target video; The first static feature and the first dynamic feature are fused to obtain the fused feature; Global pooling is performed on the fused features and the first dynamic features respectively to obtain the second static features and the second dynamic features of the target video. The second static feature and the second dynamic feature are fused to obtain the first visual feature of the target video.

4. The method according to claim 3, characterized in that, The process of performing spatiotemporal attention processing on the static visual features to obtain the first static feature of the target video includes: The static visual features are subjected to channel attention processing to obtain static channel attention weights corresponding to the static visual features. The static channel attention weights are multiplied by the static visual features to obtain the third static feature; The third static feature is subjected to max pooling, mean pooling, and dilated convolution to obtain the first spatiotemporal attention weight corresponding to the third static feature; The first static feature is obtained by multiplying the third static feature by the first spatiotemporal attention weight and then adding it to the static visual feature.

5. The method according to claim 4, characterized in that, The step of performing max pooling, mean pooling, and dilated convolution on the third static feature to obtain the first spatiotemporal attention weights corresponding to the third static feature includes: The third static feature is subjected to max pooling and mean pooling to obtain the fourth static feature; The fourth static feature is convolved using convolution kernels with different dilation rates to obtain multiple fifth static features. After summing the multiple fifth static features, normalization is performed to obtain the first spatiotemporal attention weight corresponding to the third static feature.

6. The method according to claim 3, characterized in that, The process of performing spatiotemporal attention processing on the dynamic visual features to obtain the first dynamic feature of the target video includes: The dynamic visual features are subjected to channel attention processing to obtain dynamic channel attention weights corresponding to the dynamic visual features. The dynamic channel attention weights are multiplied by the dynamic visual features to obtain the third dynamic feature; The third dynamic feature is subjected to max pooling, mean pooling, and dilated convolution to obtain the second spatiotemporal attention weight corresponding to the third dynamic feature; The first dynamic feature is obtained by multiplying the third dynamic feature by the second spatiotemporal attention weight and then adding it to the dynamic visual feature.

7. The method according to claim 6, characterized in that, The process of performing max pooling, mean pooling, and dilated convolution on the third dynamic feature to obtain the second spatiotemporal attention weights corresponding to the third dynamic feature includes: The third dynamic feature is subjected to maximum pooling and mean pooling to obtain the fourth dynamic feature; The fourth dynamic feature is convolved using convolution kernels with different dilation rates to obtain multiple fifth dynamic features. After summing the multiple fifth dynamic features, normalization is performed to obtain the second spatiotemporal attention weight corresponding to the third dynamic feature.

8. The method according to claim 3, characterized in that, The step of fusing the second static feature and the second dynamic feature to obtain the first visual feature of the target video includes: The second static feature and the second dynamic feature are subjected to mean pooling respectively. The second static feature after mean pooling and the second dynamic feature after mean pooling are then concatenated to obtain the first visual feature of the target video.

9. The method according to claim 1, characterized in that, The step of fusing the first visual feature of the target video and the second number of video frame features to obtain the target visual feature of the target video includes: The second number of video frame features are subjected to mean pooling to obtain the second visual features of the target video. The first visual feature and the second visual feature are spliced ​​together to obtain the target visual feature of the target video.

10. The method according to claim 1, characterized in that, The training methods for the second feature extraction model include: Obtain the first sample video frame from the plurality of sample video frames; The first sample video frame is augmented in two ways to obtain a first augmented video frame and a second augmented video frame. The first enhanced video frame is input into the second model, and the first sample video frame features of the first enhanced video frame are extracted by the second model. The second enhanced video frame is input into the reference model, and the reference video frame features of the second enhanced video frame are extracted through the reference model. The reference model and the second model have the same model structure. Based on the similarity between the features of the first sample video frame and the features of the reference video frame, the model parameters of the second model are adjusted. In response to the fact that the model parameters of the second model meet the second objective condition, the second model is determined as the second feature extraction model.

11. The method according to claim 10, characterized in that, Before determining the second model as the second feature extraction model in response to the second model parameters meeting the second target condition, the method further includes: Obtain a second sample video frame from the plurality of sample videos; The second sample video frame is input into the reference model, and the second sample video frame features are extracted through the reference model. The model parameters of the second model are adjusted based on the similarity between the features of the first sample video frame and the features of the second sample video frame.

12. A video feature extraction device, characterized in that, The device includes: The video frame acquisition module is used to acquire multiple video frames from the target video; A first input module is used to uniformly sample a first number of video frames to obtain a third number of video frames, the third number being less than the first number; input the third number of video frames into a first sub-model of a first feature extraction model, and input the first number of video frames into a second sub-model of the first feature extraction model to extract the first visual features of the target video. The first feature extraction model is a visual feature extraction model trained with multiple sample videos as training samples and multiple sample video labels corresponding to the multiple sample videos as supervised training. The first sub-model is used to extract the static visual features of the target video, and the second sub-model is used to extract the dynamic visual features of the target video. The second input module is used to input a second number of video frames into the second feature extraction model, and extract the second number of video frame features of the second number of video frames through the second feature extraction model. The second feature extraction model is a visual feature extraction model obtained by self-supervised training with multiple sample video frames as training samples. The feature fusion module is used to fuse the first visual features of the target video and the second number of video frame features to obtain the target visual features of the target video.

13. The apparatus according to claim 12, characterized in that, The first input module is used for: The third number of video frames are convolved using multiple convolution kernels of the first sub-model to obtain the static visual features of the target video. By using multiple convolution kernels of the second sub-model, the first number of video frames are convolved to obtain the dynamic visual features of the target video. Based on the static visual features and the dynamic visual features, the first visual features of the target video are obtained; In the first sub-model, the number of convolutional kernels is greater than the number of convolutional kernels in the second sub-model.

14. The apparatus according to claim 13, characterized in that, The first input module is used for: Spatiotemporal attention processing is performed on the static visual features to obtain the first static features of the target video; Spatiotemporal attention processing is performed on the dynamic visual features to obtain the first dynamic features of the target video; The first static feature and the first dynamic feature are fused to obtain the fused feature; Global pooling is performed on the fused features and the first dynamic features respectively to obtain the second static features and the second dynamic features of the target video. The second static feature and the second dynamic feature are fused to obtain the first visual feature of the target video.

15. The apparatus according to claim 14, characterized in that, The first input module is used for: The static visual features are subjected to channel attention processing to obtain static channel attention weights corresponding to the static visual features. The static channel attention weights are multiplied by the static visual features to obtain the third static feature; The third static feature is subjected to max pooling, mean pooling, and dilated convolution to obtain the first spatiotemporal attention weight corresponding to the third static feature; The first static feature is obtained by multiplying the third static feature by the first spatiotemporal attention weight and then adding it to the static visual feature.

16. The apparatus according to claim 15, characterized in that, The first input module is used for: The third static feature is subjected to max pooling and mean pooling to obtain the fourth static feature; The fourth static feature is convolved using convolution kernels with different dilation rates to obtain multiple fifth static features. After summing the multiple fifth static features, normalization is performed to obtain the first spatiotemporal attention weight corresponding to the third static feature.

17. The apparatus according to claim 14, characterized in that, The first input module is used for: The dynamic visual features are subjected to channel attention processing to obtain dynamic channel attention weights corresponding to the dynamic visual features. The dynamic channel attention weights are multiplied by the dynamic visual features to obtain the third dynamic feature; The third dynamic feature is subjected to max pooling, mean pooling, and dilated convolution to obtain the second spatiotemporal attention weight corresponding to the third dynamic feature; The first dynamic feature is obtained by multiplying the third dynamic feature by the second spatiotemporal attention weight and then adding it to the dynamic visual feature.

18. The apparatus according to claim 17, characterized in that, The first input module is used for: The third dynamic feature is subjected to maximum pooling and mean pooling to obtain the fourth dynamic feature; The fourth dynamic feature is convolved using convolution kernels with different dilation rates to obtain multiple fifth dynamic features. After summing the multiple fifth dynamic features, normalization is performed to obtain the second spatiotemporal attention weight corresponding to the third dynamic feature.

19. The apparatus according to claim 14, characterized in that, The first input module is used for: The second static feature and the second dynamic feature are subjected to mean pooling respectively. The second static feature after mean pooling and the second dynamic feature after mean pooling are then concatenated to obtain the first visual feature of the target video.

20. The apparatus according to claim 12, characterized in that, The feature fusion module is used for: The second number of video frame features are subjected to mean pooling to obtain the second visual features of the target video. The first visual feature and the second visual feature are spliced ​​together to obtain the target visual feature of the target video.

21. The apparatus according to claim 12, characterized in that, The training apparatus for the second feature extraction model includes: The second sample video frame acquisition module is used to acquire the first sample video frame from the plurality of sample video frames; The data augmentation module is used to perform two types of data augmentation on the first sample video frame to obtain a first augmented video frame and a second augmented video frame. The second sample input module is used to input the first enhanced video frame into the second model and extract the first sample video frame features of the first enhanced video frame through the second model. The third sample input module is used to input the second enhanced video frame into the reference model, and extract the reference video frame features of the second enhanced video frame through the reference model. The reference model and the second model have the same model structure. The second model parameter adjustment module is used to adjust the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the reference video frame. The second model acquisition module is used to determine the second model as the second feature extraction model in response to the fact that the model parameters of the second model meet the second target condition.

22. The apparatus according to claim 21, characterized in that: The second sample video frame acquisition module is further configured to acquire second sample video frames from the plurality of sample videos before determining the second model as the second feature extraction model in response to the model parameters of the second model meeting the second target condition. The third sample input module is further configured to input the second sample video frame into the reference model and extract the second sample video frame features of the second sample video frame through the reference model. The second model parameter adjustment module is further configured to adjust the model parameters of the second model based on the similarity between the features of the first sample video frame and the features of the second sample video frame.

23. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the video feature extraction method as described in any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the video feature extraction method as described in any one of claims 1 to 11.

25. A computer program product, characterized in that, The computer program product includes program code stored in a computer-readable storage medium, a processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, causing the computer device to perform the video feature extraction method as described in any one of claims 1 to 11.

Images