Video processing method, device, equipment, storage medium and program product

By combining supervised learning and self-supervised learning in the video processing model, the problem of poor generalization of video processing models in the prior art is solved, and the accuracy of the model's extraction of video features and the accuracy of subsequent operations is improved.

CN114565811BActive Publication Date: 2024-11-05TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210200063.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-02
Publication Date
2024-11-05
Estimated Expiration
2042-03-02

AI Technical Summary

Technical Problem

The existing video processing models have poor generalization and low model accuracy, which leads to insufficient accuracy in the feature extraction of the input video, affecting the accuracy of subsequent operations.

Method used

By combining supervised learning and self-supervised learning, and using video samples and labeled information to double train the model, the extraction network can be applied to supervised tasks and self-supervised tasks at the same time, thereby improving the accuracy and generalization of the model.

Benefits of technology

The accuracy and comprehensiveness of the video processing model extracts video features is improved, and the accuracy of subsequent operations based on video features is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114565811B_ABST
    Figure CN114565811B_ABST
Patent Text Reader

Abstract

The present application provides a video processing method, device, equipment, storage medium and program product, which relate to the field of machine learning technology. The method includes: obtaining a first video sample, and first annotation information for annotating the first video sample based on a first task; extracting the first sample feature of the first video sample based on a feature extraction network in a first model; processing the first sample feature based on the first task processing network and the second task processing network in the first model, respectively, to obtain a first prediction result of the first task and a second prediction result of the second task; updating the parameters of the first model in a supervised manner based on the first annotation information and the first prediction result; updating the parameters of the first model in a self-supervised manner based on the first video sample and the second prediction result. Through the above method, the model accuracy and generalization are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning, and in particular to a video processing method, apparatus, device, storage medium and program product. Background Art

[0002] With the advancement of science and technology, machine learning has been increasingly applied to various technical fields.

[0003] Among them, in the field of video processing technology, computer equipment can supervisely train video processing models based on task labels corresponding to sample videos, so as to process input videos through the video processing to perform tasks corresponding to the task labels, such as classifying input videos based on video processing models.

[0004] However, in the above process, the video processing model obtained through supervised training based on the task labels corresponding to the sample videos has poor generalization and low model accuracy. Summary of the invention

[0005] The embodiments of the present application provide a video processing method, apparatus, device, storage medium and program product, which can improve the model accuracy and generalization of the video processing model, so that the video features of the input video extracted by the video processing model are more accurate, and the accuracy of subsequent operations based on the video features is improved. The technical solution is as follows:

[0006] In one aspect, a video processing method is provided, the method being executed by a computer device, the method comprising:

[0007] Acquire a first video sample and first annotation information of the first video sample; the first annotation information is obtained by annotating the first video sample based on a first task;

[0008] Processing the first video sample based on the feature extraction network in the first model to obtain a first sample feature of the first video sample;

[0009] Processing the first sample feature based on a first task processing network in the first model to obtain a first prediction result of the first task;

[0010] Processing the first sample feature based on the second task processing network in the first model to obtain a second prediction result for the second task;

[0011] Based on the first video sample, the first annotation information, the first prediction result, and the second prediction result, the parameters of the first model are updated to obtain a first video processing model; the first annotation information and the first prediction result are used to update the parameters of the first model in a supervised manner; the first video sample and the second prediction result are used to update the parameters of the first model in a self-supervised manner;

[0012] The first video processing model is used to perform at least one of the first task and the second task on the input video.

[0013] In another aspect, a video processing method is provided, the method comprising:

[0014] Get the first target video;

[0015] Processing the first target video based on a feature extraction network in a first video processing model to obtain a first video feature of the first target video;

[0016] The first video feature is processed based on the task processing network in the first video processing model to obtain a prediction result of the first target video; the task processing network includes at least one of a first task processing network and a second task processing network; the first task processing network is used to process the first video feature to obtain a first prediction result of the first task; the second task processing network is used to process the first video feature to obtain a second prediction result of the second task; the prediction result of the first target video includes at least one of the first prediction result of the first task and the second prediction result of the second task;

[0017] Among them, the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and updating parameters based on the first video sample in a self-supervised manner; the first annotation information is obtained by annotating the first video sample based on the first task.

[0018] In another aspect, a video processing method is provided, the method comprising:

[0019] Acquire the second target video;

[0020] The second target video is processed based on the feature extraction network in the second video processing model to obtain a second video feature of the second target video; the second video processing model is constructed and trained based on the feature extraction network in the first video processing model; the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and by updating parameters in a self-supervised manner based on the first video sample;

[0021] The second video feature is processed based on the third task processing network in the second video processing model to obtain a third prediction result of the third task.

[0022] In another aspect, a video processing device is provided, the device comprising:

[0023] A first acquisition module is used to acquire a first video sample and first annotation information of the first video sample; the first annotation information is obtained by annotating the first video sample based on the first task; a first sample feature acquisition module is used to process the first video sample based on the feature extraction network in the first model to obtain a first sample feature of the first video sample;

[0024] A first result acquisition module, used to process the first sample feature based on the first task processing network in the first model to obtain a first prediction result of the first task;

[0025] A second result acquisition module, used to process the first sample feature based on the second task processing network in the first model to obtain a second prediction result of the second task;

[0026] A first parameter updating module, configured to update the parameters of the first model based on the first video sample, the first annotation information, the first prediction result, and the second prediction result to obtain a first video processing model; the first annotation information and the first prediction result are used to update the parameters of the first model in a supervised manner; the first video sample and the second prediction result are used to update the parameters of the first model in a self-supervised manner;

[0027] The first video processing model is used to perform at least one of the first task and the second task on the input video.

[0028] In a possible implementation, the feature extraction network includes a feature extraction subnetwork and a feature fusion subnetwork;

[0029] The first sample feature acquisition module includes:

[0030] A sample feature extraction submodule, configured to perform feature extraction on the first video sample based on the feature extraction subnetwork to obtain sample features of at least two dimensions of the first video sample;

[0031] The sample feature fusion submodule is used to fuse the sample features of at least two dimensions based on the feature fusion subnetwork to obtain the first sample feature of the first video sample.

[0032] In a possible implementation manner, the device further includes:

[0033] a preprocessing module, configured to preprocess the first video sample to obtain video elements of at least two dimensions of the first video sample before the first sample feature acquisition module processes the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample;

[0034] The sample feature extraction submodule is used to extract features of the video elements in at least two dimensions based on the feature extraction subnetwork to obtain sample features of the first video sample in at least two dimensions.

[0035] In a possible implementation, the video elements of at least two dimensions include: video frames and video text; the video text includes at least one of a video title text and a video content text; and the video content text refers to the text corresponding to the voice information in the input video.

[0036] In a possible implementation, the second task processing network includes at least one of a first processing network and a second processing network; the second prediction result includes at least one of a prediction result corresponding to the first processing network and a prediction result corresponding to the second processing network; the prediction result of the first processing network corresponds to a prediction result of a video frame dimension; and the prediction result of the second processing network corresponds to a prediction result of a video text dimension.

[0037] In a possible implementation manner, in response to the second task processing network including the first processing network, the apparatus further includes:

[0038] a first mask module, configured to perform a first mask process on the video frame of the first video sample before the first sample feature acquisition module processes the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample;

[0039] The second result acquisition module is used to process the first sample feature based on the second task processing network to obtain the second prediction result including a predicted video frame; the predicted video frame includes a video frame obtained by restoring the video frame in which the first video sample is missing after being processed by the first mask.

[0040] In a possible implementation manner, in response to the second task processing network including the second processing network, the apparatus further includes:

[0041] a first mask module, configured to perform a second mask process on the video text of the first video sample before the first sample feature acquisition module processes the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample;

[0042] The second result acquisition module is used to process the first sample features based on the second task processing network to obtain a second prediction result including predicted video text; the predicted video text includes the text obtained by restoring the video text that is missing from the first video sample after being processed by the second mask.

[0043] In a possible implementation, the first result acquisition module is used to process the sample features of at least two dimensions and the first sample features respectively based on the first task processing network to obtain the first prediction result; the first prediction result includes the prediction results corresponding to the video features of at least two dimensions respectively, and the prediction result corresponding to the first sample feature.

[0044] In a possible implementation, the feature fusion submodule is used to:

[0045] Splicing the sample features in at least two dimensions to obtain sample splicing features;

[0046] Based on the feature fusion subnetwork, the sample splicing feature is processed to obtain the first sample feature of the first video sample.

[0047] In a possible implementation, the first parameter updating module includes:

[0048] A first calculation submodule, used to calculate a function value of a first loss function based on the first prediction result and the first annotation information;

[0049] A second calculation submodule, used for calculating a function value of a second loss function based on the second prediction result and the first video sample;

[0050] The model updates the parameters of the first model based on the function value of the first loss function and the function value of the second loss function to obtain the first video processing model.

[0051] In a possible implementation manner, the device further includes:

[0052] A model building module, used to build a second model; the second model includes the feature extraction network after training, and a third task processing network;

[0053] A second acquisition module is used to acquire a second video sample and second annotation information of the second video sample; the second annotation information is obtained by annotating the second video sample based on the third task;

[0054] The second sample feature acquisition module processes the second video sample based on the feature extraction network to obtain a second sample feature of the second video sample;

[0055] a third result acquisition module, configured to process the second sample feature based on the third task processing network to obtain a third prediction result of the third task;

[0056] A second parameter updating module, configured to update the parameters of the second model based on the second annotation information and the third prediction result to obtain a second video processing model;

[0057] The second video processing model is used to perform the third task on the input video.

[0058] In another aspect, a video processing device is provided, the device comprising:

[0059] A first video acquisition module, used to acquire a first target video;

[0060] A first video feature acquisition module, used to process the first target video based on a feature extraction network in a first video processing model to obtain a first video feature of the first target video;

[0061] A first prediction result acquisition module is used to process the first video feature based on the task processing network in the first video processing model to obtain a prediction result of the first target video; the task processing network includes at least one of a first task processing network and a second task processing network; the first task processing network is used to process the first video feature to obtain a first prediction result of the first task; the second task processing network is used to process the first video feature to obtain a second prediction result of the second task; the prediction result of the first target video includes at least one of the first prediction result of the first task and the second prediction result of the second task;

[0062] Among them, the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and updating parameters based on the first video sample in a self-supervised manner; the first annotation information is obtained by annotating the first video sample based on the first task.

[0063] In another aspect, a video processing device is provided, the device comprising:

[0064] A second video acquisition module, used to acquire a second target video;

[0065] A second video feature acquisition module is used to process the second target video based on the feature extraction network in the second video processing model to obtain the second video feature of the second target video; the second video processing model is constructed and trained based on the feature extraction network in the first video processing model; the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and by updating parameters in a self-supervised manner based on the first video sample;

[0066] The second prediction result acquisition module is used to process the second video feature based on the third task processing network in the second video processing model to obtain a third prediction result of the third task.

[0067] On the other hand, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-mentioned video processing method.

[0068] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned video processing method.

[0069] On the other hand, a computer program product is provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor to implement the video processing method provided in the above-mentioned various optional implementations.

[0070] The technical solution provided by this application may have the following beneficial effects:

[0071] The video processing method provided in the embodiment of the present application performs supervised training on the first model based on video samples and first annotation information of the video samples, and performs self-supervised training on the first model based on the video samples, and jointly trains the first model according to the video samples, the first annotation information of the video samples, the prediction results of the supervised task and the prediction results of the self-supervised task, so as to obtain a first video model based on the trained first model, and the first video model can be used to perform at least one of the first task and the second task on the input video; that is, in the above scheme, the features extracted by the feature extraction network in the trained model can be applied to both supervised tasks and self-supervised tasks, thereby improving the model accuracy and generalization of the first video model, making the video features of the input video extracted by the first video model more accurate and comprehensive, and improving the accuracy of subsequent operations based on the video features.

[0072] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0074] Figure 1 A system structure diagram used in a video processing method provided by an exemplary embodiment of the present application is shown;

[0075] Figure 2 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown;

[0076] Figure 3 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown;

[0077] Figure 4 is a framework diagram of model training and video processing according to an exemplary embodiment;

[0078] Figure 5 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown;

[0079] Figure 6 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown;

[0080] Figure 7 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown;

[0081] Figure 8 A schematic diagram of a first network shown in an exemplary embodiment of the present application is shown;

[0082] Fig. 9 A schematic diagram of feature extraction at the ViT 384 layer shown in an exemplary embodiment of the present application is shown;

[0083] Fig.10 A schematic diagram showing a video feature shown in an exemplary embodiment of the present application is shown;

[0084] Fig.11 A schematic diagram showing a process of obtaining a first prediction result of a first task corresponding to a first video sample shown in an exemplary embodiment of the present application is shown;

[0085] Fig.12 A schematic diagram of the structure of a classification head shown in an exemplary embodiment of the present application is shown;

[0086] Fig.13 A schematic diagram of a framework of a first model training process shown in an exemplary embodiment of the present application is shown;

[0087] Fig.14 A block diagram of a video processing device shown in an exemplary embodiment of the present application is shown;

[0088] Fig.15 A block diagram of a video processing device provided by an exemplary embodiment of the present application is shown;

[0089] Fig.16 A block diagram of a video processing device provided by an exemplary embodiment of the present application is shown;

[0090] Fig.17 is a structural block diagram of a computer device according to an exemplary embodiment;

[0091] Fig.18 It is a structural block diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0092] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0093] For ease of understanding, the terms involved in this application are explained below.

[0094] 1) Artificial Intelligence (AI)

[0095] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0096] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The display device including an image acquisition component shown in this application mainly involves computer vision technology and machine learning / deep learning.

[0097] 2) Computer Vision (CV)

[0098] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (Three Dimensions) technology, virtual reality, augmented reality and map construction, as well as common biometric recognition technologies such as face recognition.

[0099] 3) Unsupervised Learning

[0100] Unsupervised learning is a training method for machine learning. It is essentially a statistical method and a training method that can discover potential structures in unlabeled data. Unsupervised learning has the following main characteristics: 1. Unsupervised learning has no clear goals. 2. Unsupervised learning does not require data labeling. 3. Unsupervised learning cannot quantify the effects.

[0101] Unsupervised learning mainly includes principal component analysis (PCA), Gaussian mixture model, manifold learning algorithm (Isomap), etc.

[0102] In an embodiment of the present application, self-supervised learning can be implemented as unsupervised learning; self-supervised learning is an unsupervised learning method in which data can provide supervisory information; in an embodiment of the present application, it is implemented by constructing a supervisory signal on the data of the video sample itself to perform unsupervised training on the first model.

[0103] 4) Supervised Learning

[0104] Supervised learning is a training method for machine learning. It uses existing training samples to train a model, and then uses this model to map all inputs to corresponding outputs. It makes simple judgments on the outputs to achieve the purpose of prediction and classification, and thus has the ability to predict and classify unknown data.

[0105] Figure 1 FIG. 1 shows a system structure diagram used in a video processing method provided by an exemplary embodiment of the present application. Figure 1 As shown, the system includes: a server 110 and a terminal 120.

[0106] The above-mentioned server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0107] The terminal 120 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a vehicle-mounted terminal, etc. with a video processing function, but is not limited thereto.

[0108] The above system includes one or more servers 110 and multiple terminals 120. The embodiment of the present application does not limit the number of servers 110 and terminals 120.

[0109] The terminal and the server are connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0110] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology. This application is not limited here.

[0111] Figure 2A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown. Figure 2 As shown, the method can be executed by a computer device, which can be implemented as a terminal or a server, and the terminal or the server can be as shown in Figure 1 The terminal or server shown, such as Figure 2 As shown, the video processing method may include the following steps:

[0112] Step 210: Obtain a first video sample and first annotation information of the first video sample; the first annotation information is obtained by annotating the first video sample based on the first task.

[0113] The first video sample may refer to at least two video samples included in a first training sample set; the first training sample set is used to train the first model, and each video sample in the first training sample set has corresponding first annotation information, and the first video annotation information corresponding to each video sample is obtained based on the annotation of each video sample by the first task.

[0114] In scenarios corresponding to different first tasks, the first annotation information may correspond to different implementation forms. Schematically, the first annotation information may be used to indicate the category to which the video sample belongs, such as the subject matter, style, etc. of the video sample; or, the first annotation information may indicate an associated object of the video sample, such as a game name, a film or TV series name, etc. The present application does not limit the implementation form of the first annotation information.

[0115] Illustratively, the first task may include but is not limited to corresponding tasks in the following scenarios: a video recommendation scenario, a video scoring scenario, a video similarity analysis scenario, a video classification scenario, and a video deduplication scenario.

[0116] Step 220: Process the first video sample based on the feature extraction network in the first model to obtain a first sample feature of the first video sample.

[0117] Step 230: Process the first sample feature based on the first task processing network in the first model to obtain a first prediction result of the first task.

[0118] The first prediction result of the first task obtained based on the first sample feature is the first prediction result of the first task corresponding to the first video sample.

[0119] Step 240: Process the first sample feature based on the second task processing network in the first model to obtain a second prediction result of the second task.

[0120] The second prediction result of the second task obtained based on the first sample feature is the second prediction result of the second task corresponding to the first video sample.

[0121] The first task processing network and the second task processing network may be part of the first model.

[0122] Step 250, based on the first video sample, the first annotation information, the first prediction result, and the second prediction result, the parameters of the first model are updated to obtain a first video processing model; the first annotation information and the first prediction result are used to update the parameters of the first model in a supervised manner; the first video sample and the second prediction result are used to update the parameters of the first model in a self-supervised manner.

[0123] The first video processing model is used to perform at least one of a first task and a second task on the input video.

[0124] During the model training process, the prediction results used to update the parameters of the first model are all prediction results corresponding to the first video sample. That is, the first prediction result refers to the first prediction result of the first task corresponding to the first video sample, and the second prediction result refers to the second prediction result of the second task corresponding to the first video sample.

[0125] In an embodiment of the present application, the first video processing model can be constructed based on the first model after training, and the first model is obtained through comprehensive training of supervised tasks and self-supervised tasks. The computer device can perform supervised training on the first model based on the difference between the first annotation information and the first prediction result; and perform self-supervised training on the first model based on the difference between the first video sample and the second prediction result.

[0126] Optionally, the first video processing model may include at least one of a first task processing network and a second task processing network, that is, the first video processing model can be used to perform at least one of a first task and a second task on the input video; when the first video processing model includes the first task processing network, the first video processing model can be used to perform the first task on the input video; when the first video processing model includes the second task processing network, the first video processing model can be used to perform the second task on the input video; when the first video processing model includes both the first task processing network and the second task processing network, the first video processing model can be used to perform the first task and the second task on the input video.

[0127] In summary, the video processing method provided in the embodiment of the present application performs supervised training on the first model based on video samples and first annotation information of the video samples, and performs self-supervised training on the first model based on the video samples, and jointly trains the first model according to the video samples, the first annotation information of the video samples, the prediction results of the supervised task and the prediction results of the self-supervised task, so as to obtain a first video model based on the trained first model, and the first video model can be used to perform at least one of the first task and the second task on the input video; that is, in the above scheme, the features extracted by the feature extraction network in the trained model can be simultaneously applicable to supervised tasks and self-supervised tasks, thereby improving the model accuracy and generalization of the first video model, making the video features of the input video extracted by the first video model more accurate, and improving the accuracy of subsequent operations based on the video features.

[0128] In one possible implementation, based on Figure 2 After the first model is trained, in the embodiment shown, in addition to generating the first video processing model, other video processing models for implementing different tasks can be constructed based on the feature extraction network in the first model after training combined with other networks; schematically, Figure 3 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The computer device can be implemented as a terminal and a server. The terminal and the server can be as follows. Figure 1 The terminal or server shown; Figure 3 As shown, the method includes:

[0129] Step 310, constructing a second model; the second model includes a trained feature extraction network and a third task processing network.

[0130] The trained feature extraction network is Figure 2 In the embodiment shown, the feature extraction network in the first model is obtained after the comprehensive training of the supervised task and the self-supervised task is completed; alternatively, the feature extraction network can also be based on Figure 2 The feature extraction network in the first video processing model obtained in the illustrated embodiment.

[0131] The feature extraction network is used to extract video features of the input video so that the third task processing network can process the video features to obtain a third prediction result of the third task corresponding to the video.

[0132] Step 320, obtaining a second video sample and second annotation information of the second video sample; the second annotation information is obtained by annotating the second video sample based on the third task.

[0133] Step 330: Process the second video sample based on the feature extraction network to obtain a second sample feature of the second video sample.

[0134] Step 340: Process the second sample feature based on the third task processing network to obtain a third prediction result of the third task.

[0135] Step 350: Based on the second annotation information and the third prediction result, update the parameters of the second model to obtain a second video processing model.

[0136] The second video processing model is used to perform a third task on the input video.

[0137] The second video processing model is used to indicate the trained second model.

[0138] Illustratively, the third task may include but is not limited to corresponding tasks in the following scenarios: video recommendation scenario, video scoring scenario, video similarity analysis scenario, video classification scenario and video deduplication scenario; taking the video classification scenario and video deduplication scenario as examples, the application of the video processing method provided by the present application in the above scenarios can be implemented as follows:

[0139] 1. Video classification scenarios.

[0140] Video classification refers to classifying the content contained in a given video clip; through the classification of video clips, the video clips can be uniformly managed and recommended; in the process of classifying video clips, the feature extraction network obtained by joint training of supervised tasks and self-supervised tasks provided in the embodiments of the present application can be combined with the video classification network (third task processing network) to construct a second model for video classification, and after fine-tuning the parameters in the second model, a second video processing model (i.e., the second model after training) is obtained, and the second video processing model is used to classify the input video; since the feature extraction network is obtained based on joint training of supervised tasks and self-supervised tasks, the feature extraction network can extract more accurate video features of the input video, thereby improving the accuracy of video classification.

[0141] 2. Video deduplication scenarios.

[0142] Video deduplication means that for a given plurality of video clips, by obtaining the similarity between the plurality of video clips, the video clips with the same content or a high degree of content overlap are screened out, and the video clips with a later release time are determined as duplicate video clips; in this process, a feature extraction network obtained by joint training of supervised tasks and self-supervised tasks provided in the embodiments of the present application can be combined with a similarity matching network (third task processing network) to construct a second model for video deduplication, and after fine-tuning the parameters in the second model, a second video processing model (i.e., the second model after training) is obtained, and the second video processing model is used to perform video deduplication on the input video; since the feature extraction network is obtained based on joint training of supervised tasks and self-supervised tasks, the feature extraction network can extract more accurate video features of the input video, and thus when performing similarity matching, the accuracy of similarity matching can be improved, thereby improving the effect of video deduplication.

[0143] The solution involved in this application includes a model training stage and a video processing stage. Figure 4 is a framework diagram of a model training and video processing according to an exemplary embodiment. Figure 4 As shown, in the model training stage, the model training device 410 performs joint training of supervised tasks and self-supervised tasks on the first model through pre-set training samples (including the first video sample and the first annotation information of the first video sample) to obtain the first model after training; after obtaining the first model after training, the computer device can generate a first video processing model based on the first model after training, and the first video processing model can be used to perform at least one of the first task and the second task; or, the computer device can also build a second model based on the feature extraction network in the first model after training, combined with the third task network, and obtain the second video processing model after fine-tuning the second model, so as to perform the third task on the input video based on the second video processing model. In the video processing stage, the video processing device 420 performs video processing on the input video based on the first video processing model or the second video processing model to perform the corresponding task.

[0144] Among them, the above-mentioned model training device 410 and video processing device 420 can be computer devices, for example, the computer device can be a terminal or a server.

[0145] Optionally, the above-mentioned model training device 410 and video processing device 420 can be the same device, or the model training device 410 and the video processing device 420 can also be different devices. Moreover, when the model training device 410 and the video processing device 420 are different devices, the model training device 410 and the video processing device 420 can be devices of the same type, such as the model training device 410 and the video processing device 420 can both be servers; or, the model training device 410 and the video processing device 420 can also be different types of devices, such as the video processing device 420 can be a terminal, and the model training device 410 can be a server, etc. The embodiment of the present application does not limit the specific types of the model training device 410 and the video processing device 420.

[0146] Figure 5 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The computer device can be implemented as a terminal or a server. The terminal or the server can be Figure 1 The terminal or server shown, such as Figure 5 As shown, the video processing method may include the following steps:

[0147] Step 510: Acquire a first target video.

[0148] In the embodiment of the present application, the video to be processed is the first target video.

[0149] Step 520: Process the first target video based on the feature extraction network in the first video processing model to obtain a first video feature of the first target video.

[0150] The first video feature is used to indicate a video feature of the first target video.

[0151] Step 530, based on the task processing network in the first video processing model, the first video feature is processed to obtain a prediction result of the first target video; the task processing network includes at least one of the first task processing network and the second task processing network; the first task processing network is used to process the first video feature to obtain a first prediction result of the first task; the second task processing network is used to process the first video feature to obtain a second prediction result of the second task; the prediction result of the first target video includes at least one of the first prediction result of the first task and the second prediction result of the second task.

[0152] In the embodiment of the present application, the prediction result obtained by processing the first video feature through the first task processing network is the first prediction result of the first target video on the first task; the prediction result obtained by processing the first video feature through the second task processing network is the second prediction result of the first target video on the second task. When the task processing network in the first video processing model includes the first task processing network, the prediction result of the first target video includes the first prediction result of the first task; when the task processing network of the first video processing model includes the second task processing network, the prediction result of the first target video includes the second prediction result.

[0153] Among them, the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and updating parameters based on the first video sample in a self-supervised manner; the first annotation information is obtained by annotating the first video sample based on the first task.

[0154] In summary, the video processing method provided in the embodiment of the present application obtains a first model through joint training of a supervised task and a self-supervised task, and obtains a first video model based on the trained first model to process the input first target video to obtain a prediction result of the first target video; since the features extracted by the feature extraction network in the trained first model can be applied to both supervised tasks and self-supervised tasks, the model accuracy and generalization of the first video model are relatively high; by processing the target video with the first video model with higher precision, the accuracy of the prediction result of the obtained target video is improved.

[0155] On the basis of obtaining the trained first model, a second model for performing other video processing tasks can be constructed based on the feature extraction network in the trained first model. After fine-tuning the second model, a second video processing model is obtained. The computer device can process the input video through the second video processing model to perform different prediction tasks. Figure 6 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The computer device can be implemented as a terminal or a server. The terminal or the server can be as follows Figure 1 The terminal or server shown; Figure 6 As shown, the method may include:

[0156] Step 610: Acquire a second target video.

[0157] Step 620, processing the second target video based on the feature extraction network in the second video processing model to obtain the second video features of the second target video; the second video processing model is constructed and trained based on the feature extraction network in the first video processing model; the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and by updating parameters in a self-supervised manner based on the first video sample.

[0158] The second video processing model includes the feature extraction network in the first video processing model and the third task network.

[0159] Step 630: Process the second video feature based on the third task processing network in the second video processing model to obtain a third prediction result of the third task.

[0160] To summarize, the video processing method provided in the embodiment of the present application, after obtaining the first model based on the joint training of the supervised task and the self-supervised task, constructs a second model based on the feature extraction network in the first model combined with the third task processing network, and after fine-tuning the second model, obtains the second video processing model, and processes the input second target video through the second video processing model to obtain a third prediction result of the second target video corresponding to the third task; since the accuracy of the trained feature extraction network is relatively high, the accuracy of the second video processing model constructed based on the feature extraction network is also relatively high, thereby improving the accuracy of the prediction result of the obtained second target video.

[0161] Taking the acquisition of the first video processing model based on the first model after training as an example, Figure 7 A flowchart of a video processing method provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The computer device can be implemented as a terminal or a server, or the terminal and the server can be used interactively. The terminal or the server can be Figure 1 The terminal or server shown, such as Figure 7 As shown, the video processing method may include the following steps:

[0162] Step 701: Obtain a first video sample and first annotation information of the first video sample; the first annotation information is obtained by annotating the first video sample based on a first task.

[0163] In a possible implementation manner, after acquiring the first video sample, the computer device may preprocess the first video sample to obtain video elements of at least two dimensions of the first video sample.

[0164] Among them, the video elements of at least two dimensions include: video frames and video text; the video text includes at least one of a video title text and a video content text; the video content text refers to the text corresponding to the voice information in the input video.

[0165] After obtaining the input video, the computer device can extract video frames from the input video, that is, extract video frames according to a target extraction interval; schematically, the target extraction interval can be 1 second, that is, the computer device can obtain M video frames of the input video by extracting one frame per second, where M is a positive integer.

[0166] When obtaining the video content text of the input video, the computer device can automatically recognize the voice information in the input video through ASR (Automatic Speech Recognition) technology, and convert the voice information in the video into text information to obtain the video content text of the video.

[0167] Step 702: Process the first video sample based on the feature extraction network in the first model to obtain a first sample feature of the first video sample.

[0168] In an embodiment of the present application, the feature extraction network in the first model may include a feature extraction subnetwork and a feature fusion subnetwork; wherein the feature extraction subnetwork is used to extract features from the input video to obtain video features of at least two dimensions of the video; and the feature fusion subnetwork is used to fuse video features of at least two dimensions to obtain fused features of the video.

[0169] Based on the composition of the feature extraction network, the process of processing the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample may include:

[0170] Perform feature extraction on the first video sample based on the feature extraction subnetwork to obtain sample features of at least two dimensions of the first video sample;

[0171] Based on the feature fusion subnetwork, sample features of at least two dimensions are fused to obtain a first sample feature of the first video sample. The first sample feature is a fusion feature obtained by fusing sample features of at least two dimensions.

[0172] On the premise that the first video sample is preprocessed to obtain video elements of at least two dimensions of the first video sample, the process of obtaining sample features of at least two dimensions of the first video sample can be implemented as follows:

[0173] Based on the feature extraction subnetwork, feature extraction is performed on video elements of at least two dimensions to obtain sample features of at least two dimensions of the first video sample.

[0174] In an embodiment of the present application, the feature extraction subnetwork may include: a first network for performing feature extraction on video frames to obtain video features corresponding to the video frames; a second network for performing feature extraction on video title text to obtain video features corresponding to the video title text; and a third network for performing feature extraction on video content text to obtain video features corresponding to the video content text.

[0175] The following embodiments schematically illustrate possible implementation forms of the first network, the second network, and the third network, but do not limit the implementation forms of the first network, the second network, and the third network:

[0176] Figure 8 A schematic diagram of a first network shown in an exemplary embodiment of the present application is shown in FIG. Figure 8 As shown, the first network 810 can be composed of a ViT 384 (Vision transformer 384) layer and a NetVLAD+ (NetVector of Locally Aggregated Descriptors) layer; wherein the ViT 384 layer is used to extract features of video frames; and the NetVLAD+ layer is used to reduce frame-level features into video-level features.

[0177] Taking the first video sample as an example, after the computer device obtains M video frames of the first video sample by extracting one frame per second, the M video frames of the first video sample are input into the first network; the ViT 384 layer in the first network extracts high-dimensional visual semantic features of the M video frames; the high-dimensional visual semantic features of the M video frames are input into the NetVLAD+ layer, and after being processed by the NetVLAD+ layer, the high-dimensional visual semantic features are converted from M*D dimensions to fixed K*D dimensions; schematically, the fixed K*D dimension can be 16*1024 dimensions; the computer device obtains the K*D dimensional features of each video frame output by the NetVLAD+ layer as the video features corresponding to the video frames of the first video sample.

[0178] Fig. 9 FIG. 4 shows a schematic diagram of feature extraction performed on the ViT 384 layer according to an exemplary embodiment of the present application, as shown in FIG. Fig. 9 As shown, after obtaining each video frame, the ViT 384 layer divides each video frame into multiple patches (blocks). Fig. 9In the embodiment, a video frame is divided into 9 patches, and the size of each patch can be specified, for example, the size of each patch can be 16*16, etc.; the computer device inputs each patch into an embedding layer 910, which can be implemented as a Linear Projection of Flattened Patches layer; after passing through the embedding layer, vectors (tokens) corresponding to the 9 patches can be obtained; a vector * for classification is added before the 9 vectors, and the dimension of the vector * is the same as the dimension of the vectors corresponding to the 9 patches; in addition, position information is added before the 10 vectors, such as Fig. 9 0 to 9 shown in . The 10 tokens are input into the Transformer Encoder 920, and then the Transformer Encoder 920 is repeatedly stacked L times, outputting the tokens for classification, and the tokens for classification are input into the MLP Head (Multilayer Perceptron Head) for classification to obtain the final classification result.

[0179] In an embodiment of the present application, the ViT 384 layer is used to extract features of the video frame dimension. In order to improve the generalization of the features of the acquired video frame dimension, the computer device can obtain the token for classification based on the ViT 384 layer, and then input the token for classification into the NetVLAD+ layer for processing; that is, the computer device can determine the features output by the Lth Transformer Encoder layer of the ViT 384 layer as the features used to express the video frame.

[0180] Take the video text containing the video title text and the video content text as an example. Fig.10 A schematic diagram showing video features shown in an exemplary embodiment of the present application is shown in FIG. Fig.10As shown, after the computer device automatically recognizes the voice information in the input video through the ASR technology, obtains the video content text of the video, and obtains the video title text by extracting the video title, the video content text and the video title text can be processed through the second network 1010 and the third network 1020 respectively, and the M*D dimensional features corresponding to the video content text and the M*D dimensional features corresponding to the video title text are obtained respectively; based on the M*D dimensional features corresponding to the video content text and the M*D dimensional features corresponding to the video title text, the computer device obtains the video features 1030 corresponding to the video content text and the video features 1040 corresponding to the video title text respectively. Among them, both the second network and the third network can be implemented as BERT (Bidirectional Encoder Representations from Transformers) networks.

[0181] Optionally, the computer device may average the features corresponding to the video title dimension and the features corresponding to the video content text, respectively, such as Fig.10 As shown, Token avg (feature average) operation is performed on the features corresponding to the video content text of M*D dimension and the features corresponding to the video title text of M*D dimension, respectively, to obtain the video features corresponding to the video title and the video features corresponding to the video content text.

[0182] After obtaining sample features of at least two dimensions of the first video sample, the process of obtaining the first sample feature of the first video sample may be implemented as follows:

[0183] Splicing sample features of at least two dimensions to obtain sample splicing features;

[0184] Based on the feature fusion sub-network, the sample splicing features are processed to obtain the first sample features of the first video sample.

[0185] The concatenation of sample features of at least two dimensions may refer to sequentially concatenating feature vectors corresponding to sample features of each dimension to obtain the sample concatenation feature. The present application does not limit the concatenation order of sample features of each dimension.

[0186] This special fusion sub-network can be implemented as a Mulit-Head network (Multi-head tag-preditionStructure, multi-head attention network).

[0187] Step 703: Process the first sample feature based on the first task processing network in the first model to obtain a first prediction result of the first task.

[0188] In an embodiment of the present application, the process of obtaining a first prediction result of a first task corresponding to a first video sample through a first task processing network includes:

[0189] Based on the first task processing network, sample features of at least two dimensions and the first sample feature are processed respectively to obtain a first prediction result; the first prediction result includes prediction results corresponding to the video features of at least two dimensions and a prediction result corresponding to the first sample feature.

[0190] Taking the sample features of at least two dimensions including the sample features corresponding to the video frame, the sample features corresponding to the video title text and the sample features corresponding to the video content text as an example, Fig.11 FIG. 1 is a schematic diagram showing a process of obtaining a first prediction result of a first task corresponding to a first video sample according to an exemplary embodiment of the present application. Fig.11 As shown, when obtaining the first prediction result, the sample features corresponding to the video frame, the sample features corresponding to the video title text, the sample features corresponding to the video content text, and the sample splicing features are respectively input into the classification head (Classification Head) 1110, wherein the first sample features of the first video sample can be obtained after the classification head processes the sample splicing features, and then, classification processing is performed based on the respective features output by the classification head to obtain the sample features corresponding to the video frame, the sample features corresponding to the video title text, the sample features corresponding to the video content text, and the prediction results corresponding to the first sample features, so as to constitute the first prediction result of the first video sample on the first task.

[0191] Fig.12 FIG. 4 shows a schematic diagram of the structure of a classification head shown in an exemplary embodiment of the present application. Fig.12 As shown, taking the sample concatenation feature as an example, the sample concatenation feature is input into the fully connected layer (FC Layer) 1210, and after being processed by the fully connected layer 1210, the first sample feature can be obtained; then, the first sample feature is input into SENet (Squeeze-and-Excitation Networks) 1220 for attention processing, and the processing result is input into the fully connected layer 1230 to obtain the prediction result corresponding to the first sample feature.

[0192] Step 704: Process the first sample feature based on the second task processing network in the first model to obtain a second prediction result of the second task.

[0193] That is to say, by processing the first sample features through the second task processing network, a second prediction result of the first video sample on the second task can be obtained.

[0194] In an embodiment of the present application, the second task processing network may include at least one of the first processing network and the second processing network; the second prediction result includes at least one of the prediction result corresponding to the first processing network and the prediction result corresponding to the second processing network; the prediction result of the first processing network corresponds to the prediction result of the video frame dimension; the prediction result of the second processing network corresponds to the prediction result of the video text dimension.

[0195] That is to say, in the embodiment of the present application, the self-supervised task performed by the second task processing network may be a task of the video frame dimension; or the self-supervised task may also be a task of the video text dimension; or the self-supervised task may include both the task of the video frame dimension and the task of the video text dimension. Among them, when the self-supervised task includes a task of the video text dimension, the self-supervised task may be a task of processing the first sample feature to restore the video title text; or the self-supervised task may also be a task of processing the first sample feature to restore the video content text; or the self-supervised task may also be a task of processing the first sample feature to restore the video title text and the video content text, and the present application does not limit this.

[0196] The first processing network may be implemented as an MFM (Masked-Frame Modeling) network; and the second processing network may be implemented as an MLM (Masked-Language Modeling) network.

[0197] In response to the second task processing network including the first processing network, before processing the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample, the method further includes:

[0198] performing a first mask process on a video frame of a first video sample;

[0199] Processing the first sample feature based on the second task processing network to obtain a second prediction result of the second task includes:

[0200] The first sample feature is processed based on the second task processing network to obtain a second prediction result including a predicted video frame; the predicted video frame includes a video frame obtained by restoring a video frame in which the first video sample is missing after being processed by the first mask.

[0201] At this time, the first sample feature is generated by pre-masking the video frame of the first video sample, extracting the video features of the video frame dimension based on the masked video frame, and fusing the video features of the video frame dimension and video features of other dimensions.

[0202] The first mask processing may refer to performing a random mask operation; or the first mask processing may refer to performing a mask processing on the video frames according to the target frame interval. In an exemplary manner, the proportion of the mask processing may be 15%, that is, the proportion of the missing video frames after the mask processing to the video frames extracted based on the first video sample is 15%.

[0203] In response to the second task processing network including the second processing network, before processing the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample, the method further includes:

[0204] Performing a second masking process on the video text of the first video sample;

[0205] Processing the first sample feature based on the second task processing network to obtain a second prediction result of the second task includes:

[0206] The first sample feature is processed based on the second task processing network to obtain a second prediction result including a predicted video frame; the predicted video frame includes a video frame obtained by restoring a video frame in which the first video sample is missing after being processed by the first mask.

[0207] At this time, the first sample feature is generated by pre-masking the video text of the first video sample, extracting the video features of the video text dimension based on the masked video text, and fusing the video features of the video text dimension and video features of other dimensions.

[0208] The second masking process may refer to random masking operations; or the second masking process may refer to masking the video text according to the target word number interval. In an exemplary manner, the masking process ratio may be 15%, that is, the missing video frames after masking process account for 15% of the initial video text.

[0209] When the second task processing network includes both the first processing network and the second processing network, it is necessary to perform the first mask processing on the video frame of the first video sample and perform the second mask processing on the video text of the first video sample.

[0210] Step 705, based on the first video sample, the first annotation information, the first prediction result, and the second prediction result, the parameters of the first model are updated to obtain a first video processing model; the first annotation information and the first prediction result are used to update the parameters of the first model in a supervised manner; the first video sample and the second prediction result are used to update the parameters of the first model in a self-supervised manner.

[0211] Optionally, based on the first video sample, the first annotation information, the first prediction result, and the second prediction result, the parameters of the first model are updated to obtain the first video processing model, which can be implemented as follows:

[0212] Calculating a function value of a first loss function based on the first prediction result and the first annotation information;

[0213] Calculating a function value of a second loss function based on the second prediction result and the first video sample;

[0214] Based on the function value of the first loss function and the function value of the second loss function, the parameters of the first model are updated to obtain a first video processing model.

[0215] In one possible implementation, the computer device updates the parameters of the feature extraction network in the first model based on the function value of the first loss function and the function value of the second loss function; the computer device updates the first task processing network in the first model based on the function value of the first loss function; the computer device updates the second task processing network in the first model based on the function value of the second loss function.

[0216] Among them, the first loss function can be implemented as a focal loss function, which can alleviate the problem of small amount of long-tail label (Tail Items) data, reduce the problem of model overfitting, and make the model training more sufficient. The second loss function can include at least one of the NCE Loss (Noise Contrastive Estimation Loss) function and the MLM Loss (Masked-Language Modeling Loss) function, wherein the NCE Loss function corresponds to the first processing network and the MLM Loss function corresponds to the second processing network; that is, when the second task processing network includes the first processing network, the second loss function includes the NCE Loss function; when the second task processing network includes the second processing network, the second loss function includes the MLM Loss function.

[0217] Optionally, taking at least two dimensions of video features including video features corresponding to video frames, video title texts, and video content texts as an example, based on the function value of the first loss function and the function value of the second loss function, the formula for calculating the total loss function can be expressed as:

[0218] L=L(tabel)*a+L(mlm)*b+L(mfm)*c

[0219] Among them, L represents the total loss function, L(tabel) represents the function value of the first loss function calculated based on the supervised task; the value of L(tabel) can be the sum or average of the function values ​​of the first loss function calculated based on the prediction results of the video features of each dimension and the first sample features and the first annotation label; L(mlm) represents the function value of the loss function calculated based on the prediction results of the video features corresponding to the video content text in the self-supervised task and the video content text without mask processing; L(mfm) ​​represents the function value of the loss function calculated based on the prediction results of the video features corresponding to the video frames in the self-supervised task and the video frames without mask processing. a, b, c represent the weights corresponding to the three tasks, and their respective values ​​can be set differently based on actual needs.

[0220] The parameters in each network of the first model are updated based on the total loss function. In the process of training the first model, the first model can be trained by repeating the above steps 701 to 705 based on at least two video samples included in the first training sample set indicated by the first video sample, so that the first prediction result obtained based on the trained first model is as close as possible to the first annotation information; the prediction result of the video frame dimension in the second prediction result obtained based on the first model is as close as possible to the video frame missing in the first video sample due to the first mask processing; the prediction result of the video text dimension in the second prediction result obtained based on the first model is as close as possible to the video text missing in the first sample video due to the second mask processing.

[0221] Optionally, when the trained first model meets the training completion condition, it is determined that the first model has been trained; wherein the training completion condition may include at least one of the accuracy of each prediction result of the trained first model reaching a respective accuracy threshold, the trained first model converges, or the number of iterations reaches a number threshold.

[0222] The computer device can directly obtain the first model after training as the first video processing model; or determine the first video processing model based on the first model, for example, obtaining the first model with the first processing network deleted as the first video processing model, or obtaining the first model with the second processing network deleted as the first video processing model.

[0223] Step 706, obtaining a first target video.

[0224] The first target video is any video to be processed.

[0225] Step 707: Process the first target video based on the feature extraction network in the first video processing model to obtain a first video feature of the first target video.

[0226] Optionally, feature extraction is performed on the first target video based on a feature extraction subnetwork in the feature extraction network to obtain video features of at least two dimensions of the first target video;

[0227] Based on the feature fusion subnetwork in the feature extraction network, video features of at least two dimensions are fused to obtain a first video feature of the first target video.

[0228] Step 708, based on the task processing network in the first video processing model, the first video feature is processed to obtain a prediction result of the first target video; the task processing network includes at least one of the first task processing network and the second task processing network; the first task processing network is used to process the first video feature to obtain a first prediction result of the first task; the second task processing network is used to process the first video feature to obtain a second prediction result of the second task; the prediction result of the first target video includes at least one of the first prediction result of the first task and the second prediction result of the second task.

[0229] Optionally, the first task can generate a predicted label for the first target video based on the first video feature of the first target video to obtain a first prediction result of the first target video on the first task; when there is a missing video frame or missing video text in the first target video, the second task can predict and generate the missing video frame of the first target video based on the first video feature of the first target video, or predict and generate the missing video text of the first target video to obtain a second prediction result of the first target video on the second task.

[0230] In summary, the video processing method provided in the embodiment of the present application performs supervised training on the first model based on video samples and first annotation information of the video samples, and performs self-supervised training on the first model based on the video samples, and jointly trains the first model according to the video samples, the first annotation information of the video samples, the prediction results of the supervised task and the prediction results of the self-supervised task, so as to obtain a first video model based on the trained first model, and the first video model can be used to perform at least one of the first task and the second task on the input video; that is, in the above scheme, the features extracted by the feature extraction network in the trained model can be simultaneously applicable to supervised tasks and self-supervised tasks, thereby improving the model accuracy and generalization of the first video model, making the video features of the input video extracted by the first video model more accurate and comprehensive, and improving the accuracy of subsequent operations based on the video features.

[0231] The method for model training by combining supervised tasks and self-supervised tasks in the embodiment of the present application can combine the advantages of supervised tasks and self-supervised tasks for model training; on the one hand, supervised tasks can reduce the difficulty of model training, and on the other hand, self-supervised tasks can enhance the generalization of the trained model, thereby improving the accuracy of feature extraction by the feature extraction network after training. When the feature extraction network is combined with other networks to construct a video processing model, so as to perform other tasks through the video processing model, the prediction accuracy of the video processing model for other tasks can be improved.

[0232] Optional, in Figure 7 In the illustrated embodiment, after executing step 705 and obtaining the trained first model, the following steps may also be performed:

[0233] Step 709, constructing a second model; the second model includes a trained feature extraction network and a third task processing network.

[0234] Step 710, obtaining a second video sample and second annotation information of the second video sample; the second annotation information is obtained by annotating the second video sample based on the third task.

[0235] The second video sample is used to indicate at least two video samples included in the second training sample set; the second training sample set is used to train the second model, and each video sample in the second training sample set has corresponding second annotation information, and the second video annotation information corresponding to each video sample is obtained based on the annotation of each video sample by the third task.

[0236] Illustratively, when the third task is a task of scoring an input video, the second annotation information may be a scoring label of the input video; when the third task is a task of classifying an input video, the second annotation information may be a classification label of the input video, and so on.

[0237] Step 711: Process the second video sample based on the feature extraction network to obtain a second sample feature of the second video sample.

[0238] Step 712: Process the second sample feature based on the third task processing network to obtain a third prediction result of the third task.

[0239] Step 713: Based on the second annotation information and the third prediction result, update the parameters of the second model to obtain a second video processing model.

[0240] The second video processing model is used to perform the third task on the input video. The second video processing model is used to indicate the second model after training.

[0241] Step 714, obtaining a second target video.

[0242] Step 715: Process the second target video based on the feature extraction network in the second video processing model to obtain a second video feature of the second target video.

[0243] Step 716: Process the second video feature based on the third task processing network in the second video processing model to obtain a third prediction result of the third task.

[0244] Therefore, the feature extraction network trained in the first model and capable of acquiring video features with high accuracy is used in combination with the third task processing network to construct a second video processing model. The second target video is processed by the second video processing model to obtain a third prediction result of the third task of the second target video, thereby improving the prediction accuracy of the second video processing model.

[0245] Taking at least two dimensions of video features including video features corresponding to video frames, video title texts, and video content texts as an example; Fig.13 A schematic diagram of a framework of a first model training process shown in an exemplary embodiment of the present application is shown in FIG. Fig.13 As shown, after obtaining the first video sample 1310, the first video sample 1310 is preprocessed to obtain the video frame 1311, the video title text 1312 and the video content text 1313 of the first video sample; the features of each video frame are extracted by the ViT 384 layer in the first network in the feature extraction subnetwork, and the features of each video frame are input into the NetVLAD+ layer for processing to obtain the features of the fixed dimension of each video frame, thereby obtaining the video features 1321 corresponding to the video frame of the first video sample; the second network in the feature extraction network (the second network can be implemented as shown in the figure) is used. Fig.13 The Bert network shown in FIG. 1 processes the video title text to obtain the video feature 1322 corresponding to the video title text of the first video sample; and the third network in the feature extraction network (the third network can be implemented as shown in FIG. 1 ) is used to extract the video title text. Fig.13 The Bert network shown in the figure processes the video content text to obtain the video feature 1323 corresponding to the video content text of the first video sample; after splicing the video feature 1321, the video feature 1322 and the video feature 1323, a sample splicing feature 1330 is obtained; the sample splicing feature 1330 is processed by the Mulit-Head network (special fusion sub-network) to obtain the sample feature of the first video sample, and the first model is jointly trained with supervised tasks and self-supervised tasks based on the sample feature to obtain the trained first model.

[0246] It can be understood that in order to improve the security and reliability of data, the video samples for model training and the target videos for video processing involved in the embodiments of the present application can be stored in the blockchain.

[0247] Fig.14 A block diagram of a video processing device shown in an exemplary embodiment of the present application is shown in FIG. Fig.14 As shown, the device comprises:

[0248] A first acquisition module 1410 is used to acquire a first video sample and first annotation information of the first video sample; the first annotation information is obtained by annotating the first video sample based on a first task;

[0249] A first sample feature acquisition module 1420, configured to process the first video sample based on a feature extraction network in a first model to obtain a first sample feature of the first video sample;

[0250] A first result acquisition module 1430, configured to process the first sample feature based on a first task processing network in the first model to obtain a first prediction result of the first task;

[0251] A second result acquisition module 1440 is used to process the first sample feature based on the second task processing network in the first model to obtain a second prediction result of the second task;

[0252] A first parameter updating module 1450 is used to update the parameters of the first model based on the first video sample, the first annotation information, the first prediction result, and the second prediction result to obtain a first video processing model; the first annotation information and the first prediction result are used to update the parameters of the first model in a supervised manner; the first video sample and the second prediction result are used to update the parameters of the first model in a self-supervised manner;

[0253] The first video processing model is used to perform at least one of the first task and the second task on the input video.

[0254] In a possible implementation, the feature extraction network includes a feature extraction subnetwork and a feature fusion subnetwork;

[0255] The first sample feature acquisition module 1420 includes:

[0256] A sample feature extraction submodule, configured to perform feature extraction on the first video sample based on the feature extraction subnetwork to obtain sample features of at least two dimensions of the first video sample;

[0257] The sample feature fusion submodule is used to fuse the sample features of at least two dimensions based on the feature fusion subnetwork to obtain the first sample feature of the first video sample.

[0258] In a possible implementation manner, the device further includes:

[0259] a preprocessing module, configured to preprocess the first video sample to obtain video elements of at least two dimensions of the first video sample before the first sample feature acquisition module 1420 processes the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample;

[0260] The sample feature extraction submodule is used to extract features of the video elements in at least two dimensions based on the feature extraction subnetwork to obtain sample features of the first video sample in at least two dimensions.

[0261] In a possible implementation, the video elements of at least two dimensions include: video frames and video text; the video text includes at least one of a video title text and a video content text; and the video content text refers to the text corresponding to the voice information in the input video.

[0262] In a possible implementation, the second task processing network includes at least one of a first processing network and a second processing network; the second prediction result includes at least one of a prediction result corresponding to the first processing network and a prediction result corresponding to the second processing network; the prediction result of the first processing network corresponds to a prediction result of a video frame dimension; and the prediction result of the second processing network corresponds to a prediction result of a video text dimension.

[0263] In a possible implementation manner, in response to the second task processing network including the first processing network, the apparatus further includes:

[0264] a first mask module, configured to perform a first mask process on the video frame of the first video sample before the first sample feature acquisition module 1420 processes the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample;

[0265] The second result acquisition module 1440 is used to process the first sample feature based on the second task processing network to obtain the second prediction result including a predicted video frame; the predicted video frame includes a video frame obtained by restoring the video frame in which the first video sample is missing after the first mask processing.

[0266] In a possible implementation manner, in response to the second task processing network including the second processing network, the apparatus further includes:

[0267] A first masking module, configured to perform a second masking process on the video text of the first video sample before the first sample feature acquisition module 1420 processes the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample;

[0268] The second result acquisition module 1440 is used to process the first sample features based on the second task processing network to obtain the second prediction result including predicted text; the predicted text includes the text obtained by restoring the video text that is missing from the first video sample after the second mask processing.

[0269] In a possible implementation, the first result acquisition module 1430 is used to process the sample features of at least two dimensions and the first sample features respectively based on the first task processing network to obtain the first prediction result; the first prediction result includes the prediction results corresponding to the video features of at least two dimensions respectively, and the prediction results corresponding to the first sample features.

[0270] In a possible implementation, the feature fusion submodule is used to:

[0271] Splicing the sample features in at least two dimensions to obtain sample splicing features;

[0272] Based on the feature fusion subnetwork, the sample splicing feature is processed to obtain the first sample feature of the first video sample.

[0273] In a possible implementation, the first parameter updating module 1450 includes:

[0274] A first calculation submodule, used to calculate a function value of a first loss function based on the first prediction result and the first annotation information;

[0275] A second calculation submodule, used for calculating a function value of a second loss function based on the second prediction result and the first video sample;

[0276] The model updates the parameters of the first model based on the function value of the first loss function and the function value of the second loss function to obtain the first video processing model.

[0277] In a possible implementation manner, the device further includes:

[0278] A model building module, used to build a second model; the second model includes the feature extraction network after training, and a third task processing network;

[0279] A second acquisition module is used to acquire a second video sample and second annotation information of the second video sample; the second annotation information is obtained by annotating the second video sample based on the third task;

[0280] The second sample feature acquisition module processes the second video sample based on the feature extraction network to obtain a second sample feature of the second video sample;

[0281] a third result acquisition module, configured to process the second sample feature based on the third task processing network to obtain a third prediction result of the third task;

[0282] A second parameter updating module, configured to update the parameters of the second model based on the second annotation information and the third prediction result to obtain a second video processing model;

[0283] The second video processing model is used to perform the third task on the input video.

[0284] In summary, the video processing device provided in the embodiment of the present application performs supervised training on the first model based on video samples and first annotation information of the video samples, and performs self-supervised training on the first model based on the video samples, and jointly trains the first model according to the video samples, the first annotation information of the video samples, the prediction results of the supervised task, and the prediction results of the self-supervised task, so as to obtain a first video model based on the trained first model, and the first video model can be used to perform at least one of the first task and the second task on the input video; that is, in the above scheme, the features extracted by the feature extraction network in the trained model can be simultaneously applicable to supervised tasks and self-supervised tasks, thereby improving the model accuracy and generalization of the first video model, making the video features of the input video extracted by the first video model more accurate and comprehensive, and improving the accuracy of subsequent operations based on the video features.

[0285] Fig.15 A block diagram of a video processing device provided by an exemplary embodiment of the present application is shown. Fig.15 As shown, the device comprises:

[0286] A first video acquisition module 1510, configured to acquire a first target video;

[0287] A first video feature acquisition module 1520, configured to process the first target video based on a feature extraction network in a first video processing model to obtain a first video feature of the first target video;

[0288] The first prediction result acquisition module 1530 is used to process the first video feature based on the task processing network in the first video processing model to obtain the prediction result of the first target video; the task processing network includes at least one of a first task processing network and a second task processing network; the first task processing network is used to process the first video feature to obtain a first prediction result of the first task; the second task processing network is used to process the first video feature to obtain a second prediction result of the second task; the prediction result of the first target video includes at least one of the first prediction result of the first task and the second prediction result of the second task;

[0289] Among them, the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and updating parameters based on the first video sample in a self-supervised manner; the first annotation information is obtained by annotating the first video sample based on the first task.

[0290] In summary, the video processing device provided in the embodiment of the present application obtains a first model based on joint training of a supervised task and a self-supervised task, and obtains a first video model based on the trained first model to process the input first target video to obtain a prediction result of the first target video; since the features extracted by the feature extraction network in the trained first model can be applied to both supervised tasks and self-supervised tasks, the model accuracy and generalization of the first video model are relatively high; by processing the target video with the first video model with higher precision, the accuracy of the prediction result of the obtained target video is improved.

[0291] Fig.16 A block diagram of a video processing device provided by an exemplary embodiment of the present application is shown. Fig.16 As shown, the device comprises:

[0292] The second video acquisition module 1610 is used to acquire a second target video;

[0293] A second video feature acquisition module 1620 is used to process the second target video based on a feature extraction network in a second video processing model to obtain a second video feature of the second target video; the second video processing model is constructed and trained based on the feature extraction network in the first video processing model; the first video processing model is a model obtained by updating parameters in a supervised manner based on a first video sample and first annotation information of the first video sample, and by updating parameters in a self-supervised manner based on the first video sample;

[0294] The second prediction result acquisition module 1630 is used to process the second video feature based on the third task processing network in the second video processing model to obtain a third prediction result of the third task.

[0295] To summarize, the video processing device provided in the embodiment of the present application, after obtaining the first model based on the joint training of the supervised task and the self-supervised task, constructs a second model based on the feature extraction network in the first model combined with the third task processing network, and after fine-tuning the second model, obtains the second video processing model, and processes the input second target video through the second video processing model to obtain a third prediction result of the second target video corresponding to the third task; since the accuracy of the trained feature extraction network is relatively high, the accuracy of the second video processing model constructed based on the feature extraction network is also relatively high, thereby improving the accuracy of the prediction result of the obtained second target video.

[0296] Fig.17 The block diagram of the computer device 1700 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as each server in the above-mentioned solution of the present application. The computer device 1700 includes a processor 1701 (such as a central processing unit (CPU)), a system memory 1704 including a random access memory (RAM) 1702 and a read-only memory (ROM) 1703, and a system bus 1705 connecting the system memory 1704 and the processor 1701. The computer device 1700 also includes a mass storage device 1706 for storing an operating system 1709, an application program 1710 and other program modules 1711.

[0297] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above. The above-mentioned system memory 1704 and mass storage device 1706 can be collectively referred to as memory.

[0298] According to various embodiments of the present application, the computer device 1700 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1700 can be connected to the network through a network interface unit 1707 connected to the system bus 1705, or the network interface unit 1707 can be used to connect to other types of networks or remote computer systems (not shown).

[0299] The memory further stores at least one computer program, and the processor 1701 implements all or part of the steps in the video processing method shown in the above embodiments by executing the at least one computer program.

[0300] Fig.18 FIG. 1 is a block diagram of a computer device 1800 according to an exemplary embodiment of the present application. The computer device 1800 may be implemented as the terminal described above.

[0301] Typically, the computer device 1800 includes a processor 1801 and a memory 1802 .

[0302] Processor 1801 may include one or more processing cores, such as a 4-core processor, an 18-core processor, etc. Processor 1801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, processor 1801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, processor 1801 may also include an AI processor, which is used to process computing operations related to machine learning.

[0303] The memory 1802 may include one or more computer-readable storage media, which may be non-transitory. The memory 1802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1802 is used to store at least one instruction, which is used to be executed by the processor 1801 to implement all or part of the steps in the video processing method shown in the method embodiment of the present application.

[0304] In some embodiments, the computer device 1800 may also optionally include: a peripheral device interface 1803 and at least one peripheral device. The processor 1801, the memory 1802 and the peripheral device interface 1803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1804, a display screen 1805, a camera assembly 1806, an audio circuit 1807 and a power supply 1808.

[0305] In some embodiments, the computer device 1800 further includes one or more sensors 1809 , including but not limited to: an acceleration sensor 1810 , a gyroscope sensor 1811 , a pressure sensor 1812 , an optical sensor 1813 , and a proximity sensor 1814 .

[0306] Those skilled in the art will understand that Fig.18The structure shown in the figure does not constitute a limitation on the computer device 1800, and the computer device 1800 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0307] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement all or part of the steps in the above-mentioned video processing method. For example, the computer-readable storage medium can be a read-only memory, a random access memory, a read-only optical disk, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0308] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor Figure 2 , Figure 3 , Figure 5 , Figure 6 or Figure 7 All or part of the steps of the video processing method shown in any embodiment shown.

[0309] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0310] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a first video sample and first annotation information of the first video sample; the first annotation information is obtained by annotating the first video sample based on a first task; Based on the feature extraction subnetwork included in the feature extraction network in the first model, feature extraction is performed on the first video sample to obtain sample features of at least two dimensions of the first video sample; based on the feature fusion subnetwork included in the feature extraction network, the sample features of the at least two dimensions are fused to obtain a first sample feature of the first video sample; Processing the first sample feature based on a first task processing network in the first model to obtain a first prediction result of the first task; Processing the first sample feature based on the second task processing network in the first model to obtain a second prediction result for the second task; Calculating a function value of a first loss function based on the first labeling information and the first prediction result, and calculating a function value of a second loss function based on the second prediction result and the first video sample; Based on the function value of the first loss function and the function value of the second loss function, the parameters of the feature extraction network are updated, the first task processing network is updated based on the function value of the first loss function, and the second task processing network is updated based on the function value of the second loss function to obtain a first video processing model; the first annotation information and the first prediction result are used to update the parameters of the first model in a supervised manner; the first video sample and the second prediction result are used to update the parameters of the first model in a self-supervised manner; The first video processing model is used to perform at least one of the first task and the second task on the input video.

2. The method according to claim 1, characterized in that Before processing the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample, the method further includes: Preprocessing the first video sample to obtain video elements of at least two dimensions of the first video sample; The feature extraction subnetwork included in the feature extraction network based on the first model performs feature extraction on the first video sample to obtain sample features of at least two dimensions of the first video sample, including: Based on the feature extraction subnetwork, feature extraction is performed on the video elements in at least two dimensions to obtain sample features in at least two dimensions of the first video sample.

3. The method according to claim 2, characterized in that The video elements of at least two dimensions include: video frames and video texts; the video texts include at least one of video title texts and video content texts; the video content texts refer to texts corresponding to voice information in the input video.

4. The method according to claim 2, characterized in that: The second task processing network includes at least one of a first processing network and a second processing network; the second prediction result includes at least one of a prediction result corresponding to the first processing network and a prediction result corresponding to the second processing network; the prediction result of the first processing network corresponds to the prediction result of the video frame dimension; the prediction result of the second processing network corresponds to the prediction result of the video text dimension.

5. The method according to claim 4, characterized in that In response to the second task processing network including the first processing network, before processing the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample, the method further includes: performing a first mask process on the video frame of the first video sample; The processing of the first sample feature by the second task processing network based on the first model to obtain a second prediction result of the second task includes: The first sample feature is processed based on the second task processing network to obtain the second prediction result including a predicted video frame; the predicted video frame includes a video frame obtained by restoring the video frame in which the first video sample is missing after being processed by the first mask.

6. The method according to claim 4, characterized in that In response to the second task processing network including the second processing network, before processing the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample, the method further includes: performing a second masking process on the video text of the first video sample; The processing of the first sample feature by the second task processing network based on the first model to obtain a second prediction result of the second task includes: The first sample features are processed based on the second task processing network to obtain the second prediction result including predicted text; the predicted text includes text obtained by restoring the video text that is missing from the first video sample after being processed by the second mask.

7. The method according to claim 1, characterized in that The processing of the first sample feature by the first task processing network based on the first model to obtain a first prediction result of the first task includes: Based on the first task processing network, the sample features of at least two dimensions and the first sample features are processed respectively to obtain the first prediction result; the first prediction result includes the prediction results corresponding to the sample features of at least two dimensions respectively, and the prediction result corresponding to the first sample feature.

8. The method according to claim 1, characterized in that The step of fusing the sample features of the at least two dimensions based on the feature fusion subnetwork included in the feature extraction network to obtain a first sample feature of the first video sample includes: Splicing the sample features of the at least two dimensions to obtain sample splicing features; Based on the feature fusion subnetwork, the sample splicing feature is processed to obtain the first sample feature of the first video sample.

9. The method according to claim 1, characterized in that: The method further comprises: Constructing a second model; wherein the second model includes the feature extraction network after training, and a third task processing network; Acquire a second video sample and second annotation information of the second video sample; the second annotation information is obtained by annotating the second video sample based on the third task; Processing the second video sample based on the feature extraction network to obtain a second sample feature of the second video sample; Processing the second sample feature based on the third task processing network to obtain a third prediction result of the third task; Based on the second annotation information and the third prediction result, updating parameters of the second model to obtain a second video processing model; The second video processing model is used to perform the third task on the input video.

10. A video processing method, characterized in that: The method comprises: Get the first target video; Processing the first target video based on a feature extraction network in a first video processing model to obtain a first video feature of the first target video; The first video feature is processed based on the task processing network in the first video processing model to obtain a prediction result of the first target video; the task processing network includes at least one of a first task processing network and a second task processing network; the first task processing network is used to process the first video feature to obtain a first prediction result of the first task; the second task processing network is used to process the first video feature to obtain a second prediction result of the second task; the prediction result of the first target video includes at least one of the first prediction result of the first task and the second prediction result of the second task; The first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and by updating parameters in a self-supervised manner based on the first video sample; the first annotation information is obtained by annotating the first video sample based on the first task; The process of training and obtaining the first video processing model includes: Based on the feature extraction subnetwork included in the feature extraction network, feature extraction is performed on the first video sample to obtain sample features of at least two dimensions of the first video sample; based on the feature fusion subnetwork included in the feature extraction network, the sample features of the at least two dimensions are fused to obtain a first sample feature of the first video sample; Calculating a function value of a first loss function based on the first labeling information and the first prediction result, and calculating a function value of a second loss function based on the second prediction result and the first video sample; Based on the function value of the first loss function and the function value of the second loss function, the parameters of the feature extraction network are updated, the first task processing network is updated based on the function value of the first loss function, and the second task processing network is updated based on the function value of the second loss function to obtain the first video processing model.

11. A video processing method, characterized in that: The method further comprises: Acquire the second target video; The second target video is processed based on the feature extraction network in the second video processing model to obtain a second video feature of the second target video; the second video processing model is constructed and trained based on the feature extraction network in the first video processing model; the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and by updating parameters in a self-supervised manner based on the first video sample; Processing the second video feature based on the third task processing network in the second video processing model to obtain a third prediction result of the third task; The process of training and obtaining the first video processing model includes: Based on the feature extraction subnetwork included in the feature extraction network, feature extraction is performed on the first video sample to obtain sample features of at least two dimensions of the first video sample; based on the feature fusion subnetwork included in the feature extraction network, the sample features of the at least two dimensions are fused to obtain a first sample feature of the first video sample; Processing the first sample feature based on a first task processing network in the first model to obtain a first prediction result of the first task; Processing the first sample feature based on the second task processing network in the first model to obtain a second prediction result for the second task; Calculating a function value of a first loss function based on the first labeling information and the first prediction result, and calculating a function value of a second loss function based on the second prediction result and the first video sample; Based on the function value of the first loss function and the function value of the second loss function, the parameters of the feature extraction network are updated, the first task processing network is updated based on the function value of the first loss function, and the second task processing network is updated based on the function value of the second loss function to obtain the first video processing model.

12. A video processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a first video sample and first annotation information of the first video sample; the first annotation information is obtained by annotating the first video sample based on a first task; The first sample feature acquisition module includes a sample feature extraction submodule and a sample feature fusion submodule; The sample feature extraction submodule is used to extract features of the first video sample based on the feature extraction subnetwork included in the feature extraction network in the first model, so as to obtain sample features of at least two dimensions of the first video sample; The sample feature fusion submodule is used to fuse the sample features of the at least two dimensions based on the feature fusion subnetwork included in the feature extraction network to obtain a first sample feature of the first video sample; A first result acquisition module, used to process the first sample feature based on the first task processing network in the first model to obtain a first prediction result of the first task; A second result acquisition module, used to process the first sample feature based on the second task processing network in the first model to obtain a second prediction result of the second task; The first parameter updating module includes: A first calculation submodule, configured to calculate a function value of a first loss function based on the first annotation information and the first prediction result; A second calculation submodule, used for calculating a function value of a second loss function based on the second prediction result and the first video sample; A module for performing the following steps: updating the parameters of the feature extraction network based on the function value of the first loss function and the function value of the second loss function, updating the first task processing network based on the function value of the first loss function, and updating the second task processing network based on the function value of the second loss function to obtain a first video processing model; the first annotation information and the first prediction result are used to update the parameters of the first model in a supervised manner; the first video sample and the second prediction result are used to update the parameters of the first model in a self-supervised manner; The first video processing model is used to perform at least one of the first task and the second task on the input video.

13. The device according to claim 12, characterized in that The device also includes: a preprocessing module, configured to preprocess the first video sample to obtain video elements of at least two dimensions of the first video sample before processing the first video sample based on the feature extraction network in the first model to obtain the first sample features of the first video sample; The sample feature extraction submodule is used to extract features of the video elements in at least two dimensions based on the feature extraction subnetwork to obtain sample features of the first video sample in at least two dimensions.

14. The device according to claim 13, characterized in that The video elements of at least two dimensions include: video frames and video texts; the video texts include at least one of video title texts and video content texts; the video content texts refer to texts corresponding to voice information in the input video.

15. The device according to claim 13, characterized in that The second task processing network includes at least one of a first processing network and a second processing network; the second prediction result includes at least one of a prediction result corresponding to the first processing network and a prediction result corresponding to the second processing network; the prediction result of the first processing network corresponds to the prediction result of the video frame dimension; the prediction result of the second processing network corresponds to the prediction result of the video text dimension.

16. The device according to claim 15, characterized in that The device also includes: a first mask module, configured to perform a first mask process on the video frame of the first video sample before the first sample feature acquisition module processes the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample; The second result acquisition module is used to process the first sample feature based on the second task processing network to obtain the second prediction result including a predicted video frame; the predicted video frame includes a video frame obtained by restoring the video frame in which the first video sample is missing after being processed by the first mask.

17. The device according to claim 15, characterized in that In response to the second task processing network including the second processing network, the apparatus further includes: a first mask module, configured to perform a second mask process on the video text of the first video sample before the first sample feature acquisition module processes the first video sample based on the feature extraction network in the first model to obtain the first sample feature of the first video sample; The second result acquisition module is used to process the first sample features based on the second task processing network to obtain the second prediction result including predicted text; the predicted text includes the text obtained by restoring the video text that is missing from the first video sample after the second mask processing.

18. The device according to claim 12, characterized in that The first result acquisition module is used to process the sample features of at least two dimensions and the first sample features respectively based on the first task processing network to obtain the first prediction result; the first prediction result includes the prediction results corresponding to the sample features of at least two dimensions respectively, and the prediction result corresponding to the first sample feature.

19. The device according to claim 12, characterized in that The feature fusion submodule is used to: Splicing the sample features of the at least two dimensions to obtain sample splicing features; Based on the feature fusion subnetwork, the sample splicing feature is processed to obtain the first sample feature of the first video sample.

20. The device according to claim 12, characterized in that The device also includes: A model building module, used to build a second model; the second model includes the feature extraction network after training, and a third task processing network; A second acquisition module is used to acquire a second video sample and second annotation information of the second video sample; the second annotation information is obtained by annotating the second video sample based on the third task; A second sample feature acquisition module, used to process the second video sample based on the feature extraction network to obtain a second sample feature of the second video sample; a third result acquisition module, configured to process the second sample feature based on the third task processing network to obtain a third prediction result of the third task; A second parameter updating module, configured to update the parameters of the second model based on the second annotation information and the third prediction result to obtain a second video processing model; The second video processing model is used to perform the third task on the input video.

21. A video processing device, characterized in that: The device comprises: A first video acquisition module, used to acquire a first target video; A first video feature acquisition module, used to process the first target video based on a feature extraction network in a first video processing model to obtain a first video feature of the first target video; A first prediction result acquisition module is used to process the first video feature based on the task processing network in the first video processing model to obtain a prediction result of the first target video; the task processing network includes at least one of a first task processing network and a second task processing network; the first task processing network is used to process the first video feature to obtain a first prediction result of the first task; the second task processing network is used to process the first video feature to obtain a second prediction result of the second task; the prediction result of the first target video includes at least one of the first prediction result of the first task and the second prediction result of the second task; The first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and by updating parameters in a self-supervised manner based on the first video sample; the first annotation information is obtained by annotating the first video sample based on the first task; The process of training and obtaining the first video processing model includes: Based on the feature extraction subnetwork included in the feature extraction network, feature extraction is performed on the first video sample to obtain sample features of at least two dimensions of the first video sample; based on the feature fusion subnetwork included in the feature extraction network, the sample features of the at least two dimensions are fused to obtain a first sample feature of the first video sample; Calculating a function value of a first loss function based on the first labeling information and the first prediction result, and calculating a function value of a second loss function based on the second prediction result and the first video sample; Based on the function value of the first loss function and the function value of the second loss function, the parameters of the feature extraction network are updated, the first task processing network is updated based on the function value of the first loss function, and the second task processing network is updated based on the function value of the second loss function to obtain the first video processing model.

22. A video processing device, characterized in that: The device comprises: A second video acquisition module, used to acquire a second target video; A second video feature acquisition module is used to process the second target video based on the feature extraction network in the second video processing model to obtain the second video feature of the second target video; the second video processing model is constructed and trained based on the feature extraction network in the first video processing model; the first video processing model is a model obtained by updating parameters in a supervised manner based on the first video sample and the first annotation information of the first video sample, and by updating parameters in a self-supervised manner based on the first video sample; A second prediction result acquisition module, used to process the second video feature based on the third task processing network in the second video processing model to obtain a third prediction result of a third task; The process of training and obtaining the first video processing model includes: Based on the feature extraction subnetwork included in the feature extraction network, feature extraction is performed on the first video sample to obtain sample features of at least two dimensions of the first video sample; based on the feature fusion subnetwork included in the feature extraction network, the sample features of the at least two dimensions are fused to obtain a first sample feature of the first video sample; Processing the first sample feature based on a first task processing network in the first model to obtain a first prediction result of the first task; Processing the first sample feature based on the second task processing network in the first model to obtain a second prediction result for the second task; Calculating a function value of a first loss function based on the first labeling information and the first prediction result, and calculating a function value of a second loss function based on the second prediction result and the first video sample; Based on the function value of the first loss function and the function value of the second loss function, the parameters of the feature extraction network are updated, the first task processing network is updated based on the function value of the first loss function, and the second task processing network is updated based on the function value of the second loss function to obtain the first video processing model.

23. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the video processing method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the video processing method according to any one of claims 1 to 11.

25. A computer program product, characterized in that The computer program product comprises at least one computer program, and the computer program is loaded and executed by a processor to implement the video processing method according to any one of claims 1 to 11.