Video processing method, machine learning model training method and related devices and equipment

Abstract

The present application discloses a video processing method, a machine learning model training method, and related devices and equipment. The method relates to artificial intelligence and includes: extracting features from a video to be processed to obtain a video feature vector; extracting features from a text to be processed to obtain a text feature vector; matching the text to be processed with the video to be processed; concatenating the text feature vector with the video feature vector to obtain a multimodal feature vector; fusing the multimodal feature vector to obtain a fused feature vector; and classifying the video to be processed based on the fused feature vector to obtain a label for the video to be processed. This method can improve the degree of fusion of video information and text information, allowing the two to complement each other and improve the accuracy of video label recognition, thereby facilitating a better understanding of the video's subject matter.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video processing method, a machine learning model training method, and related devices and equipment. Background Art

[0002] In recent years, a form of video information flow has become popular all over the world. Compared with the traditional way of disseminating information in text, video information flow has the advantages of richer information, more convenient browsing and greater impact.

[0003] In addition to professional video editors, videos generally come from self-media platforms, such as Kuaishou and Douyin, where live streamers frequently post short videos. Compared to text-based content, short videos posted by self-media platforms contain richer content and more diverse formats. Therefore, effectively understanding the themes of videos becomes a technical challenge. For short videos, we hope to summarize video information through a series of tags to facilitate the identification of the video's theme, thereby assisting upper-level business parties in distributing the video content.

[0004] When it comes to labeling short videos, people often label them from a textual or visual perspective. However, relying solely on visual or textual information to label short videos can lead to inaccurate labeling. Summary of the Invention

[0005] The embodiments of the present application provide a video processing method, a machine learning model training method, and related devices and equipment, which can improve the degree of integration of video information and text information, so that the two complement each other, improve the accuracy of video label recognition, and facilitate a better understanding of the subject of the video.

[0006] The first aspect of an embodiment of the present application discloses a video processing method, which includes: extracting features from a video to be processed to obtain a video feature vector; extracting features from a text to be processed to obtain a text feature vector; the text to be processed corresponds to the video to be processed; splicing the text feature vector with the video feature vector to obtain a multimodal feature vector; performing feature fusion on the multimodal feature vector to obtain a fused feature vector; and classifying the video to be processed based on the fused feature vector to obtain a label of the video to be processed.

[0007] This method allows the server to better integrate video and text information, making them complement each other and improving the accuracy of video tag recognition, thereby better understanding the video's subject matter. The use of text information effectively elevates model recognition from the perceptual level to the cognitive level, resulting in tags that are most relevant to the video's subject matter.

[0008] In conjunction with the first aspect, in one possible implementation, the video to be processed includes N images, the video feature vector includes N image feature vectors, the N image feature vectors correspond one-to-one to the N images, and N is a positive integer; extracting features from the video to be processed to obtain the video feature vector includes:

[0009] Features are extracted from the N images respectively to obtain the N image feature vectors.

[0010] In conjunction with the first aspect, in a possible implementation, concatenating the text feature vector and the video feature vector to obtain a multimodal feature vector includes:

[0011] The N image feature vectors are sequentially concatenated with the text feature vectors, and one row or one column of the multimodal feature vector includes an image feature vector and the text feature vector.

[0012] In conjunction with the first aspect, in another possible implementation, the text to be processed includes multiple texts, and extracting features from the text to be processed to obtain text feature vectors further includes: extracting features from the multiple texts to obtain multiple text feature vectors, wherein the text feature vectors correspond one-to-one to the texts;

[0013] The step of concatenating the text feature vector and the video feature vector to obtain a multimodal feature vector includes:

[0014] Each of the N image feature vectors is concatenated with a text feature vector corresponding to each of the image feature vectors, where a row or a column of the multimodal feature vector includes an image feature vector of an image and a text feature vector of a text corresponding to the image.

[0015] In conjunction with the first aspect, in a possible implementation, performing feature fusion on the multimodal feature vector to obtain a fused feature vector includes:

[0016] Inputting the multimodal feature vector into a fusion module to obtain a fused feature vector;

[0017] The fusion module includes at least one convolution layer and at least one pooling layer; wherein,

[0018] The convolution layer is used to extract features of input data, and the pooling layer is used to sample the input data.

[0019] This method uses convolutional neural networks to fuse video feature vectors and text feature vectors, which can effectively improve the degree of information fusion of video information and text information, give full play to the advantages of multimodal information complementarity, and improve the accuracy of video label recognition.

[0020] In combination with the first aspect, in a possible implementation, classifying the video to be processed based on the fused feature vector to obtain a label of the video to be processed includes:

[0021] Inputting the fused feature vector into a classification module to obtain a label of the video to be processed;

[0022] The classification module includes at least one fully connected layer.

[0023] In conjunction with the first aspect, in one possible implementation, the video feature vector is extracted using a first model, and the text feature vector is extracted using a second model; the training data includes a sample video, sample text corresponding to the sample video, and a label of the sample video, wherein:

[0024] The first model, the second model, the fusion module and the classification module are obtained by joint training based on the training data; or, the first model, the fusion module and the classification module are obtained by joint training based on the training data; or, the second model, the fusion module and the classification module are obtained by joint training based on the training data; or, the fusion module and the classification module are obtained by joint training based on the training data.

[0025] In conjunction with the first aspect, in a possible implementation, extracting features from the text to be processed to obtain a text feature vector includes:

[0026] Segmenting the text to be processed to obtain multiple words of the text to be processed;

[0027] Perform vector conversion on the multiple words respectively to obtain word vectors corresponding to the multiple words respectively;

[0028] The word vectors corresponding to the multiple words are input into the first model to obtain a text feature vector; the first model is used to extract the features of the input data.

[0029] The text to be processed includes at least one of the subject information, title information, subtitle information, and classification information of the video to be processed.

[0030] A second aspect of the embodiments of the present application discloses a method for training a machine learning model, wherein the model comprises:

[0031] Input the sample video into the first model to obtain a video feature vector;

[0032] Inputting a sample text into a second model to obtain a text feature vector; the sample text corresponds to the sample video;

[0033] Concatenating the text feature vector and the video feature vector to obtain a multimodal feature vector;

[0034] Inputting the multimodal feature vector into a fusion module to obtain a fused feature vector;

[0035] Inputting the fused feature vector into a classification module to obtain a predicted label of the sample video;

[0036] The model parameters of the fusion module and the classification module are adjusted based on the error between the predicted label and the labeled label of the sample video.

[0037] In conjunction with the second aspect, in a possible implementation, adjusting the model parameters of the fusion module and the classification module based on the error between the predicted label and the annotated label of the sample video includes:

[0038] Adjusting model parameters of the first model, the second model, the fusion module, and the classification module based on an error between a predicted label and an annotated label of the sample video;

[0039] or,

[0040] Adjusting model parameters of the first model, the fusion module, and the classification module based on an error between a predicted label and an annotated label of the sample video;

[0041] or,

[0042] Adjusting model parameters of the second model, the fusion module, and the classification module based on an error between the predicted label and the labeled label of the sample video;

[0043] or,

[0044] The model parameters of the fusion module and the classification module are adjusted based on the error between the predicted label and the labeled label of the sample video.

[0045] In conjunction with the second aspect, in one possible implementation, the sample video is input into the first model to obtain a video feature vector; the sample video includes N images, the video feature vector includes N image feature vectors, and the N image feature vectors correspond one-to-one to the N images, including:

[0046] N images are input into the first model to obtain the N image feature vectors.

[0047] In combination with the second aspect, in a possible implementation, the fused feature vector includes: at least one first convolution layer and at least one first pooling layer; wherein,

[0048] The first convolution layer is used to extract features of input data, and the pooling layer is used to sample the input data.

[0049] A third aspect of an embodiment of the present application discloses a video processing device, the device comprising: a first information extraction module, configured to extract features from a video to be processed to obtain a video feature vector;

[0050] A second information extraction module is used to extract features from the text to be processed to obtain a text feature vector; the text to be processed corresponds to the video to be processed;

[0051] A multimodal feature module, configured to concatenate the text feature vector and the video feature vector to obtain a multimodal feature vector;

[0052] A fusion module, configured to perform feature fusion on the multimodal feature vectors to obtain a fused feature vector;

[0053] A classification module is used to classify the video to be processed based on the fused feature vector to obtain a label of the video to be processed.

[0054] A fourth aspect of the embodiments of the present application discloses a machine learning model training device, the device comprising:

[0055] A first acquisition module, configured to input a sample video into a first model to obtain a video feature vector;

[0056] A second acquisition module is configured to input a sample text into a second model to obtain a text feature vector; the sample text corresponds to the sample video;

[0057] a second multimodal feature module, configured to concatenate the text feature vector and the video feature vector to obtain a multimodal feature vector;

[0058] A second fusion module is used to input the multimodal feature vector into a fusion module to obtain a fused feature vector;

[0059] A second classification module, configured to input the fused feature vector into a classification module to obtain a predicted label for the sample video;

[0060] An information processing module is used to adjust the model parameters of the fusion module and the classification module based on the error between the predicted label and the labeled label of the sample video.

[0061] A fifth aspect of the embodiments of the present application discloses a computer device, the computer device comprising: a processor and a memory;

[0062] The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method of the first aspect above.

[0063] A sixth aspect of an embodiment of the present application discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the first aspect mentioned above.

[0064] A seventh aspect of the present application discloses a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a server reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the server to perform the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0066] Figure 1 1 is a schematic diagram of the architecture of a video processing system 100 provided in an embodiment of the present application;

[0067] Figure 2 This is a structural diagram of a video processing device or electronic device provided in an embodiment of the present application;

[0068] Figure 3A This is a flow chart of a machine learning model training method provided in an embodiment of the present application;

[0069] Figure 3B This is a flow chart of a method for training a machine learning model based on a sample video set and a sample data set, provided in an embodiment of the present application;

[0070] Figure 3C This is a flow chart of a method for obtaining a text feature vector provided in an embodiment of the present application;

[0071] Figure 4A Schematic diagram of a method for concatenating video feature vectors and text feature vectors provided in an embodiment of the present application;

[0072] Figure 4B is a schematic diagram of another method for splicing video feature vectors and text feature vectors provided in an embodiment of the present application;

[0073] Figure 5 It is a schematic diagram of the structure of two fusion modules provided in the embodiment of the present application;

[0074] Figure 6A and Figure 6B This is a flow chart of a video processing method provided in an embodiment of the present application;

[0075] Figure 7 is a structural diagram of a video processing device provided in an embodiment of the present application;

[0076] Figure 8 It is a structural diagram of a machine learning model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0078] First, the technical terms involved in the embodiments of this application are introduced:

[0079] (1) Artificial Intelligence

[0080] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also encompasses the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0081] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0082] Among them, the solutions provided in the embodiments of the present application mainly involve natural language processing (NLP) technology and machine learning (ML) technology of artificial intelligence.

[0083] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0084] Machine learning is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to imbue computers with intelligence. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0085] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0086] (2) Modality, multimodal fusion, and multimodal machine learning

[0087] Modality: Every source or form of information can be called a modality. For example, humans have touch, hearing, vision, and smell; information media include voice, video, and text; and various sensors, such as radar, infrared, and accelerometers, can be considered a modality. Modality can also be defined very broadly. For example, two different languages ​​can be considered two modalities, and even two datasets collected under two different circumstances can be considered two modalities.

[0088] Multimodal fusion is responsible for combining information from multiple modalities to perform target prediction (classification or regression).

[0089] Multimodal machine learning (MMML) aims to achieve the ability to process and understand multi-source modal information through machine learning methods.

[0090] (3) Neural Network

[0091] Neural network (NN): Artificial neural network (ANN), also known as neural network or neural network-like network, is a mathematical model or computational model that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) in the field of machine learning and cognitive science. It is used to estimate or approximate functions.

[0092] ① Language model training method

[0093] Language model training method (bidirectional encoder representations from transformers, BERT): A training method that uses massive amounts of text and is widely used in various natural language processing tasks, such as text classification, text matching, and machine reading comprehension.

[0094] ②Model parameters

[0095] Model parameter: A quantity that uses universal variables to establish the relationship between functions and variables. In artificial neural networks, model parameters are usually real number matrices.

[0096] ③Model training

[0097] Model training: Perform multi-classification learning on image datasets. This model can be built using deep learning frameworks such as TensorFlow and Torch, using multiple layers of neural network layers such as CNN to form a multi-classification model. The model input is a three-channel or raw channel matrix of images read using tools such as OpenCV. The model output is multi-classification probabilities, which are ultimately output as webpage categories using algorithms such as softmax. During training, the model approaches the correct trend using objective functions such as cross entropy.

[0098] ④ Convolutional Neural Network

[0099] Convolutional neural networks (CNNs) are a type of feed-forward neural network with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. CNNs possess representation learning capabilities and can perform shift-invariant classification on input information based on their hierarchical structure.

[0100] ⑤ Recurrent Neural Network

[0101] Recurrent neural networks (RNNs) are a type of neural network designed to process sequential data. Unlike other neural networks, RNNs can better process sequential information, recognizing relationships between previous and subsequent inputs. In natural language processing (NLP), understanding a sentence requires more than simply understanding the individual words that make up the sentence; we need to holistically process the entire sequence of words connected by them.

[0102] ⑥ Long Short-Term Memory Network

[0103] Long short-term memory (LSTM) neural networks are a special type of recurrent neural network that can capture long-term dependencies. They are specifically designed to avoid the problem of long-term dependencies. Remembering information for a long time has become their default behavior, without the need for special training.

[0104] ⑦ Bidirectional Attention Neural Network

[0105] The bidirectional encoder representation from transformers (BERT) model architecture is based on multi-layer bidirectional transform decoding. Because the decoder cannot obtain the information to be predicted, the main innovation of the model lies in the pre-training method, namely using Masked LM and NextSentence Prediction to capture word-level and sentence-level features respectively.

[0106] The "bidirectional" part means that when the model processes a word, it can simultaneously utilize information from both the previous word and the following word. This "bidirectional" feature comes from the fact that BERT, unlike traditional language models, does not predict the most likely current word based on all previous words. Instead, it randomly masks some words and uses all unmasked words for prediction.

[0107] ⑧Deep Residual Network

[0108] The deep residual network (ResNet) uses a skip architecture as its basic structure, overcoming the problem of reduced learning efficiency and ineffective accuracy improvement (also known as network degradation) caused by increasing network depth. By superimposing layers where y = x (called identity mappings) on top of a shallow network, the network can be made to maintain its stability as depth increases.

[0109] ⑨Encoder-Decoder

[0110] The encoder-decoder architecture is a commonly used network structure in machine translation. It consists of two parts: an encoder and a decoder. The encoder converts the input text into a series of context vectors that represent the characteristics of the input text. The decoder receives the encoder's output as its input and outputs the corresponding text sequence in another language.

[0111] In existing technologies, whether predicting topic tags based on text or entity tags based on video content, the two modules operate independently, ignoring the complementary nature of visual and textual information. Clearly, visual information complements textual information in topic tag classification, providing more perceptual information, while textual information is more cognitive.

[0112] The solution provided in the embodiment of the present application relates to artificial intelligence video processing technology, which is specifically illustrated by the following embodiment: The embodiment of the present application provides a video processing method, a machine learning model training method and related devices and equipment, the method comprising: extracting features from the video to be processed to obtain a video feature vector. Extracting features from the text to be processed to obtain a text feature vector; wherein the text to be processed corresponds to the video to be processed. Splicing the text feature vector and the video feature vector to obtain a multimodal feature vector. Fusing the multimodal feature vector to obtain a fused feature vector. Classifying the video to be processed based on the fused feature vector to obtain a label for the video to be processed.

[0113] This method can improve the degree of integration of video information and text information, making the two complement each other, and improve the accuracy of video label recognition, so as to better understand the theme of the video.

[0114] See also Figure 1 , Figure 1 This is an architectural diagram of a video processing system 100 provided in an embodiment of the present application. The system may include: a server 200, a network 300 and a terminal 400 (taking terminal 400-1 and terminal 400-2 as examples), wherein the terminal 400 (taking terminal 400-1 and terminal 400-2 as examples) is connected to the server 200 via the network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0115] Terminal 400-1 belongs to the publisher of the online video and is used to upload the video to be processed containing video text information to the background server of the video processing system 100, that is, server 200, so as to publish the video to be processed to other terminals on the network 300, such as terminal 400-2, through server 200.

[0116] Server 200 is the backend server of video processing system 100 and belongs to the administrator of video processing system 100. It is used to receive the video to be processed uploaded by terminal 400-1 and store it in database 500. It then performs image feature extraction on the video to be processed to obtain a video feature vector corresponding to the video to be processed; performs text feature extraction on the text to be processed in the video to obtain a text feature vector corresponding to the text to be processed; concatenates the video feature vector with the text feature vector to obtain a multimodal fusion feature vector corresponding to the video to be processed; performs feature fusion on the multimodal fusion feature vector to obtain a fused feature vector; and classifies the video to be processed based on the fused feature vector to obtain a label for the video to be processed. Furthermore, server 200 can use the label of the video to be processed to display and recommend the video to be processed on the online video platform.

[0117] Terminal 400 - 2 can access the network video platform 100 through the client application 410 of the network video platform 100 , display the video on the client application 410 , and perform operations such as clicking and collecting on the processed video through the client application 410 , and accordingly play or collect the video on the client application 410 .

[0118] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0119] In an embodiment of the present application, the server for machine learning model training and the server for video processing may be the same or different, and this application does not impose any restrictions on this.

[0120] Terminal 400 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. It can also be various types of terminals such as laptops, tablet computers, desktop computers, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, etc., and can also be implemented as a server.

[0121] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of a video processing device or electronic device provided in an embodiment of the present application. The video processing device can be the above-mentioned Figure 1 The server 200 or terminal 400 in the embodiment includes: at least one processor 210, a memory 220, at least one network interface 230, and a user interface 240. The various components in the server 200 are coupled together via a bus system 250. It is understood that the bus system 250 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 250 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 250 is not described in detail. Figure 2 Various buses are labeled as bus system 250 .

[0122] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the video processing device or electronic device. In other embodiments of the present application, the video processing device or electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0123] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0124] The memory 220 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 220 may optionally include one or more storage devices that are physically remote from the processor 210.

[0125] The memory 220 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 220 described in the embodiments of the present application is intended to include any suitable type of memory.

[0126] In some embodiments, the memory 220 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0127] Operating system 221, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0128] A network communication module 222 for reaching other computing devices via one or more (wired or wireless) network interfaces 230 , exemplary network interfaces 230 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0129] a presentation module 223 for enabling presentation of information via one or more output devices 241 (e.g., a display screen, a speaker, etc.) associated with a user interface 240 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0130] The input processing module 224 is configured to detect input or interaction via one or more input devices 242 and to translate the detected input or interaction.

[0131] Figure 2 The models and modules stored in the memory 220 are shown, which can be software in the form of programs and plug-ins, including the following software modules: a first model 2251, a second model 2252, a multimodal feature module 2253, a fusion module 2254 and a classification module 2255. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0132] The first model 2251 is used to extract features from the video to be processed to obtain a video feature vector.

[0133] The second model 2252 is used to extract features from the text to be processed to obtain a text feature vector; the text to be processed corresponds to the video to be processed.

[0134] The multimodal feature module 2253 is configured to concatenate the text feature vector and the video feature vector to obtain a multimodal feature vector.

[0135] The fusion module 2254 is used to perform feature fusion on the multimodal vector matrix to obtain a fused feature vector.

[0136] The classification module 2255 is configured to classify the video to be processed based on the fused feature vector to obtain a label of the video to be processed.

[0137] The user interface 240 includes one or more output devices 241 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 240 also includes one or more input devices 242, including user interface components that facilitate object input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0138] In other embodiments, the modules or models provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0139] The video processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0140] Example 1

[0141] The following describes a method for training a machine learning model according to an embodiment of the present application.

[0142] Example 1 introduces a training method for a machine learning model involved in the embodiment of this application. The machine learning method can be Figure 1 and Figure 2 This is achieved by the server 200 in .

[0143] Specifically, if Figure 3A As shown in the example, it is a flowchart of the method for training a machine learning model, which may include but is not limited to some or all of the following steps:

[0144] S301: Based on a sample video set, obtain a sample data set.

[0145] A sample video set may include one or more sample videos.

[0146] Sample videos can be found from Figure 1 The publisher terminal 400-1 of the online video can be a professional video editor or a self-media, such as the major anchors who often publish videos on daily short videos such as Kuaishou and Douyin.

[0147] The sample data set, also known as the training data set, includes one or more sample data. Each sample data may include, but is not limited to, one or a combination of the following: a label of a sample video, a sample text corresponding to the sample video, a sample video, audio information corresponding to the sample video, etc. In some embodiments, the sample data (sample) includes a sample text (text) corresponding to the sample video, a sample video (image), and a label (label) of the sample video. The format of each sample data in the sample data set is as follows: Sample

[0148] ={"text":I heard eating meat can make it trending, so I'll try it,"image":url,"label":[pig's trotter, beauty, food]}

[0149] Here, "url" refers to the address for obtaining the sample video. Based on this address, the sample video can be obtained.

[0150] S302: Training a machine learning model based on the sample video set and the sample data set.

[0151] Specifically, if Figure 3B As shown in the example, S302 may include some or all of steps S3021-S3026:

[0152] S3021: Input the sample video in the sample data into the first model to obtain a video feature vector.

[0153] In some embodiments, the entire sample video may be input into a first model for extracting video modality features to perform feature extraction to obtain a video feature vector, which may also be referred to as a video modality feature.

[0154] In other embodiments, N images may be selected from a sample video and input into a first model for feature extraction to obtain N image feature vectors, each of which corresponds one-to-one to the N images. In this embodiment, the sample video includes N images, and the video feature vector includes N image feature vectors, where N is a positive integer.

[0155] The embodiment of the present application does not limit the method of selecting N images from the sample video. The following describes one of the methods as an example.

[0156] The sample video can be divided into one or more video segments, and at least one image is selected from each of the one or more video segments according to a certain time interval; at least one image constitutes an image set; one image set corresponds to one video segment; one or more image sets include N images.

[0157] In some embodiments, the first model can be a deep residual network (ResNet). The server can obtain video feature vectors based on ResNet, and ResNet can effectively improve the accuracy of model training as the network depth increases. Specifically, in the embodiment of the present application, a sample video is used as an example of N selected images. The present application can also obtain video feature vectors through other network models, such as recurrent neural networks and long short-term memory networks, without limitation.

[0158] The ResNet can include at least one convolutional layer and at least one pooling layer. The convolutional layer is used to extract features of input data; the pooling layer is used to sample the input data. Both the convolutional layer and the pooling layer include activation functions.

[0159] In an embodiment of the present application, the convolution layer can be used to extract the initial features of N input images. Specifically, in the first step, the N images are vector-converted to obtain N image vectors, and the N image vectors can be combined into an image vector matrix; in the second step, the image vector matrix is ​​input into the convolution layer, and the convolution kernel is used to perform a convolution operation with the image vector matrix, that is, the image vector matrix and the convolution kernel are performed as an inner product operation to obtain the convolution result corresponding to the image vector matrix; next, the convolution result is nonlinearly transformed based on the activation function, and the bias vector is added to obtain the initial feature vector; in the third step, the initial feature vector is input into the pooling layer, and the initial feature vector can be sampled; then, the convolution result is nonlinearly transformed based on the activation function, and the bias vector is added to obtain the video feature vector. Video (image) Video feature vector V extracted by Resnet152 img It can be expressed as:

[0160] V img =Resnet152(image) S3022: Input the sample text corresponding to the sample video into the second model to obtain a text feature vector.

[0161] In an embodiment of the present application, the sample text includes at least one of the subject information, title information, subtitle information, classification information, etc. of the sample video.

[0162] Here's how to get the text feature vector:

[0163] Specifically, if Figure 3C As shown in the example, S3022 may include some or all of steps S30221-S30223:

[0164] S30221: Segment the sample text to obtain multiple words of the sample text.

[0165] S30222: Perform vector conversion on each of the multiple words to obtain word vectors corresponding to the multiple words.

[0166] S30223: Inputting the word vectors corresponding to the plurality of words into a second model for extracting text modal features to obtain a text feature vector; the text feature vector may also be referred to as a text modal feature.

[0167] The second model is used to extract the features of the input text, which are also called text modal features. In the embodiment of the present application, the second model is used to extract the features of the sample text. The method of obtaining the text feature vector by the second model can refer to the method of obtaining the video feature vector by the first model in S3021, which will not be repeated here. In the embodiment of the present application, the second model can be a bidirectional attention neural network model (bidirectional encoder representation from transformers, BERT), or other network models, which is not limited by the present application. The text feature vector V extracted by BERT is text It can be expressed as:

[0168] V text =BERT(text)S3023: Concatenate the text feature vector and the video feature vector to obtain a multimodal feature vector.

[0169] like Figure 4A As shown, in some embodiments, the sample text includes a text, which corresponds to a text feature vector, and the video feature vector includes N image feature vectors. The N image feature vectors can be concatenated with the text feature vector in sequence, and the resulting multimodal feature vector includes an image feature vector and a text feature vector in a row or column. The multimodal feature vectors of the N images are respectively expressed as:

[0170]

[0171] Among them, the N image feature vectors are represented as The text feature vector is represented as V text .

[0172] For example, the video feature vector is an N1×1024 feature vector matrix, and the text feature vector is a one-dimensional vector containing 768 elements. The text feature vector can be directly spliced ​​behind or in front of each image feature vector in sequence to form an N1×(1024+768) feature vector matrix, which is a multimodal feature vector.

[0173] like Figure 4BAs shown, in other embodiments, the sample text includes multiple texts, and features can be extracted from each of the multiple texts to obtain multiple text feature vectors, with the text feature vectors corresponding to the texts one-to-one. Each image feature vector in the N image feature vectors can be concatenated with the text feature vector corresponding to each image feature vector, resulting in a row or column of multimodal feature vectors including an image feature vector for an image and a text feature vector for the text corresponding to the image.

[0174] Here we take M text feature vectors as an example, and there is a corresponding relationship between M text feature vectors and N image feature vectors. For example, the image feature vector includes N image feature vectors, and the text feature vector includes M text feature vectors. N and M are both positive integers, and the two can be equal or unequal. When M=N, the text feature vector corresponds to the image feature vector one-to-one. In one possible implementation, the first image feature vector corresponds to the first text feature vector; the second image feature vector corresponds to the second text feature vector; and the third image feature vector corresponds to the third text feature vector. In this case, the server can splice the first text feature vector behind or in front of the first image feature vector, and the second text feature vector behind or in front of the second image feature vector, and so on, and splice the N image feature vectors with the corresponding text feature vectors to form a multimodal feature vector. The multimodal feature vectors of N images and M texts are respectively expressed as:

[0175]

[0176] …

[0177]

[0178] Among them, the N image feature vectors are represented as The M text feature vectors are expressed as

[0179] It should be understood that the image feature vector and the text feature vector may be spliced ​​in other ways, and this application does not limit this.

[0180] S3024: Input the multimodal feature vector into the fusion module to obtain a fused feature vector.

[0181] In some embodiments, the server may input the multimodal feature vector of the sample video into a fusion module to obtain a fused feature vector. The fusion module may be a convolutional neural network, such as Figure 5 As shown, Figure 5Figure 1 is a schematic diagram of the fusion module. The convolutional network can include at least one convolutional layer and at least one pooling layer. The convolutional layer is used to extract features from the multimodal feature vector to obtain an initial feature vector, and the pooling layer is used to sample the initial feature vector to obtain a more accurate fused feature vector. Both the convolutional layer and the pooling layer include activation functions.

[0182] Specifically, the server inputs the multimodal feature vector into the convolution layer, and uses the convolution kernel to perform a convolution operation with the multimodal feature vector, that is, the multimodal feature vector and the convolution kernel are inner-producted to obtain the convolution result corresponding to the multimodal feature vector; next, the convolution result is nonlinearly transformed based on the activation function, and the bias vector is added to obtain the initial feature vector; in the third step, the initial feature vector is input into the pooling layer, and the initial feature vector can be sampled; then, the convolution result is nonlinearly transformed based on the activation function, and the bias vector is added to obtain the fused feature vector.

[0183] In other embodiments, the server may also obtain the fused feature vector in other ways, and the fusion module may also be other network models such as a recurrent neural network, a deep residual network, etc. This application does not impose any restrictions on this.

[0184] S3025: Input the fused feature vector into the classification module to obtain the predicted label of the sample video.

[0185] In some embodiments, the classification module may include at least one fully connected layer, which can classify the sample to obtain a predicted label for the sample video. In addition, the fully connected layer may include an activation function, which includes a weight matrix and a bias constant.

[0186] Specifically, the server can input the fused feature vector into the fully connected layer, perform a nonlinear transformation on the fused feature vector based on the weight matrix and bias vector of the activation function, and then normalize it to obtain the probability that the sample video belongs to each label, that is, the predicted label.

[0187] S3026: Adjust the model parameters of the initial machine learning model based on the error between the predicted label of the sample video and the label of the sample video.

[0188] In implementation 1, the initial machine learning model includes a first model, a second model, a fusion module, and a classification module. Model parameters of the first model, the second model, the fusion module, and the classification module are adjusted based on the error between the predicted label of the sample video and the annotated label of the sample video.

[0189] Exemplarily, a loss function is constructed to represent the error between the predicted label of the sample video and the annotated label of the sample video, the loss is iteratively calculated by the gradient descent method, and when the loss meets the model convergence condition, the final machine learning model is output.

[0190] In implementation 2, the initial machine learning model includes a first model, a fusion module, and a classification module. During training, the model parameters of the first model, the fusion module, and the classification module can be adjusted based on the error between the predicted label of the sample video and the annotated label of the sample video.

[0191] In this embodiment, the second model can be a pre-trained bidirectional attention neural network BERT, or other network models, such as a long short-term memory neural network, a deep residual network, etc.

[0192] For example, the second model is used to obtain a text feature vector. The sample data set used in the process of training the second model may include one or more sample data, and one sample data may include a sample text and an annotated label of the sample text. The sample text is input into the text classification model to obtain a predicted label for the sample text. The text classification model includes a second model and a first classification unit. The second model is used to extract the features of the input sample text to obtain its text feature vector; the first classification unit is used to predict the label of the sample text based on the text feature vector of the sample text. The training method is to adjust the model parameters of the text classification model based on the error between the predicted label and the annotated label of the sample text. Furthermore, the second model in the obtained text classification model can be trained to extract the text feature vector of the text.

[0193] In implementation 3, the initial machine learning model includes a second model, a fusion module, and a classification module. During training, the model parameters of the second model, the fusion module, and the classification module can be adjusted based on the error between the predicted label of the sample video and the label of the sample video.

[0194] In this embodiment, the first model can be a pre-trained deep residual network ResNet, or other network models, such as a long short-term memory neural network.

[0195] Exemplarily, the first model is used to obtain a video feature vector, and the sample data set used in the process of training the first model may include one or more sample data, and a sample data may include a sample video and an annotated label of the sample video. The sample video is input into the video classification model to obtain a predicted label of the sample video. The text classification model includes a first model and a second classification unit, wherein the first model is used to extract the features of the input sample video to obtain its video feature vector; the second classification unit is used to predict the label of the sample video based on the video feature vector of the sample video. The training method is to adjust the model parameters of the video classification model based on the error between the predicted label and the annotated label of the sample video. Furthermore, the first model in the obtained video classification model can be trained to extract the video feature vector of the video.

[0196] In Implementation 4, the initial machine learning model may also include only a fusion module and a classification module. During training, the model parameters of the fusion module and the classification module can be adjusted based on the error between the predicted label and the label of the sample video. In this case, the first model and the second model are both pre-trained, as described in Implementation 2 and Implementation 3 above, and will not be further described here.

[0197] The trained machine learning model can be applied to video classification to predict video labels. For details, please refer to the relevant description in the following embodiment 2.

[0198] Example 2

[0199] The following describes a video processing method involved in an embodiment of the present application.

[0200] In some embodiments, a video processing method provided by the embodiment of the present application can be Figure 1 and Figure 2 It is implemented by the server 200 or terminal 400 in.

[0201] In the embodiment of the present application, the execution entity is taken as server 200 as an example.

[0202] Specifically, if Figure 6A and Figure 6B As shown in the example, it is a flowchart of a video processing method, which may include but is not limited to some or all of the following steps:

[0203] S601: The server extracts features from the video to be processed to obtain a video feature vector.

[0204] The video to be processed can come from Figure 1 The publisher terminal 400-1 of the online video can be a professional video editor or a major anchor who publishes videos on a self-media platform.

[0205] In some embodiments, the entire video to be processed can be input into the first model, that is, the video modality feature extraction module, to extract features and obtain a video feature vector. The training method of the first model can refer to the relevant description in the above embodiment 1 and will not be repeated here.

[0206] In other embodiments, N images may be selected from the video to be processed and input into the first model, i.e., the video modality feature extraction module, for feature extraction to obtain N image feature vectors, each of which corresponds one-to-one to the N images. In this embodiment, the sample video includes N images, and the video feature vectors include N image feature vectors, where N is a positive integer. The training method of the first model can be found in the relevant description of the first embodiment above and will not be repeated here.

[0207] The embodiment of the present application does not limit the method of selecting N images from the video to be processed. The following is an introduction using one of the methods as an example.

[0208] The video to be processed can be divided into one or more video segments, and at least one image is selected from each of the one or more video segments according to a certain time interval; at least one image constitutes an image set; one image set corresponds to one video segment; one or more image sets include N images.

[0209] In some embodiments, the first model can be a deep residual network (ResNet). The server can obtain video feature vectors based on ResNet, which can effectively improve the accuracy of model training as the network depth increases. Specifically, in the embodiment of the present application, the video to be processed is taken as N selected images as an example. The present application can also obtain video feature vectors through other network models, without limitation.

[0210] The server can input the N images into a ResNet network. The ResNet network can include at least one convolutional layer and at least one pooling layer. The convolutional layer is used to extract features of the input data; the pooling layer is used to sample the input data. Both the convolutional layer and the pooling layer include activation functions.

[0211] In an embodiment of the present application, the convolution layer can be used to extract the initial features of N input images. Specifically, in the first step, the N images are vector-converted to obtain N image vectors, and the N image vectors can be combined into an image vector matrix. In the second step, the image vector matrix is ​​input into the convolution layer, and the convolution kernel is used to perform a convolution operation with the image vector matrix, that is, the image vector matrix and the convolution kernel are performed as an inner product operation to obtain the convolution result corresponding to the image vector matrix. Next, the convolution result is nonlinearly transformed based on the activation function, and the bias vector is added to obtain the initial feature vector. In the third step, the initial feature vector is input into the pooling layer, and the initial feature vector can be feature sampled. Then, the convolution result is nonlinearly transformed based on the activation function, and the bias vector is added to obtain the video feature vector.

[0212] S602: The server extracts features from the text to be processed to obtain a text feature vector; the text to be processed corresponds to the video to be processed.

[0213] In an embodiment of the present application, the text to be processed includes at least one of the subject information, title information, subtitle information, and classification information of the video to be processed.

[0214] Regarding how to obtain the text feature vector, please refer to the description of obtaining the text feature vector from the sample video in Example 1, which will not be repeated here.

[0215] The training method of the second model can be found in the relevant description in the above embodiment 1, which will not be repeated here.

[0216] The second model is used to extract features of the input data. In an embodiment of the present application, the second model is used to extract features of the sample text. The method for obtaining a text feature vector by the second model can refer to the method for obtaining a video feature vector by the first model in S601, which will not be repeated here. In an embodiment of the present application, the second model can be a bidirectional attention neural network model (bidirectional encoder representation from transformers, BERT) or other network models, which is not limited by the present application.

[0217] S603: The server concatenates the text feature vector and the video feature vector to obtain a multimodal feature vector.

[0218] In some embodiments, the text to be processed includes a text line, the text line corresponds to a text feature vector, and the video feature vector includes N image feature vectors. The N image feature vectors can be sequentially concatenated with the text feature vector, resulting in a multimodal feature vector having a row or a column including an image feature vector and a text feature vector.

[0219] For example, the video feature vector is an N2×1024 feature vector matrix, and the text feature vector is a one-dimensional vector containing 768 elements. The text feature vector can be directly spliced ​​behind or in front of each image feature vector in sequence to form an N2×(1024+768) feature vector matrix, which is a multimodal feature vector.

[0220] In other embodiments, the text to be processed includes multiple texts, and features can be extracted from each of the multiple texts to obtain multiple text feature vectors, with the text feature vectors corresponding to the texts. Each of the N image feature vectors can be concatenated with the text feature vector corresponding to each image feature vector, resulting in a row or column of multimodal feature vectors including an image feature vector for an image and a text feature vector for the text corresponding to the image.

[0221] Here, we take M text feature vectors as an example. There is a corresponding relationship between M text feature vectors and N image feature vectors. For example, the image feature vector includes N image feature vectors, and the text feature vector includes M text feature vectors. N and M are both positive integers, and the two can be equal or unequal. When M=N, the text feature vector corresponds to the image feature vector one-to-one. In one possible implementation, the first image feature vector corresponds to the first text feature vector; the second image feature vector corresponds to the second text feature vector; and the third image feature vector corresponds to the third text feature vector. In this case, the server can splice the first text feature vector behind or in front of the first image feature vector; splice the second text feature vector behind or in front of the second image feature vector; and so on, splicing the N image feature vectors with the corresponding text feature vectors to form a multimodal feature vector.

[0222] It should be understood that the image feature vector and the text feature vector may be spliced ​​in other ways, and this application does not limit this.

[0223] S604: The server performs feature fusion on the multimodal feature vectors to obtain a fused feature vector.

[0224] In some embodiments, the server may input the multimodal feature vector into the fusion module to obtain a fused feature vector. The training method of the fusion module can refer to the relevant description in the above embodiment 1 and will not be repeated here.

[0225] The fusion module can be a convolutional neural network, which can include at least one convolutional layer and at least one pooling layer. The convolutional layer is used to extract features from the multimodal feature vector to obtain an initial feature vector, and the pooling layer is used to sample the initial feature vector to obtain a more accurate fused feature vector. Both the convolutional layer and the pooling layer include an activation function.

[0226] Specifically, the server inputs the multimodal feature vector into the convolution layer, and uses the convolution kernel to perform a convolution operation with the multimodal feature vector, that is, the multimodal feature vector and the convolution kernel are inner-producted to obtain the convolution result corresponding to the multimodal feature vector. Next, the convolution result is nonlinearly transformed based on the activation function, and the bias vector is added to obtain the initial feature vector. In the third step, the initial feature vector is input into the pooling layer, and feature sampling of the initial feature vector can be performed. Then, the convolution result is nonlinearly transformed based on the activation function, and the bias vector is added to obtain the fused feature vector.

[0227] In other embodiments, the server may also obtain the fused feature vector in other ways, and the fusion module may also be other network models such as a recurrent neural network, a deep residual network, etc. This application does not impose any restrictions on this.

[0228] S605: The server classifies the video to be processed based on the fused feature vector to obtain a label of the video to be processed.

[0229] In some embodiments, the server can input the fused feature vector into a classification module to obtain a label for the video to be processed. The classification module can include at least one fully connected layer, which can classify the rows to be processed to obtain a label for the video to be processed. In addition, the fully connected layer includes an activation function, which includes a weight matrix and a bias constant. The training method of the classification module can be found in the relevant description of the above embodiment 1 and will not be repeated here.

[0230] Specifically, the server can input the fused feature vector into the fully connected layer, and perform a nonlinear transformation on the fused feature vector based on the weight matrix and bias vector of the activation function to obtain the prediction score of each feature in the fused feature vector. When the prediction score of each feature is greater than a first threshold, the feature is marked as the main feature. For example, the first threshold can be 60 points. Next, the main features are normalized to obtain the predicted probability that each feature in the video to be processed is the label of the video to be processed. When the predicted probability of the feature is greater than a second threshold, the feature can be determined as the label of the video to be processed. For example, the second threshold can be 0.5 points.

[0231] For example, the fused feature vector contains five features: "pig's trotter, beautiful woman, delicious food, table, and water cup." The predicted scores for these five features are 80, 60, 70, 50, and 40, respectively. If the first threshold is 60, then after comparing the predicted scores of the five features with the first threshold, it can be determined that pig's trotter, delicious food, and beautiful woman are the main features, with the predicted probabilities of these three main features being 0.8, 0.7, and 0.6, respectively. If the second threshold is 0.5, then after comparing the predicted probabilities of these three main features with the second threshold, it can be determined that pig's trotter, delicious food, and beautiful woman are the labels for the video to be processed.

[0232] It should be understood that N and M in the first embodiment and the second embodiment may be different.

[0233] The video processing method provided in the embodiment of the present application can obtain the label of the video to be processed by fusing multimodal information. In the embodiment of the present application, video information and text information are taken as an example. By obtaining the label of the video to be processed through the video processing method, the following three technical effects can be achieved:

[0234] (1) By fusing multimodal information, the complementary advantages of each modality can be fully utilized.

[0235] (2) Using text information can effectively elevate model recognition from the perceptual level to the cognitive level, thereby better sorting video tags and obtaining the tags that are most relevant to the main theme of the video content.

[0236] (3) Based on the convolution operation, the features contained in the multimodal information are fused to achieve the purpose of enhancing the interaction between graphic and text information.

[0237] Further, see Figure 7 , Figure 7 1 is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application, wherein the video processing device may include: a first information extraction module 10, a second information extraction module 20, a multimodal feature module 30, a fusion module 40, and a classification module 50.

[0238] A first information extraction module 10 is used to extract features from the video to be processed to obtain a video feature vector;

[0239] A second information extraction module 20 is configured to extract features from the text to be processed to obtain a text feature vector; the text to be processed corresponds to the video to be processed;

[0240] A multimodal feature module 30, configured to concatenate the text feature vector and the video feature vector to obtain a multimodal feature vector;

[0241] A fusion module 40 is configured to perform feature fusion on the multimodal feature vectors to obtain a fused feature vector;

[0242] The classification module 50 is configured to classify the video to be processed based on the fused feature vector to obtain a label of the video to be processed.

[0243] In some embodiments, the first information extraction module 10 may be configured to extract features from the N images respectively to obtain the N image feature vectors.

[0244] In some embodiments, the multimodal feature module 30 is configured to sequentially concatenate the N image feature vectors with the text feature vectors, wherein a row or a column of the multimodal feature vectors includes an image feature vector and the text feature vector.

[0245] In other embodiments, the second information extraction module 20 can be used to extract features from the multiple texts to obtain multiple text feature vectors, wherein the text feature vectors correspond one-to-one to the texts.

[0246] The multimodal feature module 30 can also be used to splice each of the N image feature vectors with the text feature vector corresponding to each of the image feature vectors, where a row or a column of the multimodal feature vector includes an image feature vector of an image and a text feature vector of the text corresponding to the image.

[0247] The fusion module 40 may be configured to input the multimodal feature vector into a fusion module to obtain a fused feature vector.

[0248] The fusion module includes at least one first convolution layer and at least one first pooling layer; wherein,

[0249] The first convolution layer is used to extract features of input data, and the pooling layer is used to sample the input data.

[0250] The classification module 50 can be used to input the fused feature vector into the classification module to obtain the label of the video to be processed; the classification module includes at least one first fully connected layer.

[0251] The second information extraction module 20 may be used to segment the text to be processed to obtain multiple words of the text to be processed;

[0252] Perform vector conversion on the multiple words respectively to obtain word vectors corresponding to the multiple words respectively;

[0253] Inputting the word vectors corresponding to the multiple words into the first model to obtain a text feature vector;

[0254] The text feature vector includes word vectors corresponding to the multiple words respectively;

[0255] The first model is used to extract features of input data.

[0256] The text to be processed includes at least one of video theme information, video title information, video subtitle information, and video classification information.

[0257] See Figure 8 , Figure 8 1 is a schematic diagram of the structure of a machine learning model training device provided in an embodiment of the present application. The video processing device may include: a first acquisition module 10, a second acquisition module 20, a second multimodal feature module 30, a second fusion module 40, a second classification module 50, and an information processing module 60.

[0258] The first acquisition module 10 can be used to input the sample video into the first model to obtain a video feature vector.

[0259] The second acquisition module 20 can be used to input the sample text into the second model to obtain a text feature vector; the sample text corresponds to the sample video.

[0260] The second multimodal feature module 30 may be configured to concatenate the text feature vector and the video feature vector to obtain a multimodal feature vector.

[0261] The second fusion module 40 may be configured to input the multimodal feature vector into a fusion module to obtain a fused feature vector.

[0262] The second classification module 50 can be used to input the fused feature vector into a classification module to obtain a predicted label of the sample video.

[0263] The information processing module 60 can be used to adjust the model parameters of the fusion module and the classification module based on the error between the predicted label and the annotated label of the sample video. Specifically, it can be divided into the following three situations:

[0264] Adjusting model parameters of the first model, the second model, the fusion module, and the classification module based on an error between a predicted label and an annotated label of the sample video;

[0265] or,

[0266] Adjusting model parameters of the first model, the fusion module, and the classification module based on an error between a predicted label and an annotated label of the sample video;

[0267] or,

[0268] Adjusting model parameters of the second model, the fusion module, and the classification module based on an error between the predicted label and the labeled label of the sample video;

[0269] or,

[0270] The model parameters of the fusion module and the classification module are adjusted based on the error between the predicted label and the labeled label of the sample video.

[0271] An embodiment of the present application further provides a computer storage medium having program instructions stored therein. When the program is executed, the program may include some or all of the steps of the method in embodiment 1 or embodiment 2.

[0272] It is understandable that in the specific implementation of this application, when it comes to sample videos, sample texts, videos to be processed, texts to be processed and other related data, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0273] It should be noted that for the aforementioned various method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0274] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0275] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a server reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the server to perform the steps performed in the above-described method embodiments.

[0276] The above is a detailed introduction to a video processing method, a machine learning model training method and related devices and equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video processing method, characterized in that: include: Extract features from the video to be processed to obtain a video feature vector; Extract features from the text to be processed to obtain a text feature vector; the text to be processed corresponds to the video to be processed; Concatenating the text feature vector and the video feature vector to obtain a multimodal feature vector; Performing feature fusion on the multimodal feature vector to obtain a fused feature vector; Determine a prediction score for each feature in the fused feature vector, determine a main feature based on the prediction score of each feature and a first threshold, determine a probability that each main feature is a label for the video to be processed, and determine a label for the video to be processed from the labels corresponding to the main features based on the probability that each main feature is a label for the video to be processed and a second threshold, where the number of labels for the video to be processed is one or more.

2. The method according to claim 1, characterized in that The video to be processed includes N images, the video feature vector includes N image feature vectors, the N image feature vectors correspond one-to-one to the N images, and N is a positive integer; The step of extracting features from the video to be processed to obtain a video feature vector includes: Features are extracted from the N images respectively to obtain the N image feature vectors.

3. The method according to claim 2, characterized in that The step of concatenating the text feature vector and the video feature vector to obtain a multimodal feature vector includes: The N image feature vectors are sequentially concatenated with the text feature vectors, and one row or one column of the multimodal feature vector includes an image feature vector and the text feature vector.

4. The method according to claim 2, characterized in that The text to be processed includes a plurality of texts, and extracting features from the text to be processed to obtain text feature vectors includes: extracting features from the plurality of texts respectively to obtain a plurality of text feature vectors, wherein the text feature vectors correspond to the texts one by one; The step of concatenating the text feature vector and the video feature vector to obtain a multimodal feature vector includes: Each of the N image feature vectors is concatenated with a text feature vector corresponding to each of the image feature vectors, where a row or a column of the multimodal feature vector includes an image feature vector of an image and a text feature vector corresponding to the image.

5. The method according to any one of claims 1 to 4, characterized in that The step of fusing the multimodal feature vectors to obtain a fused feature vector includes: Inputting the multimodal feature vector into a fusion module to obtain a fused feature vector; The fusion module includes at least one convolution layer and at least one pooling layer; wherein, The convolution layer is used to extract features of input data, and the pooling layer is used to sample the input data.

6. The method according to claim 5, characterized in that The method further comprises: Inputting the fused feature vector into a classification module to obtain a label of the video to be processed; The classification module includes at least one fully connected layer.

7. The method according to claim 6, characterized in that The video feature vector is extracted by the first model, and the text feature vector is extracted by the second model; the training data includes a sample video, a sample text corresponding to the sample video, and a label of the sample video, wherein: The first model, the second model, the fusion module and the classification module are obtained by joint training based on the training data; or, The first model, the fusion module and the classification module are jointly trained based on training data; or, The second model, the fusion module and the classification module are jointly trained based on training data; or, The fusion module and the classification module are jointly trained based on training data.

8. The method according to any one of claims 1 to 3, characterized in that The step of extracting features from the text to be processed to obtain a text feature vector includes: Segmenting the text to be processed to obtain multiple words of the text to be processed; Perform vector conversion on the multiple words respectively to obtain word vectors corresponding to the multiple words respectively; The word vectors corresponding to the multiple words are input into the first model to obtain a text feature vector; the first model is used to extract the features of the input data.

9. The method according to any one of claims 1 to 4, characterized in that The text to be processed includes at least one of subject information, title information, subtitle information, and classification information of the video to be processed.

10. A machine learning model training method, characterized in that: include: Input the sample video into the first model to obtain a video feature vector; Inputting a sample text into a second model to obtain a text feature vector; the sample text corresponds to the sample video; Concatenating the text feature vector and the video feature vector to obtain a multimodal feature vector; Inputting the multimodal feature vector into a fusion module to obtain a fused feature vector; Inputting the fused feature vector into a classification module, determining a prediction score for each feature in the fused feature vector, determining a primary feature based on the prediction score of each feature and a first threshold, determining a probability that each primary feature is a label for the sample video, and determining a predicted label for the sample video from the labels corresponding to the primary feature based on the probability that each primary feature is a label for the sample video and a second threshold, where the number of predicted labels for the sample video is one or more; The model parameters of the fusion module and the classification module are adjusted based on the error between the predicted label and the labeled label of the sample video.

11. The method according to claim 10, characterized in that The adjusting the model parameters of the fusion module and the classification module based on the error between the predicted label and the labeled label of the sample video includes: Adjusting model parameters of the first model, the second model, the fusion module, and the classification module based on an error between a predicted label and an annotated label of the sample video; or, Adjusting model parameters of the first model, the fusion module, and the classification module based on an error between a predicted label and an annotated label of the sample video; or, Adjusting model parameters of the second model, the fusion module, and the classification module based on an error between the predicted label and the labeled label of the sample video; or, The model parameters of the fusion module and the classification module are adjusted based on the error between the predicted label and the labeled label of the sample video.

12. A video processing device, characterized in that: include: A first information extraction module is used to extract features from the video to be processed to obtain a video feature vector; A second information extraction module is used to extract features from the text to be processed to obtain a text feature vector; the text to be processed corresponds to the video to be processed; a first multimodal feature module, configured to concatenate the text feature vector and the video feature vector to obtain a multimodal feature vector; A first fusion module is used to perform feature fusion on the multimodal feature vectors to obtain a fused feature vector; The first classification module is used to determine the prediction score of each feature in the fused feature vector, determine the main features based on the prediction score of each feature and the first threshold, determine the probability that each main feature is the label of the video to be processed, and determine the label of the video to be processed from the labels corresponding to the main features based on the probability that each main feature is the label of the video to be processed and the second threshold, where the number of labels of the video to be processed is one or more.

13. A machine learning model training device, characterized in that include: A first acquisition module, configured to input a sample video into a first model to obtain a video feature vector; A second acquisition module is configured to input a sample text into a second model to obtain a text feature vector; the sample text corresponds to the sample video; a second multimodal feature module, configured to concatenate the text feature vector and the video feature vector to obtain a multimodal feature vector; A second fusion module is used to input the multimodal feature vector into a fusion module to obtain a fused feature vector; a second classification module, configured to input the fused feature vector into a classification module, determine a prediction score for each feature in the fused feature vector, determine a primary feature based on the prediction score of each feature and a first threshold, determine a probability that each primary feature is a label for the sample video, and determine a predicted label for the sample video from the labels corresponding to the primary feature based on the probability that each primary feature is a label for the sample video and a second threshold, where the number of predicted labels for the sample video is one or more; An information processing module is used to adjust the model parameters of the fusion module and the classification module based on the error between the predicted label and the labeled label of the sample video.

14. A computer device, characterized in that: include: processor and memory; The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 11.

16. A computer program product or a computer program, characterized in that The computer program product or computer program includes computer instructions, which are stored in a computer-readable storage medium. A processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that a computer device having the processor executes the method according to any one of claims 1 to 11.

Images