Model training method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a model training method and device, electronic equipment and storage medium, relates to the field of artificial intelligence, in particular to the technical field of neural networks, big data and the like. The specific implementation scheme is: inputting a video sample into a multi-modal feature extraction model to be trained to obtain video features and text features; constructing positive samples by using the video features and the text features of the same video sample, and constructing negative samples by using the video features and the text features of different video samples; training the multi-modal feature extraction model based on the positive and negative samples; then extracting the video features and the text features by using the trained multi-modal feature extraction model; and fine-tuning the network model of a target task by using the fusion features of the text features and the video features. In the present disclosure, the positive and negative samples are automatically labeled, the multi-modal feature extraction model can be trained by using massive data, the training of the network model of the target task downstream can be completed by fine-tuning, the training efficiency can be improved, and resources can be saved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to the fields of neural networks and big data. Background Technology

[0002] With the rapid development of internet technology, massive amounts of video resources have been uploaded to the network, greatly increasing the difficulty of recommendation and search technologies. To effectively meet and improve the search experience for different users, it is necessary to provide users with high-quality resources. The ability to efficiently and accurately understand massive amounts of video resources directly affects the accuracy of recommendation and search strategies, ultimately impacting user experience and user retention rates.

[0003] Currently, the industry mainly uses supervised deep learning to train models in small batches, solving a specific video problem from only one dimension or direction. Different video problems require repeating the same operations, resulting in low training efficiency and wasted processing resources. Summary of the Invention

[0004] This disclosure provides a model training method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of this disclosure, a model training method is provided, comprising:

[0006] Multiple video samples are input into the multimodal feature extraction model to be trained to obtain the video features and text features of each video sample;

[0007] Positive samples are constructed using video features and text features from the same video sample, while negative samples are constructed using video features and text features from different video samples, thus obtaining a sample set.

[0008] Based on the sample set, the model parameters of the multimodal feature extraction model to be trained are adjusted, and the trained multimodal feature extraction model is obtained when the training convergence condition is met.

[0009] For the target task, a trained multimodal feature extraction model is used to extract video and text features from the training samples of the target task;

[0010] The text features and video features of the training samples are fused to obtain fused features;

[0011] The network model for the target task is trained based on fusion features.

[0012] According to another aspect of this disclosure, a model training apparatus is provided, comprising:

[0013] The input module is used to input multiple video samples into the multimodal feature extraction model to be trained, and obtain the video features and text features of each video sample;

[0014] The construction module is used to construct positive samples using video features and text features of the same video sample, and to construct negative samples using video features and text features of different video samples, thus obtaining a sample set;

[0015] The adjustment module is used to adjust the model parameters of the multimodal feature extraction model to be trained based on the sample set, so as to obtain the trained multimodal feature extraction model when the training convergence condition is met.

[0016] The extraction module is used to extract video and text features from the training samples of the target task using a trained multimodal feature extraction model.

[0017] The fusion module is used to fuse the text features and video features of the training samples to obtain fused features.

[0018] The training module is used to train a network model for the target task based on fused features.

[0019] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0020] At least one processor; and

[0021] The memory is communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0023] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0024] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0025] In this embodiment, positive and negative samples are automatically labeled, which can support the training of multimodal feature extraction models with massive amounts of data, and the training can be completed by fine-tuning the network model of the downstream target task, thereby improving training efficiency and saving resources.

[0026] It should be understood that the above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0027] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout multiple drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application. Wherein:

[0028] Figure 1 This is a flowchart of a model training method according to an embodiment of the present disclosure;

[0029] Figure 2 This is a flowchart illustrating the extraction of video features using a model training method according to another embodiment of the present disclosure;

[0030] Figure 3 This is a flowchart illustrating the extraction of text features according to another embodiment of the model training method of this disclosure;

[0031] Figure 4 This is an overall framework diagram of feature extraction according to another embodiment of the model training method of this disclosure;

[0032] Figure 5(a) is a flowchart illustrating the determination of cross-entropy loss in a model training method according to another embodiment of the present disclosure;

[0033] Figure 5(b) is an example of determining cross-entropy loss in a model training method according to another embodiment of the present disclosure;

[0034] Figure 6 This is a schematic diagram of the overall framework of a model training method according to another embodiment of the present disclosure;

[0035] Figure 7 This is a schematic diagram of the structure of a model training apparatus according to another embodiment of the present disclosure;

[0036] Figure 8 This is a block diagram of an electronic device used to implement the model training method of the embodiments of this disclosure. Detailed Implementation

[0037] The detailed description and technical content of this application are illustrated below with reference to the accompanying drawings. However, the accompanying drawings are provided for reference and illustration only and are not intended to limit this application.

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0039] The terms "first," "second," etc., used in the embodiments of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0040] Current technical solutions typically employ supervised learning, using a small amount of labeled data to train the target model for feature extraction from the video being processed. However, the manual annotation cost of existing solutions is too high, and the results are poor when transferred to different downstream tasks. Furthermore, different video problems require repeating the same operations to train a suitable target model; simultaneously solving multiple video problems incurs significantly higher costs, leading to low training efficiency and wasted processing resources. Therefore, reducing annotation costs and extracting highly expressive video features applicable to different video problems could save processing resources and improve training efficiency while maintaining training effectiveness.

[0041] Based on this technical concept, this disclosure proposes a model training method, which includes constructing a sample set by extracting video and text features from a large number of video samples, and then using this sample set to train a multimodal feature extraction model in a self-supervised manner. The trained multimodal feature extraction model can be applied to video feature extraction for different target tasks. Therefore, for different video problems, there is no need to repeatedly train the multimodal feature extraction model; only the model for downstream tasks needs to be fine-tuned. Based on this technical concept, such as... Figure 1 As shown, the model training method in this embodiment includes the following:

[0042] S101, input multiple video samples into the multimodal feature extraction model to be trained to obtain the video features and text features of each video sample.

[0043] The multimodal feature extraction model to be trained can be any model capable of extracting video features and text features from video samples.

[0044] S102, positive samples are constructed using video features and text features of the same video sample, and negative samples are constructed using video features and text features of different video samples, thus obtaining a sample set.

[0045] Based on the video and text features of each video sample obtained in S101, video and text features from the same video sample and video and text features from different video samples are distinguished and used to construct positive and negative samples, respectively, so as to adjust the model parameters of the multimodal feature extraction model to be trained.

[0046] S103. Based on the sample set, adjust the model parameters of the multimodal feature extraction model to be trained, and obtain the trained multimodal feature extraction model when the training convergence condition is met.

[0047] In this process, based on the positive and negative samples in the sample set, the correspondence between video features and text features in the positive samples and the correspondence between video features and text features in the negative samples can be determined by adjusting the model parameters of the multimodal feature extraction model to be trained. Based on the determined correspondence, the model parameters of the multimodal feature extraction model to be trained are adjusted again to obtain the trained multimodal feature extraction model. Ultimately, when using this trained multimodal feature extraction model to extract video and text features from video samples, more accurate video and text features that represent the video samples can be obtained.

[0048] S104. For the target task, a trained multimodal feature extraction model is used to extract video and text features from the training samples of the target task.

[0049] The target task may include specific downstream tasks such as video classification and retrieval. It may also include downstream tasks for generating video description information. Of course, the specific target task is not specifically limited in this embodiment.

[0050] S105, the text features and video features of the training samples are fused to obtain fused features.

[0051] S106, a network model for training the target task based on fused features.

[0052] In this embodiment, video features and text features of the same video sample are used as the multimodal features of that video sample. Multimodal features can comprehensively describe video samples, thus possessing greater expressive power. By constructing positive samples from the video features and text features of the same video sample, and negative samples from different video samples, self-supervised training of the multimodal feature extraction model to be trained can be achieved. Training of this model requires no labeling work, enabling self-supervised training using massive amounts of data. Therefore, the multimodal feature extraction model trained in this embodiment can learn feature extraction capabilities from massive amounts of unlabeled data, allowing it to extract features applicable to different video problems. Furthermore, for any downstream target task, only supervised training is needed to fine-tune the network model parameters for the downstream task. This embodiment, through a combination of self-supervised and supervised training, can more efficiently solve multiple video problems. For different video problems, only the corresponding network model needs to be adjusted, eliminating the need to repeatedly train the multimodal feature extraction model, thus improving model training efficiency and saving processing resources.

[0053] In some embodiments, to better describe the video, this disclosure requires fusing the text features and video features of the training samples to obtain fused features. The fusion process can be implemented as follows:

[0054] Fusion Method 1) The text features and video features of the training samples are weighted and averaged to obtain the fused features.

[0055] In this embodiment of the disclosure, compared to describing video samples using both video features and text features separately, a weighted average method can reduce the feature dimensionality and be used by downstream edge tasks.

[0056] Fusion Method 2) The text features and video features of the training samples are concatenated to obtain the fused features.

[0057] In implementation, the Concat concatenation function can be used to concatenate the text features and video features of the training samples. Of course, other concatenation layers can also be selected as needed, all of which are applicable to the embodiments of this disclosure.

[0058] In this embodiment, the text features and video features of the training samples are fused, and the fused features are used as the features of the video samples, instead of using only the text features or only the video features of the training samples as the features of the video samples. This makes the features of the final determined video samples more closely resemble the true features of the video samples, which is beneficial for training the multimodal feature extraction model and for downstream target tasks, enabling better achievement of the objective based on the fused features.

[0059] In some embodiments, training a network model for the target task based on fused features can be implemented as follows:

[0060] When the target task is a classification task, a multi-layer perceptron (MLP) is used as the network model for the target task, and the network model for the target task is trained based on the training labels of the classification task.

[0061] For example, in a video classification task, videos are categorized into categories such as children's, sports, and science. These categories can then be used as training labels to train the network model for the classification task.

[0062] When the target task includes both classification and regression tasks, the classification layer and regression layer are determined as the network model for the target task, and the network model for the target task is trained based on the classification label and location label of the target task.

[0063] The network model for the target task can include a neural network model. Currently, there are many variations of neural network models, such as backpropagation (BP) neural networks, probabilistic neural networks, convolutional neural networks (CNNs – suitable for image recognition), and long-short-term memory networks (LSTMs – suitable for speech recognition). Based on biological neuron models, the basic structure of a multilayer perceptron (MLP) can be obtained. The most typical MLP includes three layers: an input layer, a hidden layer, and an output layer. Different layers in an MLP neural network model are fully connected (any neuron in one layer is connected to all neurons in the next layer).

[0064] Neural network models primarily consist of three basic elements: weights, biases, and activation functions. Weights represent the strength of connections between neurons, and their magnitude indicates the probability of a connection. Bias is a crucial parameter in the model, ensuring that the output value calculated from the input does not arbitrarily activate any neurons, thus guaranteeing correct sample classification. Activation functions act as non-linear mappings, limiting the output amplitude of neurons to a certain range, typically between -1 and 1, or 0 and 1. The most commonly used activation function is the Sigmoid function, which maps numbers from (-∞, +∞) to the range (0 to 1). Other activation functions include tanh and ReLU. Tanh is a variation of the Sigmoid function, with a mean of 0, and often performs better than Sigmoid in practical applications. ReLU is a more recently popular activation function; when the input signal is less than 0, the output is 0; when the input signal is greater than 0, the output equals the input.

[0065] In the embodiments of this disclosure, a suitable network model can be constructed based on the classification task, a network model for the target task can be trained based on training labels, or a neural network model for the target task can be trained based on the classification labels and location labels of the target task.

[0066] In this embodiment, the MLP possesses strong adaptive learning capabilities and can handle complex multi-input, multi-output nonlinear problems. Using a network model trained on the target task based on fused features facilitates the application of the multimodal feature extraction model to feature extraction in downstream tasks, thereby improving feature extraction quality.

[0067] In some embodiments, the multimodal feature extraction model to be trained includes an image encoder and a video encoder. For each video sample, a method such as... Figure 2 The video features are extracted in the manner shown, including the following:

[0068] S201, downsample the video sample to obtain a frame sequence.

[0069] For downsampling, image frames can be extracted from the video sample at fixed intervals or fixed frame numbers to obtain a frame sequence composed of a series of image frames.

[0070] S202, use an image encoder to extract image features for each frame in the frame sequence.

[0071] The image encoder may include tools such as the jpeg_axi image codec that can encode images.

[0072] S203: Input the image features of the frame sequence into the video encoder to obtain the video features of the video samples.

[0073] The video encoder may include a tool that can obtain video features by utilizing image features, consisting of a dedicated audio and video compression codec chip, data and alarm input and output channels, network interface, audio and video interface (such as HDMI, VGA, HD-SDI), RS232 serial interface control, protocol interface control, embedded software, etc.

[0074] In this embodiment, an image encoder is first used to extract image features, and then the image features of the frame sequence are input into a video encoder to obtain video features. This makes the extracted video features more expressive and is beneficial for training a multimodal feature extraction model.

[0075] In some embodiments, the multimodal feature extraction model to be trained includes a speech recognition model, a text summarizer, and a text encoder. Multiple video samples are input into the multimodal feature extraction model to be trained to obtain the text features of each video sample, such as... Figure 3 As shown, it can be implemented as follows:

[0076] S301, acquire the audio of each video sample from multiple video samples.

[0077] S302, input the audio of each video sample into the speech recognition model to obtain the audio text of each video sample.

[0078] The speech recognition model can include, for example, dynamic time warping, vector quantization, and hidden Markov models. This model converts the audio in each video sample into audio-text.

[0079] S303, input the audio text of each video sample into the text summarizer to obtain the video summary text of each video sample.

[0080] Based on the acquired audio text, a text summarizer is used to extract audio text summaries, which are the video summary texts of each video sample. The text summarizer can employ traditional methods such as extractive summarization or generative summarization to extract the video summary text. Extractive summarization generates video summary text by extracting and concatenating key sentences from the audio text. Generative summarization, on the other hand, automatically organizes language based on the important content expressed in the audio text to summarize it; the entire process is end-to-end, similar to translation and dialogue tasks.

[0081] Video summarization text can also be extracted using neural network models applied to text summarization, including extractive models, generative models, and compression models. Extractive models primarily model the problem as two tasks: sequence labeling and sentence ranking, encompassing sequence labeling methods, sentence ranking methods, and seq2seq (sequence-to-sequence). Generative models are mainly based on seq2seq and transformer models, incorporating various auxiliary information. Compression models are primarily based on the information bottleneck and can also be considered a hybrid of extractive and generative models.

[0082] S304. Input the video summary text of each video sample into the text encoder to obtain the text features of each video sample.

[0083] For ease of understanding, such as Figure 4The diagram illustrates the process of generating a sample set in this embodiment of the present disclosure. The video samples can be segmented into a sequence of video frames per second using ffmpeg (fast forward moving picture expert group, a multimedia video processing tool), and audio can be extracted from the video samples. Then, based on the audio of the video samples, a speech recognition model is used to obtain audio text, followed by the extraction of text summaries using a summarization model (e.g., MatchSum model). Finally, a text encoder (e.g., BERT) is used to obtain audio text vectors. Alternatively, based on video frames, an image encoder (e.g., ViT) is used to convert each video frame into an image vector, and a video encoder (e.g., Transformer) is used to obtain video vectors.

[0084] In this embodiment, a speech recognition model is first used to obtain the audio text of each video sample, and then a text summarizer is used to process the audio text to obtain the video summary text of each video sample. This makes the text features more expressive and can better describe the video samples, which in turn helps to accelerate the training of the multimodal feature extraction model.

[0085] In other embodiments, in addition to Figure 3 As shown, besides extracting text features from audio, text features from video samples can also be extracted from various sources. Accordingly, in this embodiment, the multimodal feature extraction model to be trained includes a text summarizer and a text encoder. Multiple video samples are input into the multimodal feature extraction model to be trained to obtain the text features of each video sample, which can be implemented as follows:

[0086] Step A1: Obtain the text information of each video sample. The text information includes the text corresponding to the audio of the video sample, the text in the image of the video sample, and the text description information of the video sample.

[0087] The text description information of the video samples may include text information in the video file, video title, category tags, and comment list, etc.

[0088] Step A2: Input the text information of each video sample into the text summarizer to obtain the video summary text of each video sample. Input the video summary text of each video sample into the text encoder to obtain the text features of each video sample.

[0089] In some embodiments, text information can also be extracted from image frames in each video sample using an optical character recognition (OCR) model. Both audio text and OCR text can be used as video text.

[0090] In this embodiment of the disclosure, multiple video-related text descriptions are used to determine the source of text features, which can more comprehensively obtain text features and enable the text features to describe video samples well, thereby accelerating the training of the multimodal feature extraction model.

[0091] Based on the preceding description, we have introduced how to construct positive and negative samples, and how to extract video features and text features respectively. To better utilize a self-supervised approach to train the multimodal feature extraction model, in this embodiment, the model parameters of the multimodal feature extraction model to be trained are adjusted based on the sample set to obtain a trained multimodal feature extraction model, as shown in Figure 5(a):

[0092] S501, for each sample to be processed in the training sample set, determine the feature similarity between video features and text features in the sample to be processed.

[0093] In cases where video and text features in the sample to be processed are not on the same dimension, the video and text features are first converted to the same dimension before calculating the feature similarity between them. This same dimension can be a preset dimension. Specifically, the video and text features in the sample can be converted into video vectors and text vectors respectively, and then the cosine similarity between the video and text vectors is calculated. That is, the cosine of the angle between the two vectors in the vector space is used to measure the similarity between video / text features and video features.

[0094] S502, determine the cross-entropy loss based on the feature similarity of each sample to be processed and the sample label of each sample to be processed. The sample label can be either a positive or negative sample.

[0095] In some embodiments, the dimension of the label matrix can be determined according to the number of samples to be processed, where samples from the same video are denoted as 1, and samples from different videos are denoted as 0. Simultaneously, the feature similarity between each sample to be processed is calculated to obtain a similarity matrix for each sample. By comparing the label matrix and the similarity matrix, and based on the feature similarity and sample labels of each sample, the cross-entropy loss is obtained.

[0096] The smaller the cross-entropy loss, the more accurately the multimodal feature extraction model can extract video features. The larger the cross-entropy loss, the more the multimodal feature extraction model needs further training. In this case, the model parameters of the multimodal feature extraction model to be trained can be adjusted based on the cross-entropy loss to reduce the cross-entropy loss.

[0097] For ease of understanding, Figure 5(b) illustrates the process of obtaining cross-entropy loss in this embodiment of the present disclosure. For example, if there are three video samples: Video 1, Video 2, and Video 3, a 3×3 similarity matrix can be generated. By comparing this similarity matrix with the label matrix, the cross-entropy loss can be obtained.

[0098] Finally, in step S503, if the training convergence condition is met, training ends, and the trained multimodal feature extraction model is obtained. If the training convergence condition is not met, the process returns to step S101, where multiple video samples are input into the multimodal feature extraction model to be trained to obtain the video features and text features of each video sample, until the multimodal feature extraction model converges.

[0099] In this embodiment, the cross-entropy loss is obtained by calculating the feature similarity between video features and text features in the sample to be processed, and comparing this feature similarity with the sample label. Finally, the model parameters of the multimodal feature extraction model to be trained can be adjusted based on the cross-entropy loss. Feature similarity can describe the differences between multimodal features within the same video, as well as the differences between multimodal features across different videos. Using feature similarity to determine the cross-entropy loss allows the multimodal feature extraction model to extract text and video features not independently, but rather to mutually understand the content of the same video, achieving mutual learning between text and video features of the same video. This enables the extracted video and text features to better describe the video. Furthermore, it allows for the differentiation of features from different videos, leading to a better understanding of the feature differences between them. Therefore, the feature extraction capability of the multimodal feature extraction model is improved.

[0100] To facilitate understanding of the overall scheme of this disclosure, in conjunction with Figure 6 Please provide an explanation. For example... Figure 6 The diagram illustrates the overall model training process in this embodiment. In the upstream model training, a multimodal feature extraction model is trained using a self-supervised method. Once this model converges, training is complete, and its parameters remain fixed. In the training of the downstream target task network model (such as an MLP for classification), the multimodal feature extraction model is applied to different downstream tasks, and the network model for each task is fine-tuned. For downstream tasks, such as action classification and video classification, only the newly added task header parameters need to be fine-tuned according to the specific downstream task.

[0101] Based on the same technical concept, this disclosure also provides a model training apparatus, such as... Figure 7 As shown, it includes:

[0102] The input module is used to input multiple video samples into the multimodal feature extraction model to be trained, and obtain the video features and text features of each video sample;

[0103] The construction module is used to construct positive samples using video features and text features of the same video sample, and to construct negative samples using video features and text features of different video samples, thus obtaining a sample set;

[0104] The adjustment module is used to adjust the model parameters of the multimodal feature extraction model to be trained based on the sample set, so as to obtain the trained multimodal feature extraction model when the training convergence condition is met.

[0105] The extraction module is used to extract video and text features from the training samples of the target task using the trained multimodal feature extraction model.

[0106] The fusion module is used to fuse the text features and video features of the training sample to obtain fused features.

[0107] The training module is used to train the network model for the target task based on the fusion features.

[0108] In some embodiments, the input module includes:

[0109] The sampling submodule is used to downsample the video of each video sample to obtain a frame sequence.

[0110] The extraction submodule is used to extract image features for each frame in the frame sequence using the image encoder.

[0111] The first input submodule is used to input the image features of the frame sequence into the video encoder to obtain the video features of the video sample.

[0112] In some embodiments, the input module further includes:

[0113] The acquisition submodule is used to acquire the audio of each video sample from the multiple video samples;

[0114] The second input submodule is used to input the audio of each video sample into the speech recognition model to obtain the audio text of each video sample.

[0115] The third input submodule is used to input the audio text of each video sample into the text summarizer to obtain the video summary text of each video sample;

[0116] The fourth input submodule is used to input the video summary text of each video sample into the text encoder to obtain the text features of each video sample.

[0117] In some embodiments, the extraction module includes:

[0118] The first determination submodule is used to determine the feature similarity between video features and text features in each sample to be processed in the training sample set.

[0119] The second determination submodule is used to determine the cross-entropy loss based on the feature similarity corresponding to each sample to be processed and the sample label of each sample to be processed; the sample label is either a positive sample or a negative sample.

[0120] The adjustment submodule is used to adjust the model parameters of the multimodal feature extraction model to be trained based on the cross-entropy loss;

[0121] The first execution submodule is used to terminate training and obtain the trained multimodal feature extraction model when the training convergence condition is met.

[0122] The second execution submodule is used to return to the step of inputting multiple video samples into the multimodal feature extraction model to be trained to obtain the video features and text features of each video sample if the training convergence condition is not met.

[0123] In some embodiments, the fusion module includes:

[0124] The fused feature is obtained by taking a weighted average of the text features and video features of the training sample; or,

[0125] The text features and video features of the training sample are concatenated to obtain the fused feature.

[0126] In some embodiments, the training module includes:

[0127] The first training submodule is used to train the network model of the target task based on the training labels of the classification task, using the multilayer perceptron as the network model of the target task.

[0128] The second training submodule is used to determine the classification layer and regression layer as the network model for the target task when the target task package includes classification and regression tasks, and to train the network model for the target task based on the classification label and location label of the target task.

[0129] In some embodiments, the input module includes:

[0130] The acquisition submodule is used to acquire the text information of each video sample. The text information includes the text corresponding to the audio of the video sample, the text in the image of the video sample, and the text description information of the video sample.

[0131] The abstract extraction submodule is used to input the text information of each video sample into the text abstract extractor to obtain the video abstract text of each video sample;

[0132] The encoding submodule is used to input the video summary text of each video sample into the text encoder to obtain the text features of each video sample.

[0133] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0134] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0135] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0136] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0138] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as model training methods. For example, in some embodiments, the model training method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the model training method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform model training methods by any other suitable means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable model training device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0145] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0146] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model training method, comprising: Multiple video samples are input into the multimodal feature extraction model to be trained, and the video features and text features of each video sample are obtained, including: Downsample the video of each video sample to obtain a frame sequence; For each frame in the frame sequence, the image encoder of the multimodal feature extraction model is used to extract image features; The image features of the frame sequence are input into the video encoder of the multimodal feature extraction model to obtain the video features of the video sample; Obtain the text information of each video sample, including text information in the video file, video title, category tags, and comment list; The text information of each video sample is input into the text summarizer of the multimodal feature extraction model to obtain the video summary text of each video sample; The video summary text of each video sample is input into the text encoder of the multimodal feature extraction model to obtain the text features of each video sample; Positive samples are constructed using video features and text features from the same video sample, while negative samples are constructed using video features and text features from different video samples, thus obtaining a sample set. Based on the sample set, the multimodal feature extraction model is trained in a self-supervised manner to adjust the model parameters of the multimodal feature extraction model to be trained. Under the condition that the training convergence condition is met, the trained multimodal feature extraction model is obtained. For the target task, the trained multimodal feature extraction model is used to extract video and text features from the training samples of the target task; The text features and video features of the training samples are fused to obtain fused features; The network model for the target task is trained in a supervised manner based on the fusion features.

2. The method according to claim 1, wherein, The step of adjusting the model parameters of the multimodal feature extraction model to be trained based on the sample set, and obtaining the trained multimodal feature extraction model under the condition of meeting the training convergence condition, includes: For each sample to be processed in the sample set, determine the feature similarity between the video features and text features of the sample to be processed; The cross-entropy loss is determined based on the feature similarity of each sample to be processed and the sample label of each sample to be processed; the sample label is either a positive sample or a negative sample. The model parameters of the multimodal feature extraction model to be trained are adjusted based on the cross-entropy loss. If the training convergence condition is met, the training ends, and the trained multimodal feature extraction model is obtained. If the training convergence condition is not met, return to the step of inputting multiple video samples into the multimodal feature extraction model to be trained to obtain the video features and text features of each video sample.

3. The method according to claim 1 or 2, wherein, The process of fusing the text features and video features of the training samples to obtain fused features includes: The fused feature is obtained by weighted averaging of the text features and video features of the training samples; or, The text features and video features of the training samples are concatenated to obtain the fused features.

4. The method according to claim 1, wherein, The process of training the network model for the target task based on the fused features includes: When the target task is a classification task, a multilayer perceptron is used as the network model for the target task, and the network model for the target task is trained based on the training labels of the classification task. In the case where the target task package includes classification and regression tasks, the classification layer and regression layer are determined as the network model for the target task, and the network model for the target task is trained based on the classification label and location label of the target task.

5. A model training device, comprising: The input module is used to input multiple video samples into the multimodal feature extraction model to be trained, and obtain the video features and text features of each video sample, including: The sampling submodule is used to downsample the video of each video sample to obtain a frame sequence; The extraction submodule is used to extract image features from each frame in the frame sequence using the image encoder of the multimodal feature extraction model; The first input submodule is used to input the image features of the frame sequence into the video encoder of the multimodal feature extraction model to obtain the video features of the video sample; The acquisition submodule is used to acquire the text information of each video sample, including the text information in the video file, the video title, the category tags, and the comment list; The abstract extraction submodule is used to input the text information of each video sample into the text abstract extractor of the multimodal feature extraction model to obtain the video abstract text of each video sample; The encoding submodule is used to input the video summary text of each video sample into the text encoder of the multimodal feature extraction model to obtain the text features of each video sample. The construction module is used to construct positive samples using video features and text features of the same video sample, and to construct negative samples using video features and text features of different video samples, thus obtaining a sample set; The adjustment module is used to train the multimodal feature extraction model in a self-supervised manner based on the sample set, so as to adjust the model parameters of the multimodal feature extraction model to be trained, and obtain the trained multimodal feature extraction model when the training convergence condition is met. The extraction module is used to extract video and text features of the training samples of the target task using the trained multimodal feature extraction model. The fusion module is used to fuse the text features and video features of the training samples to obtain fused features; The training module is used to perform supervised training on the network model for the target task based on the fused features.

6. The apparatus according to claim 5, wherein, The extraction module includes: The first determining submodule is used to determine the feature similarity between video features and text features in each sample to be processed in the sample set. The second determining submodule is used to determine the cross-entropy loss based on the feature similarity corresponding to each sample to be processed and the sample label of each sample to be processed; the sample label is a positive sample or a negative sample. The adjustment submodule is used to adjust the model parameters of the multimodal feature extraction model to be trained based on the cross-entropy loss. The first execution submodule is used to terminate training and obtain the trained multimodal feature extraction model when the training convergence condition is met. The second execution submodule is used to return to the step of inputting multiple video samples into the multimodal feature extraction model to be trained to obtain the video features and text features of each video sample if the training convergence condition is not met.

7. The apparatus according to claim 5 or 6, wherein, The fusion module includes: The fused feature is obtained by weighted averaging of the text features and video features of the training samples; or, The text features and video features of the training samples are concatenated to obtain the fused features.

8. The apparatus according to claim 5, wherein, The training module includes: The first training submodule is used to train the network model of the target task based on the training labels of the classification task, using a multilayer perceptron as the network model of the target task when the target task is a classification task. The second training submodule is used to determine the classification layer and regression layer as the network model of the target task when the target task package includes classification and regression tasks, and to train the network model of the target task based on the classification label and location label of the target task.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4.

Images