Pre-training method and device of multi-modal video classification model, and video classification method and device

Abstract

The embodiment of the application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving, and particularly relates to a pre-training method and device of a multi-modal video classification model and a video classification method. The method is that: first, the text feature extraction network is subjected to multi-round self-supervised training to obtain an initial video feature extraction network after training, then the text feature extraction network is subjected to multi-round self-supervised training to obtain an initial text feature extraction network after training, and then a pre-training sample data set is used to perform multi-round supervised iterative training on an initial multi-modal video classification model including the initial video feature extraction network and the initial text feature extraction network to obtain a pre-trained multi-modal video classification model. In this way, the completed pre-training process can be specifically divided into two training stages, can learn based on video feature information and text feature information, and realizes efficient classification of videos.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a pre-training method for a multimodal video classification model, a video classification method, and an apparatus. Background Technology

[0002] With the development of information technology, video, as a collection of multimodal information, has gradually become an important medium for carrying information. Therefore, in order to effectively classify the massive amount of video data added every day, specific video classification tasks can usually be achieved based on pre-trained models with multimodal dual-stream structures obtained through pre-training.

[0003] However, in the current process of training pre-trained models of multimodal dual-stream structures, self-supervised training is usually achieved by constructing video-text pairs. This limits the accuracy of the pre-trained model to the sample quality of the video-text pairs. Furthermore, the training process of the pre-trained model only focuses on learning to establish the correspondence between videos and text. Therefore, in subsequent specific video classification tasks, a lot of resources are needed to adjust the obtained pre-trained model, which makes it impossible to achieve efficient video classification. Summary of the Invention

[0004] This application provides a pre-training method for a multimodal video classification model, a video classification method, and an apparatus to solve the problem that existing technologies cannot efficiently classify videos.

[0005] Firstly, a pre-training method for a multimodal video classification model is proposed, including:

[0006] Obtain a pre-training sample data set; each pre-training sample data set includes a sample video stream, as well as sample text information and sample classification labels associated with the sample video stream;

[0007] Using the sample video streams included in each sample data in the pre-trained sample dataset, the video feature extraction network is trained in multiple rounds of self-supervised training to obtain the initial video feature extraction network after training.

[0008] Using the sample text information associated with each sample video stream in the pre-trained sample dataset, the text feature extraction network is trained in multiple rounds of self-supervised training to obtain the initial text feature extraction network after training.

[0009] Using the pre-trained sample data set, the initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network, is subjected to multiple rounds of supervised training to obtain the pre-trained multimodal video classification model.

[0010] Secondly, a video classification method is proposed, including:

[0011] Acquire the video stream to be processed and determine the text information to be processed associated with the video stream to be processed;

[0012] The data stream to be processed and the text information to be processed are input into the target multimodal video classification model that has been trained, and the video classification result output by the target multimodal video classification model is obtained. The target multimodal video classification model is obtained by fine-tuning the pre-trained multimodal video classification model for any of the above-mentioned first aspects.

[0013] Optionally, the target multimodal video classification model includes at least a target video feature extraction network, a target text feature extraction network, and a target classification network; the step of inputting the data stream to be processed and the text information to be processed into the trained target multimodal video classification model to obtain the video classification label output by the target multimodal video classification model includes:

[0014] The video stream to be processed is input into the target video feature extraction network, and the text information to be processed is input into the target text feature extraction network;

[0015] The target video feature extraction network is used to split the video stream to be processed into a specified number of video frames to be processed, and to extract the video feature information included in each video frame to be processed. The target text feature extraction network is used to extract the text feature information in the text to be processed.

[0016] Using the target classification network, based on the text feature information and the video feature information concatenated in chronological order, the video classification result is output.

[0017] Optionally, when the pre-trained multimodal video classification model includes at least a pre-trained video feature extraction network, a pre-trained text feature extraction network, and a pre-trained classification network, if it is determined that the video classification label configured in the fine-tuning stage is consistent with the classification label configured in the pre-trained classification network, the pre-trained multimodal video classification model is fine-tuned, and the following operations are performed to obtain the target multimodal video classification model:

[0018] Obtain a set of fine-tuning sample data, each set of fine-tuning sample data including a fine-tuning sample video stream, as well as fine-tuning sample text information and fine-tuning sample classification labels associated with the fine-tuning sample video stream;

[0019] Using the fine-tuned sample data set, the pre-trained multimodal video classification model is subjected to multiple rounds of fine-tuning training to obtain the fine-tuned target multimodal video classification model.

[0020] Optionally, determining the text information associated with the video stream to be processed includes:

[0021] Obtain the title information associated with the video stream to be processed;

[0022] For each video frame to be processed, a text recognition algorithm is used to identify the video frame text information included in the video frame to be processed;

[0023] The title information and the identified text information of each video frame are used as the text information to be processed associated with the video stream to be processed.

[0024] Thirdly, a pre-training device for a multimodal video classification model is proposed, comprising:

[0025] The acquisition unit is used to acquire a pre-training sample data set; each pre-training sample data in the pre-training sample data set includes a sample video stream, as well as sample text information and sample classification labels associated with the sample video stream;

[0026] The first training unit is used to perform multiple rounds of self-supervised training on the video feature extraction network using the sample video streams included in each sample data in the pre-training sample data set, so as to obtain the initial video feature extraction network after training.

[0027] The second training unit is used to perform multiple rounds of self-supervised training on the text feature extraction network using the sample text information associated with each sample video stream in the pre-trained sample data set, so as to obtain the initial text feature extraction network after training.

[0028] The third training unit is used to perform multiple rounds of supervised training on the initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network, using the pre-trained sample data set, to obtain the pre-trained multimodal video classification model.

[0029] Optionally, when acquiring the pre-trained sample data set, the acquisition unit is used for:

[0030] Based on the historical video data corresponding to each video source, a subset of historical video data corresponding to each video source is constructed; wherein, each historical video data includes a historical video stream, as well as historical text information and original classification labels associated with the historical video stream;

[0031] For each video source, a target pseudo-classification model corresponding to the video source is obtained by supervised training based on the historical video data corresponding to the video source.

[0032] For each historical video stream, based on each target pseudo-classification model, determine each pseudo-classification result associated with the historical video stream, and take the union of the original classification label associated with the historical video stream and each pseudo-classification result as the sample classification label associated with the historical video stream.

[0033] Furthermore, based on the historical video stream and the sample classification labels and historical text information associated with the historical video stream, a pre-training sample data is generated; based on each pre-training sample data, the pre-training sample data set is generated.

[0034] Optionally, when constructing a subset of historical video data corresponding to each video source based on the historical video data corresponding to each video source, the acquisition unit is used for:

[0035] Based on the initial video data from each video source whose release time is within a preset time range, an initial video data set is generated, wherein each initial video data set includes an initial video stream, as well as historical text information and original classification tags associated with the initial video stream;

[0036] One initial video data point is read sequentially from the initial video data set, and the following deduplication process is performed based on the read initial video data until there is no unread initial video data in the initial video data set:

[0037] The video feature similarity between the initial video stream in the read initial video data and the initial video stream in other initial video data in the initial video data set is determined; and other initial video data in the initial video data set whose video feature similarity with the read initial video data meets the similarity threshold is deleted.

[0038] The initial video data in the deduplicated initial video data set is taken as historical video data; and according to the video source to which each historical video data belongs, a subset of historical video data corresponding to each video source is obtained by clustering based on each historical video data.

[0039] Optionally, when determining the pseudo-classification results associated with the historical video stream based on each target pseudo-classification model, the acquisition unit is used for:

[0040] The target video source corresponding to the historical video stream is determined, the target pseudo-classification model corresponding to the target video source is determined, and other target pseudo-classification models corresponding to each other video source other than the target video source are obtained.

[0041] Various other target pseudo-classification models are used to determine the pseudo-classification results associated with the historical video stream.

[0042] Optionally, the pseudo-classification result includes a pseudo-classification label and a classification prediction value obtained based on the pseudo-classification label;

[0043] After determining the pseudo-classification results associated with the historical video stream, the acquisition unit is further configured to:

[0044] Obtain the initial intersection between the pseudo-classification labels included in each pseudo-classification result and the original classification labels associated with the historical video stream, and determine the classification prediction value corresponding to each classification label in the initial intersection;

[0045] From the original classification labels associated with the historical video stream, delete the classification labels whose classification prediction values ​​in the initial intersection meet the label filtering conditions to obtain the processed original classification labels;

[0046] Next, for the processed original classification labels, the step of taking the union of the original classification labels associated with the historical video stream and each pseudo-classification result as the sample classification labels associated with the historical video stream is performed.

[0047] Optionally, during a round of self-supervised training of the video feature extraction network, the first training unit is used to perform the following operations:

[0048] A specified number of sample video streams are read from the pre-trained sample data set, and a specified number of sample video frames with associated time-series information are extracted from each of the read sample video streams through the video feature extraction network.

[0049] The video feature extraction network is controlled to generate positive sample groups based on each sample video frame belonging to the same sample video stream, and to generate negative sample groups based on each sample video frame belonging to different sample video streams, wherein a sample group includes two sample video frames.

[0050] The video feature extraction network is trained in multiple rounds of self-supervised iterative training based on the positive sample group and the negative sample group until the preset single-round convergence condition is met.

[0051] Optionally, when extracting a specified number of sample video frames with associated time-series information from each of the read sample video streams, the first training unit is configured to perform any one of the following operations:

[0052] For each sample video stream read, a target time point is set according to the duration information corresponding to the sample video stream, and a specified number of sample video frames with associated time sequence information are extracted starting from the sample video frame corresponding to the target time point on the sample video stream.

[0053] For each sample video stream read, the sample video stream is divided into a specified number of sample video sub-streams according to the duration information corresponding to the sample video stream, and a sample video frame is randomly extracted from each sample video sub-stream to obtain a specified number of sample video frames associated with time sequence information.

[0054] Fourthly, a video classification device is proposed, comprising:

[0055] Acquisition unit: used to acquire the video stream to be processed and determine the text information to be processed associated with the video stream to be processed;

[0056] The input unit is used to input the data stream to be processed and the text information to be processed into the target multimodal video classification model that has been trained, and to obtain the video classification result output by the target multimodal video classification model. The target multimodal video classification model is obtained by fine-tuning the pre-trained multimodal video classification model according to any one of the first aspects above.

[0057] Optionally, the target multimodal video classification model includes at least a target video feature extraction network, a target text feature extraction network, and a target classification network; when the data stream to be processed and the text information to be processed are input into the trained target multimodal video classification model to obtain the video classification label output by the target multimodal video classification model, the input unit is used for:

[0058] The video stream to be processed is input into the target video feature extraction network, and the text information to be processed is input into the target text feature extraction network;

[0059] The target video feature extraction network is used to split the video stream to be processed into a specified number of video frames to be processed, and to extract the video feature information included in each video frame to be processed. The target text feature extraction network is used to extract the text feature information in the text to be processed.

[0060] Using the target classification network, based on the text feature information and the video feature information concatenated in chronological order, the video classification result is output.

[0061] Optionally, when the pre-trained multimodal video classification model includes at least a pre-trained video feature extraction network, a pre-trained text feature extraction network, and a pre-trained classification network, if it is determined that the video classification label configured in the fine-tuning stage is consistent with the classification label configured in the pre-trained classification network, the pre-trained multimodal video classification model is fine-tuned, and the input unit performs the following operations to obtain the target multimodal video classification model:

[0062] Obtain a set of fine-tuning sample data, each set of fine-tuning sample data including a fine-tuning sample video stream, as well as fine-tuning sample text information and fine-tuning sample classification labels associated with the fine-tuning sample video stream;

[0063] Using the fine-tuned sample data set, the pre-trained multimodal video classification model is subjected to multiple rounds of fine-tuning training to obtain the fine-tuned target multimodal video classification model.

[0064] Optionally, when the pre-trained multimodal video classification model includes at least a pre-trained video feature extraction network, a pre-trained text feature extraction network, and a pre-trained classification network, if it is determined that the video classification labels configured during the fine-tuning stage are inconsistent with the classification labels configured in the pre-trained classification network, the pre-trained multimodal video classification model is fine-tuned, and the input unit performs the following operations to obtain the target multimodal video classification model:

[0065] Obtain a set of fine-tuning sample data, wherein each set of fine-tuning sample data includes a fine-tuning sample video stream, as well as fine-tuning sample text information and fine-tuning sample classification labels associated with the fine-tuning sample video stream;

[0066] Based on the video classification labels during the fine-tuning phase, adjust the classification labels configured in the pre-trained classification network.

[0067] Using the aforementioned fine-tuned sample data set, the adjusted multimodal video classification model is trained through multiple rounds of fine-tuning to obtain the fine-tuned target multimodal video classification model.

[0068] Optionally, the acquisition unit is used to perform the following operations to determine the text information associated with the video stream to be processed:

[0069] Obtain the title information associated with the video stream to be processed;

[0070] For each video frame to be processed, a text recognition algorithm is used to identify the video frame text information included in the video frame to be processed;

[0071] The title information and the identified text information of each video frame are used as the text information to be processed associated with the video stream to be processed.

[0072] Optionally, the device further includes a pushing unit, the pushing unit being used for:

[0073] Based on the video classification results associated with each video stream to be processed, the video streams to be processed are clustered and presented, wherein the video classification results include corresponding classification labels;

[0074] In response to a video push request triggered by a target object for a target category tag, the target video associated with the target category tag is pushed to the target object.

[0075] Fifthly, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in either the first or second aspect above.

[0076] In a sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the method described in either the first or second aspect above.

[0077] In a seventh aspect, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements the method described in either the first or second aspect above.

[0078] The beneficial effects of this application are as follows:

[0079] This application provides a pre-training method, video classification method, and apparatus for a multimodal video classification model. During pre-training, self-supervised training is first performed separately on the video feature extraction network and the text feature extraction network. Then, supervised training is performed on an initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network obtained from the self-supervised training. This allows the pre-training process to be divided into two training stages. The initial self-supervised training process independently learns the ability to extract single-modal features from single-modal video streams and text information. Combined with the subsequent supervised training process, it learns the ability to classify videos based on video and text feature information. This enables the pre-trained multimodal video classification model to fully learn the data information provided by the pre-training sample dataset, learn more about the prior information included in the pre-training sample data, and learn the combined effects of various modalities, thus improving model performance and video modeling capabilities.

[0080] Furthermore, during the video classification process, after obtaining the video stream to be processed and the associated text information to be processed, a target multimodal video classification model is obtained by fine-tuning the pre-trained multimodal video classification model. Based on the video stream to be processed and the text information to be processed, the video classification result is output.

[0081] Thus, for the multimodal video classification model obtained by adopting a phased pre-training method, the target multimodal video classification model can be obtained by fine-tuning the pre-trained multimodal video classification model, which has fully learned prior knowledge during the pre-training process. This can greatly improve the efficiency of video classification and achieve effective processing of video data. Attached Figure Description

[0082] Figure 1 This is a schematic diagram illustrating one possible application scenario in the embodiments of this application;

[0083] Figure 2 This is a schematic diagram of the process of pre-training a multimodal video classification model by the processing device in an embodiment of this application;

[0084] Figure 3 This is a schematic diagram illustrating the process by which the processing device generates a pre-training sample data set based on historical video data from multiple video sources in an embodiment of this application.

[0085] Figure 4 This is a schematic diagram of one process of deleting redundant data in an embodiment of this application;

[0086] Figure 5 This is a schematic diagram of the process for generating pseudo-classification results in the embodiments of this application;

[0087] Figure 6 This is a schematic diagram illustrating the process of performing one round of self-supervised training on the video feature extraction network in an embodiment of this application;

[0088] Figure 7 This is a schematic diagram illustrating the division of a video stream into multiple segments in an embodiment of this application;

[0089] Figure 8 This is a schematic diagram of the self-supervised training process within the video feature extraction network in this application.

[0090] Figure 9 This is a schematic diagram of the structure of the text feature extraction network implemented in this application;

[0091] Figure 10 This is a schematic diagram illustrating the stages of training the target multimodal video classification model in the embodiments of this application;

[0092] Figure 11 This is a schematic diagram illustrating a round of supervised training in an embodiment of this application;

[0093] Figure 12 This is a schematic diagram illustrating the process of video classification performed by the processing device in an embodiment of this application;

[0094] Figure 13 This is a schematic diagram illustrating the video classification process in an embodiment of this application;

[0095] Figure 14 This is a schematic diagram illustrating the presentation of video content in response to a search operation on a relevant object in an embodiment of this application.

[0096] Figure 15 This is a schematic diagram of the logical structure of the pre-training device for the multimodal video classification model in the embodiments of this application;

[0097] Figure 16 This is a schematic diagram of the logical structure of the video classification device in the embodiments of this application;

[0098] Figure 17 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application;

[0099] Figure 18 This is a schematic diagram of the structure of a computing device in an embodiment of this application. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0101] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0102] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0103] Pre-training: This involves training a pre-trained model through self-supervised or semi-supervised learning, enabling the model to learn general or prior knowledge for application in downstream tasks. In this application example, the pre-training process specifically involves first performing self-supervised training on the video feature extraction network to obtain an initial video feature extraction network; then performing self-supervised training on the text feature extraction network to obtain an initial text feature extraction network; and finally, performing supervised training on the initial multimodal video classification model, which includes at least the initial video and text feature extraction networks, to obtain the pre-trained multimodal video classification model, thus completing the pre-training process.

[0104] Fine-tuning training refers to adaptive training of the pre-trained model based on the actual task processing needs before the model is used to perform downstream tasks. In this embodiment, it refers to adaptive training of the pre-trained multimodal video classification model when performing video classification tasks.

[0105] Self-supervised training refers to a training method that uses unlabeled sample corpora to train a model or network structure. In this embodiment, it refers to using unlabeled sample video streams to train a video feature extraction network and using unlabeled sample text information to train a text feature extraction network.

[0106] Multimodal: In this embodiment of the application, it refers to the content information of multiple existence states included in the video data. In the processing of this application, only the video stream and the text information determined based on the video stream are considered as multimodal content for the video.

[0107] Multimodal (cross-stream) video classification model: In the embodiments of this application, the model used to implement the video classification task provides dual channels for modal feature extraction during the specific processing, extracting video feature information and text feature information respectively, and realizing video classification based on the fused video feature information and text feature information.

[0108] Video classification results: In this embodiment of the application, the video classification results can be used to determine the classification label corresponding to the video, and the classification label corresponds to the classification category divided for the video content; in other words, the video classification results can be used to characterize the core content included in the video. For example, for a news video about migratory birds, the corresponding classification label may be: "nature", "migratory birds", "migration", etc.

[0109] The design concept of the embodiments of this application is briefly introduced below:

[0110] In order to effectively classify the massive amount of new videos added every day, a pre-trained model with a multimodal dual-stream structure can be used to perform specific video classification tasks.

[0111] However, pre-trained models obtained using existing technologies only focus on learning to establish the correspondence between videos and text, and therefore cannot achieve efficient video classification in specific video classification tasks.

[0112] In view of this, this application proposes a pre-training method, a video classification method, and an apparatus for a multimodal video classification model. After obtaining a pre-training sample data set, the text feature extraction network is first trained in multiple rounds using sample video streams from the pre-training sample data set to obtain an initial video feature extraction network after training. Then, the text feature extraction network is trained in multiple rounds using sample text information from the pre-training sample data set to obtain an initial text feature extraction network after training. Finally, the initial multimodal video classification model, including the initial video feature extraction network and the initial text feature extraction network, is trained in multiple rounds using the pre-training sample data set to obtain a pre-trained multimodal video classification model.

[0113] In this way, by performing self-supervised training separately for the video feature extraction network and the text feature extraction network during pre-training, and then performing supervised training on the initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network obtained from the self-supervised training, the pre-training process can be specifically divided into two training stages. The first self-supervised training process enables the model to independently learn the ability to extract single-modal features from single-modal video streams and text information; combined with the subsequent supervised training process, it can learn the ability to classify videos based on video and text feature information. This allows the pre-trained multimodal video classification model to fully learn the data information provided by the pre-training sample dataset, learn more about the prior information included in the pre-training sample data, and learn the joint effects of various modal data, which helps to improve model performance and video modeling capabilities.

[0114] Furthermore, during the video classification process, after obtaining the video stream to be processed and the associated text information to be processed, a target multimodal video classification model is obtained by fine-tuning the pre-trained multimodal video classification model. Based on the video stream to be processed and the text information to be processed, the video classification result is output.

[0115] Thus, for the multimodal video classification model obtained by adopting a phased pre-training method, the target multimodal video classification model can be obtained by fine-tuning the pre-trained multimodal video classification model, which has fully learned prior knowledge during the pre-training process. This can greatly improve the efficiency of video classification and achieve effective processing of video data.

[0116] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0117] See Figure 1 The diagram shown is a possible application scenario illustration in the embodiments of this application. The application scenario illustration includes a processing device 110 and a terminal device 120 (including terminal device 1201, terminal device 1202... terminal device 120n).

[0118] It should be noted that, Figure 1 The number of processing devices 110 and terminal devices 120 shown in the schematic diagram of the application scenario is for illustrative purposes only. The processing devices 110 and terminal devices 120 can communicate with each other through wired or wireless networks.

[0119] In this embodiment, the processing device 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. It can also be an electronic device, including but not limited to mobile phones, computers, smart voice interaction devices, e-book readers, smart home appliances, vehicle terminals, and aircraft, etc., which are computer devices with certain computing capabilities; or it can be an electronic device with image acquisition capabilities. This embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving scenarios.

[0120] Terminal device 120 can specifically be any one or a combination of mobile phones, computers, smart voice interaction devices, e-book readers, smart home appliances, vehicle terminals, aircraft, cameras, and other device terminals capable of capturing images or videos.

[0121] The technical solution proposed in this application can perform targeted processing based on the obtained video data in various application scenarios. The video classification method proposed in this application can be applied to processing devices or terminal devices. The following description will only take the processing device as the main body for video classification processing as an example.

[0122] Scenario 1: Applied to classifying live videos uploaded by broadcasters.

[0123] Specifically, in various live streaming scenarios, including game live streaming, the processing device acquires the live video stream uploaded by the streamer on the terminal device; and uses a speech recognition algorithm to identify the speech and text information in the speech; and based on the live video stream and text information, generates a fine-tuned sample data set for the live streaming scenario, and then determines the video classification result corresponding to the live video stream based on the target multimodal video classification model obtained by fine-tuning.

[0124] Scenario 2: Applied to processing self-made video data from terminal devices.

[0125] The processing device can receive videos shot by the terminal device itself and classify and process the videos. The video streams received by the processing device may be news video streams, video streams corresponding to self-shot short videos, etc.

[0126] Specifically, taking the processing of news video streams as an example, the processing device can receive news video streams sent by news terminals and title information configured for the news video streams; and use text recognition algorithms to identify the video frame text information included in each news video frame in the news video stream; then, based on the news video stream, as well as the title information associated with the news video stream and the text information of each video frame, it uses a fine-tuned sample data set generated for the application scenario, and finally fine-tunes the target multimodal video classification model to determine the video classification result corresponding to the news video stream.

[0127] Scenario 3: Applied to processing security video data.

[0128] Specifically, when the security video stream includes subtitles, text recognition can be performed on each security video frame in the security video stream to obtain the associated text information; when the security video stream does not include subtitles, the processing device can also acquire the collected audio and extract the audio-text information from the audio, that is, convert the audio associated with the security video stream into audio-text information. The security video stream can be promotional data for safe driving, or traffic video streams released by traffic management departments, etc.

[0129] Furthermore, based on the security video stream and text information, a fine-tuned sample data set generated for the corresponding scenario is used to fine-tune the target multimodal video classification model to determine the video classification result corresponding to the security video stream. Here, converting speech into text information is a conventional technique in this field, and this application will not impose specific limitations on it.

[0130] It should be noted that in some possible application scenarios of this application, the target multimodal video classification model can be installed on the processing device 110, so that the processing device 110 can directly use the target multimodal video classification model to process the video stream obtained directly or indirectly. The target multimodal video classification model can be trained by the processing device 110 itself, or it can be trained by other devices and provided to the processing device 110. The video recognized by the processing device 110 may be collected by itself, or it may be obtained from other devices.

[0131] The following explanation will use the training of a target multimodal video classification model using a processing device as an example to illustrate the relevant training process in detail.

[0132] In addition, in this embodiment of the application, the training of the target multimodal video classification model by the processing device can be a periodic process according to the actual processing needs. The training samples can be regenerated periodically, and a pre-training process and a fine-tuning training process can be performed to obtain the target multimodal video classification model.

[0133] The pre-training process before generating the target multimodal video classification model in this embodiment will be described below with reference to the accompanying drawings:

[0134] See Figure 2 As shown, this is a flowchart illustrating the pre-training process of the multimodal video classification model in an embodiment of this application. The following is a detailed explanation in conjunction with the attached diagram. Figure 2 The process of pre-training the multimodal video classification model using the processing equipment is explained below:

[0135] Step 201: The processing device acquires the pre-training sample data set.

[0136] In this embodiment of the application, when the processing device obtains the pre-training sample data set, it can specifically obtain the pre-training sample data set constructed by itself or other devices.

[0137] The following two possible methods for generating pre-training sample datasets illustrate the process of how the processing device automatically generates pre-training sample datasets:

[0138] Method 1: The processing device generates a pre-training sample data set based on historical video data from a single video source.

[0139] The processing device can use historical video data from a single video source as pre-training sample data to generate a pre-training sample data set, wherein each pre-training sample data set includes a sample video stream, as well as sample text information and sample classification labels associated with the sample video stream.

[0140] It should be noted that a video source specifically refers to the origin of video data, and the video source itself has classification tags for classifying video streams. For example, a video source can specifically refer to a video playback application that maintains a collection of video data.

[0141] In addition, the number of sample classification labels can be set according to actual processing needs. Considering that the technical solution proposed in this application aims to achieve detailed classification of video data so as to effectively define the content in the video stream, the number of sample classification labels can be set to multiple.

[0142] For example, assuming the processing device itself is an associated device of a video playback application X, then the processing device can acquire the video data it maintains; and then the processing device can generate a pre-training sample data set based on the video data in the video playback application X.

[0143] It should be noted that, depending on the actual processing needs, the historical text information associated with the historical video stream may specifically include at least one or a combination of the title information, subtitle information, and information obtained from speech conversion associated with the historical video stream.

[0144] In some possible embodiments of this application, subtitle information in the video stream can be used as the associated text information. When acquiring subtitle information, the processing device can use an Optical Character Recognition (OCR) algorithm to process each historical video frame in the historical video stream and identify the subtitle information included in the historical video frames. Then, the identified subtitle information can be determined as the text information associated with the historical video stream. Subsequently, when generating pre-training sample data, a historical video stream is used as a sample video stream, and the historical text information associated with the historical video stream and the original classification label are used as the sample text information and sample classification label associated with the sample video stream.

[0145] In addition, since OCR algorithms are mostly used to recognize subtitle information in frame images, the text information that can be recognized at this time is the subtitle information in the video frame. Therefore, in the scenario where this application only processes video streams with subtitles, only the OCR algorithm can be used to effectively extract the text information.

[0146] Optionally, considering that the text information may be the same in different historical video frames, redundant text information can be identified and deleted by using a duplicate text recognition algorithm based on relevant technologies. Specifically, the duplicate text recognition algorithm can be the simhash algorithm.

[0147] Specifically, in scenarios where the video stream does not include subtitles, speech-to-text algorithms based on relevant technologies can be used to extract text information from the speech associated with the video stream.

[0148] Furthermore, after using an OCR algorithm to recognize the text information in each video frame, considering that different video frames may contain duplicate text information, a duplicate text recognition algorithm can be used to remove redundant text information. In addition, the method for determining text information illustrated in Method 1 is also applicable to the process of determining text information in Method 2 below, and will not be specifically described further in this application.

[0149] Method 2: The processing device generates a pre-training sample data set based on historical video data from multiple video sources.

[0150] See Figure 3 As shown, this is a schematic diagram illustrating the process by which the processing device generates a pre-training sample data set based on historical video data from multiple video sources in an embodiment of this application. The following is a detailed explanation in conjunction with the attached diagram. Figure 3 The process of generating a pre-training sample dataset using processing equipment is described below:

[0151] Step 2011: The processing device constructs a subset of historical video data corresponding to each video source based on the historical video data corresponding to each video source.

[0152] Specifically, the processing device obtains historical video data from each video source and clusters the historical video data belonging to the same video source into a subset of historical video data; thus, it obtains each subset of historical video data obtained from each video source, wherein each historical video data includes a historical video stream, as well as the historical text information and original classification label associated with the historical video stream.

[0153] Optionally, in this embodiment of the application, in order to avoid redundant data in the subsequently generated pre-training sample data set, the processing device can generate an initial video data set based on the initial video data obtained from each video source; then, it can perform deduplication processing on each initial video data in the initial video data set to delete redundant data in the initial video data set. The criterion for determining redundant data can be: the video feature similarity between the initial video stream in one initial video data read from the initial video data set and the initial video streams in each of the other initial video data in the initial video data set reaches a preset similarity threshold, wherein the similarity threshold is set according to actual processing needs.

[0154] When the processing device performs the operation of removing redundant data in the initial video data set, the processing device can first generate an initial video data set based on each initial video data in each video source whose release time is within a preset time range. Each initial video data set includes an initial video stream, as well as historical text information and original classification tags associated with the initial video stream.

[0155] Specifically, when the processing device obtains initial video data from various video sources, it can set a preset time range for determining the initial video data according to the actual processing needs, and then obtain video data whose release time is within the preset time range from each video source as the initial video data.

[0156] Then, the processing device sequentially reads one initial video data from the initial video data set and performs the following deduplication process based on the read initial video data until there is no unread initial video data in the initial video data set: determining the video feature similarity between the initial video stream in the read initial video data and the initial video stream in other initial video data in the initial video data set; and deleting other initial video data in the initial video data set whose video feature similarity with the read initial video data meets the similarity threshold.

[0157] In this embodiment, before calculating video feature similarity, the processing device needs to calculate the video feature information of the initial video stream. Specifically, the processing device can use a preset frame feature fusion algorithm to fuse the feature information of each initial video frame included in each initial video stream in the initial video data, and obtain the embedding result of the initial video stream in the high-dimensional space, that is, obtain the video feature information corresponding to the initial video stream. The frame feature fusion algorithm used can be Net Vector of Locally Aggregated Descriptors (Netvlad), Nextvlad, etc.

[0158] In addition, when calculating the video feature similarity between video feature information, the processing device can use conventional similarity calculation methods to calculate the video feature similarity between video feature information. Furthermore, when performing deduplication based on the currently read initial video data, other initial video data whose video feature similarity with the read initial video data meets a similarity threshold are deleted from the initial video data set. These other initial video data, whose video feature similarity with the read initial video data reaches a preset similarity threshold, can be considered redundant data of that initial video data. The similarity threshold is set according to actual processing needs. The method for calculating the video feature similarity between video features can be: calculating the cosine similarity between video features, or Euclidean distance, etc.

[0159] Furthermore, the processing device uses the initial video data in the deduplicated initial video data set as historical video data; and according to the video source to which each historical video data belongs, it clusters the historical video data to obtain the historical video data subset corresponding to each video source.

[0160] Specifically, the processing device takes the initial video data remaining after removing redundant data from the initial video data set as historical video data; then, according to the video source to which the initial video data belongs, it clusters each historical video data to obtain a historical video data subset corresponding to each video source, where each historical video data subset corresponds to one video source.

[0161] For example, see Figure 4 The diagram shown illustrates a single step of the process for deleting redundant data in an embodiment of this application. Assuming the similarity threshold set for video feature similarity is 0.9, according to... Figure 4 As illustrated, assuming the initial video streams corresponding to each initial video data in the currently established initial video dataset are: initial video stream 1, initial video stream 2, initial video stream 3... initial video stream 10; assuming that initial video stream 1 has been read, then calculating the video feature similarity between initial video stream 1 and initial video streams 2-10 respectively will yield... Figure 4 The similarity of the features of each video stream shown is calculated. It is determined that the similarity of the video features between initial video streams 4, 8, and 9 and initial video stream 1 reaches a set threshold of 0.9, meaning that initial video streams 4, 8, and 9 can be considered redundant data of initial video stream 1. The initial video data corresponding to initial video streams 4, 8, and 9 are then deleted from the initial video data set. Subsequently, one initial video stream can be read from each of the remaining unread initial video streams, and the above operation can be repeated until there are no unread initial video streams in the deduplicated initial video data set.

[0162] Thus, considering that the initial video data obtained from various data sources may contain initial video data with the same content, resulting in a large amount of redundant data in the initial video data, it is possible to filter the obtained initial video streams by calculating the video feature similarity between them, so as to avoid reducing the training efficiency of the relevant models due to redundant sample video data in the subsequently generated pre-training video data set.

[0163] It should be noted that the content on deleting redundant data shown in Method 2 of this application can also be applied to the processing in Method 1. Furthermore, in the process of calculating video feature similarity by reading the initial video stream, the video feature similarity between the same two initial video streams can be avoided by not calculating it repeatedly. Specifically, when calculating the video feature similarity between initial video streams, the video feature similarity between the currently read initial video stream and each of the remaining unread initial video streams can be calculated, thus avoiding redundant calculations.

[0164] This can reduce the amount of computation to some extent, improve processing efficiency, and reduce computation time to some extent, thus saving computing resources.

[0165] Step 2012: For each video source, the processing device performs supervised training based on the historical video data corresponding to the video source to obtain the target pseudo-classification model corresponding to the video source.

[0166] In this embodiment of the application, after the processing device obtains a subset of historical video data from each video source, it performs supervised iterative training on each subset of historical video data to obtain the corresponding target pseudo-classification model. Each historical video data includes a historical video stream, as well as historical text information and original classification labels associated with the historical video stream; one video source corresponds to one target pseudo-classification model.

[0167] Specifically, for each video source, the processing device acquires historical video data from the corresponding historical video data subset, and performs supervised training on the initial pseudo-classification model based on the acquired historical video data to obtain the trained target pseudo-classification model. Here, a target pseudo-classification model is trained from a subset of historical video data of a video source.

[0168] It should be noted that, in the embodiments of this application, the initial pseudo-classification model may adopt a single-modal structure, such as Nextvlad or Swin Transformer 3d, or it may adopt the multi-modal structure proposed in this application. No specific limitations are imposed here. Therefore, when performing supervised training on the initial pseudo-classification model to obtain the target pseudo-classification model, the corresponding sample data is obtained according to the actual model architecture. For example, for a single-modal initial pseudo-classification model, training can be completed based solely on the single-modal historical video stream and the associated original classification labels.

[0169] In this embodiment of the application, when the processing device configures an initial pseudo-classification model for a subset of historical video data, it first analyzes the original classification labels associated with each historical video stream in the subset of historical video data to determine the total number and content of the classification labels for each historical video stream; then, it configures the total number of classification labels and each classification label in the classification network of an initial pseudo-classification model to obtain the configured initial pseudo-classification model.

[0170] Then, the processing device uses a subset of historical video data used when configuring the initial pseudo-classification model to train the corresponding initial pseudo-classification model until the preset convergence condition is met, thus obtaining the trained target pseudo-classification model.

[0171] It should be noted that the preset convergence condition can be any of the following: the number of training rounds reaches a preset first value; the number of times the calculated loss value is continuously lower than a second value reaches a third value. The loss value used to calculate the loss function is determined according to the actual model structure adopted. In addition, considering that the initial pseudo-classification model is constructed using a conventional model structure in the art in this embodiment, the specific loss function used is not specifically limited here.

[0172] For example, suppose the processing device generates three historical video data subsets based on video data acquired from three video sources: Video Source 1 - Historical Video Data Subset A, Video Source 2 - Historical Video Data Subset B, and Video Source 3 - Historical Video Data Subset C. Then, by analyzing the original classification labels associated with each historical video stream in Historical Video Data Subset A, the processing device determines the total number of categories and corresponding classification labels for classifying the historical video streams in Video Source 1. It then assigns the determined total number of categories and classification labels to an initial pseudo-classification model with a Nextvlad structure and trains the initial pseudo-classification model using Historical Video Data Subset A to obtain the trained target pseudo-classification model 1. Similarly, corresponding target pseudo-classification models can be generated for Video Source 2 and Video Source 3 respectively.

[0173] In this way, by training corresponding target pseudo-classification models for each subset of historical video data, the trained target pseudo-classification models can classify video streams according to the classification methods in the corresponding video sources, which is equivalent to learning the ability to classify according to the classification labels of different video sources.

[0174] Step 2013: For each historical video stream, the processing device determines the pseudo-classification results associated with each historical video stream based on each target pseudo-classification model, and uses the union of the original classification labels associated with the historical video stream and each pseudo-classification result as the sample classification label associated with the historical video stream.

[0175] For details, please refer to Figure 5 As shown, this is a flowchart illustrating the process of generating pseudo-classification results in an embodiment of this application. The following explanation, without distinguishing the subset of historical video data to which a historical video stream belongs, uses the determination of each pseudo-classification result associated with each historical video stream as an example to illustrate the process of generating a pseudo-classification result associated with a historical video stream:

[0176] Step 2013-a: The processing device determines the target video source corresponding to the historical video stream, determines the target pseudo-classification model corresponding to the target video source, and obtains other target pseudo-classification models corresponding to each other video source other than the target video source.

[0177] Specifically, after the processing device determines the target video source corresponding to the historical video stream, it determines the target pseudo-classification model corresponding to the target video source; then, among the target pseudo-classification models corresponding to each video source, it determines each pseudo-classification model other than the target pseudo-classification model corresponding to the target video source.

[0178] Step 2013-b: The processing device uses various other target pseudo-classification models to determine the pseudo-classification results associated with the historical video stream.

[0179] Specifically, after determining the pseudo-classification models for each other target, the processing device uses these models to determine the pseudo-classification results corresponding to the historical video stream. Then, each pseudo-classification result determined for the historical video stream is used as a pseudo-classification result associated with that historical video stream. Each pseudo-classification result includes a pseudo-classification label and a predicted classification value obtained based on the pseudo-classification label.

[0180] It should be noted that when the processing device uses various other target pseudo-classification models to determine the pseudo-classification labels corresponding to the historical video stream, it can determine the form of the input data based on the model structure of the target pseudo-classification model.

[0181] For example, assuming that the classification of the target pseudo-classification model depends on the simultaneous input of video streams and text information, then text information can be extracted based on historical video streams to determine the text information associated with the historical video streams; then the determined text information and historical video streams can be input together into the target pseudo-classification model to obtain the corresponding pseudo-classification result.

[0182] In this way, by using the target pseudo-classification models corresponding to other video sources, when classifying historical video streams belonging to a video source, it is possible to comprehensively classify the historical video stream according to the classification methods corresponding to other video sources, refine the classification content of the historical video stream, and help obtain more tag information to describe the content in the historical video stream.

[0183] Furthermore, the processing device takes the union of the pseudo-classification results associated with a historical video stream and the original classification labels as the sample classification labels associated with that historical video stream.

[0184] Optionally, considering that some noise is inevitable during data processing, in order to minimize the possibility of incorrectly labeled original classification tags in historical video streams, after determining the pseudo-classification results corresponding to the historical video stream, the processing device can filter the original classification tags associated with the historical video stream based on the pseudo-classification results associated with the historical video stream.

[0185] Specifically, the processing device obtains the initial intersection between the pseudo-classification labels included in each pseudo-classification result and the original classification labels associated with the historical video stream, and determines the classification prediction value corresponding to each classification label in the initial intersection; then, from the original classification labels associated with the historical video stream, it deletes the classification labels whose classification prediction values ​​in the initial intersection meet the label selection criteria, and obtains the processed original classification labels; then, for the processed original classification labels, it performs the step of taking the union of the original classification labels associated with the historical video stream and each pseudo-classification result as the sample classification label associated with the historical video stream.

[0186] In this embodiment, the processing device can use pseudo-classification results as a reference when processing the original classification labels associated with historical video streams. After determining the initial intersection between each pseudo-classification label associated with the historical video stream and the original label, the device determines the classification prediction value corresponding to each classification label in the initial intersection. Then, by determining the matching between the classification prediction value and the label filtering condition, the device deletes the classification labels whose corresponding classification prediction values ​​meet the label filtering condition from the original classification labels associated with the historical video stream, and obtains the processed original classification labels. The label filtering condition can be that the classification prediction value is lower than a set threshold value.

[0187] Thus, considering the accuracy of the classification labels associated with historical video streams, the performance of subsequent models on the pre-trained video dataset will be greatly affected, thereby impacting the learning and classification judgment of subsequent models. The pseudo-label filtering mechanism adopted in this application can filter out all classification labels in the original classification labels whose corresponding classification prediction values ​​are lower than a set threshold; it also deletes all classification labels in the original classification labels that are less likely to be identified by the target pseudo-classification model, making the remaining original classification labels more reliable and reducing the interference caused by mislabeled classification labels to a certain extent.

[0188] Furthermore, the processing device takes the union of each pseudo-classification result associated with the historical video stream and the processed original classification label as the sample classification label associated with the historical video stream.

[0189] In this way, the sample classification labels generated for historical video streams integrate classification labels of different granularities corresponding to each video source, which can comprehensively associate the classification labels corresponding to the historical video streams as much as possible, and to a certain extent compensate for the incompleteness and one-sidedness of the label information of a single video source.

[0190] Step 2014: The processing device generates a pre-training sample data based on each historical video stream, as well as the sample classification labels and historical text information associated with that historical video stream, and generates a pre-training sample data set based on each pre-training sample data.

[0191] Specifically, without distinguishing the video stream to which the historical video stream belongs, the processing device generates a corresponding pre-training sample data based on each historical video stream, as well as the sample classification labels and historical text information associated with that historical video stream.

[0192] Furthermore, the processing device generates a pre-training sample data set based on the generated pre-training sample data.

[0193] In this way, when obtaining historical video data from multiple video sources, corresponding target pseudo-classification models are trained and generated using the historical video data from each video source, gaining the ability to classify videos according to the classification methods of each video source; and in the process of generating different pseudo-classification results for each historical video stream using each target pseudo-classification model, different video classification methods are integrated, so that the fused pre-training sample data can eliminate the classification differences between video data obtained from different video sources; thus achieving diverse classification of video streams and providing a basis for the supervisory role played in the subsequent pre-training process.

[0194] Step 202: The processing device uses the sample video streams included in each sample data in the pre-trained sample dataset to perform multiple rounds of self-supervised training on the video feature extraction network to obtain the initial video feature extraction network after training.

[0195] In this embodiment, after obtaining the pre-training sample data set, the processing device uses each sample video stream in the pre-training sample data to perform multiple rounds of self-supervised training on the video feature extraction network to obtain the initial trained video feature extraction network. Here, one round of self-supervised training on the video feature extraction network refers to the training process of inputting a batch of sample video streams into the video feature extraction network. The number of sample video streams input in a batch is set according to actual processing needs, and this application does not make a specific limitation.

[0196] See Figure 6 As shown, this is a flowchart illustrating a round of self-supervised training of the video feature extraction network in an embodiment of this application. The following is a detailed explanation in conjunction with the attached diagram. Figure 6 A schematic diagram illustrating the operations performed by the processing device during one round of training of the initial video feature extraction network in this embodiment of the application:

[0197] Step 2021: The processing device reads a specified number of sample video streams from the pre-trained sample data set, and extracts a specified number of sample video frames with associated time-series information from each of the read sample video streams through a video feature extraction network.

[0198] Specifically, the processing device reads a specified number of sample video streams from the pre-trained sample dataset based on the number of sample video streams included in a batch, and determines the duration and frame rate information corresponding to each sample video stream. The duration information is used to indicate the total duration of the video stream, and the frame rate information is used to indicate the number of frames at a specified time granularity. For example, if the unit of frame rate is f / s, it represents the number of video frames included in 1 second.

[0199] Then, the processing device uses the initially established video feature extraction network to perform frame extraction for each sample video stream in the current batch input, thereby extracting a specified number of sample video frames with associated time-series information from each of the read sample video streams.

[0200] Specifically, the processing device can use a video feature extraction network to extract a specified number of sample video frames with associated time-series information from each of the obtained sample video streams. The video feature extraction network can select an appropriate frame extraction method for processing based on actual processing needs, taking into account the duration information of the video stream and the specific downstream tasks. The following two possible frame extraction methods are used as examples to illustrate the process of implementing frame extraction processing.

[0201] It should be noted that in different rounds of iterative training in this application embodiment, the number of sample video frames extracted from the sample video stream may be different. In other words, in this application embodiment, a multi-scale temporal input method is adopted when extracting sample video frames. Different numbers of frames can be randomly extracted in each iteration to increase the diversity of input samples. Here, multi-scale temporal input means that the number of sample video frames extracted from the sample video stream in each iteration is diverse. This enables the video feature extraction network to learn to flexibly process sample video streams of various durations and to perform targeted processing on sample video streams with different numbers of extracted frames in different iteration rounds. The number of frames that may be extracted is set according to the actual processing needs, such as any number from 4, 8, 12, 16, 24, 32, etc.

[0202] Frame extraction method one: continuously extract a specified number of sample video frames starting from the target time point.

[0203] Specifically, for each sample video stream read, the processing device sets a target time point for that sample video stream based on its duration information, and extracts a specified number of sample video frames with associated time sequence information, starting from the sample video frame corresponding to the target time point in the sample video stream.

[0204] It should be noted that when using frame extraction method one, frame extraction can start from a specified part of each sample video stream, or it can start from the same playback time position of each sample video stream.

[0205] For example, suppose a batch contains 5 sample video streams, and the duration of sample video stream 1 is 2 minutes and 26 seconds; the duration of sample video stream 2 is 3 minutes and 26 seconds; the duration of sample video stream 3 is 1 minute and 12 seconds; the duration of sample video stream 4 is 2 minutes and 27 seconds; and the duration of sample video stream 5 is 1 minute and 19 seconds. Then, when extracting sample video frames from the middle part of the sample video streams for analysis, the target time point can be determined as 1 minute and 13 seconds for sample video stream 1; 1 minute and 56 seconds for sample video stream 2; 36 seconds for sample video stream 3; approximately 1 minute and 13 seconds for sample video stream 4; and approximately 39 seconds for sample video stream 5.

[0206] For example, assuming the video data frame rate is 32 frames per second, and the number of sample video frames extracted is 32, there are 5 sample video streams in one batch. The duration of sample video stream 1 is 2 minutes and 26 seconds; the duration of sample video stream 2 is 3 minutes and 26 seconds; the duration of sample video stream 3 is 1 minute and 12 seconds; the duration of sample video stream 4 is 2 minutes and 27 seconds; and the duration of sample video stream 5 is 1 minute and 19 seconds. If the same target time point is set for each sample video stream, then 1 second of video stream needs to be acquired. Furthermore, considering that the shortest duration of a sample video stream is 1 minute and 19 seconds, the maximum target time point is 1 minute and 18 seconds. Therefore, the time before 1 minute and 18 seconds can be determined as the target time point, such as 25 seconds.

[0207] In this way, by first randomly selecting a certain time point in the sample video stream and then continuously acquiring sample video frames, the extracted sample video frames can represent the local information in the sample video stream, which is beneficial for generating rich self-supervised training samples.

[0208] Frame extraction method two: split the sample video stream into a specified number of sample video sub-streams, and randomly extract a sample image frame from each sample video sub-stream.

[0209] Specifically, for each sample video stream read, the processing device divides the sample video stream into a specified number of sample video sub-streams according to the duration information corresponding to the sample video stream, and randomly extracts a sample video frame from each sample video sub-stream to obtain a specified number of sample video frames associated with time sequence information.

[0210] It should be noted that the processing device can use a sampling method of temporal-segment-networks (TSN) sampling to divide the sample video stream into multiple clips and randomly extract a frame from each clip.

[0211] For example, see Figure 7 As shown, this is a schematic diagram of dividing a sample video stream into multiple segments in an embodiment of this application. Assuming the duration of a sample video stream 3 is 1 minute 12 seconds, and the specified number of segments for the current batch input is set to 4, then we can obtain... Figure 7 The image shows each clip, and the corresponding playback time range for each clip; then, from each... Figure 7 Within each playback time range shown, a sample video frame is extracted.

[0212] In addition, depending on the actual processing needs, the duration of the playback time range of each clip included in the sample video stream may be different for each sample video stream, which will not be specifically described in this application.

[0213] In this way, by extracting video frames from each clip separately and by combining the method of randomly extracting video frames from each clip, the extracted video frames are distributed across various playback time ranges, thus enabling the extracted video frames to reflect the global characteristics of the sample video stream.

[0214] Step 2022: The processing device controls the video feature extraction network to generate positive sample groups based on each sample video frame belonging to the same sample video stream, and to generate negative sample groups based on each sample video frame belonging to different sample video streams.

[0215] Specifically, in this embodiment of the application, when performing self-supervised training on the video feature extraction network, the pre-training task is as follows: the processing device controls the video feature extraction network to extract frames from each sample video stream in a batch, and then generates a positive sample group based on two sample video frames belonging to the same sample video stream, and generates a negative sample group based on two sample video frames belonging to different sample video streams, wherein a sample group includes two sample video frames.

[0216] Step 2023: The processing device performs multiple rounds of self-supervised iterative training on the video feature extraction network based on the positive sample group and the negative sample group until the preset single-round convergence condition is met.

[0217] Specifically, after the processing device acquires the positive and negative sample groups constructed by the video feature extraction network, it controls the construction of the video feature extraction network to undergo multiple rounds of iterative training based on the positive and negative sample groups until the preset single-round convergence condition is met.

[0218] It should be noted that the single-round convergence condition refers to the condition for determining the end of iterative training based on a batch of input sample video streams during self-supervised training. Specifically, the single-round convergence condition may be: the number of self-supervised training rounds reaches a fourth preset value; the number of times the loss calculated during self-supervised training is continuously lower than a fifth preset value reaches a sixth preset value. The specific values ​​of the fourth, fifth, and sixth preset values ​​are set according to actual processing needs, and this application does not impose specific limitations.

[0219] It should be noted that, in the embodiments of this application, the network structure of the video feature extraction network can specifically be MOBY, Momentum Contrast (MOCO), MOCOV2, and BYOL, etc.

[0220] The following section uses a MOBY architecture as an example to illustrate the training process of a video feature extraction network:

[0221] See Figure 8 As shown, this is a schematic diagram of the self-supervised training process within the video feature extraction network in this application. The following is a related illustration. Figure 8 The relevant self-monitoring process is explained.

[0222] During the self-supervised training of the video feature extraction network, the processing device utilizes the two encoders included in the MOBY structure video feature extraction network: an online encoder and a target encoder. Specifically, the online encoder corresponds to... Figure 8 The encoder and argument encoder in the text correspond to... Figure 8 The two encoders consist of a momentum encoder. Both encoders include a backbone network and a multilayer perceptron (MLP) head. Additionally, the target encoder includes a prediction MLP for prediction to maintain the asymmetry between the online encoder and the target encoder, while obtaining the shift value of the parameter update through momentum updates in each iteration.

[0223] When using MOBY as the network structure for the video feature extraction network, the frame extraction method proposed in the second frame extraction method above can be specifically adopted to control the video feature extraction network to randomly extract sample video frames from a clip of a sample video stream, and then input them into two encoders. For example... Figure 8 As illustrated, the current batch input consists of h sample video streams, and m frames are extracted for each sample video stream. Here, h and m are positive integers set according to actual processing needs. The backbone network in the encoder can be understood as a network structure used for video feature extraction.

[0224] During the self-supervised training process, the processing device controls the video feature extraction network to generate positive and negative sample groups based on the extracted sample video frames. Then, the network outputs corresponding video features based on two sample video frames in a sample group and performs self-supervised training by calculating the contrastive loss between the two video feature information.

[0225] It should be noted that in the embodiments of this application, the positive sample group can be generated based on sample video frames extracted from different clips of the same video; the negative sample group can be generated by continuously adding the obtained keys to a queue for storage during the training process, thus being able to be generated based on sample video frames from different sample video sources. Since the use of the MOBY architecture for self-supervised training is a mature technology in this field, this application will not provide a detailed explanation here.

[0226] Step 203: The processing device uses the sample text information associated with each sample video stream in the pre-trained sample dataset to perform multiple rounds of self-supervised training on the text feature extraction network to obtain the initial text feature extraction network after training.

[0227] Specifically, the processing device can adopt a network structure such as Bidirectional Encoder Representations from Transformers (BERT) or a network structure including a longform structure for text feature extraction to establish a text feature extraction network. The self-supervised training method used in the self-supervised training process of the text feature extraction network is a mature technology in this field, and will not be described in detail in this application.

[0228] For example, see Figure 9 As shown, this is a schematic diagram of the structure of the text feature extraction network in this application. When the processing device uses sample text information from the pre-trained text dataset to perform multiple rounds of self-supervised training on the text feature extraction network, the pre-training task followed can be any one or a combination of the following: randomly covering part of the text content to achieve prediction; combining sample text information belonging to the same sample video stream to generate a positive sample group; and combining sample text information belonging to different sample video streams to generate a negative sample group.

[0229] exist Figure 9 In the illustration, the Masked Language Model (Masked LM) randomly overlays text content, enabling the BERT model to learn and understand contextual relationships. The Next Sentence Prediction (NSP) task allows the BERT model to learn these contextual relationships, where [CLS] represents the sentence label and [SEP] is used to separate sentences. The pre-training techniques related to the BERT model are mature technologies in this field and will not be discussed further in this application.

[0230] It should be noted that the execution order between steps 202 and 203 is flexible in this embodiment. Depending on the actual processing needs, step 203 can be executed first to obtain the initial trained text feature extraction network; then step 202 can be executed to obtain the initial trained video feature extraction network.

[0231] Step 204: The processing device uses a pre-trained sample data set to perform multiple rounds of supervised training on the initial multimodal video classification model, which includes at least an initial video feature extraction network and an initial text feature extraction network, to obtain the pre-trained multimodal video classification model.

[0232] Specifically, after obtaining the initial video feature extraction network and the initial text feature extraction network, the processing device continues to use the pre-trained sample data set to perform multiple rounds of supervised training on the initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network, to obtain the pre-trained multimodal video classification model.

[0233] See Figure 10 As shown, this is a schematic diagram illustrating the stages of training the pre-trained multimodal video classification model in an embodiment of this application. According to... Figure 10 As shown, in the specific training process, sample video streams are first used to perform multiple rounds of iterative self-supervised training on the constructed video feature extraction network to obtain the initial video feature extraction network after self-supervised training. At the same time, sample text information is used to perform multiple rounds of iterative self-supervised training on the constructed text feature extraction network to obtain the initial text feature extraction network after self-supervised training. Then, for the initial multimodal video classification model, which includes at least the initial text feature extraction network and the initial video feature extraction network, multiple rounds of supervised iterative training are performed using pre-training sample data, including sample video streams, sample text information, and sample classification labels, to obtain the pre-trained multimodal video classification model.

[0234] In this embodiment, since the constructed initial multimodal video classification model is used to achieve video classification, the network structure in the constructed initial multimodal video classification model specifically includes: an initial video feature extraction network, an initial text feature extraction network, a feature fusion network for fusing text feature information and video feature information, and a classification network. Specifically, the feature fusion network can be a network structure that implements concat, or a cross-attention network structure, etc.; the classification network may employ a network structure that includes fully connected and sigmoid structures to achieve multi-class classification.

[0235] See Figure 11As shown, it is a schematic diagram of a round of supervised training implemented in an embodiment of this application. Figure 11 As shown, the initial video feature extraction network is the Swin Transformer network; the initial text feature extraction network is the BERT network; the feature fusion network used to fuse text and video feature information is the Cross-attention network; and the classification network is a fully connected and sigmoid structure used to achieve multi-class classification.

[0236] So, combining Figure 11 As shown in this embodiment, during supervised training of the initial multimodal video classification model, assuming a batch contains four pre-training sample data, four sample video streams and four sample text information are read from the pre-training sample set as input information for the initial multimodal video classification model. Specifically, the four sample video streams are input into the initial video feature extraction network (Swin Transformer); and the four sample text information are input into the initial text feature extraction network (BERT). Then, the concatenation result of the video feature information output by the Swin Transformer, along with the text feature information output by the BERT, is input into the feature fusion network (Cross-attention) to obtain fused video and text feature information. The feature fusion result output by the feature fusion network is then input into the classification network to obtain the predicted classification result for each sample video data. Finally, combining the difference between the predicted classification result and the corresponding sample classification label, the BCE loss function is used to calculate the loss value, and the model parameters of the initial multimodal video classification model are adjusted based on backpropagation of the loss value.

[0237] Furthermore, when the processing device determines that the number of supervised training rounds has reached the seventh preset value, or when the number of times the calculated loss value is continuously lower than the eighth preset value has reached the ninth preset value, the initial multimodal video classification model is determined to have converged, and the pre-trained multimodal video classification model is obtained.

[0238] In this way, by performing self-supervised training separately for the video feature extraction network and the text feature extraction network during pre-training, and then performing supervised training on the initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network obtained from the self-supervised training, the pre-training process can be specifically divided into two training stages. The first self-supervised training process enables the model to independently learn the ability to extract single-modal features from single-modal video streams and text information; combined with the subsequent supervised training process, it can learn the ability to classify videos based on video and text feature information. This allows the pre-trained multimodal video classification model to fully learn the data information provided by the pre-training sample dataset, learn more about the prior information included in the pre-training sample data, and learn the joint effects of various modal data, which helps to improve model performance and video modeling capabilities.

[0239] See Figure 12 As shown, this is a schematic diagram of the process of video classification performed by the processing device in an embodiment of this application. The following is a description of the process in conjunction with the attached diagram. Figure 12 The process of implementing video classification in specific applications is explained below:

[0240] Step 1201: The processing device acquires the video stream to be processed and determines the text information to be processed associated with the video stream.

[0241] In this embodiment of the application, the processing device obtains the corresponding video stream to be processed according to the specific application scenario, and then determines the text information to be processed associated with the video stream to be processed.

[0242] Specifically, when the acquired video stream to be processed is associated with title information, the processing device can obtain the title information associated with the video stream to be processed when it obtains the text information to be processed; then, for each video frame to be processed, a text recognition algorithm is used to identify the video frame text information included in the video frame to be processed; and then the title information and the identified text information of each video frame are used as the text information to be processed associated with the video stream to be processed.

[0243] In addition, when the acquired video stream to be processed does not contain title information, the processing device can directly use a text recognition algorithm to identify the video frame text information contained in each video frame to be processed for each video frame to be processed; and at least use the text information of each video frame as the text information to be processed associated with the video stream to be processed. The text recognition algorithm used can be an OCR algorithm.

[0244] Optionally, considering that the text information may be the same in different video frames, redundant text information can be identified and deleted by using a duplicate text recognition algorithm under relevant technology. Specifically, the duplicate text recognition algorithm can be the simhash algorithm.

[0245] Specifically, depending on the actual processing needs, the processing device can identify the audio associated with the video stream to be processed and extract the audio-text information from the audio. In this case, the text information processing method described in step 201 above can be used to determine the text information associated with the video stream to be processed.

[0246] In this embodiment of the application, the processing device can generate text information to be processed based on at least one or a combination of voice text information, text information of each video frame, and title information, according to actual processing needs.

[0247] In this way, by recognizing the text content in the video frame, the text information associated with the video frame to be processed can be determined, providing a processing basis for subsequent multimodal fusion analysis.

[0248] Step 1202: The processing device inputs the data stream to be processed and the text information to be processed into the target multimodal video classification model that has been trained, and obtains the video classification result output by the target multimodal video classification model.

[0249] It should be noted that, in the embodiments of this application, before performing the actual processing task, the processing device can fine-tune the pre-trained multimodal video classification model obtained in the above process according to the actual business processing needs, so as to obtain the fine-tuned target multimodal video classification model.

[0250] Considering the different classification needs in real-world application scenarios, there may be situations where the pre-trained classification network can meet the classification needs of the actual application scenario, and situations where the pre-trained classification network cannot meet the classification needs of the actual application scenario. Therefore, in the actual fine-tuning training process, there are the following two possible fine-tuning training methods:

[0251] It should be noted that, in the embodiments of this application, the pre-trained multimodal video classification network includes at least a pre-trained video feature extraction network, a pre-trained text feature extraction network, and a pre-trained classification network. The pre-trained video feature extraction network is trained from the initial video feature extraction network during the supervised training process in the pre-training stage; the pre-trained text feature extraction network is trained from the initial text feature extraction network during the supervised training process in the pre-training stage; and the pre-trained classification network is trained based on the initial classification network in the initial multimodal video classification model during the supervised training process in the pre-training stage.

[0252] Fine-tuning training method one: Fine-tuning training is performed without changing the structure of the pre-trained multimodal video classification model.

[0253] In the implementation of this application, it is determined that the total number of classification labels and the content of each classification label remain unchanged in actual application compared to the classification labels configured in the pre-training stage. That is, when it is determined that the video classification labels in the fine-tuning stage are consistent with the classification labels in the pre-trained classification network, fine-tuning training can be carried out without modifying the model structure of the pre-trained multimodal video classification model.

[0254] The processing device acquires a set of fine-tuned sample data, wherein each set of fine-tuned sample data includes a fine-tuned sample video stream, as well as the fine-tuned sample text information and fine-tuned sample classification label associated with the fine-tuned sample video stream; then, the fine-tuned sample data set is used to perform multiple rounds of fine-tuning training on the acquired pre-trained multimodal video classification model to obtain the fine-tuned target multimodal video classification model.

[0255] In possible embodiments of this application, the fine-tuning sample data in the fine-tuning sample data set may be obtained from the pre-training sample data set; or, it may be regenerated specifically. The method of generating fine-tuning sample data can refer to the sample data generation process described in step 202, and this application does not impose specific limitations here.

[0256] In this way, while ensuring that the total number and content of classification labels in practical applications are consistent with those constructed during pre-training, fine-tuning can be performed according to actual needs to obtain the target multimodal video classification model for subsequent video classification. Based on this, it can be determined that, based on the pre-trained multimodal video classification model obtained during the pre-training process, only a relatively small number of iterations are needed to obtain a target multimodal video classification model that meets practical requirements.

[0257] The second method of fine-tuning training involves adaptively adjusting the target classification network in the pre-trained multimodal video classification model before fine-tuning the training.

[0258] In this embodiment of the application, when the processing device determines that the total number of classification labels and the content of each classification label have changed in actual application compared to the classification labels configured in the pre-training stage, that is, when it is determined that the video classification labels in the fine-tuning stage are inconsistent with the classification labels in the pre-trained classification network, the processing device needs to perform fine-tuning training while modifying the model structure in the pre-trained multimodal video classification model to obtain the target multimodal video classification model.

[0259] Specifically, the processing device acquires a set of fine-tuning sample data, in which each fine-tuning sample data includes a fine-tuning sample video stream, as well as the fine-tuning sample text information and fine-tuning sample classification label associated with the fine-tuning sample video stream. The processing device then adjusts the classification network in the pre-trained multimodal video classification model based on the classification labels of each video in the fine-tuning stage. Then, using the set of fine-tuning sample data, the adjusted multimodal video classification model is subjected to multiple rounds of fine-tuning training to obtain the target multimodal video classification model after fine-tuning training.

[0260] After determining that the total number and content of classification labels in actual applications have changed, the processing device can generate fine-tuning sample data based on the total number and content of classification labels in actual applications; and adaptively adjust the classification network in the pre-trained multimodal video classification model according to the total number and content of classification labels actually needed; then, using the fine-tuning sample data, perform multiple rounds of fine-tuning training on the adjusted pre-trained multimodal video classification model to obtain the fine-tuned target multimodal video classification model.

[0261] In this way, the pre-trained multimodal video classification model can be adaptively adjusted to meet the needs of actual task processing. Moreover, since the backbone network of the pre-trained multimodal video classification model has already been pre-trained during the pre-training process, that is, the network functions used for extracting video feature information and text feature information have been pre-trained, the subsequent fine-tuning training process only needs to train the mapping ability for new classification labels. This gives the pre-trained multimodal video classification model in this application a very strong generalization ability and simplifies the fine-tuning training process.

[0262] Furthermore, after obtaining the target multimodal video classification model for video classification during actual processing, the processing device inputs the data stream to be processed and the text information to be processed into the target multimodal video classification model to obtain the video classification result output by the target multimodal video classification model.

[0263] Specifically, when the target multimodal video classification model includes at least a target video feature extraction network, a target text feature extraction network, and a target classification network, the processing device inputs the video stream to be processed into the target video feature extraction network and the text information to be processed into the target text feature extraction network. Then, the target video feature extraction network splits the video stream into a specified number of video frames and extracts the video feature information contained in each video frame. The target text feature extraction network also extracts the text feature information from the text. Finally, the target classification network outputs the video classification result based on the text feature information and the video feature information concatenated in temporal order.

[0264] It should be noted that, in the embodiments of this application, during actual task processing, while the target video feature extraction network is performing video processing based on the received video stream to be processed, the processing device controls the target video feature extraction network to splice the various video frames to be processed belonging to the video stream to be processed, thereby splicing the corresponding video feature information. The specific method of extracting each video frame to be processed from the video stream to be processed is the same as the frame extraction method in the pre-training process described above, and this application does not impose specific restrictions here.

[0265] In this way, by leveraging the target multimodal video classification model, the final video classification result can be determined based on the multimodal video stream and text information to be processed.

[0266] Similarly, the processing device can repeatedly perform the above processing procedure on each of the obtained video streams to be processed, and obtain the video classification results corresponding to each video stream to be processed.

[0267] Furthermore, the processing device can cluster and present each video to be processed according to its corresponding video classification results, where the video classification results include corresponding classification tags; then, in response to a video push request triggered by the target object for the target classification tag, the target video associated with the target classification tag is pushed to the target object.

[0268] Thus, the video classification method proposed in this application can be extended to specific video presentation and video push scenarios according to specific application needs, enabling more intuitive classification and presentation of video content. From a visualization perspective, this increases the understandability of video content and helps relevant parties quickly find the video content they need.

[0269] In addition, when testing the technical solution in this application, the applicant compared the performance difference between the existing video classification model directly trained without using the pre-training method proposed in this application and the target multimodal video classification model trained based on the pre-training method proposed in this application, and obtained the comparison results shown in Table 1 below.

[0270] Table 1

[0271] Menthod Precision Recall Existing video classification models 70％ 37.8％ Target multimodal video classification model 70％ 48.73％

[0272] When the applicant uses a test set to test the existing video classification model and the target multimodal video classification model respectively, while ensuring that the precision of the two models is the same, it can be determined that the recall of the existing video classification model is lower than the recall of the target multimodal video classification model of this application. Hereinafter, recall is a common indicator for measuring model performance that is known to those skilled in the art, and will not be described in detail here.

[0273] See Figure 13 As shown, this is a schematic diagram of the video classification process in an embodiment of this application. The following is a description of the process in conjunction with the attached diagram. Figure 13 The process of video classification based on the obtained target multimodal video classification model is explained:

[0274] Suppose the processing device currently obtains a news video stream about the impact of global warming on the survival of polar bears. Then, this news video stream can be used as the video stream to be processed. The device can obtain the title information associated with the video stream to be processed. Then, using OCR technology, the device can perform text recognition on each video frame to be processed to obtain OCR subtitles. The extracted OCR subtitles and titles can then be used as the text information associated with the video stream to be processed.

[0275] Furthermore, after the processing device inputs the video stream to be processed into the target video extraction network of the target multimodal video classification model, it performs frame extraction processing on the video stream to be processed through the target feature extraction network. Assuming that the number of video frames to be processed is set to be 3, then 3 video frames to be processed are extracted from the video stream to be processed. Then, video feature extraction is performed on the extracted video frames to be processed to obtain the corresponding video feature information 1-3.

[0276] At the same time, the processing device inputs the text information to be processed into the target text feature extraction network of the target multimodal video classification model. Through the target text feature extraction network, text features are extracted from the title information and OCR subtitle information included in the text information to be processed, and the text feature information corresponding to the text information to be processed is obtained.

[0277] Then, the processing device inputs the video feature information splicing result obtained by splicing video feature information 1-3, along with the obtained text feature information, into the target feature fusion network to obtain the fusion result of the video feature information splicing result and the text feature information. Afterwards, the processing device inputs the obtained fusion result into the target classification network to obtain the video stream to be processed based on the video modality, and the text information to be processed based on the text modality, determining the video classification results: animals, polar bears, Arctic, glaciers, global warming.

[0278] Further, see Figure 14 As shown, this is a schematic diagram illustrating the presentation of video content in response to a search operation on a related object in an embodiment of this application. Figure 13 The processed video classification results can be used for Figure 13 The news video streams related to the impact of global warming on polar bear survival are categorized with the tags: "animals, polar bears, Arctic, glaciers, global warming". Subsequently, when relevant parties initiate video push requests based on the input target category tags, the system can push video streams associated with those target category tags to those parties.

[0279] according to Figure 14 The content shown, assuming the relevant object is based on "animals," triggers a video search request. The processing device can then retrieve video streams associated with the category tag "animals." Thus, in... Figure 13 News video streams that are tagged "animals" in the final classification results of the processing flow can be displayed on the operation page of the relevant object.

[0280] Based on the same inventive concept, see [reference] Figure 15 As shown, this is a schematic diagram of the logical structure of a pre-training device for a multimodal video classification model in an embodiment of this application. The pre-training device 1500 for the multimodal video classification model includes an acquisition unit 1501, a first training unit 1502, a second training unit 1503, and a third training unit 1504.

[0281] The acquisition unit 1501 is used to acquire a pre-training sample data set; each pre-training sample data set includes a sample video stream, sample text information associated with the sample video stream, and sample classification labels;

[0282] The first training unit 1502 is used to perform multiple rounds of self-supervised training on the video feature extraction network using the sample video streams included in each sample data in the pre-training sample data set, so as to obtain the initial video feature extraction network after training.

[0283] The second training unit 1503 is used to perform multiple rounds of self-supervised training on the text feature extraction network using the sample text information associated with each sample video stream in the pre-training sample dataset, so as to obtain the initial text feature extraction network after training.

[0284] The third training unit 1504 is used to perform multiple rounds of supervised training on the initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network, using a pre-trained sample data set, to obtain the pre-trained multimodal video classification model.

[0285] Optionally, when acquiring the pre-training sample data set, the acquisition unit 1501 is used for:

[0286] Based on the historical video data corresponding to each video source, a subset of historical video data corresponding to each video source is constructed; wherein, each historical video data includes a historical video stream, as well as historical text information associated with the historical video stream and original classification labels;

[0287] For each video source, a pseudo-classification model for the target video source is obtained by supervised training based on the historical video data corresponding to the video source.

[0288] For each historical video stream, based on each target pseudo-classification model, determine each pseudo-classification result associated with the historical video stream, and take the union of the original classification label associated with the historical video stream and each pseudo-classification result as the sample classification label associated with the historical video stream.

[0289] Additionally, a pre-training sample data set is generated based on historical video streams, sample classification labels associated with historical video streams, and historical text information; a pre-training sample data set is generated based on each pre-training sample data set.

[0290] Optionally, when constructing a subset of historical video data corresponding to each video source based on the historical video data corresponding to each video source, the acquisition unit 1501 is used for:

[0291] Based on the initial video data from each video source that were published within a preset time range, an initial video data set is generated. Each initial video data set includes an initial video stream, as well as historical text information associated with the initial video stream and original classification tags.

[0292] Read one initial video data point at a time from the initial video dataset, and perform the following deduplication process based on the read initial video data until there are no unread initial video data points left in the initial video dataset:

[0293] Determine the video feature similarity between the initial video stream in the read initial video data and the initial video stream in other initial video data in the initial video data set; and delete other initial video data in the initial video data set whose video feature similarity with the read initial video data meets the similarity threshold.

[0294] The initial video data in the deduplicated initial video data set is used as historical video data; and based on the video source to which each historical video data belongs, a subset of historical video data corresponding to each video source is obtained by clustering.

[0295] Optionally, when determining the pseudo-classification results associated with historical video streams based on various target pseudo-classification models, the acquisition unit 1501 is used for:

[0296] Determine the target video source corresponding to the historical video stream, determine the target pseudo-classification model corresponding to the target video source, and obtain other target pseudo-classification models corresponding to each other video source besides the target video source;

[0297] Various pseudo-classification models for different targets are used to determine the pseudo-classification results associated with the historical video stream.

[0298] Optionally, the pseudo-classification results include pseudo-classification labels and classification predictions obtained based on the pseudo-classification labels;

[0299] After determining the pseudo-classification results associated with the historical video stream, the acquisition unit 1501 is also used for:

[0300] Obtain the initial intersection between the pseudo-classification labels included in each pseudo-classification result and the original classification labels associated with the historical video stream, and determine the classification prediction value corresponding to each classification label in the initial intersection;

[0301] From the original classification labels associated with historical video streams, delete the classification labels whose classification prediction values ​​in the initial intersection meet the label selection criteria, and obtain the processed original classification labels;

[0302] Next, for the processed original classification labels, the process of taking the union of the original classification labels associated with the historical video streams and each pseudo-classification result is performed as the sample classification labels associated with the historical video streams.

[0303] Optionally, during a round of self-supervised training of the video feature extraction network, the first training unit 1502 is used to perform the following operations:

[0304] Read a specified number of sample video streams from the pre-trained sample data set, and extract a specified number of sample video frames with associated time sequence information from each of the read sample video streams through a video feature extraction network.

[0305] The control video feature extraction network generates positive sample groups based on sample video frames belonging to the same sample video stream, and generates negative sample groups based on sample video frames belonging to different sample video streams. Each sample group includes two sample video frames.

[0306] The video feature extraction network is trained through multiple rounds of self-supervised iterative training based on positive and negative sample groups until the preset single-round convergence condition is met.

[0307] Optionally, when extracting a specified number of sample video frames with associated time-series information from each of the read sample video streams, the first training unit 1502 is used to perform any one of the following operations:

[0308] For each sample video stream read, a target time point is set according to the duration information of the sample video stream, and a specified number of sample video frames with associated time sequence information are extracted starting from the sample video frame corresponding to the target time point in the sample video stream.

[0309] For each sample video stream read, the sample video stream is divided into a specified number of sample video sub-streams according to the duration information corresponding to the sample video stream, and a sample video frame is randomly extracted from each sample video sub-stream to obtain a specified number of sample video frames with associated time sequence information.

[0310] Based on the same inventive concept, see [reference] Figure 16 As shown, this is a schematic diagram of the logical structure of a video classification device in an embodiment of this application. The video classification device 1600 includes an acquisition unit 1601 and an input unit 1602, wherein...

[0311] Acquisition unit 1601: used to acquire the video stream to be processed and determine the text information to be processed associated with the video stream to be processed;

[0312] The input unit 1602 is used to input the data stream to be processed and the text information to be processed into the target multimodal video classification model that has been trained, and to obtain the video classification result output by the target multimodal video classification model. The target multimodal video classification model is obtained by fine-tuning the pre-trained multimodal video classification model for any of the above-mentioned first aspects.

[0313] Optionally, the target multimodal video classification model includes at least a target video feature extraction network, a target text feature extraction network, and a target classification network; when inputting the data stream to be processed and the text information to be processed into the trained target multimodal video classification model to obtain the video classification label output by the target multimodal video classification model, the input unit 1602 is used for:

[0314] The video stream to be processed is input into the target video feature extraction network, and the text information to be processed is input into the target text feature extraction network.

[0315] The target video feature extraction network splits the video stream to be processed into a specified number of video frames to be processed and extracts the video feature information contained in each video frame to be processed. The target text feature extraction network extracts the text feature information in the text to be processed.

[0316] A target classification network is used to output video classification results based on text feature information and video feature information spliced ​​in chronological order.

[0317] Optionally, when the pre-trained multimodal video classification model includes at least a pre-trained video feature extraction network, a pre-trained text feature extraction network, and a pre-trained classification network, if it is determined that the video classification labels configured during the fine-tuning stage are consistent with the classification labels configured in the pre-trained classification network, the pre-trained multimodal video classification model is fine-tuned. The input unit 1602 is used to perform the following operations to obtain the target multimodal video classification model:

[0318] Obtain a set of fine-tuning sample data, each of which includes a fine-tuning sample video stream, as well as the fine-tuning sample text information associated with the fine-tuning sample video stream and the fine-tuning sample classification label;

[0319] By using a fine-tuning sample dataset, the pre-trained multimodal video classification model is subjected to multiple rounds of fine-tuning training to obtain the fine-tuned target multimodal video classification model.

[0320] Optionally, when the pre-trained multimodal video classification model includes at least a pre-trained video feature extraction network, a pre-trained text feature extraction network, and a pre-trained classification network, if it is determined that the video classification labels configured during the fine-tuning stage are inconsistent with the classification labels configured in the pre-trained classification network, the pre-trained multimodal video classification model is fine-tuned. The input unit 1602 is used to perform the following operations to obtain the target multimodal video classification model:

[0321] Obtain a set of fine-tuned sample data, wherein each fine-tuned sample data includes a fine-tuned sample video stream, as well as fine-tuned sample text information and fine-tuned sample classification labels associated with the fine-tuned sample video stream;

[0322] Based on the video classification labels during the fine-tuning phase, adjust the classification labels configured in the pre-trained classification network.

[0323] By using a fine-tuned sample dataset, the adjusted multimodal video classification model is trained through multiple rounds of fine-tuning to obtain the target multimodal video classification model after fine-tuning.

[0324] Optionally, the text information associated with the video stream to be processed is determined, and the acquisition unit 1601 is used to perform the following operations:

[0325] Obtain the title information associated with the video stream to be processed;

[0326] For each video frame to be processed, a text recognition algorithm is used to identify the video frame text information included in the video frame to be processed;

[0327] The title information and the identified text information of each video frame are used as the text information to be processed associated with the video stream to be processed.

[0328] Optionally, the video classification device further includes a push unit, the push unit 1603 being used for:

[0329] Based on the video classification results associated with each video stream to be processed, the video streams to be processed are clustered and presented, where the video classification results include the corresponding classification labels;

[0330] In response to a video push request triggered by a target object based on a target category tag, the target video associated with the target category tag will be pushed to the target object.

[0331] After introducing the pre-training method, video classification method and apparatus for multimodal video classification model according to exemplary embodiments of this application, the electronic device according to another exemplary embodiment of this application will be introduced next.

[0332] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0333] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device, see reference. Figure 17 As shown, this is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device 1700 may include at least a processor 1701 and a memory 1702. The memory 1702 stores program code, which, when executed by the processor 1701, causes the processor 1701 to execute the pre-training method of any of the above-mentioned multimodal video classification models and the steps of the video classification method.

[0334] In some possible implementations, the computing device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps of the pre-training method for the multimodal video classification model according to the various exemplary embodiments of this application described above, as well as the steps of the video classification method. For example, the processor may perform actions such as... Figure 2 , Figure 12 The steps are shown in the figure.

[0335] See Figure 18 As shown, it is a schematic diagram of the structure of a computing device in an embodiment of this application. The following refers to... Figure 18 To describe a computing device 1800 according to this embodiment of the present application. For example... Figure 18 As shown, the computing device 1800 is presented in the form of a general-purpose computing device. The components of the computing device 1800 may include, but are not limited to: at least one processing unit 1801, at least one storage unit 1802, and a bus 1803 connecting different system components (including storage unit 1802 and processing unit 1801).

[0336] Bus 1803 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus architectures.

[0337] Storage unit 1802 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 18021 and / or cache memory unit 18022, and may further include read-only memory (ROM) 18023.

[0338] Storage unit 1802 may also include a program / utility 18025 having a set (at least one) of program modules 18024, such program modules 18024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0339] The computing device 1800 can also communicate with one or more external devices 1804 (e.g., keyboard, pointing device, etc.), one or more devices that enable objects to interact with the computing device 1800, and / or any device that enables the computing device 1800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1805. Furthermore, the computing device 1800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1806. As shown, network adapter 1806 communicates with other modules used in the computing device 1800 via bus 1803. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 1800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0340] Based on the same inventive concept as the above-described method embodiments, the pre-training method for the multimodal video classification model and various aspects of the video classification method provided in this application can also be implemented as a program product, which includes program code. When the program product is run on an electronic device, the program code is used to cause the electronic device to execute the steps in the pre-training method for the multimodal video classification model according to the various exemplary embodiments of this application described above, as well as the video classification method. For example, the electronic device can execute... Figure 2 , Figure 12 The steps are shown in the figure.

[0341] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0342] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0343] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A pre-training method for a multimodal video classification model, characterized in that, include: For each historical video stream, based on the pseudo-classification models of each target trained for each corresponding video source, determine the pseudo-classification results associated with the historical video stream; Obtain the initial intersection between the pseudo-classification labels included in each pseudo-classification result and the original classification labels associated with the historical video stream, and determine the classification prediction value corresponding to each classification label in the initial intersection; and delete the classification labels whose classification prediction values ​​satisfy the label selection conditions from the original classification labels associated with the historical video stream, to obtain the processed original classification labels, and take the union of the original classification labels associated with the historical video stream and each pseudo-classification result as the sample classification labels associated with the historical video stream; generate a pre-training sample data based on the historical video stream, the sample classification labels associated with the historical video stream, and historical text information; Based on each pre-training sample data, a pre-training sample data set is generated; each pre-training sample data in the pre-training sample data set includes a historical video stream as a sample video stream, as well as sample text information and sample classification labels associated with the sample video stream; Using the sample video streams included in each sample data in the pre-trained sample dataset, the video feature extraction network is trained in multiple rounds of self-supervised training to obtain the initial video feature extraction network after training. Using the sample text information associated with each sample video stream in the pre-trained sample dataset, the text feature extraction network is trained in multiple rounds of self-supervised training to obtain the initial text feature extraction network after training. Using the pre-trained sample data set, the initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network, is subjected to multiple rounds of supervised training to obtain the pre-trained multimodal video classification model.

2. The method as described in claim 1, characterized in that, The respective target pseudo-classification models are constructed in the following manner: Based on the historical video data corresponding to each video source, a subset of historical video data corresponding to each video source is constructed; wherein, each historical video data includes a historical video stream, as well as historical text information and original classification labels associated with the historical video stream; For each video source, a target pseudo-classification model corresponding to the video source is obtained by supervised training based on the historical video data corresponding to the video source.

3. The method as described in claim 2, characterized in that, The process of constructing a subset of historical video data for each video source based on the historical video data corresponding to each video source includes: Based on the initial video data from each video source whose release time is within a preset time range, an initial video data set is generated, wherein each initial video data set includes an initial video stream, as well as historical text information and original classification tags associated with the initial video stream; One initial video data point is read sequentially from the initial video data set, and the following deduplication process is performed based on the read initial video data until there is no unread initial video data in the initial video data set: The video feature similarity between the initial video stream in the read initial video data and the initial video stream in other initial video data in the initial video data set is determined; and other initial video data in the initial video data set whose video feature similarity with the read initial video data meets the similarity threshold is deleted. The initial video data in the deduplicated initial video data set is taken as historical video data; and according to the video source to which each historical video data belongs, a subset of historical video data corresponding to each video source is obtained by clustering based on each historical video data.

4. The method as described in claim 1, characterized in that, The process of determining the pseudo-classification results associated with the historical video stream based on the pseudo-classification models trained for each corresponding video source includes: The target video source corresponding to the historical video stream is determined, the target pseudo-classification model corresponding to the target video source is determined, and other target pseudo-classification models corresponding to each other video source other than the target video source are obtained. Various other target pseudo-classification models are used to determine the pseudo-classification results associated with the historical video stream.

5. The method as described in claim 1, characterized in that, During a round of self-supervised training of the video feature extraction network, the following operations are performed: A specified number of sample video streams are read from the pre-trained sample data set, and a specified number of sample video frames with associated time-series information are extracted from each of the read sample video streams through the video feature extraction network. The video feature extraction network is controlled to generate positive sample groups based on each sample video frame belonging to the same sample video stream, and to generate negative sample groups based on each sample video frame belonging to different sample video streams, wherein a sample group includes two sample video frames. The video feature extraction network is trained in multiple rounds of self-supervised iterative training based on the positive sample group and the negative sample group until the preset single-round convergence condition is met.

6. The method as described in claim 5, characterized in that, When extracting a specified number of sample video frames with associated time-series information from each of the read sample video streams, perform any one of the following operations: For each sample video stream read, a target time point is set according to the duration information corresponding to the sample video stream, and a specified number of sample video frames with associated time sequence information are extracted starting from the sample video frame corresponding to the target time point on the sample video stream. For each sample video stream read, the sample video stream is divided into a specified number of sample video sub-streams according to the duration information corresponding to the sample video stream, and a sample video frame is randomly extracted from each sample video sub-stream to obtain a specified number of sample video frames associated with time sequence information.

7. A video classification method, characterized in that, include: Acquire the video stream to be processed and determine the text information to be processed associated with the video stream to be processed; The video stream to be processed and the text information to be processed are input into the target multimodal video classification model that has been trained, and the video classification result output by the target multimodal video classification model is obtained. The target multimodal video classification model is obtained by fine-tuning the multimodal video classification model trained by the pre-training method according to any one of claims 1-6.

8. The method as described in claim 7, characterized in that, When the pre-trained multimodal video classification model includes at least a pre-trained video feature extraction network, a pre-trained text feature extraction network, and a pre-trained classification network, if it is determined that the video classification labels configured during the fine-tuning stage are inconsistent with the classification labels configured in the pre-trained classification network, the pre-trained multimodal video classification model is fine-tuned, and the following operations are performed to obtain the target multimodal video classification model: Obtain a set of fine-tuning sample data, wherein each set of fine-tuning sample data includes a fine-tuning sample video stream, as well as fine-tuning sample text information and fine-tuning sample classification labels associated with the fine-tuning sample video stream; Based on the video classification labels during the fine-tuning phase, adjust the classification labels configured in the pre-trained classification network. Using the aforementioned fine-tuned sample data set, the adjusted multimodal video classification model is trained through multiple rounds of fine-tuning to obtain the fine-tuned target multimodal video classification model.

9. The method as described in claim 7 or 8, characterized in that, Also includes: Based on the video classification results associated with each video stream to be processed, the video streams to be processed are clustered and presented, wherein the video classification results include corresponding classification labels; In response to a video push request triggered by a target object for a target category tag, the target video associated with the target category tag is pushed to the target object.

10. A pre-training device for a multimodal video classification model, characterized in that, include: The acquisition unit is used to determine, for each historical video stream, the pseudo-classification results associated with the historical video stream based on the pseudo-classification models trained for each corresponding video source. The process involves: obtaining the initial intersection between the pseudo-classification labels included in each pseudo-classification result and the original classification labels associated with the historical video stream; determining the classification prediction value corresponding to each classification label in the initial intersection; deleting classification labels whose classification prediction values ​​satisfy the label selection criteria from the original classification labels associated with the historical video stream to obtain the processed original classification labels; and using the union of the original classification labels associated with the historical video stream and each pseudo-classification result as the sample classification label associated with the historical video stream; generating a pre-training sample data set based on the historical video stream, the sample classification labels associated with the historical video stream, and historical text information; and generating a pre-training sample data set based on each pre-training sample data set, wherein each pre-training sample data set includes a historical video stream as a sample video stream, sample text information associated with the sample video stream, and sample classification labels; The first training unit is used to perform multiple rounds of self-supervised training on the video feature extraction network using the sample video streams included in each sample data in the pre-training sample data set, so as to obtain the initial video feature extraction network after training. The second training unit is used to perform multiple rounds of self-supervised training on the text feature extraction network using the sample text information associated with each sample video stream in the pre-trained sample data set, so as to obtain the initial text feature extraction network after training. The third training unit is used to perform multiple rounds of supervised training on the initial multimodal video classification model, which includes at least the initial video feature extraction network and the initial text feature extraction network, using the pre-trained sample data set, to obtain the pre-trained multimodal video classification model.

11. A video classification device, characterized in that, include: Acquisition unit: used to acquire the video stream to be processed and determine the text information to be processed associated with the video stream to be processed; The input unit is used to input the video stream to be processed and the text information to be processed into the target multimodal video classification model that has been trained, and to obtain the video classification result output by the target multimodal video classification model. The target multimodal video classification model is obtained by fine-tuning the multimodal video classification model trained according to the pre-training method of any one of claims 1-6.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

Images