Multimodal pre-training method and system based on unlabeled network video data
Through iterative video dataset refinement and noise control loss function, combined with AdaTaiLr loss function, data quality and scalability problems in the video-text pretrained model are solved, and the understanding ability and performance of the video-language big model are improved.
Patent Information
- Application Number
- CN202411196066.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-29
AI Technical Summary
The existing video-text pretrained model cannot take into account the data quantity, diversity and quality at the same time, resulting in noise accumulation and lack of scalability.
Using multimodal pre-training method based on unlabeled network video data, the iterative video data set refinement and noise control loss function improves the data set labeling quality, and using AdaTaiLr loss function as a metric to reduce the assumption of noise distribution and improves the Model video-language comprehension ability.
It effectively breaks through the problem that data quantity, diversity and quality cannot be taken into account at the same time, improves the scalability and noise control effect of the video-language model, and enhances the performance of the video-language understanding task.
Smart Images

Figure CN118711168B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video data processing, and in particular relates to a multimodal pre-training method and system based on unlabeled network video data. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] Video-Language Pre-training technology is widely used in video retrieval, video question answering, video content understanding and generation, and other fields. Given the complexity and diversity of video and text data, many researchers are committed to developing more efficient and robust video-text pre-training models. These models usually use advanced natural language processing and computer vision neural network architectures, such as GPT, ViT, etc. Although current model technology has made significant progress, video-text pre-training has the following problems in terms of data: Currently, there is an impossible triangle of quantity, diversity, and quality in video-text pre-training datasets (that is, it is impossible to take all of them into account at the same time).
[0004] For the impossible triangle of pre-training datasets, existing methods are committed to collecting Internet videos and extracting descriptive sentences as video annotations based on the multimodal information of the video (such as video frames, titles, tags, and speech recognition results). The "extraction" process usually uses large language models and image-language models to generate text descriptions. Although these methods have shown potential, there are still the following difficulties:
[0005] (1) Noise accumulation: Reducing noise in generated text remains a difficult problem because current denoising techniques in visual language pre-training rely on assumptions about noise distribution, which are often inconsistent with real-world data. Therefore, existing dataset optimization methods usually use pre-trained text-video retrieval models to filter out low-quality annotations, but this process reduces the diversity of annotations.
[0006] (2) No scalability: Even if the noise is eliminated, the performance of the pre-trained model is still limited by the capabilities of the image-language model and the text-video retrieval model. That is, as the number and computing power investment increase, the model performance approaches but cannot surpass the existing models. Summary of the invention
[0007] In order to solve the above technical problems, the present invention provides a multimodal pre-training method and system based on unlabeled network video data, which is achieved by iteratively improving the annotation quality of the dataset under a noise control algorithm, aiming to break through the problem that the quantity, diversity and quality of data cannot be taken into account at the same time, and solve the problems of noise accumulation and lack of scalability of existing methods.
[0008] In order to achieve the above object, the present invention adopts the following technical solution:
[0009] A first aspect of the present invention provides a multimodal pre-training method based on unlabeled network video data.
[0010] In one or more embodiments, a multimodal pre-training method based on unlabeled network video data is provided, including:
[0011] Obtain a video data set, use a preset large language model and an image-text model to preliminarily refine the video data set to generate a preliminary training data set, and then introduce a noise control loss function to preliminarily train a video-language large model as the video-language large model obtained by the previous stage of training;
[0012] Determine whether the video-language large model obtained by the previous stage of training meets the set requirements. If so, end the training; if not, use the video-language large model obtained by the previous stage of training to iteratively refine the video data set, update the training data set, and then introduce the same noise control loss function to iteratively train the video-language large model until the video-language large model that meets the set requirements is obtained through training, so as to use the finally trained video-language large model to identify the correspondence between speech and subtitles in the video content.
[0013] The advantage of the above technical solution is that it uses iterative video data set refinement to improve the annotation quality of the training data set, and then combines the noise control loss function to train the video-language large model, thereby improving the effect of video language pre-training.
[0014] As an implementation method, the noise control loss function is characterized by using the AdaTaiLr loss function.
[0015] The advantage of the above technical solution is that, in the process of training the large video-language model, the noise control loss function is introduced, which effectively breaks the limitation of the "impossible triangle" and is significantly better than the existing technology in terms of scalability in video-language understanding tasks.
[0016] As an implementation method, the AdaTaiLr loss function is set to :
[0017]
[0018]
[0019] in, is the probability distribution of the training data set, The text in the video-text pairs sampled for the training dataset, Refers to the text The number of the individual unit elements, A vector representing the probability of all individual unit elements predicted by the model; The model predicts The independent unit element is The probability of Elements in a vector; is a 0-1 vector, only At 1, is the number of all independent unit elements in the vocabulary; The first independent unit elements; is the probability distribution of the training data set expectations; The length of the text in the sampled video-text pair; is a constant; represents the 1-norm function; Indicates an intermediate parameter.
[0020] The advantage of the above technical solution is that the AdaTaiLr loss function uses the total variation distance as a metric, has fewer assumptions about noise distribution, and has a smaller approximation error under theoretical guarantee, thereby improving the effect of video language pre-training.
[0021] As an implementation method, in the process of preliminary training of the video-language large model, a basic model is first obtained, and then the basic model is fine-tuned to obtain a preliminary trained video-language large model.
[0022] The advantage of the above technical solution is that by fine-tuning the basic model through instruction fine-tuning data, the model can acquire general video-language understanding capabilities, thereby improving the performance of the model in identifying the correspondence between speech and subtitles in video content.
[0023] As an implementation method, in the process of iteratively training the video-language large model, in each iteration, the corresponding basic model is first obtained, and then the corresponding basic model is fine-tuned to obtain the video-language large model obtained by the current iterative training.
[0024] The advantage of the above technical solution is that the basic model is fine-tuned by using instruction fine-tuning data, thereby improving the model's video-language understanding ability.
[0025] As an implementation manner, the video data set includes video clips and text information attached to the video clips; the text information includes titles, tags, and speech recognition results.
[0026] As an implementation method, the step of generating a preliminary training data set includes:
[0027] Uniformly extract several frames from the video clips in the video dataset and use the image-text model to generate a description of each frame;
[0028] Using the large language model, the title, tags and descriptions of all frames of the corresponding video clip are summarized to form a description of the entire corresponding video clip, which is used as the annotation of the corresponding video clip to generate a preliminary training dataset.
[0029] As an implementation method, the step of iteratively refining the video dataset using the video-language large model trained in the previous stage includes:
[0030] Uniformly extract a number of frames from the video clips of the video dataset and encode them by the video encoder;
[0031] Arrange the titles, tags and speech recognition results of the corresponding video clips according to the text prompt words;
[0032] The video-language large model trained in the previous stage is used to jointly process the video encoded by the video encoder and the arranged text prompt words to generate video description content and obtain the corresponding training data set.
[0033] A second aspect of the present invention provides a multimodal pre-training system based on unlabeled network video data.
[0034] In one or more embodiments, a multimodal pre-training system based on unlabeled network video data includes:
[0035] An initial training module is used to obtain a video data set, preliminarily refine the video data set using a preset large language model and an image-text model, generate a preliminary training data set, and then preliminarily train a video-language large model under a noise control loss function as the video-language large model obtained by the previous stage of training;
[0036] The iterative training module is used to determine whether the video-language large model obtained by the previous stage of training meets the set requirements. If so, the training is terminated; if not, the video-language large model obtained by the previous stage of training is used to iteratively refine the video data set, and update the training data set, and then introduce the same noise control loss function to iteratively train the video-language large model until the video-language large model that meets the set requirements is obtained through training, so as to use the finally trained video-language large model to identify the correspondence between speech and subtitles in the video content.
[0037] As an implementation method, the noise control loss function is characterized by the AdaTaiLr loss function; the AdaTaiLr loss function is set to :
[0038]
[0039]
[0040] in, is the probability distribution of the training data set, The text in the video-text pairs sampled for the training dataset, Refers to the text The number of the individual unit elements, A vector representing the probability of all individual unit elements predicted by the model; The model predicts The independent unit element is The probability of Elements in a vector; is a 0-1 vector, only At 1, is the number of all independent unit elements in the vocabulary; The first independent unit elements; is the probability distribution of the training data set expectations; The length of the text in the sampled video-text pair; is a constant; represents the 1-norm function; Indicates an intermediate parameter.
[0041] As an implementation manner, the video data set includes video clips and text information attached to the video clips; the text information includes titles, tags, and speech recognition results.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] (1) The present invention utilizes iterative video dataset refinement to improve the annotation quality of the training dataset, and then combines it with the noise control loss function to jointly train the video-language large model. This achieves the iterative improvement of the dataset annotation quality under the noise control algorithm, breaking through the problem that the quantity, diversity and quality of data cannot be taken into account at the same time, solving the noise accumulation and lack of scalability problems of the existing methods, and improving the video-language understanding ability of the video-language large model.
[0044] (2) The present invention introduces the AdaTaiLr loss function and takes advantage of its fewer assumptions on noise distribution and smaller approximation error under theoretical guarantee to iteratively optimize the annotation quality of the video dataset, effectively breaking the limitation of the "impossible triangle" and enhancing the scalability in video-language understanding tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0046] Figure 1 is a flow chart of a multimodal pre-training method based on unlabeled network video data according to an embodiment of the present invention;
[0047] Figure 2 is a schematic diagram of a training data set generation process according to an embodiment of the present invention;
[0048] Figure 3 is a schematic diagram of the structure of a multimodal pre-training system based on unlabeled network video data according to an embodiment of the present invention;
[0049] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0051] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0053] Terminology explanation:
[0054] Video-Language Pre-training Technology:
[0055] The goal is to pre-train a deep learning model by leveraging large-scale video and text datasets to learn the deep association between video content and text descriptions. This technology builds a cross-modal representation space so that visual elements in the video (such as scenes, actions, objects) can be mapped and corresponded to semantic information in the text.
[0056] AdaTaiLr algorithm:
[0057] The AdaTaiLr algorithm was introduced in the pre-training stage of the iterative dataset annotation quality improvement framework. Through a new loss function AdaTaiLr, the total variation distance is used instead of the Kullback–Leibler divergence as the error measure between the model and the data distribution. This measure makes fewer assumptions about the distribution of noisy data, and can filter the noisy data distribution more efficiently, effectively reducing the adverse effects of data noise on model training.
[0058] Figure 1 is a flow chart of a multimodal pre-training method based on unlabeled network video data in an embodiment of the present invention, such as Figure 1 The multimodal pre-training method based on unlabeled network video data in the embodiment shown may include two stages, namely an initial improvement stage and an iterative improvement stage.
[0059] Initial improvement stage: obtain a video data set, use the preset large language model and image-text model to preliminarily refine the video data set, generate a preliminary training data set, and then introduce the noise control loss function to preliminarily train the video-language large model as the video-language large model trained in the previous stage;
[0060] Iterative improvement stage: determine whether the video-language large model trained in the previous stage meets the set requirements. If so, end the training; if not, use the video-language large model trained in the previous stage to iteratively refine the video data set, update the training data set, and then introduce the same noise control loss function to iteratively train the video-language large model until the video-language large model that meets the set requirements is obtained through training, so as to use the finally trained video-language large model to identify the correspondence between speech and subtitles in the video content.
[0061] This embodiment uses iterative video data set refinement to improve the annotation quality of the training data set, and then combines it with the noise control loss function to train the video-language large model, thereby improving the effect of video language pre-training.
[0062] In the process of preliminary training of the video-language large model, the basic model is first obtained, and then the basic model is fine-tuned by instructions to obtain the preliminary trained video-language large model. In this way, by fine-tuning the basic model with instruction fine-tuning data, the model can acquire general video-language understanding capabilities and improve the performance of the model in identifying the corresponding relationship between speech and subtitles in video content.
[0063] Specifically, in the initial improvement stage, the preset large language model and image-text model are used to perform preliminary refinement on the video dataset to generate a preliminary training dataset, which can be marked as dataset 0. Then, the noise control loss function is introduced to preliminarily train the video-language large model to obtain the basic model 0. After fine-tuning the instructions of the basic model 0, the preliminary trained video-language large model is obtained, which is marked as video-language large model 0.
[0064] In the initial boosting phase, the large language model and the image-language model are used to initially refine the dataset. First, the labeling errors in the dataset are corrected by giving hints to the large language model and the image-language model. The refined dataset is then used to train the initial video-language model under noise control for further refinement.
[0065] In the process of iterative training of the video-language large model, in each iteration, the corresponding basic model is first obtained, and then the instruction fine-tuning is performed on the corresponding basic model to obtain the video-language large model obtained by the current iteration training. In this way, the basic model is fine-tuned using the instruction fine-tuning data, thereby improving the model's video-language understanding ability.
[0066] Specifically, the video data set is further refined using the video-language big model 0 to generate a training data set, recorded as data set 1; then, under the introduction of the noise control loss function, the video-language big model is pre-trained using data set 1 to obtain the basic model 1, and then the basic model 1 is fine-tuned to obtain the video-language big model 1, and this iteration is repeated until the video-language big model is trained.
[0067] Among them, the iterative stopping condition for completing the training of the video-language large model can be that the iteration reaches a set number of times, or the performance error of the video-language large model after two iterations is within a set range. Technical personnel in this field can make specific settings based on actual conditions, which will not be described in detail here.
[0068] In the iterative improvement stage, the video-language model trained in the current stage is used to iteratively refine the video dataset to obtain the training dataset for the next stage of training. Through the video understanding ability of the video-language model, the correspondence between speech and subtitles in the video content can be more accurately identified, thereby further improving the annotation quality of the dataset.
[0069] In this embodiment, the noise control loss function is characterized by the AdaTaiLr loss function. The AdaTaiLr loss function is set to :
[0070] ;
[0071] ;
[0072] in, is the probability distribution of the training data set, The text in the video-text pairs sampled for the training dataset, Refers to the text The number of the individual unit elements, A vector representing the probability of all individual unit elements predicted by the model; The model predicts The independent unit element is The probability of Elements in a vector; is a 0-1 vector, only At 1, is the number of all independent unit elements in the vocabulary; The first independent unit elements; is the probability distribution of the training data set expectations; The length of the text in the sampled video-text pair; is a constant; represents the 1-norm function; Indicates an intermediate parameter.
[0073] The independent unit element here can be a word, a number or a symbol, which can be represented by a token.
[0074] In the process of training the large video-language model, the noise control loss function is introduced, which effectively breaks the limitation of the "impossible triangle" and significantly outperforms the existing technology in terms of scalability in video-language understanding tasks. In addition, the AdaTaiLr loss function uses the total variation distance as a metric, has fewer assumptions about noise distribution, and has a smaller approximation error under theoretical guarantee, which improves the effect of video language pre-training.
[0075] The video data set of this embodiment includes video clips and text information attached to the video clips; the text information includes titles, tags and speech recognition results.
[0076] according to Figure 2 ,The steps to generate a preliminary training data set include:
[0077] Uniformly extract several frames from the video clips in the video dataset and use the image-text model to generate a description of each frame;
[0078] Using the large language model, the title, tags and descriptions of all frames of the corresponding video clip are summarized to form a description of the entire corresponding video clip, which is used as the annotation of the corresponding video clip to generate a preliminary training dataset.
[0079] The image-text model and large language model here are both existing neural network model structures and will not be described in detail here.
[0080] according to Figure 2 The step of iteratively refining the video dataset using the video-language large model trained in the previous stage includes:
[0081] Uniformly extract a number of frames from the video clips of the video dataset and encode them by the video encoder;
[0082] Arrange the titles, tags and speech recognition results of the corresponding video clips according to the text prompt words;
[0083] The video-language large model trained in the previous stage is used to jointly process the video encoded by the video encoder and the arranged text prompt words to generate video description content and obtain the corresponding training data set.
[0084] The frequency encoder here is an existing structure and will not be described in detail here.
[0085] Figure 3 is a schematic diagram of the structure of a multimodal pre-training system based on unlabeled network video data in an embodiment of the present invention. Figure 1 The multimodal pre-training method based on unlabeled network video data corresponds to Figure 3 As shown, the multimodal pre-training system based on unlabeled network video data in this embodiment may include:
[0086] The initial training module 301 is used to obtain a video data set, preliminarily refine the video data set using a preset large language model and an image-text model to generate a preliminary training data set, and then preliminarily train a video-language large model under a noise control loss function as the video-language large model obtained by the previous stage of training;
[0087] The iterative training module 302 is used to determine whether the video-language large model obtained by the previous stage of training meets the set requirements. If so, the training is terminated; if not, the video-language large model obtained by the previous stage of training is used to iteratively refine the video data set, and update the training data set, and then introduce the same noise control loss function to iteratively train the video-language large model until the video-language large model that meets the set requirements is obtained through training, so as to use the finally trained video-language large model to identify the correspondence between speech and subtitles in the video content.
[0088] In this embodiment, the noise control loss function is characterized by the AdaTaiLr loss function; the AdaTaiLr loss function is set to :
[0089]
[0090]
[0091] in, is the probability distribution of the training data set, The text in the video-text pairs sampled for the training dataset, Refers to the text The number of the individual unit elements, A vector representing the probability of all individual unit elements predicted by the model; The model predicts The independent unit element is The probability of Elements in a vector; is a 0-1 vector, only At 1, is the number of all independent unit elements in the vocabulary; The first independent unit elements; is the probability distribution of the training data set expectations; The length of the text in the sampled video-text pair; is a constant; represents the 1-norm function; Indicates an intermediate parameter.
[0092] It should be noted here that Figure 3 The initial training module 301 and the iterative training module 302 in the multimodal pre-training system based on unlabeled network video data in Figure 1 The specific implementation process of each step in the multimodal pre-training method based on unlabeled network video data is the same and will not be repeated here.
[0093] Reference Figure 4 , a schematic diagram of an electronic device is given. It should be noted that, Figure 4 The electronic device 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0094] like Figure 4As shown, the electronic device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403. In the RAM 403, various programs and data required for system operation are also stored. The central processing unit 401, the ROM 402 and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0095] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.
[0096] When the central processing unit 401 in the electronic device of this embodiment executes the program, the following is achieved: Figure 1 The steps in the multimodal pre-training method based on unlabeled network video data are shown.
[0097] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, the computer program including a computer program for executing Figure 1 In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the apparatus of the present application are executed.
[0098] in, Figure 1 The computer program instructions corresponding to the method shown may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0099] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A multimodal pre-training method based on unlabeled network video data, characterized in that: include: Obtain a video data set, use a preset large language model and an image-text model to preliminarily refine the video data set to generate a preliminary training data set, then introduce a noise control loss function to preliminarily train a video-language large model as the video-language large model obtained by training in the previous stage; Determine whether the video-language model obtained by the previous stage of training meets the set requirements. If so, end the training; if not, use the video-language model obtained by the previous stage of training to iteratively refine the video data set, update the training data set, and then introduce the same noise control loss function to iteratively train the video-language model until the video-language model that meets the set requirements is obtained through training, so as to use the finally trained video-language model to identify the correspondence between speech and subtitles in the video content; The step of generating a preliminary training data set includes: Uniformly extract several frames from the video clips in the video dataset and use the image-text model to generate a description of each frame; Using a large language model, summarize the title, label and description of all frames of the corresponding video clip to form a description of the entire corresponding video clip as the annotation of the corresponding video clip to generate a preliminary training data set; The step of iteratively refining the video data set using the video-language large model trained in the previous stage includes: Uniformly extract a number of frames from the video clips of the video dataset and encode them by the video encoder; Arrange the titles, tags and speech recognition results of the corresponding video clips according to the text prompt words; The video-language model trained in the previous stage is used to jointly process the video encoded by the video encoder and the arranged text prompt words to generate video description content and obtain the corresponding training data set; The noise control loss function is characterized by the AdaTaiLr loss function; the AdaTaiLr loss function is set to : in, is the probability distribution of the training data set, The text in the video-text pairs sampled for the training dataset, Refers to the text The number of the individual unit elements, A vector representing the probability of all individual unit elements predicted by the model; The model predicts The independent unit element is The probability of Elements in a vector; is a 0-1 vector, only At 1, is the number of all independent unit elements in the vocabulary; The first independent unit elements; is the probability distribution of the training data set expectations; The length of the text in the sampled video-text pair; is a constant; represents the 1-norm function; Indicates an intermediate parameter.
2. The multimodal pre-training method based on unlabeled network video data according to claim 1, characterized in that: In the process of preliminary training of the video-language large model, a basic model is first obtained, and then the basic model is fine-tuned to obtain the preliminary trained video-language large model.
3. The multimodal pre-training method based on unlabeled network video data according to claim 1, characterized in that: In the process of iterative training of the video-language large model, in each iteration, the corresponding basic model is first obtained, and then the corresponding basic model is fine-tuned to obtain the video-language large model obtained by the current iterative training.
4. The multimodal pre-training method based on unlabeled network video data according to claim 1, characterized in that: The video data set includes video clips and text information attached to the video clips; the text information includes titles, tags and speech recognition results.
5. A multimodal pre-training system based on unlabeled network video data, characterized in that: include: An initial training module is used to obtain a video data set, preliminarily refine the video data set using a preset large language model and an image-text model, generate a preliminary training data set, and then introduce a noise control loss function to preliminarily train a video-language large model as the video-language large model obtained by the previous stage of training; An iterative training module is used to determine whether the video-language model obtained by the previous stage of training meets the set requirements. If so, the training is terminated; if not, the video-language model obtained by the previous stage of training is used to iteratively refine the video data set, and update the training data set, and then introduce the same noise control loss function to iteratively train the video-language model until the video-language model that meets the set requirements is obtained through training, so as to use the finally trained video-language model to identify the correspondence between speech and subtitles in the video content; The step of generating a preliminary training data set includes: Uniformly extract several frames from the video clips in the video dataset and use the image-text model to generate a description of each frame; Using a large language model, summarize the title, label and description of all frames of the corresponding video clip to form a description of the entire corresponding video clip as the annotation of the corresponding video clip to generate a preliminary training data set; The step of iteratively refining the video data set using the video-language large model trained in the previous stage includes: Uniformly extract a number of frames from the video clips of the video dataset and encode them by the video encoder; Arrange the titles, tags and speech recognition results of the corresponding video clips according to the text prompt words; The video-language model trained in the previous stage is used to jointly process the video encoded by the video encoder and the arranged text prompt words to generate video description content and obtain the corresponding training data set; The noise control loss function is characterized by the AdaTaiLr loss function; the AdaTaiLr loss function is set to : in, is the probability distribution of the training data set, The text in the video-text pairs sampled for the training dataset, Refers to the text The number of the individual unit elements, A vector representing the probability of all individual unit elements predicted by the model; The model predicts The independent unit element is The probability of Elements in a vector; is a 0-1 vector, only At 1, is the number of all independent unit elements in the vocabulary; The first independent unit elements; is the probability distribution of the training data set expectations; The length of the text in the sampled video-text pair; is a constant; represents the 1-norm function; Indicates an intermediate parameter.
6. The multimodal pre-training system based on unlabeled network video data according to claim 5, characterized in that: The video data set includes video clips and text information attached to the video clips; the text information includes titles, tags and speech recognition results.
