Dataset construction method and system for large visual language model training

Abstract

The application provides a dataset construction method and system for large visual language model training, and belongs to the technical field of dataset construction. The method comprises the following steps: acquiring a plurality of videos, sending part of the videos to a target device to enable the target device to feed back frame-level labels and video-level labels corresponding to each video in the part of the videos; determining a basic training dataset based on the part of the videos, the frame-level labels and the video-level labels corresponding to each video in the part of the videos; training a random forest model based on the basic training dataset to obtain a target random forest model, and labeling the remaining videos based on the target random forest model; and constructing the part of the videos, the frame-level labels and the video-level labels corresponding to each video in the part of the videos, and the labeled remaining videos into a target dataset for large visual language model training. The application can construct a video dataset with fine-grained labels while reducing human input.

Description

Technical Field

[0001] This application belongs to the field of dataset construction technology, and more specifically, relates to a method and system for constructing datasets for training large-scale visual language models. Background Technology

[0002] In recent years, with the rapid development of diffusion models, video generation models, and multimodal generation models, AI-generated videos have continuously improved in terms of visual quality, motion coherence, and scene consistency, gradually approaching the quality of realistically shot videos. This technology has significant application value in film and television production, digital content creation, and virtual reality, but it can also be used to create false information, tamper with video evidence, and mislead public opinion, posing potential risks to cybersecurity and social governance.

[0003] To address these challenges, researchers have proposed various AI-generated video detection methods. Early methods primarily relied on image forensics techniques, analyzing texture features, noise distribution, and compression artifacts in videos to identify forged content. However, with advancements in generative modeling techniques, these methods have become increasingly ineffective at detecting high-quality generated videos. In recent years, large-scale visual-language models have been increasingly applied to generated content detection tasks. By integrating visual understanding and language reasoning capabilities, these models can perform multimodal reasoning in complex scenarios, thereby identifying anomalous patterns in generated content.

[0004] However, most current large-scale visual language models treat AI-generated video detection tasks as a simple binary classification problem, meaning that the training data for these models only contains coarse-grained true / false labels. This results in significant limitations in traditional evaluation methods, which can only achieve coarse-grained true / false identification. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for constructing datasets for training large-scale visual language models, so as to construct video datasets with fine-grained annotations while reducing human input.

[0006] A first aspect of this application provides a method for constructing a dataset for training large-scale visual language models, comprising: Multiple videos are acquired, and a portion of these videos is sent to a target device. The target device then provides frame-level and video-level tags for each video in the selected portion. The frame-level tags are used to characterize the authenticity of the corresponding video from multiple dimensions at the video frame level, and the video-level tags are used to characterize the authenticity of the corresponding video from multiple dimensions at the video level. The basic training dataset is determined based on a subset of videos, frame-level labels and video-level labels corresponding to each video in the subset of videos; The random forest model is trained based on the basic training dataset to obtain the target random forest model. The remaining videos in multiple videos are then labeled with frame-level and video-level labels based on the target random forest model. The target dataset for training a large visual language model is constructed by combining a portion of the videos, frame-level and video-level labels for each video in the portion of the videos, and the remaining videos after annotation.

[0007] A second aspect of this application provides a dataset construction system for training large-scale visual language models, comprising: The tag acquisition module is used to acquire multiple videos and send a portion of the videos to the target device, so that the target device can return the frame-level tag and video-level tag corresponding to each video in the portion of the videos. The frame-level tag is used to characterize the authenticity of the corresponding video from multiple dimensions at the video frame level, and the video-level tag is used to characterize the authenticity of the corresponding video from multiple dimensions at the video level. The training dataset determination module is used to determine the basic training dataset based on a portion of videos, frame-level labels and video-level labels corresponding to each video in the portion of videos; The model training and annotation module is used to train the random forest model based on the basic training dataset to obtain the target random forest model, and to annotate the remaining videos in multiple videos with frame-level labels and video-level labels based on the target random forest model. The dataset construction module is used to construct a target dataset for training large-scale visual language models by combining a portion of the videos, the frame-level labels and video-level labels corresponding to each video in the portion of the videos, and the remaining labeled videos.

[0008] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for constructing a dataset for training a large visual language model.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for constructing a dataset for training a large visual language model.

[0010] The beneficial effects of the dataset construction method and system for training large-scale visual language models provided in this application are as follows: This application provides fine-grained labeled data for large-scale visual language models by acquiring frame-level and video-level multi-dimensional true / false labels from a subset of videos. This replaces the coarse-grained labels found only in traditional training data, providing fine-grained labeled data that matches the training requirements of multimodal inference in video authenticity detection tasks. This ensures the quality of the training set's labeling and solves the problem of insufficient label granularity in traditional training sets. Furthermore, this application trains a random forest model based on this fine-grained labeled data, enabling automated fine-grained labeling of the remaining videos. This avoids the significant human resource investment required for full-scale manual fine-grained labeling, thus solving the problem of high manpower consumption in fine-grained labeling. This application also constructs a target dataset by integrating manually labeled samples and automatically labeled samples, achieving large-scale training set construction while maintaining high-quality fine-grained labeling. Therefore, this application balances the quality of fine-grained labeling in the training set with the efficiency of large-scale construction, efficiently building a high-quality dataset suitable for training large-scale visual language models and providing data support for improving the fine-grained video authenticity detection capabilities of large-scale visual language models. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a method for constructing a dataset for training a large visual language model, as provided in an embodiment of this application; Figure 2 A structural block diagram of a dataset construction system for training large visual language models provided in one embodiment of this application; Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0015] Please refer toFigure 1 , Figure 1 This is a flowchart illustrating a method for constructing a dataset for training a large visual language model, provided in an embodiment of this application. The method can be executed by an electronic device and may include: S101-S104.

[0016] S101: Acquire multiple videos, and send a portion of the videos to the target device so that the target device can provide the frame-level and video-level tags corresponding to each video in the selected portion.

[0017] In this embodiment, multiple videos include both real and generated videos. The real videos primarily originate from a real-scene video dataset, containing a large number of actually filmed video clips covering human activities, natural environments, urban street scenes, and everyday life scenarios. The generated videos can be AI-generated content, including video content generated by various video generation models, such as diffusion models. To ensure data diversity, the collected generated videos cover different visual scenes, including: scenes of human behavior, scenes of animals and nature, urban traffic scenes, and scenes of object movement.

[0018] In this embodiment, to ensure data consistency, all videos can undergo unified preprocessing before being sent to the target device. This includes: removing video and audio information, trimming the video length to 1–5 seconds, maintaining the original aspect ratio, and limiting the video resolution to between 360p and 1080p. This processing ensures that videos from different sources have a consistent data format during the evaluation process. After preprocessing, the preprocessed videos are manually screened. The screening process mainly considers the following factors: video quality, scene realism, and detection difficulty. First, low-resolution, heavily compressed, or visually poor videos are removed. Second, obviously fake or easily identifiable generated videos are removed. This process retains video samples with high visual realism that are difficult to identify.

[0019] In this embodiment, a portion of the pre-processed and manually screened videos (which may be a small portion of multiple videos, such as 5%) can be sent to a target device. The target device may be the computer corresponding to the annotators. There are multiple annotators, and they independently complete the annotation of the aforementioned portion of the videos through the target device. That is, each annotator annotates every video in the portion of the videos. Each video in the portion of the videos has multiple frame-level tags and video-level tags. Different frame-level tags for the same video correspond to different annotators, and different video-level tags for the same video correspond to different annotators.

[0020] In this embodiment, the target device feeds back frame-level tags and video-level tags for each video. The frame-level tags are used to characterize the authenticity of the corresponding video from multiple dimensions at the video frame level, and the video-level tags are used to characterize the authenticity of the corresponding video from multiple dimensions at the video level.

[0021] In this embodiment, frame-level tags are a set of tags that digitally represent the authenticity attributes of a single frame of video based on multiple preset authenticity representation dimensions. Each frame-level tag corresponds to a unique frame-level authenticity representation dimension, and the tag content is the authenticity judgment result of the video frame under that dimension, providing a more refined annotation of the authenticity of local visual features in the video. Video-level tags refer to a set of tags that digitally represent the overall authenticity attributes of a video at the level of a complete video sequence, based on multiple preset authenticity representation dimensions. Each video-level tag corresponds to a unique video-level authenticity representation dimension, and the tag content is the authenticity judgment result of the complete video under that dimension, providing a systematic annotation of the authenticity of global video features.

[0022] In this embodiment, frame-level annotation dimensions may include the following 10 dimensions: texture realism, edge and contour sharpness, consistency of material and physical properties, local forgery traces, depth of field and background plausibility, text and symbol readability, compositional naturalness, color consistency, lighting and shadow consistency, and reflection, refraction, and parallax relationship. When annotating video frames, annotators should evaluate each frame according to these 10 dimensions. That is, for the same video frame, there are 10 frame-level labels, each categorized as "true" or "false".

[0023] Video-level dimensions can include the following five dimensions: inter-frame consistency, continuity of characters and actions, temporal logical rationality, physical interaction rationality, and consistency with real-world logic. The above dimensional classification is merely an example; relevant testers or annotators can set their own dimensions according to their preferences, but a unified standard should be maintained when annotating.

[0024] S102: Determine the basic training dataset based on a subset of videos, the frame-level labels and video-level labels corresponding to each video in the subset of videos.

[0025] In this embodiment, the basic training dataset is determined based on a subset of videos, frame-level labels, and video-level labels corresponding to each video within that subset, including: For each video in the selected video set, perform the following tag determination operation: Statistical analysis is performed on multiple frame-level labels under each frame-level annotation dimension of the video. For each frame-level annotation dimension, the frame-level label with the most frame-level labels under that frame-level annotation dimension is taken as the final frame-level label for that frame-level annotation dimension. We statistically analyze multiple video-level tags under each video-level annotation dimension of the video. For each video-level annotation dimension, the video-level tag with the most video-level tags under that video-level annotation dimension is taken as the final video-level tag for that video-level annotation dimension. The video, the corresponding final frame-level label, and the corresponding final video-level label are associated and stored as a basic training sample. For each video in the subset of videos, perform the label determination operation as described above to obtain multiple basic training samples; integrate the multiple basic training samples into a basic training dataset.

[0026] In this embodiment, due to potential differences in judgment among different annotators, a statistical method is used to aggregate the annotation results. All frame-level tags for the same video are categorized according to a preset frame-level annotation dimension, ensuring that tags within the same frame-level annotation dimension correspond to the same true / false judgment direction, avoiding confusion between tags from different dimensions. Frequency counting is performed on multiple frame-level tags within each frame-level annotation dimension, counting the occurrences of each tag within that dimension to clarify the frequency differences among tags. The tag with the highest occurrence frequency is selected as the final frame-level tag for that annotation dimension. Essentially, this uses a majority voting method to integrate the judgment results of multiple annotators, using the consensus of the majority of annotators as the final frame-level tag. This minimizes the subjective bias of a single annotator, ensuring the objectivity, consistency, and accuracy of the frame-level tags, providing a unified annotation benchmark for the model to learn frame-level true / false features. In this embodiment, the logic for determining the final video-level tag is consistent with the logic for determining the final frame-level tag, and therefore will not be elaborated further.

[0027] In this embodiment, if there exists a frame-level annotation dimension or video-level annotation dimension where the difference between the number of true and false tags after annotation by various annotators is less than a preset review threshold, then the video frame or video can be sent to the annotation device corresponding to the expert, who will then annotate that dimension.

[0028] In this embodiment, the raw data of a single video (video frame sequence, video metadata, etc.) is associated one-to-one with the final frame-level labels corresponding to all frame-level annotation dimensions and the final video-level labels corresponding to all video-level annotation dimensions of that video, establishing a structured mapping relationship between video data, frame-level labels, and video-level labels. The associated video and label data are stored to obtain basic training samples. The label determination operation described above is performed on each video to obtain multiple basic training samples, and a basic training dataset is composed of these multiple basic training samples. In the basic training dataset, each video has a unique value under each frame-level annotation dimension and each video-level annotation dimension.

[0029] S103: Train the random forest model based on the basic training dataset to obtain the target random forest model. Based on the target random forest model, annotate the remaining videos in multiple videos with frame-level labels and video-level labels.

[0030] In this embodiment, feature extraction can be performed on the basic training samples in the basic training dataset to obtain frame-level features and video-level features. Then, the random forest model is trained based on the frame-level and video-level labels from the basic training samples to obtain the target random forest model. Specifically, for each video in the basic training dataset, the frame-level and video-level features can be extracted using the corresponding expert model. Similarly, when labeling the remaining videos with frame-level and video-level labels, the features input to the target random forest model should also be features obtained after feature extraction by the corresponding expert model.

[0031] In this embodiment, the remaining video refers to the video remaining after excluding the portion sent to the target device for annotation from a plurality of videos. Since the target random forest model has already been trained on a labeled base training dataset, it can automatically perform frame-level and video-level labeling for the remaining videos.

[0032] S104: Construct a target dataset for training a large visual language model by combining a portion of the videos, the frame-level labels and video-level labels corresponding to each video in the portion of the videos, and the remaining labeled videos.

[0033] In this embodiment, some videos from multiple videos are manually labeled, while the remaining videos are labeled using a target random forest model. This means the target dataset contains multiple videos, along with frame-level and video-level labels for each video. The target dataset is adapted to the visual local feature learning requirements of large-scale visual language models, supporting these models in performing multi-dimensional feature learning for video authenticity detection.

[0034] As can be seen from the above, this application's embodiments obtain frame-level and video-level multi-dimensional true / false labels from a portion of the videos, replacing the coarse-grained true / false labels found only in traditional training data. This provides fine-grained labeled data for large-scale visual language models, matching the training requirements of multimodal inference in video authenticity detection tasks, ensuring the labeling quality of the training set, and solving the problem of insufficient label granularity in traditional training sets. Simultaneously, this application's embodiments train a random forest model based on this fine-grained labeled data, achieving automated fine-grained labeling of the remaining videos, avoiding the large human resource investment required for full-scale manual fine-grained labeling, and solving the problem of high human resource consumption in fine-grained labeling. This application's embodiments also construct a target dataset by integrating manually labeled samples and automatically labeled samples, achieving large-scale construction of the training set while retaining the high quality of fine-grained labeling. Therefore, this application balances the quality of fine-grained labeling of the training set with the efficiency of large-scale construction, efficiently constructing a high-quality dataset suitable for training large-scale visual language models, providing data support for improving the model's fine-grained video authenticity detection capabilities.

[0035] In one embodiment of this application, a random forest model is trained based on a basic training dataset to obtain a target random forest model, including: The average percentage of different label types under each frame-level annotation dimension for each video in the statistical part of the video is calculated; the average percentage of different label types under each video-level annotation dimension for each video in the statistical part of the video is calculated; the average percentage of different label types under each frame-level annotation dimension is used to characterize the recognition difficulty of the corresponding frame-level dimension; the average percentage of different label types under each video-level annotation dimension is used to characterize the recognition difficulty of the corresponding video-level dimension. The parameter configuration of the random forest model is determined based on the average proportion of different label types under each frame-level annotation dimension and the average proportion of different label types under each video-level annotation dimension. The random forest model is trained based on the basic training dataset and parameter configuration to obtain the target random forest model.

[0036] In this embodiment, the average proportion of different label types under each frame-level annotation dimension refers to the average of the number of each type of label (in this embodiment, "true" and "false") in all frame-level labels of each video in the aforementioned 10 frame-level annotation dimensions, representing the proportion of each type of label to the total number of frame-level labels in that dimension. For example, in the frame-level texture authenticity dimension, if a certain video in a subset of videos has 60 "false" labels and 40 "true" labels, then the proportion of "false" labels in that dimension is 60% and the proportion of "true" labels is 40%. This proportion is the quantitative result of the label type distribution in that dimension. The same applies to the video-level annotation dimension.

[0037] In this embodiment, the reuse of manually labeled data is taken into account. That is, the difficulty of recognition of each dimension is indirectly represented by the quantification of the label proportion. If the average proportion of different label types under a certain label dimension is closer (e.g., the proportions of "true" and "false" are close to 50%), it means that the labelers have greater disagreement on the true and false judgments of this dimension during the manual labeling process. That is, the visual / temporal features of this dimension are more difficult to distinguish, and the corresponding recognition difficulty is higher. If the proportion of different label types under a certain label dimension is more divergent (e.g., the proportion of "false" label is 70% and the proportion of "true" label is 30%), it means that the labelers have a higher consensus on the true and false judgments of this dimension. The features of this dimension are easier to distinguish, and the corresponding recognition difficulty is lower.

[0038] In one embodiment, the parameter configuration of the random forest model is determined based on the average proportion of different label types under each frame-level annotation dimension and the average proportion of different label types under each video-level annotation dimension, including: For each frame-level annotation dimension and each video-level annotation dimension, calculate the deviation value of the average proportion of different label types under that dimension; Frame-level and video-level annotation dimensions with deviation values ​​less than a preset deviation threshold are classified as high-difficulty dimensions; frame-level and video-level annotation dimensions with deviation values ​​greater than or equal to the preset deviation threshold are classified as low-difficulty dimensions. Determine the parameter configuration for the random forest model during training for dimensions with high recognition difficulty, and determine the parameter configuration for the random forest model during training for dimensions with low recognition difficulty.

[0039] In this embodiment, the deviation value refers to the absolute difference between the average proportions of different label types within a single frame-level / video-level annotation dimension. It is the core indicator for quantifying the balance of label distribution within that dimension. Since this embodiment uses binary labels, the deviation value = |average proportion of one type of label - average proportion of the other type of label|. The smaller the deviation value, the more balanced the distribution of the two types of labels within that dimension; the larger the deviation value, the more unbalanced the distribution of the two types of labels. If the deviation value of a certain frame-level / video-level annotation dimension is less than a preset deviation threshold (e.g., the preset deviation threshold is 10%), then that dimension is determined to be a high-difficulty dimension; if the deviation value of a certain frame-level / video-level annotation dimension is greater than or equal to the preset deviation threshold, then that dimension is determined to be a low-difficulty dimension.

[0040] In this embodiment, the smaller the deviation value, the more balanced the distribution of "true" and "false" labels under this dimension, which means that during the manual annotation process, the annotators have greater disagreements on the true and false judgments of this dimension (it is difficult to reach a unified consensus), indirectly reflecting that the video features (frame-level visual features / video-level temporal features) corresponding to this dimension are more difficult to distinguish, and therefore it is judged as a high-difficulty dimension; the larger the deviation value, the more unbalanced the distribution of "true" and "false" labels under this dimension, the higher the consensus of the annotators on the true and false judgments of this dimension, and the easier it is to distinguish the video features corresponding to this dimension, and therefore it is judged as a low-difficulty dimension.

[0041] In this embodiment, the parameter configuration of the random forest model may include: sample sampling parameter configuration, feature weight allocation parameter configuration, and decision tree structure parameter configuration.

[0042] In one embodiment, determining the parameter configuration of the random forest model for dimensions with high recognition difficulty includes: For dimensions with high recognition difficulty, the following sampling parameter configurations are implemented: oversampling is performed on label samples where the proportion of samples in this high recognition difficulty dimension is lower than a first proportion, and undersampling is performed on label samples where the proportion of samples in this dimension is higher than a second proportion. Additionally, the parameter configurations for increasing the feature selection weights of the corresponding features in the random forest decision tree splitting process are also implemented. Furthermore, the parameter configurations for increasing the construction depth of a single decision tree in the random forest and increasing the sampling proportion of the corresponding features in the feature subset of the decision tree are also implemented. The first and second proportions can be set manually, for example, the first proportion can be set to 20%, and the second proportion to 80%.

[0043] In this embodiment, the sample sampling parameter configuration is used to optimize the distribution balance of training samples with high recognition difficulty dimensions during the sample input stage of random forest model training. The feature weight allocation parameter configuration is used to enhance the contribution of high recognition difficulty dimension features in model decision-making during the feature selection stage of random forest decision tree splitting nodes. The decision tree structure parameter configuration refers to the construction rules for each decision tree in the random forest, with the goal of improving the fitting ability of a single decision tree to high recognition difficulty dimension features, and may include the decision tree construction depth and the feature subset sampling ratio.

[0044] In this embodiment, feature selection weight refers to the probability weight coefficient of a certain dimension feature being selected as the splitting feature when a random forest decision tree splits a node. The higher the weight, the greater the probability that the feature will be preferentially selected during the decision tree splitting process. It is an indicator that determines the contribution of a feature to the model's judgment result. The construction depth of a single decision tree refers to the maximum number of levels from the root node to the leaf node in a single decision tree in the random forest. It is used to measure the complexity of the decision tree. The higher the construction depth, the richer the feature details that the decision tree can capture, and the more adaptable it is to complex feature patterns with high recognition difficulty dimensions. The sampling proportion of the decision tree feature subset refers to the probability that a certain dimension feature is included in the feature subset during the random forest construction of a single decision tree when randomly sampling feature subsets from all frame-level and video-level feature sets. Increasing this proportion can increase the frequency of the target dimension feature participating in the construction of the decision tree.

[0045] To address the issues of low discriminative power and sample distribution imbalance in high-difficulty features, this embodiment configures differentiated parameters across three dimensions: sample distribution optimization (sample layer), feature weight enhancement (feature layer), and model structure adaptation (structure layer). Through end-to-end parameter optimization, it maximizes the learning ability and accuracy of the random forest model in judging true and false features in high-difficulty dimensions. Specifically: At the sample layer, for dimensions with high recognition difficulty, the distribution of sample quantity is adjusted by oversampling and undersampling to ensure that the model can learn the feature patterns of "true" and "false" labels in this dimension equally. This eliminates the extreme imbalance of sample distribution in dimensions with high recognition difficulty and avoids "biased learning" due to the excessive proportion of majority class samples. This lays a balanced sample foundation for parameter optimization in the feature layer and structure layer.

[0046] At the feature layer, for dimensions with high recognition difficulty, the feature selection weight of their corresponding features in the decision tree splitting process is increased. This ensures that the random forest model prioritizes the core features of dimensions with high recognition difficulty. When the decision tree calculates the information gain / Gini coefficient of each feature, the weight coefficient of that dimension feature is included in the calculation. This ensures that even if the information gain is slightly lower than that of low-difficulty dimension features, it can still be selected as the splitting feature. This solves the problem that high-difficulty dimension features are easily masked by low-difficulty dimension features due to low discriminative power, and ensures that the dimension feature becomes the core basis for model decision-making.

[0047] At the structural layer, by adjusting the decision tree construction depth and the sampling ratio of feature subsets, individual decision trees are allowed to learn more fully the subtle feature differences in high-difficulty dimensions, increasing the basic construction depth of individual decision trees in the random forest. This allows decision trees to split to finer levels and avoids insufficient feature learning due to insufficient decision tree depth. Simultaneously, by increasing the complexity of the decision trees and the sampling frequency of high-difficulty dimension features, individual decision trees are allowed to deeply fit the complex feature patterns of that dimension, improving the model's recognition accuracy for high-difficulty dimension features.

[0048] In this embodiment, the specific adjustment step size of the parameter configuration can be set manually through multiple experiments or experience. For dimensions with low recognition difficulty, the default parameter configuration of the random forest model can be used without adjustment.

[0049] Corresponding to the dataset construction method for training large visual language models in the above embodiments, Figure 2 This is a structural block diagram of a dataset construction system for training large-scale visual language models, provided in one embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2 The dataset construction system 20 for training large-scale visual language models includes: a label acquisition module 21, a training dataset determination module 22, a model training and annotation module 23, and a dataset construction module 24.

[0050] The tag acquisition module 21 is used to acquire multiple videos and send a portion of the videos to the target device so that the target device can return the frame-level tag and video-level tag corresponding to each video in the portion of the videos. The frame-level tag is used to characterize the authenticity of the corresponding video from multiple dimensions at the video frame level, and the video-level tag is used to characterize the authenticity of the corresponding video from multiple dimensions at the video level. The training dataset determination module 22 is used to determine the basic training dataset based on a portion of videos, frame-level labels and video-level labels corresponding to each video in the portion of videos; The model training and annotation module 23 is used to train the random forest model based on the basic training dataset to obtain the target random forest model, and to annotate the remaining videos in multiple videos with frame-level labels and video-level labels based on the target random forest model. The dataset construction module 24 is used to construct a target dataset for training a large visual language model by combining a portion of the videos, the frame-level labels and video-level labels corresponding to each video in the portion of the videos, and the remaining labeled videos.

[0051] In one embodiment of this application, each video in a subset of videos has multiple frame-level tags and multiple video-level tags; different frame-level tags for the same video correspond to different annotators, and different video-level tags for the same video correspond to different annotators. Training dataset determination module 22 is specifically used to perform the following label determination operation for each video in a subset of videos: Statistical analysis is performed on multiple frame-level labels under each frame-level annotation dimension of the video. For each frame-level annotation dimension, the frame-level label with the most frame-level labels under that frame-level annotation dimension is taken as the final frame-level label for that frame-level annotation dimension. We statistically analyze multiple video-level tags under each video-level annotation dimension of the video. For each video-level annotation dimension, the video-level tag with the most video-level tags under that video-level annotation dimension is taken as the final video-level tag for that video-level annotation dimension. The video, the corresponding final frame-level label, and the corresponding final video-level label are associated and stored as a basic training sample. For each video in the subset of videos, perform the label determination operation as described above to obtain multiple basic training samples; integrate the multiple basic training samples into a basic training dataset.

[0052] In one embodiment of this application, the model training and annotation module 23 is specifically used to calculate the average proportion of different label types under each frame-level annotation dimension corresponding to each video in the partial video; and to calculate the average proportion of different label types under each video-level annotation dimension corresponding to each video in the partial video; wherein, the average proportion of different label types under each frame-level annotation dimension is used to characterize the recognition difficulty of the corresponding frame-level dimension; and the average proportion of different label types under each video-level annotation dimension is used to characterize the recognition difficulty of the corresponding video-level dimension. The parameter configuration of the random forest model is determined based on the average proportion of different label types under each frame-level annotation dimension and the average proportion of different label types under each video-level annotation dimension. The random forest model is trained based on the basic training dataset and parameter configuration to obtain the target random forest model.

[0053] In one embodiment of this application, the model training and annotation module 23 is further configured to calculate the deviation value of the average proportion of different label types under each frame-level annotation dimension and each video-level annotation dimension. Frame-level and video-level annotation dimensions with deviation values ​​less than a preset deviation threshold are classified as high-difficulty dimensions; frame-level and video-level annotation dimensions with deviation values ​​greater than or equal to the preset deviation threshold are classified as low-difficulty dimensions. Determine the parameter configuration for the random forest model during training for dimensions with high recognition difficulty, and determine the parameter configuration for the random forest model during training for dimensions with low recognition difficulty.

[0054] In one embodiment of this application, the parameter configuration includes: sample sampling parameter configuration; model training and annotation module 23, which is further configured to perform oversampling on label samples whose sample quantity ratio is lower than a first ratio for high recognition difficulty dimension, and to perform undersampling on label samples whose sample quantity ratio is higher than a second ratio.

[0055] In one embodiment of this application, the parameter configuration includes: feature weight allocation parameter configuration; model training and annotation module 23, which is further configured to improve the feature selection weight of the corresponding feature in the random forest decision tree splitting process for a high recognition difficulty dimension.

[0056] In one embodiment of this application, the parameter configuration includes: decision tree structure parameter configuration; model training and annotation module 23, which is further configured to improve the construction depth of a single decision tree in the random forest and increase the sampling ratio of the feature corresponding to the high recognition difficulty dimension in the feature subset of the decision tree for the parameter configuration.

[0057] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above system embodiments, for example... Figure 2 The functions of the label acquisition module 21, training dataset determination module 22, model training and annotation module 23, and dataset construction module 24 are shown.

[0058] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0059] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0060] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0061] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the dataset construction method for training large-scale visual language models provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0062] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0063] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0064] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0065] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0066] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0067] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0068] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0069] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing a dataset for training large-scale visual language models, characterized in that, include: Multiple videos are acquired, and a portion of the videos are sent to a target device so that the target device can provide a frame-level tag and a video-level tag for each video in the portion of videos. The frame-level tag is used to characterize the authenticity of the corresponding video from multiple dimensions at the video frame level, and the video-level tag is used to characterize the authenticity of the corresponding video from multiple dimensions at the video level. The basic training dataset is determined based on the aforementioned partial videos, the frame-level labels and video-level labels corresponding to each video in the aforementioned partial videos; The random forest model is trained based on the basic training dataset to obtain the target random forest model. The remaining videos in the multiple videos are then labeled with frame-level tags and video-level tags based on the target random forest model. The selected videos, the frame-level labels and video-level labels corresponding to each video in the selected videos, and the remaining labeled videos are used to construct a target dataset for training a large visual language model.

2. The dataset construction method for training large-scale visual language models as described in claim 1, characterized in that, Each video in the aforementioned video contains multiple frame-level tags and video-level tags; different frame-level tags for the same video correspond to different annotators, and different video-level tags for the same video correspond to different annotators. The process of determining the basic training dataset based on the partial videos, the frame-level labels and video-level labels corresponding to each video in the partial videos, includes: For each video in the aforementioned video set, perform the following tag determination operation: The multiple frame-level labels under each frame-level annotation dimension of the video are statistically analyzed. For each frame-level annotation dimension, the frame-level label with the most frame-level labels under that frame-level annotation dimension is taken as the final frame-level label for that frame-level annotation dimension. The video-level tags under each video-level annotation dimension are statistically analyzed. For each video-level annotation dimension, the video-level tag with the most video-level tags under that video-level annotation dimension is taken as the final video-level tag for that video-level annotation dimension. The video, the final frame-level label corresponding to the video, and the final video-level label corresponding to the video are associated and stored as a basic training sample; For each video in the aforementioned portion of the videos, perform the label determination operation as described above to obtain multiple basic training samples; integrate the multiple basic training samples into the basic training dataset.

3. The dataset construction method for training large-scale visual language models as described in claim 1, characterized in that, The step of training the random forest model based on the aforementioned basic training dataset to obtain the target random forest model includes: The average percentage of different label types under each frame-level annotation dimension for each video in the aforementioned video set is calculated; the average percentage of different label types under each video-level annotation dimension for each video in the aforementioned video set is calculated; wherein, the average percentage of different label types under each frame-level annotation dimension is used to characterize the recognition difficulty of the corresponding frame-level dimension; the average percentage of different label types under each video-level annotation dimension is used to characterize the recognition difficulty of the corresponding video-level dimension. The parameter configuration of the random forest model is determined based on the average proportion of different label types under each frame-level annotation dimension and the average proportion of different label types under each video-level annotation dimension. The random forest model is trained based on the basic training dataset and the parameter configuration to obtain the target random forest model.

4. The dataset construction method for training large-scale visual language models as described in claim 3, characterized in that, The process of determining the parameter configuration of the random forest model based on the average proportion of different label types under each frame-level annotation dimension and the average proportion of different label types under each video-level annotation dimension includes: For each frame-level annotation dimension and each video-level annotation dimension, calculate the deviation value of the average proportion of different label types under that dimension; Frame-level and video-level annotation dimensions with deviation values ​​less than a preset deviation threshold are classified as high-difficulty dimensions; frame-level and video-level annotation dimensions with deviation values ​​greater than or equal to the preset deviation threshold are classified as low-difficulty dimensions. Determine the parameter configuration of the random forest model during training for the high recognition difficulty dimension, and determine the parameter configuration of the random forest model during training for the low recognition difficulty dimension.

5. The dataset construction method for training large-scale visual language models as described in claim 4, characterized in that, The parameter configuration includes: sample sampling parameter configuration; Determining the parameter configuration of the random forest model for the high recognition difficulty dimension includes: For the high recognition difficulty dimension, the sampling parameter configuration performs oversampling on the label samples whose sample quantity ratio under the high recognition difficulty dimension is lower than the first ratio, and performs undersampling on the label samples whose sample quantity ratio is higher than the second ratio.

6. The dataset construction method for training large-scale visual language models as described in claim 4, characterized in that, The parameter configuration includes: feature weight allocation parameter configuration; Determining the parameter configuration of the random forest model for the high recognition difficulty dimension includes: For the high recognition difficulty dimension, the parameter configuration of the feature selection weight of the corresponding feature in the random forest decision tree splitting process is improved.

7. The dataset construction method for training large-scale visual language models as described in claim 4, characterized in that, The parameter configuration includes: decision tree structure parameter configuration; Determining the parameter configuration of the random forest model for the high recognition difficulty dimension includes: For the configuration of the high recognition difficulty dimension, the construction depth of a single decision tree in the random forest is increased, and the sampling ratio of the feature corresponding to the high recognition difficulty dimension in the feature subset of the decision tree is improved.

8. A dataset construction system for training large-scale visual language models, characterized in that, include: The tag acquisition module is used to acquire multiple videos and send a portion of the videos to a target device, so that the target device can return a frame-level tag and a video-level tag corresponding to each video in the portion of videos; wherein, the frame-level tag is used to characterize the authenticity of the corresponding video from multiple dimensions at the video frame level, and the video-level tag is used to characterize the authenticity of the corresponding video from multiple dimensions at the video level. The training dataset determination module is used to determine the basic training dataset based on the partial videos, the frame-level labels and video-level labels corresponding to each video in the partial videos; The model training and annotation module is used to train the random forest model based on the basic training dataset to obtain the target random forest model, and to annotate the remaining videos in the multiple videos with frame-level labels and video-level labels based on the target random forest model. The dataset construction module is used to construct a target dataset for training a large-scale visual language model by combining the partial videos, the frame-level labels and video-level labels corresponding to each video in the partial videos, and the remaining labeled videos.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images