Data set acquisition method, device and equipment and readable storage medium

Abstract

The invention discloses a data set acquisition method, device and equipment and a readable storage medium, and the method comprises the steps: obtaining a video set, and marking a text for a video in the video set; deleting the videos of which the video contents conform to the deletion condition in the video set to obtain a first video set; calculating relevance scores of the videos in the first video set and the annotated texts of the videos; based on the relevance score, deleting videos with inaccurate text annotations in the first video set to obtain a second video set; and storing videos in the second video set according to a sample data format of the video generation model. The technical effect of the invention is that multi-angle cleaning is carried out on the videos of the video set, so that the high-quality data set for training the video generation model is obtained.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data set acquisition method, device, equipment and readable storage medium. Background Art

[0002] The quality and scale of the pre-training dataset are crucial to the training quality of the model. Therefore, the development of video generation models is constrained by the lack of high-quality video-image-text datasets. Compared with image-text data, high-quality and large-scale video-text data is more difficult to obtain and more expensive to obtain.

[0003] Existing video-text datasets can be divided into two categories: one is mainly based on label annotation, which is mostly based on label annotation. Since it is manually annotated, it is usually accurate and of high quality, but the quantity is relatively small. The other is mainly based on video description. This type of annotation can be automatically annotated, but in different solutions, it still cannot fully meet the training requirements of video generation models due to various reasons.

[0004] In summary, how to obtain a pre-training data set that meets the training requirements of a video generation model is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of this application is to provide a data set acquisition method, device, equipment and readable storage medium, which can acquire a data set that meets the training requirements of a video generation model.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] A method for obtaining a data set, comprising:

[0008] Obtain a video set, and annotate texts on the videos in the video set;

[0009] Deleting videos in the video set whose video contents meet a deletion condition to obtain a first video set; wherein the deletion condition is preset with reference to a training requirement of a video generation model;

[0010] Calculating a correlation score between the videos in the first video set and the annotated texts thereof;

[0011] Based on the relevance score, deleting videos with inaccurate text annotations in the first video set to obtain a second video set;

[0012] The videos in the second video set are saved in a sample data format of the video generation model; wherein the data format is that a sample data includes a video, key frames in the video, and annotated text of the video.

[0013] Preferably, when the deletion condition corresponds to at least two conditions, deleting the videos in the video set whose video contents meet the deletion condition to obtain a first video set includes:

[0014] Determine the current deletion condition from the deletion conditions according to the deletion order;

[0015] Using a plurality of threads corresponding to the current deletion conditions, the videos meeting the current deletion conditions are detected from the video set and deleted;

[0016] After deleting the video that meets the current deletion condition, determining whether the current deletion condition is the last deletion condition;

[0017] If yes, determining the current video set as the first video set;

[0018] If not, the current deletion condition is redetermined from the deletion conditions according to the deletion order.

[0019] Preferably, the deletion condition includes a static video judgment condition; accordingly, deleting the video in the video set whose video content meets the deletion condition includes:

[0020] Calculating the optical flow of the videos in the video set, and performing statistics on the optical flow to obtain a statistical value;

[0021] When the statistical value meets the static video judgment condition, the corresponding video is deleted from the video set.

[0022] Preferably, the deletion condition includes a condition for determining a video with too many words. Accordingly, deleting the video in the video set whose video content meets the deletion condition includes:

[0023] Performing text recognition on image frames corresponding to the videos in the video set to obtain text amounts in the image frames in the videos;

[0024] When the text amount of an image frame is greater than the text threshold, determining that the image frame is a multi-text frame;

[0025] When the ratio of multi-text frames in a video meets the condition for determining a video with too many texts, the video is deleted from the video set.

[0026] Preferably, the deletion condition includes a single-person speech video judgment condition, and accordingly, deleting the video in the video set whose video content meets the deletion condition includes:

[0027] Using a facial detection library, performing portrait recognition on image frames corresponding to the videos in the video set;

[0028] Count the portrait recognition results to get the proportion of image frames that are mainly faces or upper bodies in the video;

[0029] When the proportion of image frames mainly featuring the face or the upper body in the video meets the single-person speech video judgment condition, the video is deleted from the video set.

[0030] Preferably, the deletion condition includes a bad video judgment condition, and accordingly, deleting the video in the video set whose video content meets the deletion condition includes:

[0031] Using a bad information database to detect bad information on image frames of videos in the video set;

[0032] When the detection result of the video meets the bad video judgment condition, the video is deleted from the video set.

[0033] Preferably, the deletion condition includes a special marked video judgment condition, and accordingly, deleting the video in the video set whose video content meets the deletion condition includes:

[0034] Obtain key frames of videos in the video set;

[0035] Detecting whether the key frame has a mark content that meets the special mark video judgment condition,

[0036] After detecting the logo content, determining a positioning frame covering the logo content;

[0037] The video to which the key frame belongs is cropped based on the positioning frame to obtain a video without the marked content;

[0038] The video with the marked content is deleted from the video collection, and the video without the marked content is added to the video collection.

[0039] Preferably, obtaining a video set includes:

[0040] Get keywords in different languages;

[0041] Searching video websites in different languages ​​based on the keywords;

[0042] Collect original videos whose lengths are within a specified range from the retrieval results;

[0043] Segmenting the original video;

[0044] Storing the videos obtained by video segmentation into the video set;

[0045] The step of segmenting the original video includes:

[0046] Segmenting the original video according to fixed lengths, and / or segmenting the original video according to scenes;

[0047] The step of segmenting the original video according to a fixed length includes:

[0048] Determine whether the original video is greater than the fixed length;

[0049] If yes, start from the beginning and segment the original video one by one with the fixed length as the interval;

[0050] If not, retaining the original video;

[0051] The step of segmenting the original video according to the scenes includes:

[0052] Detecting scenes of the original video;

[0053] In the case where only a single scene is detected, retaining the original video;

[0054] When at least two scenes are detected, the original video is segmented at the scene switching position.

[0055] A video processing device, comprising:

[0056] A video set acquisition module, used to acquire a video set and annotate texts on the videos in the video set;

[0057] A first cleaning module is used to delete videos in the video set whose video contents meet a deletion condition to obtain a first video set; the deletion condition is preset with reference to a training requirement of a video generation model;

[0058] A relevance analysis module, used to calculate the relevance score between the videos in the first video set and their annotated texts;

[0059] A second cleaning module is used to delete videos with inaccurate text annotations in the first video set based on the relevance score to obtain a second video set;

[0060] The sample saving module is used to save the videos in the second video set according to the sample data format of the video generation model; wherein the data format is that a sample data includes a video, key frames in the video and annotated text of the video.

[0061] Preferably, the first cleaning module is specifically used to determine the current deletion condition from the deletion conditions in a deletion order when the deletion condition corresponds to at least two conditions; use multiple threads corresponding to the current deletion condition to detect videos that meet the current deletion condition from the video set and delete them; after deleting the videos that meet the current deletion condition, determine whether the current deletion condition is the last deletion condition; if so, determine the current video set as the first video set; if not, redetermine the current deletion condition from the deletion conditions in the deletion order.

[0062] An electronic device, comprising:

[0063] Memory for storing computer programs;

[0064] A processor is used to implement the steps of the above-mentioned data set acquisition method when executing the computer program.

[0065] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data set acquisition method are implemented.

[0066] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned data set acquisition method.

[0067] Apply the method provided in the embodiment of the present application to obtain a video set and annotate texts on the videos in the video set; delete the videos in the video set whose video contents meet the deletion conditions to obtain a first video set; wherein the deletion conditions are pre-set with reference to the training requirements of a video generation model; calculate the correlation scores between the videos in the first video set and their annotated texts; based on the correlation scores, delete the videos in the first video set with inaccurate text annotations to obtain a second video set; save the videos in the second video set in a sample data format of a video generation model; wherein the data format is a sample data including a video, key frames in the video, and annotated text of the video.

[0068] First, a large number of video sets are obtained, and then the videos in the video set are annotated with text. In order to make the final data set meet the training requirements of the video generation model, the videos in the video set need to be cleaned. Specifically, the deletion conditions of the videos can be set in advance according to the training requirements of the video generation model, and then, after obtaining the video set, the videos that meet the deletion conditions can be deleted, thereby obtaining the first video set corresponding to the training requirements of the video generation model. After obtaining the first video set that meets the training requirements of the video generation model, the relevance of the videos and their annotated texts in the first video set is scored. Based on the relevance score, the videos with inaccurate text annotations in the first video set can be deleted, thereby obtaining a second video set with a stronger correlation between the video and the annotated text. Then, the second video set is saved in the sample data format of the video generation model, that is, a high-quality data set for training the video generation model is obtained.

[0069] The technical effect of the present application is that the video quality can be effectively improved by cleaning the video set from multiple angles. In other words, the present application can obtain a high-quality data set for training a video generation model.

[0070] Accordingly, the embodiments of the present application also provide a data set acquisition device, apparatus, readable storage medium and computer program product corresponding to the above-mentioned data set acquisition method, which have the above-mentioned technical effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] FIG1 is a flowchart of a method for acquiring a data set in an embodiment of the present application;

[0073] FIG2 is a schematic diagram of the structure of a data set acquisition device in an embodiment of the present application;

[0074] FIG3 is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0075] FIG. 4 is a schematic diagram of the specific structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0077] Please refer to FIG. 1 , which is a flow chart of a method for obtaining a data set in an embodiment of the present application. The method includes the following steps:

[0078] S101: Obtain a video set, and annotate texts on the videos in the video set.

[0079] In the present application, the pre-stored video collection can be directly read from the storage medium, or the video collection can be obtained from the network through online acquisition.

[0080] In a specific implementation of the present application, obtaining a video set includes:

[0081] Step 1: Get keywords in different languages;

[0082] Step 2: Search video websites in different languages ​​based on keywords;

[0083] Step 3: Collect original videos whose lengths are within a specified range from the search results;

[0084] Step 4: Segment the original video;

[0085] Step 5, storing the videos obtained by video segmentation into a video set;

[0086] Among them, step 4 segments the original video, including:

[0087] Segmenting the original video according to fixed lengths, and / or segmenting the original video according to scenes;

[0088] The original video is segmented according to a fixed length, including:

[0089] Step 1: Determine whether the original video is longer than a fixed length;

[0090] Step 2, if yes, start from the beginning and split the original video one by one at intervals of fixed length;

[0091] Step 3. If no, keep the original video;

[0092] The original video is segmented according to the scene, including:

[0093] Step 1: Detect the scene of the original video;

[0094] Step 2: When only a single scene is detected, keep the original video;

[0095] Step 3: When at least two scenes are detected, the original video is segmented at the scene switching position.

[0096] For ease of description, the above steps are combined for explanation below.

[0097] In practical applications, keywords in different languages ​​can be set so that the video generation model can learn the generation knowledge of videos in different language types. In practical applications, videos can be collected from Chinese and English video websites using M Chinese and English keywords, and then the video length can be controlled within L. min and L max between.

[0098] Specifically, in order to control the relevant content, language and proportion of the video, the video can be obtained by keyword retrieval and the length of the video can be limited.

[0099] Among them, keywords include but are not limited to: news, variety shows, celebrities, sports, film and television, beauty, pets, etc. The corresponding high-definition videos are obtained through keyword retrieval, and the proportion of the video is calculated. Keywords include Chinese and English. For Chinese keywords, you can search on videos or short video websites that are mainly in Chinese; for English keywords, you can search on video websites that are mainly in English. The original video source will be in a mixed form of Chinese and English. In addition to the video itself, the collected information also includes the video title, subtitle, keywords, subtitles, video link, etc.

[0100] The length of the video is controlled within L min and L max The purpose of controlling the minimum length is to prevent the content in the video from being too simple, while the purpose of controlling the maximum length is to prevent too many short videos from coming from the same video source after the video is segmented.

[0101] The original video can be split into two ways: sub , or switch splits by scene.

[0102] Given the current computing power and task requirements, the video generation model cannot read in too long a video at one time, so the original video needs to be cut. sub When cutting, if the video length is greater than L sub , then start from the beginning according to L sub Split the video one by one at intervals; when the video length is less than L subWhen the video is split according to scene switching, an open source tool is used to detect the video scene and split the video according to the scene switching.

[0103] Fixed length L sub The method is relatively simple and may include multiple scenes, which is conducive to the video model learning the scene switching method, but the beginning and end of the video may be abrupt. When segmenting by scene switching, the length of the obtained videos varies, but it is more convenient for video annotation and model learning changes in a single scene. Both methods can be retained when segmenting videos, that is, when segmenting videos, they can be segmented based on length or scene.

[0104] After obtaining the video collection, you can annotate the videos in the video collection with text. The specific annotation method can identify the video content and use the text corresponding to the video content as the annotated text of the video. You can also use the source-related information of the video, such as which website it comes from, its corresponding download volume, reading volume, etc. as the annotated text.

[0105] For specific information on how to annotate the video with text information, please refer to the relevant technical solutions, which will not be described in detail here.

[0106] S102: Delete videos in the video set whose video contents meet the deletion condition to obtain a first video set.

[0107] Among them, the deletion condition is pre-set with reference to the training requirements of the video generation model.

[0108] In this application, the deletion conditions can be set in advance according to the training requirements of the video generation model. For example, if static data or data with low motion is added to the data set, it will greatly damage the generation effect of the video generation model, and many frames of data will be wasted. Therefore, the deletion condition can be set as a static video judgment condition. In other words, when the static video judgment condition is met, it indicates that the video is a static video and needs to be deleted. According to the specific training requirements of the video generation model, one or more deletion conditions can be set.

[0109] After obtaining the video set, the videos in the video set can be matched with the deletion condition, so as to find the videos that meet the deletion condition and delete them. In this embodiment, for the sake of distinction, the video set after deleting all the videos that meet the deletion condition is called the first video set.

[0110] In a specific implementation of the present application, in order to achieve large-scale acquisition of a data set, when the deletion condition corresponds to at least two conditions, videos in the video set whose video content meets the deletion condition are deleted to obtain a first video set, including:

[0111] According to the deletion order, determine the current deletion condition from the deletion conditions;

[0112] Using multiple threads corresponding to the current deletion conditions, the videos meeting the current deletion conditions are detected from the video set and deleted;

[0113] After deleting the video that meets the current deletion condition, determine whether the current deletion condition is the last deletion condition;

[0114] If yes, the current video set is determined as the first video set;

[0115] If not, the current deletion condition is redetermined from the deletion conditions according to the deletion order.

[0116] For ease of description, the above steps are combined for explanation below.

[0117] There can be multiple deletion conditions, that is, multiple types of videos need to be deleted.

[0118] When multiple types of videos need to be deleted, the order of the videos to be deleted can be set according to the actual situation. For example, the deletion order of the videos of a certain type with the largest estimated number can be placed in the front, and the deletion order of the videos of a certain type with the smallest estimated number can be placed in the back.

[0119] According to the deletion order, the current deletion condition can be determined from the deletion conditions, and then, multiple threads corresponding to the current deletion conditions are used to execute in parallel to retrieve videos that meet the current deletion condition from the video set and delete them. After deleting the videos that meet the current deletion condition, if the current deletion condition is the last deletion condition, it means that the deletion of the videos that meet the deletion condition has been completed, and the current video set can be determined as the first video set. If the current deletion condition is not the last deletion condition, it means that the deletion operation of all videos that meet the deletion condition has not been completed. At this time, it is necessary to re-determine the current deletion condition from the deletion conditions according to the deletion order until the deletion of all videos that meet the deletion condition is completed.

[0120] In a specific implementation of the present application, when executing step S102, deleting videos in the video set whose video contents meet the deletion condition, obtaining a first video set includes:

[0121] Perform frame rate compression on videos in the video set;

[0122] Detect whether the video after frame rate compression matches the deletion condition;

[0123] If yes, the original video before the compression frame rate is deleted from the video set.

[0124] In order to speed up the detection and deletion of videos that meet the deletion conditions, in this embodiment, the video frame rate can be compressed, and then the compressed video frame rate can be detected to see if it matches the deletion conditions. The specific compression degree can be set and adjusted according to the actual computing resources and task volume. It is only necessary to compress the video frame rate to ensure that the compressed video can replace the original video to detect videos that meet the deletion conditions.

[0125] If there is a match, the original video corresponding to the compressed frame rate video in the video set needs to be deleted. In this way, the amount of calculation can be greatly reduced, which can not only speed up the calculation, but also expand the scale of the processed data set.

[0126] In a specific implementation of the present application, the deletion condition includes a static video judgment condition; accordingly, deleting videos in the video set whose video content meets the deletion condition includes:

[0127] Calculate the optical flow of the videos in the video set, and perform statistics on the optical flow to obtain statistical values;

[0128] When the statistical value meets the static video judgment condition, the corresponding video is deleted from the video set.

[0129] For ease of description, the above steps are combined for explanation below.

[0130] If static data or data with low motion is added to the dataset, it will have a great impact on the generation effect of the model, and a lot of frames of data will be wasted. Therefore, optical flow can be used to detect and clean up static videos.

[0131] You can use the open source optical flow estimation model to process the video. To save computational effort, you can first compress the video frame rate (FPS) to FPS. sample , and then calculate the average optical flow of the compressed video Sum mean difference , and then keep the videos that meet the following criteria (that is, the videos that do not meet the following criteria are still videos):

[0132] Greater than a threshold ;

[0133] when Less than threshold hour, Less than a certain threshold R;

[0134] when When it is greater than the threshold R, Greater than a threshold ;

[0135] The specific value of each threshold value can be set or adjusted according to actual needs and is not specifically limited here.

[0136] The static video judgment condition is that the video does not meet the above-mentioned standards related to optical flow.

[0137] In a specific implementation of the present application, the deletion condition includes a judgment condition of a video with too many words. Accordingly, deleting videos in the video set whose video content meets the deletion condition includes:

[0138] Perform text recognition on image frames corresponding to the videos in the video set to obtain the text quantity of the image frames in the videos;

[0139] When the text amount of an image frame is greater than the text threshold, determining that the image frame is a multi-text frame;

[0140] When the ratio of multi-text frames in a video meets the criteria for determining a video with too much text, the video is deleted from the video set.

[0141] For ease of description, the above steps are combined for explanation below.

[0142] Videos with too much text mostly correspond to the cast and crew list of film and television works, course PPT explanations, etc., which usually lack general video elements. Therefore, in the embodiment, an open source OCR model can be used to detect and then remove videos with too much text.

[0143] In order to save computational effort, the frame rate (FPS) of the video can be compressed to F sample Then, calculate the text ratio of the frames in the compressed video. And delete the videos that meet the following requirements (judgment conditions for videos with too much text):

[0144] After OCR detection, the amount of text in the frame is greater than a certain threshold ;

[0145] The number of frames that meet the above annotations is greater than the number of all frames .

[0146] In a specific implementation of the present application, the deletion condition includes a single-person speech video judgment condition. Accordingly, deleting videos in the video set whose video content meets the deletion condition includes:

[0147] Use the face detection library to perform portrait recognition on the image frames corresponding to the videos in the video set;

[0148] Count the portrait recognition results to get the proportion of image frames that are mainly faces or upper bodies in the video;

[0149] If the proportion of image frames dominated by the face or the upper body in the video meets the single-person speech video judgment condition, the video is deleted from the video set.

[0150] For ease of description, the above steps are combined for explanation below.

[0151] Videos with only one person speaking are usually news broadcasts or video bloggers' commentary. Most of them have the problem that the text annotations do not match the actual content of the video. Therefore, if such samples are collected in the original video, they will be removed in this step.

[0152] In order to save computational effort, the video frame rate (FPS) is compressed to F sample , and then use the open source face detection library to process the frames in the video. Calculate the proportion of frames in the compressed video that are mainly faces or upper bodies. And delete videos that meet the following requirements (single-person speech video judgment conditions):

[0153] According to the model detection, the proportion of the face or the upper body in the frame is greater than a certain threshold ;

[0154] The number of frames that meet the above annotations is greater than the number of all frames .

[0155] In a specific implementation of the present application, the deletion condition includes a bad video judgment condition, and accordingly, deleting videos in the video set whose video content meets the deletion condition includes:

[0156] Using the bad information database to detect bad information on image frames of videos in the video set;

[0157] When the detection result of the video meets the bad video judgment condition, the video is deleted from the video set.

[0158] For ease of description, the above steps are combined for explanation below.

[0159] In order to save computational effort, the frame rate (FPS) of the video can be compressed to F sample Then, use the open source nsfw library (Not Suitable For Work, a bad information library) to process the frames in the video. And delete the videos that meet the following requirements (bad video judgment conditions):

[0160] After model detection, the nsfw value of the frame is greater than a certain threshold ;

[0161] As long as a frame is detected that is larger than the threshold, the entire video is deleted.

[0162] In a specific implementation of the present application, the deletion condition includes a special marking video judgment condition, and accordingly, deleting videos in the video set whose video content meets the deletion condition includes:

[0163] Get the key frames of the videos in the video set;

[0164] Check whether the key frame has the mark content that meets the special mark video judgment conditions,

[0165] After detecting the logo content, determining a positioning frame covering the logo content;

[0166] The video to which the key frame belongs is cropped based on the positioning frame to obtain a video without logo content;

[0167] Remove videos with flagged content from the video collection and add videos without flagged content to the video collection.

[0168] For ease of description, the above steps are combined for explanation below.

[0169] In order to save computational effort, the middle frame of the video can be used as the key frame to detect the logos and watermarks at the corners and other signs corresponding to the special marking video judgment conditions. Since the watermarks and logos of the video are usually at the four corners or the bottom of the video, an open source text detection model can be used to locate a certain proportion of the top of the video. A certain proportion to the bottom If the keyframe contains text, the corresponding text in the video will be or Cut it off.

[0170] S103: Calculate the relevance scores between the videos in the first video set and their annotated texts.

[0171] Specifically, a contrastive learning model may be used to score the relevance of a video to its annotated text, thereby obtaining a relevance score.

[0172] S104: Based on the relevance scores, delete the videos with inaccurate text annotations in the first video set to obtain a second video set.

[0173] In this embodiment, a relevance threshold may be set, and videos with a relevance threshold lower than the relevance threshold may be determined as videos with inaccurate text annotations and deleted, thereby obtaining a second video set.

[0174] S105: Save the videos in the second video set according to the sample data format of the video generation model.

[0175] The data format is that a sample data includes a video, key frames in the video and annotated text of the video.

[0176] The key frames may be the first frame, the middle frame, and the last frame of the video.

[0177] After obtaining the second video set, the videos in the second video set can be saved according to the sample data format corresponding to the video generation model. The data format is that a sample data includes a video, a key frame of the video, and annotated text of the video.

[0178] It should be noted that based on the saved data set, text-video data pairs, picture-video data pairs, and text-picture-video data pairs can be composed by reading different contents for use in different model training.

[0179] The method provided by the embodiment of the present application is applied to obtain a video set, and annotate texts on the videos in the video set; delete videos in the video set whose video content meets the deletion condition to obtain a first video set; wherein the deletion condition is preset with reference to the training requirements of the video generation model; calculate the correlation score between the videos in the first video set and their annotated texts; based on the correlation score, delete the videos in the first video set with inaccurate text annotations to obtain a second video set; save the videos in the second video set according to the sample data format of the video generation model; wherein the data format is that a sample data includes a video, a key frame in the video and the annotated text of the video; when the deletion condition corresponds to at least two conditions, delete the videos in the video set whose video content meets the deletion condition to obtain the first video set, including: determining the current deletion condition from the deletion conditions in a deletion order; using multiple threads corresponding to the current deletion condition to detect videos that meet the current deletion condition from the video set and delete them; after deleting the videos that meet the current deletion condition, determine whether the current deletion condition is the last deletion condition; if yes, determine the current video set as the first video set; if not, redetermine the current deletion condition from the deletion conditions in a deletion order.

[0180] First, a large number of video sets are obtained, and then the videos in the video sets are annotated with text. In order to make the final data set meet the training requirements of the video generation model, the videos in the video set need to be cleaned. Specifically, the deletion conditions of the videos can be set in advance according to the training requirements of the video generation model, and then, after the video set is obtained, the videos that meet the deletion conditions can be deleted, thereby obtaining the first video set corresponding to the training requirements of the video generation model. In order to obtain a large-scale data set, when cleaning the video, multi-threading is used to process the videos that meet the deletion conditions in parallel by type. Multi-threading can effectively accelerate the detection and deletion of videos that meet a single deletion condition, and process the videos that meet the deletion conditions by type. In the case of multiple deletion conditions, the number of subsequent deletion condition detections can be gradually reduced, and the computing power consumption can be reduced, thereby processing large-scale videos. After obtaining a large-scale first video set that meets the training requirements of the video generation model, the relevance of the videos and their annotated texts in the first video set is scored. Based on the relevance score, the videos with inaccurate text annotations in the first video set can be deleted, thereby obtaining a second video set with a stronger relevance between the videos and the annotated texts. Then, the second video set is saved in the sample data format of the video generation model, thereby obtaining a high-quality data set for training the video generation model.

[0181] The technical effect of this application is that by cleaning the videos in the video set from multiple angles, the video quality can be effectively improved. When cleaning the videos, parallel acceleration processing can be performed one by one, which can save computing power and achieve large-scale data acquisition. In other words, this application can obtain high-quality, large-scale data sets for training video generation models.

[0182] Corresponding to the above method embodiment, the embodiment of the present application further provides a data set acquisition device. The data set acquisition device described below and the data set acquisition method described above can refer to each other.

[0183] As shown in FIG2 , the device includes the following modules:

[0184] The video set acquisition module 101 is used to acquire a video set and annotate texts on the videos in the video set;

[0185] The first cleaning module 102 is used to delete the videos in the video set whose video contents meet the deletion condition to obtain the first video set; the deletion condition is pre-set with reference to the training requirements of the video generation model;

[0186] A relevance analysis module 103 is used to calculate the relevance scores between the videos in the first video set and their annotated texts;

[0187] A second cleaning module 104 is used to delete videos with inaccurate text annotations in the first video set based on the relevance score to obtain a second video set;

[0188] The sample saving module 105 is used to save the videos in the second video set according to the sample data format of the video generation model; wherein the data format is that a sample data includes a video, a key frame in the video and annotated text of the video;

[0189] Using the device provided in the embodiment of the present application, a video set is obtained, and text is annotated on the videos in the video set; videos in the video set whose video content meets the deletion condition are deleted to obtain a first video set; wherein the deletion condition is pre-set with reference to the training requirements of the video generation model; the correlation score between the videos in the first video set and their annotated text is calculated; based on the correlation score, the videos in the first video set with inaccurate text annotations are deleted to obtain a second video set; the videos in the second video set are saved in a sample data format of the video generation model; wherein the data format is a sample data including a video, key frames in the video, and annotated text of the video.

[0190] First, a large number of video sets are obtained, and then the videos in the video set are annotated with text. In order to make the final data set meet the training requirements of the video generation model, the videos in the video set need to be cleaned. Specifically, the deletion conditions of the videos can be set in advance according to the training requirements of the video generation model, and then, after obtaining the video set, the videos that meet the deletion conditions can be deleted, thereby obtaining the first video set corresponding to the training requirements of the video generation model. After the first video set that meets the training requirements of the video generation model, the relevance of the videos and their annotated texts in the first video set is scored. Based on the relevance score, the videos with inaccurate text annotations in the first video set can be deleted, thereby obtaining a second video set with a stronger correlation between the video and the annotated text. Then, the second video set is saved in the sample data format of the video generation model, that is, a high-quality data set for training the video generation model is obtained.

[0191] The technical effect of this application is that by cleaning the videos in the video set from multiple angles, the video quality can be effectively improved. When cleaning the videos, parallel acceleration processing can be performed one by one, which can save computing power and achieve large-scale data acquisition. In other words, this application can obtain high-quality, large-scale data sets for training video generation models.

[0192] In a specific embodiment of the present application, the first cleaning module is specifically used to determine the current deletion condition from the deletion conditions in the deletion order when the deletion condition corresponds to at least two conditions; use multiple threads corresponding to the current deletion condition to detect videos that meet the current deletion condition from the video set and delete them; after deleting the videos that meet the current deletion condition, determine whether the current deletion condition is the last deletion condition; if so, determine the current video set as the first video set; if not, redetermine the current deletion condition from the deletion conditions in the deletion order.

[0193] That is, in order to obtain a large-scale data set, when performing video cleaning, multi-threading is used to process videos that meet the deletion conditions in parallel by type. Multi-threading can effectively accelerate the detection and deletion of videos that meet a single deletion condition. Processing videos that meet the deletion conditions by type can gradually reduce the number of subsequent deletion condition detections when there are multiple deletion conditions, reduce computing power consumption, and thus process large-scale videos.

[0194] In a specific implementation of the present application, the deletion condition includes a static video judgment condition; accordingly, the first cleaning module is specifically used to calculate the optical flow of the video in the video set, and to perform statistics on the optical flow to obtain a statistical value;

[0195] When the statistical value meets the static video judgment condition, the corresponding video is deleted from the video set.

[0196] In a specific implementation of the present application, the deletion condition includes a video judgment condition of too much text, and accordingly, the first cleaning module is specifically used to perform text recognition on the image frames corresponding to the videos in the video set to obtain the text amount in the image frames in the video;

[0197] When the text amount of an image frame is greater than the text threshold, determining that the image frame is a multi-text frame;

[0198] When the ratio of multi-text frames in a video meets the criteria for determining a video with too much text, the video is deleted from the video set.

[0199] In a specific implementation of the present application, the deletion condition includes a single-person speech video judgment condition, and accordingly, the first cleaning module is specifically used to use a face detection library to perform portrait recognition on image frames corresponding to the videos in the video set;

[0200] Count the portrait recognition results to get the proportion of image frames that are mainly faces or upper bodies in the video;

[0201] If the proportion of image frames dominated by the face or the upper body in the video meets the single-person speech video judgment condition, the video is deleted from the video set.

[0202] In a specific implementation of the present application, the deletion condition includes a bad video judgment condition, and accordingly, the first cleaning module is specifically used to detect bad information on image frames of videos in the video set using the bad information library;

[0203] When the detection result of the video meets the bad video judgment condition, the video is deleted from the video set.

[0204] In a specific implementation of the present application, the deletion condition includes a special marking video judgment condition, and accordingly, the first cleaning module is specifically used to obtain key frames of videos in the video set;

[0205] Check whether the key frame has the mark content that meets the special mark video judgment conditions,

[0206] After detecting the logo content, determining a positioning frame covering the logo content;

[0207] The video to which the key frame belongs is cropped based on the positioning frame to obtain a video without logo content;

[0208] Remove videos with flagged content from the video collection and add videos without flagged content to the video collection.

[0209] In a specific implementation of the present application, the video set acquisition module is specifically used to acquire keywords in different languages;

[0210] Search video websites in different languages ​​based on keywords;

[0211] Collect original videos whose lengths are within a specified range from the retrieval results;

[0212] Segment the original video;

[0213] The videos obtained by video segmentation are stored in a video set;

[0214] The original video is segmented, including:

[0215] Segmenting the original video according to fixed lengths, and / or segmenting the original video according to scenes;

[0216] The original video is segmented according to a fixed length, including:

[0217] Determine whether the original video is greater than a fixed length;

[0218] If yes, start from the beginning and split the original video one by one at intervals of fixed length;

[0219] If no, keep the original video;

[0220] The original video is segmented according to the scene, including:

[0221] Detect scenes in raw videos;

[0222] In the case where only a single scene is detected, the original video is retained;

[0223] When at least two scenes are detected, the original video is segmented at the scene switching position.

[0224] Corresponding to the above method embodiment, the embodiment of the present application further provides an electronic device. The electronic device described below and the data set acquisition method described above can refer to each other.

[0225] As shown in FIG3 , the electronic device includes:

[0226] A memory 332, for storing computer programs;

[0227] The processor 322 is configured to implement the steps of the data set acquisition method of the above method embodiment when executing a computer program.

[0228] Specifically, please refer to FIG. 4, which is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.

[0229] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0230] The steps in the data set acquisition method described above can be implemented by the structure of an electronic device.

[0231] Corresponding to the above method embodiment, the embodiment of the present application further provides a readable storage medium. The readable storage medium described below and the data set acquisition method described above can refer to each other.

[0232] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data set acquisition method of the above method embodiment are implemented.

[0233] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0234] Corresponding to the above method embodiment, the embodiment of the present application further provides a computer program product. The computer program product described below and the data set acquisition method described above can refer to each other.

[0235] A computer program product includes a computer program / instruction. When the computer program / instruction is executed by a processor, the steps of the above-mentioned data set acquisition method are implemented.

[0236] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0237] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0238] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0239] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms include, include or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0240] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for acquiring a data set, characterized in that: include: Obtain a video set, and annotate texts on the videos in the video set; Deleting videos in the video set whose video contents meet a deletion condition to obtain a first video set; wherein the deletion condition is preset with reference to a training requirement of a video generation model; Calculating a correlation score between the videos in the first video set and the annotated texts thereof; Based on the relevance score, deleting videos with inaccurate text annotations in the first video set to obtain a second video set; The videos in the second video set are saved in a sample data format of the video generation model; wherein the data format is that a sample data includes a video, key frames in the video, and annotated text of the video.

2. The method according to claim 1, characterized in that include: In the case where the deletion condition corresponds to at least two conditions, deleting videos in the video set whose video contents meet the deletion condition to obtain a first video set, including: Determine the current deletion condition from the deletion conditions according to the deletion order; Using a plurality of threads corresponding to the current deletion conditions, the videos meeting the current deletion conditions are detected from the video set and deleted; After deleting the video that meets the current deletion condition, determining whether the current deletion condition is the last deletion condition; If yes, determining the current video set as the first video set; If not, the current deletion condition is redetermined from the deletion conditions according to the deletion order.

3. The method according to claim 1, characterized in that The deletion condition includes a static video judgment condition; accordingly, deleting the video in the video set whose video content meets the deletion condition includes: Calculating the optical flow of the videos in the video set, and performing statistics on the optical flow to obtain a statistical value; When the statistical value meets the static video judgment condition, the corresponding video is deleted from the video set.

4. The method according to claim 1, characterized in that: The deletion condition includes a judgment condition for videos with too many words; accordingly, deleting videos in the video collection whose video content meets the deletion condition includes: Performing text recognition on image frames corresponding to the videos in the video set to obtain text amounts in the image frames in the videos; When the text amount of an image frame is greater than the text threshold, determining that the image frame is a multi-text frame; When the ratio of multi-text frames in a video meets the condition for determining a video with too many texts, the video is deleted from the video set.

5. The method according to claim 1, characterized in that The deletion condition includes a single-person speech video judgment condition. Accordingly, deleting the video in the video set whose video content meets the deletion condition includes: Using a facial detection library, performing portrait recognition on image frames corresponding to the videos in the video set; Count the portrait recognition results to get the proportion of image frames that are mainly faces or upper bodies in the video; When the proportion of image frames mainly featuring the face or the upper body in the video meets the single-person speech video judgment condition, the video is deleted from the video set.

6. The method according to claim 1, characterized in that The deletion condition includes a bad video judgment condition. Accordingly, deleting the video in the video set whose video content meets the deletion condition includes: Using a bad information database to detect bad information on image frames of videos in the video set; When the detection result of the video meets the bad video judgment condition, the video is deleted from the video set.

7. The method according to claim 1, characterized in that The deletion condition includes a special marked video judgment condition. Accordingly, deleting the video in the video set whose video content meets the deletion condition includes: Obtain key frames of videos in the video set; Detecting whether the key frame has a mark content that meets the special mark video judgment condition, After detecting the logo content, determining a positioning frame covering the logo content; The video to which the key frame belongs is cropped based on the positioning frame to obtain a video without the marked content; The video with the marked content is deleted from the video collection, and the video without the marked content is added to the video collection.

8. The method according to any one of claims 1 to 7, characterized in that: Get a collection of videos including: Get keywords in different languages; Searching video websites in different languages ​​based on the keywords; Collect original videos whose lengths are within a specified range from the retrieval results; Segmenting the original video; Storing the videos obtained by video segmentation into the video set; The step of segmenting the original video includes: Segmenting the original video according to fixed lengths, and / or segmenting the original video according to scenes; The step of segmenting the original video according to a fixed length includes: Determine whether the original video is greater than the fixed length; If yes, start from the beginning and segment the original video one by one with the fixed length as the interval; If not, retaining the original video; The step of segmenting the original video according to the scenes includes: Detecting a scene of the original video; In the case where only a single scene is detected, retaining the original video; When at least two scenes are detected, the original video is segmented at the scene switching position.

9. A video processing device, characterized in that: include: A video set acquisition module, used to acquire a video set and annotate texts on the videos in the video set; A first cleaning module is used to delete videos in the video set whose video contents meet a deletion condition to obtain a first video set; the deletion condition is preset with reference to a training requirement of a video generation model; A relevance analysis module, used to calculate the relevance score between the videos in the first video set and their annotated texts; A second cleaning module is used to delete videos with inaccurate text annotations in the first video set based on the relevance score to obtain a second video set; The sample saving module is used to save the videos in the second video set according to the sample data format of the video generation model; wherein the data format is that a sample data includes a video, key frames in the video and annotated text of the video.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for acquiring a data set according to any one of claims 1 to 8 when executing the computer program.

11. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data set acquisition method according to any one of claims 1 to 8 are implemented.

Images