Data set construction method for film frame insertion

Abstract

The invention discloses a data set construction method for movie frame insertion, which relates to the technical field of information and comprises the following steps: acquiring input video data from a movie video; decoding the input video data to obtain a video frame sequence; performing picture quality evaluation and motion quality evaluation on the video frame sequence, and screening out video samples meeting a preset standard; and performing data enhancement processing on the video sample meeting the preset standard, identifying and cutting video post-production content, and generating a final data sample. According to the method, millions of effective data can be collected, the effective data exceed more than ten times of an existing frame insertion data set, the development of a follow-up film frame insertion algorithm can be better supported, and a final video scheme can have higher frame insertion precision and a better frame insertion effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of information technology, and in particular relates to a data set construction method for movie interpolation. Background Art

[0002] The current mainstream video interpolation algorithms rely on deep learning and generative networks, which have extremely high requirements for training data:

[0003] The dataset should be large enough to ensure that the algorithm can learn sufficiently rich and complex motion types and color styles, preferably with a data size of millions;

[0004] The motion distribution of data in the dataset should be uniform and stable, excluding static, transition, and non-motion-related scenes, such as dynamic textures and CG rendering content;

[0005] The dataset should be collected from movies and ensure that the image quality is high enough, without blurring, distortion, noise, and other image problems.

[0006] Currently, mainstream datasets primarily come from academia. Typical examples include Vimeo Septuplet and X4K1000FPS. However, these datasets are typically based on live, unrestricted online videos, with data sizes of less than 100,000, and relatively simple motion. Therefore, we propose a method for constructing a dataset for movie interpolation to address the following issues:

[0007] Automated Collection: Data sets of millions of units cannot be collected, filtered, and enhanced manually. Therefore, an automated collection solution must be designed to complete the dataset construction.

[0008] Data screening: The solution collects data from completed films. The data samples collected cannot guarantee meaningful motion: the images may be static, dynamic textures, or CG. Furthermore, due to the diversity of data sources, we cannot guarantee the image quality of the samples. The collected samples may contain noise, blur, and other image quality issues.

[0009] Data augmentation: Movies undergo post-production, and subtitles, logos, black borders, and other post-production content are pressed into the screen. These contents are inconsistent with the movement of the picture and will introduce misleading information to the network during training. Summary of the Invention

[0010] The purpose of the present invention is to provide a method for constructing a dataset for movie interpolation, which solves the problem that previous video interpolation datasets only use small-scale datasets and lack discussion and screening of video image quality and inter-frame motion quality. Only through algorithms, ultra-large-scale datasets are automatically collected to better support the development of interpolation algorithms.

[0011] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0012] A method for constructing a dataset for movie interpolation mainly includes:

[0013] The method includes obtaining input video data from a movie video, decoding the input video data to obtain a video frame sequence, performing picture quality assessment and motion quality assessment on the video frame sequence, and screening out video samples that meet preset standards, performing data enhancement processing on the video samples that meet the preset standards, identifying and cropping video post-production content, and generating final data samples.

[0014] Furthermore, the decoding process of the input video data to obtain a video frame sequence includes: using a preset decoding tool to perform a decoding operation on the input video data to generate a video frame sequence containing multiple consecutive frames; for the video frame sequence, segmenting it according to a preset segmentation method to obtain multiple data segments, each data segment containing a preset number of video frames; and providing basic data for subsequent picture quality evaluation and motion quality evaluation by frame-by-frame analysis of the data segments.

[0015] Furthermore, the video frame sequence is evaluated for picture quality to select video samples that meet preset standards, including: for each data segment in the video frame sequence, obtaining image features of each video frame therein; judging whether there is a preset picture quality problem based on the image features; if the picture quality problem exists, eliminating the corresponding data segment; and generating a set of video samples that meet the picture quality standards by summarizing the remaining data segments.

[0016] Furthermore, the motion quality assessment of the video frame sequence is performed to screen out video samples that meet preset standards, including: calculating inter-frame optical flow data for each data segment in the video frame sequence; judging whether there is a static or abnormal motion situation based on the statistical characteristics of the inter-frame optical flow data; if the static or abnormal motion situation exists, eliminating the corresponding data segment; and generating a set of video samples that meet the motion quality standards by analyzing the remaining data segments.

[0017] Furthermore, the data enhancement processing is performed on the video samples that meet the preset standards to identify and crop the video post-production content, including: for the video samples that meet the preset standards, obtaining difference data between each video frame; generating an initial mask based on the difference data and preset optical flow information; processing the initial mask through a preset neural network model to generate a post-production content mask; and locating and cropping the post-production content area in the video frame based on the post-production content mask.

[0018] Furthermore, the video samples that meet the preset standards are subjected to data enhancement processing to generate final data samples, including: obtaining a cropped video frame area for the video samples that meet the preset standards; searching and determining the largest image block that does not contain post-production content based on the cropped video frame area; generating the final data sample by extracting and integrating the largest image block; and storing and classifying the final data samples to provide data support for subsequent video interpolation algorithm training.

[0019] Furthermore, the video frame sequence is subjected to picture quality assessment and motion quality assessment to screen out video samples that meet preset standards, including: obtaining picture quality assessment results and motion quality assessment results for the video frame sequence; eliminating data segments with preset quality defects based on the picture quality assessment results; eliminating data segments with preset motion anomalies based on the motion quality assessment results; and generating a set of video samples that meet both picture quality and motion quality standards by integrating the remaining data segments.

[0020] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for constructing a data set for movie interpolation.

[0021] Another object of the present invention is to provide an electronic device comprising a memory and a processor, wherein the memory is used to store a computer program and the processor runs the computer program to enable the electronic device to execute the data set construction method for movie interpolation.

[0022] Beneficial effects:

[0023] The present invention discloses a method for constructing a dataset for movie interpolation. The method first decodes the input video to obtain a video frame sequence, and divides it into data segments in a preset manner. Then, the picture quality and motion quality of each data segment are evaluated, and segments that do not meet the standards are eliminated. For the screened video samples, an initial mask is generated by calculating the inter-frame difference and optical flow information, and then a neural network model is used to process it to obtain a post-production content mask for locating and cropping the post-production content area. Finally, the largest image block that does not contain post-production content is searched in the cropped area, and the final data sample is generated by extraction and integration. It can automatically collect large-scale data sets and can collect millions of valid data, which is more than ten times the existing traditional interpolation data sets, and can better support the development of subsequent movie interpolation algorithms.

[0024] The present invention determines the picture quality and motion continuity of movie videos and performs data screening based on their characteristics, thereby reducing abnormal data in the video interpolation dataset, improving the training efficiency and algorithm stability of the video interpolation algorithm for movies, and enabling the final video solution to have higher interpolation accuracy and better interpolation effect.

[0025] The present invention identifies and crops the post-production content of movies, ensuring the motion consistency of the collected data samples. It solves the problem of algorithm stability degradation caused by data fitting when the algorithm encounters moving objects with similar post-production content after training, such as blinds and billboards. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a method for constructing a data set for movie interpolation according to the present invention;

[0027] Figure 2 This is a data enhancement flow chart of the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0029] like Figure 1 As shown, a method for constructing a dataset for movie interpolation in this embodiment may specifically include:

[0030] The method includes obtaining input video data from a movie video, decoding the input video data to obtain a video frame sequence, performing picture quality assessment and motion quality assessment on the video frame sequence, and screening out video samples that meet preset standards, performing data enhancement processing on the video samples that meet the preset standards, identifying and cropping video post-production content, and generating final data samples.

[0031] The process of obtaining input video data from a movie video mainly refers to extracting original video materials suitable for subsequent processing from existing movie video resources.

[0032] For example.

[0033] In one possible implementation, the target movie video file can be obtained by accessing an online video platform or a locally stored movie file. For example, a 120-minute HD movie with an MP4 file format and a resolution of 1920x1080 is selected. This acquisition method ensures diverse input data, covering videos with different compression qualities and encoding formats, providing a rich sample for subsequent processing.

[0034] It's important to note that when acquiring input video data, we prioritize files that haven't undergone multiple transcoding cycles to minimize image quality loss and provide a good foundation for subsequent decoding and analysis. This approach maximizes the preservation of original video details and improves the accuracy of subsequent processing. Decoding the input video data to obtain a video frame sequence involves restoring the compressed video file to frame-by-frame image data for subsequent analysis and processing.

[0035] For example.

[0036] In one embodiment, assume that we use a video decoding tool to decode the MP4 movie file, setting the decoding parameters to 30 frames per second, and ultimately obtaining a video frame sequence containing 216,000 frames. During the decoding process, different decoding strategies can be selected based on hardware performance, such as using GPU-accelerated decoding to improve efficiency.

[0037] It's important to note that the decoded video frame sequence is stored in image format, with each frame containing complete pixel information, facilitating subsequent image quality testing and motion analysis. This processing method not only ensures data integrity but also provides clear input for subsequent algorithms, significantly improving the training effectiveness and stability of the video interpolation algorithm.

[0038] The input video data is decoded to obtain a video frame sequence: for example.

[0039] In one possible implementation, a common decoding tool, such as FFmpeg, can be used to decode the input movie video data. The decoding tool decomposes the compressed video file into raw image data, forming a video frame sequence consisting of multiple consecutive frames. For example, if the input is a 2-minute high-definition movie clip, decoding may generate a sequence of 3,600 frames, each with a resolution of 1920x1080 pixels. This converts complex video data into an image sequence that can be analyzed frame by frame, providing basic data support for subsequent quality assessment. Decoding also ensures the integrity of the video data, preventing data loss due to differences in compression formats. The video frame sequence is segmented according to a preset segmentation method to obtain multiple data segments.

[0040] For example, in practice, a video frame sequence can be segmented into units of five frames, resulting in multiple data segments, each containing five consecutive video frames. Assuming a 3,600-frame sequence, segmentation yields 720 data segments. This segmentation approach is based on the fact that a five-frame length can reflect short-term motion changes without increasing data processing complexity due to excessively long segments. After segmentation, each segment can be analyzed independently, facilitating subsequent assessment of image and motion characteristics while also facilitating parallel processing and improving efficiency. Frame-by-frame analysis of data segments provides basic data for subsequent image quality and motion quality assessments, for example.

[0041] In one embodiment, frame-by-frame analysis can include examining each frame's clarity, brightness distribution, and inter-frame motion vector variations. For example, in a five-frame clip, the analysis examines each frame for blur or noise, while also calculating pixel displacement between adjacent frames to assess motion continuity. This approach provides intuitive support for subsequent image quality assessments, such as eliminating blurry frames. Furthermore, motion quality assessments can also identify suitable clips for interpolation training based on inter-frame difference data. This frame-by-frame analysis lays the foundation for dataset optimization, ensuring the accuracy and stability of the final interpolation algorithm.

[0042] Step S104, performing picture quality assessment on the video frame sequence and screening out video samples that meet preset standards, includes: obtaining image features of each video frame in each data segment in the video frame sequence; judging whether there is a preset picture quality problem based on the image features; if the picture quality problem exists, removing the corresponding data segment; and generating a set of video samples that meet the picture quality standards by aggregating the remaining data segments.

[0043] For each data segment in the video frame sequence, image features of each video frame are obtained.

[0044] For example, when processing a video clip from a movie, image processing techniques can be used to extract characteristic parameters such as brightness, contrast, and sharpness for each frame. These characteristics reflect the basic quality of the image. For example, brightness can be used to determine whether the image is too dark or overexposed, while contrast reveals the clarity of details. Extracting these features provides a data foundation for subsequent image quality assessment. This aims to fully understand the visual performance of the video clip, laying the foundation for further screening. Based on the image characteristics, it is determined whether there are any pre-defined image quality issues.

[0045] For example, after analyzing the extracted features, certain quality thresholds can be set. For example, images with a clarity below a certain standard value of 60 are considered blurry, or images with a brightness value below 30 are considered too dark. If a frame or video clip exhibits these problems, such as blurriness or excessive noise, it is marked as unqualified. The advantage of this judgment method is that it can quickly identify video clips that do not meet the requirements, thus avoiding the use of low-quality data for subsequent training or analysis, and improving the overall quality of the final dataset. If there are any image quality issues, the corresponding data clip is discarded.

[0046] For example, if blur is detected across multiple consecutive frames in a video clip, the entire clip can be removed from the candidate dataset. This removal mechanism ensures the purity of the dataset and prevents low-quality data from interfering with subsequent algorithm training. This removal effectively reduces the impact of abnormal data on the model, thereby improving the accuracy and stability of the video interpolation algorithm. By aggregating the remaining data segments, a collection of video samples that meet image quality standards is generated.

[0047] For example, after all clips are screened, those that have passed quality assessment are integrated to form a collection of high-quality video samples. These samples can be used to train the film video interpolation model, ensuring that the model learns from data with high image quality and motion coherence. The benefit of this is that the generated sample collection is more suitable for the characteristics of film videos, ultimately improving the smoothness and visual quality of the interpolation effect.

[0048] Furthermore, picture quality assessment and motion quality assessment are performed on the video frame sequence.

[0049] The performing of picture quality assessment on the video frame sequence and screening out video samples that meet preset standards includes: obtaining image features of each video frame in each data segment in the video frame sequence; judging whether there is a preset picture quality problem based on the image features; if the picture quality problem exists, removing the corresponding data segment; and generating a set of video samples that meet the picture quality standard by aggregating the remaining data segments.

[0050] For each data segment in the video frame sequence, image features of each video frame are obtained.

[0051] For example, when processing a video clip from a movie, image processing techniques can be used to extract characteristic parameters such as brightness, contrast, and sharpness for each frame. These characteristics reflect the basic quality of the image. For example, brightness can be used to determine whether the image is too dark or overexposed, while contrast reveals the clarity of details. Extracting these features provides a data foundation for subsequent image quality assessment. This aims to fully understand the visual performance of the video clip, laying the foundation for further screening. Based on the image characteristics, it is determined whether there are any pre-defined image quality issues.

[0052] For example, after analyzing the extracted features, certain quality thresholds can be set. For example, images with a clarity below a certain standard value of 60 are considered blurry, or images with a brightness value below 30 are considered too dark. If a frame or video clip exhibits these problems, such as blurriness or excessive noise, it is marked as unqualified. The advantage of this judgment method is that it can quickly identify video clips that do not meet the requirements, thus avoiding the use of low-quality data for subsequent training or analysis, and improving the overall quality of the final dataset. If there are any image quality issues, the corresponding data clip is discarded.

[0053] For example, if blur is detected across multiple consecutive frames in a video clip, the entire clip can be removed from the candidate dataset. This removal mechanism ensures the purity of the dataset and prevents low-quality data from interfering with subsequent algorithm training. This removal effectively reduces the impact of abnormal data on the model, thereby improving the accuracy and stability of the video interpolation algorithm. By aggregating the remaining data segments, a collection of video samples that meet image quality standards is generated.

[0054] After all clips are screened, the clips that have passed the quality assessment are integrated to form a high-quality video sample collection.

[0055] In this example, to construct a large-scale dataset, we needed to collect samples from completed film videos. These videos undergo compression and processing by different uploaders, resulting in varying video quality. Furthermore, due to the filming content and expressive requirements, some films contain out-of-focus, overexposed, or underexposed content. This relatively low-quality content does not meet the practical requirements of video interpolation algorithm development, and therefore needs to be screened out and removed during dataset construction.

[0056] We first decode the captured movie videos to obtain image frames, then segment them into segments. Each segmented data sample contains N frames. To identify and process this content, this solution proposes video quality assessment. During training, this video quality assessment module uses a manually selected dataset of high-quality images but only 50,000 data samples:

[0057] I={I0,I1,…,I N}

[0058] N is the number of frames in each video sample. We artificially add four types of image quality issues, including lens blur, overexposure, underexposure, and negative effects of video compression, to obtain:

[0059] I d =D(I,d)

[0060] Among them, I d is the sample after adding negative effects, D(·) is the function of adding negative effects, and d is the selected negative effect. The video quality assessment module converts (I,I d ) is input into the network and its negative effects are classified into the four aforementioned negative effects and no negative effects. After training, we use the trained image quality assessment module to determine the negative effects of the collected video samples. If the algorithm indicates that the current sample has any of the four negative effects, we will remove it.

[0061] Furthermore, the motion quality assessment of the video frame sequence is performed to screen out video samples that meet preset standards, including: calculating inter-frame optical flow data for each data segment in the video frame sequence; judging whether there is a static or abnormal motion situation based on the statistical characteristics of the inter-frame optical flow data; if the static or abnormal motion situation exists, eliminating the corresponding data segment; and generating a set of video samples that meet the motion quality standards by analyzing the remaining data segments.

[0062] For each data segment in the video frame sequence, inter-frame optical flow data is calculated.

[0063] For example, when processing a movie clip, an optical flow map can be generated by analyzing the pixel displacement between adjacent frames, reflecting the trajectory and speed of objects in the image. This optical flow data can intuitively display dynamic information such as running characters and moving vehicles in the image, providing a basis for subsequent evaluation.

[0064] It should be noted that optical flow calculations typically rely on the assumption of pixel brightness variations, inferring motion by comparing the brightness differences between previous and subsequent frames. This is suitable for high-quality images of most movie scenes. Based on the statistical characteristics of inter-frame optical flow data, it can determine whether there is stillness or abnormal motion.

[0065] For example, in a video, if the average value of the optical flow data is close to 0, it indicates that there is almost no motion in the image, which may be a static scene, such as a fixed shot of a character dialogue. Such clips are not suitable for the interpolated frame dataset and should be eliminated. On the other hand, if the optical flow data has abnormally large values ​​or discontinuous jumps, it may indicate jitter or special effects interference in the image, such as the rapid switching of explosion scenes. This abnormal motion also does not meet the standard. Through this statistical analysis, we can effectively filter out clips with reasonable motion characteristics, ensuring the quality of the dataset. If there is stillness or abnormal motion, the corresponding data clip is eliminated.

[0066] For example, when processing a clip from an action movie, if the mean optical flow value of a segment is detected to be below 0.1, it is judged as a static scene and directly removed from the candidate data. If the optical flow value fluctuation of a segment exceeds three times the normal range, it is judged to be abnormal and also removed. This elimination mechanism ensures the motion consistency of the remaining data, providing reliable training material for the subsequent frame insertion algorithm. By analyzing the remaining data segments, a set of video samples that meet the motion quality standards is generated.

[0067] For example, after screening, qualified clips are classified by motion complexity, generating a collection of examples of different motion types, including slow-speed walking, medium-speed running, and high-speed chasing. This classification analysis not only increases the diversity of the dataset but also meets the needs of different interpolation scenarios, significantly improving the algorithm's adaptability and accuracy in practical applications.

[0068] In this embodiment, in order to train a stable and efficient video interpolation algorithm, we will analyze and evaluate the motion quality of the video samples after the picture evaluation.

[0069] Specifically, given a sample I that has undergone video quality evaluation, we will first calculate the optical flow between frames within the sample:

[0070] F f ={F 0→1 ,F 1→2 ,F 2→3 ,…,F N-1→N}

[0071] F b ={F N→N-1 ,…,F 3→2 ,F 2→1 ,F 1→0}

[0072] After obtaining the inter-frame optical flow, we evaluate the video motion quality based on two criteria. First, high-quality video motion should be stable, without static (inter-frame optical flow is 0 or all small values), sudden and violent fluctuations (optical flow is all small values, and some maximum values ​​suddenly appear), etc. In order to evaluate the video motion based on this criterion, we calculate the mean value of the inter-frame optical flow (m f ,m b ) and variance (v f ,v b )

[0073] m f +δ·v f >C

[0074] C is a constant. The above formula allows us to determine whether a data sample is stationary. Then, for each moment of forward / backward optical flow, we use the following formula to determine its motion quality:

[0075] where δ1, δ2, δ3 and δ'1, δ'2, δ'3 are two sets of constants obtained through experiments. Assuming the above equations hold, it is guaranteed that the inter-frame optical flow within the video sequence will not have extreme values, that is, no transitions or abnormal motion will occur. This embodiment eliminates samples that do not meet the requirements to obtain data samples that meet the motion quality requirements.

[0076] Furthermore, the data enhancement processing is performed on the video samples that meet the preset standards to identify and crop the video post-production content, including: for the video samples that meet the preset standards, obtaining difference data between each video frame; generating an initial mask based on the difference data and preset optical flow information; processing the initial mask through a preset neural network model to generate a post-production content mask; and locating and cropping the post-production content area in the video frame based on the post-production content mask.

[0077] For video samples that meet the preset criteria, obtain the difference data between each video frame.

[0078] For example, when processing a film clip containing dynamic subtitles, the system analyzes the pixel changes between adjacent frames, identifying significant differences in the subtitle area caused by scrolling text. This difference data can reflect the dynamic characteristics of post-production content such as subtitles or special effects, providing a basis for subsequent processing.

[0079] It should be noted that the purpose of obtaining difference data is to preliminarily locate areas where post-production content may exist, avoiding misinterpreting natural scene motion as post-production content, thereby improving recognition accuracy. An initial mask is generated based on the difference data and preset optical flow information.

[0080] For example, in the above-mentioned movie clip, the system combines optical flow information, that is, by analyzing the displacement trajectory of pixels between adjacent frames, it further confirms the movement direction and speed of the subtitle area and generates a rough initial mask that covers the approximate range of the subtitle scrolling.

[0081] It's important to note that optical flow information can capture the motion patterns of objects in a video, helping to distinguish post-production content from background motion. The generated initial mask lays the foundation for subsequent refined processing, effectively reducing misjudgments and improving the targeting of data processing. The initial mask is processed using a pre-set neural network model to generate a post-production content mask.

[0082] For example, a trained convolutional neural network model is used to optimize the initial mask, remove noisy areas, and accurately outline the specific boundaries of the subtitles to form a more accurate post-production content mask.

[0083] It should be noted that the neural network model learns the characteristics of post-production content from a large amount of labeled data. It can identify complex post-production elements such as dynamic special effects or billboards, ensuring high-precision masks. Its role is to provide a reliable basis for cropping. Based on the post-production content mask, the post-production content area in the video frame is located and cropped.

[0084] For example, in the final processing stage, the system accurately locates the subtitle area based on the generated mask and crops it from the video frame, preserving the original scene content and ensuring the motion consistency of the data samples.

[0085] In this embodiment, combined with Figure 2 To identify and crop post-production content such as subtitles, logos, and black borders introduced during video post-production, and to ensure consistent motion in the collected data samples, this embodiment uses prior motion information of post-production content, rather than existing text recognition algorithms, to process video post-production content.

[0086] Take the difference of each frame in I (the video frame is the remaining samples after the deletion in step 2),

[0087] Right now

[0088] Diff={I1-I0,I2-I1,…,I N -I N-1}

[0089] The estimated optical flow is used to perform pixel warping and the difference is made with the warped pixel frame:

[0090] Diff'={Warp(I1,f 0→1 )-I0,…,Warp(I N ,fN-1→N )-I N-1}

[0091] The post-processing content mask will be calculated for Diff, Diff':

[0092] M=Diff <Diff'

[0093] We take M as the initial information and input it into CNN together with the input frame I to obtain the post-processing content mask.

[0094] This design is mainly based on the prior information of motion coherence, that is, assuming that the error of directly subtracting the input frames is smaller than the error of pixel distortion using optical flow, it can be proved that there is a gap in the optical flow information of the corresponding area.

[0095] It should be noted that the benefit of doing this is to avoid post-production content interfering with the training effect of the video interpolation algorithm, such as preventing the algorithm from misjudging subtitle scrolling as scene movement, thereby improving the stability of the algorithm in practical applications, especially when generating high frame rate content in film post-production.

[0096] Finally, after data enhancement processing is performed on the video samples that meet the preset standards, the final data sample is generated. According to the generated post-content mask M, the algorithm will gradually expand the search in the length and width dimensions from the upper left to obtain the largest image block that does not include the post-content mask as the final data sample.

[0097] The method includes: obtaining a cropped video frame area for the video sample that meets the preset criteria; searching and determining the largest image block that does not contain post-production content based on the cropped video frame area; generating a final data sample by extracting and integrating the largest image block; and storing and classifying the final data sample to provide data support for subsequent video interpolation algorithm training.

[0098] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for constructing a movie interpolation-oriented dataset described in Embodiment 1 is implemented.

[0099] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for constructing a data set for movie interpolation.

[0100] Optionally, the above-mentioned electronic device may be a server.

[0101] Embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] Any content not described in detail in this specification is prior art known to those skilled in the art. It should be noted that the above description is intended to help those skilled in the art understand the present invention, but does not limit the scope of protection of the present invention. Any equivalent substitution, modification, improvement, or simplification of the above description that does not depart from the essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A method for constructing a dataset for movie interpolation, characterized in that: include: Get input video data from movie video; Decoding the input video data to obtain a video frame sequence; Performing picture quality assessment and motion quality assessment on the video frame sequence to select video samples that meet preset standards; Data enhancement processing is performed on the video samples that meet the preset standards, and the video post-production content is identified and cropped to generate the final data samples.

2. The method for constructing a movie frame interpolation dataset according to claim 1, wherein: The decoding process of the input video data to obtain a video frame sequence includes: Using a preset decoding tool to perform a decoding operation on the input video data to generate a video frame sequence including a plurality of consecutive frames; Segmenting the video frame sequence according to a preset segmentation method to obtain a plurality of data segments, each data segment containing a preset number of video frames; By analyzing the data segments frame by frame, basic data is provided for subsequent picture quality evaluation and motion quality evaluation.

3. The method for constructing a movie frame insertion dataset according to claim 1, wherein: The performing picture quality evaluation on the video frame sequence to select video samples that meet preset standards includes: For each data segment in the video frame sequence, obtaining image features of each video frame therein; Determining whether a preset picture quality problem exists based on the image features; If the picture quality problem exists, the corresponding data segment is removed; By summarizing the remaining data segments, a set of video samples that meet the picture quality standards is generated.

4. The method for constructing a movie frame interpolation dataset according to claim 1, wherein: The performing motion quality assessment on the video frame sequence to select video samples that meet a preset standard includes: For each data segment in the video frame sequence, calculating inter-frame optical flow data; Determining whether there is a stationary or abnormal motion situation based on the statistical characteristics of the inter-frame optical flow data; If the static or abnormal motion situation exists, the corresponding data segment is removed; By analyzing the remaining data segments, a set of video samples that meet the motion quality standards is generated.

5. The method for constructing a movie frame insertion dataset according to claim 1, wherein: The performing data enhancement processing on the video samples meeting the preset standards, and identifying and cropping the video post-production content, includes: For the video samples that meet the preset standard, obtaining difference data between each video frame; generating an initial mask according to the difference data and preset optical flow information; Processing the initial mask through a preset neural network model to generate a post-production content mask; The post-production content area in the video frame is located and cropped according to the post-production content mask.

6. The method for constructing a movie frame insertion dataset according to claim 1, wherein: The performing data enhancement processing on the video samples meeting the preset standards to generate final data samples includes: For the video sample meeting the preset standard, obtaining a cropped video frame area; Searching and determining a largest image block that does not contain post-production content based on the cropped video frame area; Generating a final data sample by extracting and integrating the largest image block; The final data samples are stored and classified to provide data support for subsequent video interpolation algorithm training.

7. The method for constructing a movie frame insertion dataset according to claim 1, wherein: The performing picture quality assessment and motion quality assessment on the video frame sequence to select video samples that meet preset standards includes: Obtaining a picture quality assessment result and a motion quality assessment result for the video frame sequence; Eliminating data segments with preset quality defects based on the picture quality evaluation result; Eliminating data segments with preset motion anomalies based on the motion quality assessment result; By integrating the remaining data segments, a set of video samples that meet both picture quality and motion quality standards is generated.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the method for constructing a dataset for movie interpolation according to any one of claims 1 to 7 is implemented.

9. An electronic device comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for constructing a movie interpolation-oriented dataset according to any one of claims 1 to 7.

Images