Data processing method for 3D film frame insertion and computer equipment

Abstract

The invention discloses a data processing method for 3D movie frame insertion and computer equipment, and the method comprises the steps: obtaining a high-frame-rate 2D movie source, carrying out the data collection and motion processing of the 2D movie source, and constructing a first type of data containing inter-frame motion information; acquiring a low-frame-rate 3D film source, performing data acquisition on the 3D film source, and constructing a second type of data including spatial information; a 3D high-frame-rate film source is obtained, a third type of data containing motion information and space information is obtained according to the 3D high-frame-rate film source, and the number of the third type of data is smaller than that of the first type of data and that of the second type of data; and fusing the first type of data, the second type of data and the third type of data to obtain the data set for training the 3D movie frame insertion model. According to the invention, reasonable data set support can be provided for the frame insertion work of the 3D movie, and the consistency and stability of the frame insertion process of the 3D movie are improved.

Description

Technical Field

[0001] This application relates to the technical field of video data processing, and in particular to a data processing method and computer device for 3D movie frame interpolation. Background Technology

[0002] With advancements in film production technology, 3D movies offer audiences an immersive viewing experience. However, limited by shooting costs and production complexity, most 3D movies are produced and released at lower frame rates (such as 24fps), which can lead to stuttering or blurring in fast-moving scenes, affecting the visual experience. To address this issue, video frame interpolation techniques are widely used to improve video frame rates and smoothness. Among these, deep learning-based frame interpolation methods, trained end-to-end on massive pairs of low-frame-rate to high-frame-rate videos, have achieved significant success in the 2D video field.

[0003] However, the development of 3D movie frame interpolation technology faces many challenges, especially in data collection and construction. Movie frame interpolation techniques in related technologies are often based on deep learning, which learns from massive amounts of "3D low frame rate input - 3D high frame rate output" data pairs to interpolate frames from low frame rate movies. However, limited by the current availability of high frame rate 3D movies, it is difficult to train the network using conventional "input-output" data pairs. Forcing training with only a small amount of 3D high frame rate data will lead to severe overfitting, poor generalization ability, and an inability to handle diverse movie content and motion patterns. Furthermore, if alternative data schemes are used (such as directly applying 3D data after training), the model cannot acquire and understand the unique spatial geometric constraints of 3D, resulting in inconsistent frame interpolation results in stereoscopic vision.

[0004] Therefore, given the scarcity of real-world high-frame-rate 3D data, how to construct an effective training dataset and design a suitable training method to achieve high-quality 3D movie frame interpolation is a pressing technical problem in this field. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method and computer device for 3D movie frame interpolation, which can obtain an effective training dataset, provide reasonable data support for 3D movie frame interpolation, and improve the consistency and stability of the 3D movie frame interpolation process.

[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a data processing method for 3D movie frame interpolation, including: The process involves: acquiring high-frame-rate 2D movie sources; performing data acquisition and motion processing on the 2D movie sources to construct a first type of data containing inter-frame motion information; acquiring low-frame-rate 3D movie sources; performing data acquisition on the 3D movie sources to construct a second type of data containing spatial information; acquiring high-frame-rate 3D movie sources; obtaining a third type of data containing both motion and spatial information based on the high-frame-rate 3D movie sources, wherein the amount of the third type of data is less than that of the first and second types of data; fusing the first, second, and third types of data to obtain a dataset for training a 3D movie frame interpolation model; wherein the process of training the 3D movie frame interpolation model includes at least: using the first type of data to train the motion estimation capability of the 3D movie frame interpolation model, using the second type of data to train the spatial information estimation capability of the 3D movie frame interpolation model, and using the third type of data for fine-tuning.

[0007] For example, data acquisition of the 2D movie source includes: obtaining the 2D movie source of different movie types to obtain the original data source; capturing video from the original data source; and decoding the captured video to obtain video frame data.

[0008] For example, motion processing of the 2D movie source includes: scaling the video frame data using bilinear sampling; estimating the video frame data based on an optical flow estimation algorithm to obtain optical flow data; and performing motion cleaning on the video frame data based on the optical flow data to remove data that does not meet preset conditions, so as to obtain the first type of data.

[0009] For example, motion cleaning of the video frame data based on the optical flow data to remove data that does not meet preset conditions includes: confirming inter-frame motion information based on the optical flow data, wherein the inter-frame motion information includes an average inter-frame motion value and an inter-frame motion amplitude; comparing the average inter-frame motion value with a minimum average threshold and a maximum average threshold, and comparing the inter-frame motion amplitude with an amplitude threshold; removing video frame data whose average inter-frame motion value is less than the minimum average threshold and whose inter-frame motion amplitude is less than the amplitude threshold; and removing video frame data whose average inter-frame motion value is greater than the maximum average threshold.

[0010] For example, data is collected from the 3D movie source to construct a second type of data containing spatial information, including: extracting left and right video frame pairs corresponding to the same moment from the 3D movie source with a low frame rate, as the second type of data.

[0011] For example, the 3D movie frame interpolation model includes an interpolation network and a spatial information estimation module; wherein, the interpolation network is used to perform inter-frame interpolation on a single video signal, and the interpolation network includes at least an optical flow estimation module, a pixel capture module and a pixel fusion module; the spatial information estimation module is used to estimate the 3D spatial information between the left and right video frame pairs and embed the 3D spatial information into the interpolation network.

[0012] For example, when training the motion estimation capability of the 3D movie frame interpolation model using the first type of data, the method further includes: disconnecting the spatial information estimation module, setting the output of the spatial information estimation module to zero, and causing the frame interpolation network to share the video signals used to process the left and right eyes.

[0013] For example, when using the second type of data to train the spatial information estimation capability of the 3D movie frame interpolation model, the method further includes: fixing the parameters of the trained frame interpolation network and enabling the spatial information estimation module to output the estimated 3D spatial information for training.

[0014] For example, when fine-tuning using the third type of data, the method further includes: updating the parameters of the interpolation network and spatial information estimation module of the 3D movie interpolation model at a learning rate lower than that of the previous training phase.

[0015] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method for 3D movie frame interpolation as described above.

[0016] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a data processing method and computer device for 3D movie frame interpolation. It decomposes the target data—high frame rate 3D movies—into two dimensions: motion information and spatial information. These are replaced by readily available high frame rate 2D movie data and widely available low frame rate 3D movie data, respectively, to construct a large-scale dataset suitable for training. By constructing a second type of data containing stable spatial information and establishing an independent spatial information estimation stage during training, the model can deeply learn the geometric structure of 3D scenes, avoiding flickering and ghosting in 3D effects caused by parallax errors, and improving the stereoscopic visual stability of the generated results. Through a segmented training strategy, it solves the problems of mode conflict and performance degradation that easily occur during joint training with multi-source data, ensuring the stability of the training process. This provides reasonable dataset support for 3D movie frame interpolation, improving the consistency and stability of the 3D movie frame interpolation process. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a data processing method for 3D movie frame interpolation in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the training process in the motion information estimation stage of this application embodiment.

[0020] Figure 3 This is a schematic diagram of the training process in the spatial information estimation stage of this application embodiment.

[0021] Figure 4 This is a schematic diagram of the training process during the fine-tuning stage in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 As shown in the figure, this application provides a data processing method for 3D movie frame interpolation, the method including the following steps: S110. Acquire high frame rate 2D movie sources, perform data acquisition and motion processing on the 2D movie sources, and construct the first type of data containing inter-frame motion information.

[0025] S120. Acquire low-frame-rate 3D movie sources, collect data from the 3D movie sources, and construct a second type of data containing spatial information.

[0026] S130. Obtain a 3D high frame rate movie source. Based on the 3D high frame rate movie source, obtain a third type of data containing motion information and spatial information, wherein the amount of the third type of data is less than that of the first type of data and the second type of data.

[0027] S140. A dataset for training the 3D movie frame interpolation model is obtained by fusing the first type of data, the second type of data, and the third type of data; wherein, the process of training the 3D movie frame interpolation model includes at least: using the first type of data to train the motion estimation ability of the 3D movie frame interpolation model, using the second type of data to train the spatial information estimation ability of the 3D movie frame interpolation model, and using the third type of data for fine-tuning.

[0028] This application provides a data processing method for 3D movie frame interpolation, aiming to address the problem that the limited number of existing high frame rate 3D movies cannot meet the needs of deep learning-based algorithm development. By decomposing the target data of high frame rate 3D movies into two dimensions—motion information and spatial information—and using readily available high frame rate 2D movie data and widely available low frame rate 3D movie data as substitutes, a large-scale dataset suitable for training is constructed. By constructing a second type of data containing stable spatial information and establishing an independent spatial information estimation stage during training, the model can deeply learn the geometric structure of 3D scenes, avoiding 3D flickering and ghosting caused by parallax errors, and improving the stereoscopic visual stability of the generated results. Through a segmented training strategy, the method solves the problems of mode conflict and performance degradation that easily occur during joint training with multi-source data, ensuring the stability of the training process. This provides reasonable dataset support for 3D movie frame interpolation, improving the consistency and stability of the 3D movie frame interpolation process.

[0029] For example, in step S110 above, data acquisition of 2D movie sources includes the following steps: obtaining 2D movie sources of different film types to obtain raw data sources; capturing video from the raw data sources and decoding the captured video to obtain video frame data. The different types of 2D movie sources can be data from a predetermined year, which can be confirmed according to actual circumstances, such as setting the predetermined year to 5 years, thereby capturing high frame rate (e.g., 48, 60 frames per second, etc.) 2D movie sources. This data is easier to obtain and can provide motion information by extracting frames from high frame rate movies. The video frame data can be obtained through... The representation is given, where 1-N represent the sequence of frames in the corresponding video.

[0030] In step S110 above, motion processing of the 2D movie source includes the following steps: scaling the video frame data using bilinear sampling; estimating the video frame data based on an optical flow estimation algorithm to obtain optical flow data; and performing motion cleaning on the video frame data based on the optical flow data to remove data that does not meet preset conditions, thus obtaining the first type of data.

[0031] Specifically, in this method, the decoded video frame data is scaled down using bilinear sampling, so that the short side of the video is 1080, and the long side is scaled down proportionally. Then, the optical flow estimation algorithm (RAFT) is used to estimate the inter-frame motion of consecutive video frames to obtain optical flow data. To ensure that the acquired 2D high frame rate movie videos contain reasonable motion, the data needs to be filtered and processed to complete motion cleaning.

[0032] Specifically, motion cleaning is performed on video frame data based on optical flow data to remove data that does not meet preset conditions. This includes: confirming inter-frame motion information based on optical flow data, whereby the inter-frame motion information includes the average inter-frame motion value and the inter-frame motion amplitude.

[0033] The average inter-frame motion value is compared with the minimum average threshold and the maximum average threshold, and the inter-frame motion amplitude is compared with the amplitude threshold. The values ​​of the minimum average threshold, the maximum average threshold, and the amplitude threshold can be determined according to the actual situation. In some embodiments, they are 10, 200, and 50, respectively.

[0034] Video frames with average inter-frame motion values ​​less than the lowest average threshold and inter-frame motion amplitude less than the amplitude threshold are removed; video frames with average inter-frame motion values ​​greater than the highest average threshold are also removed.

[0035] That is, the optical flow obtained through calculation For input frames and To identify and filter the motion between them: if and The inter-frame motion is minimal, so this portion of the video frame data samples is discarded. If... If there is excessive inter-frame motion, this portion of the video frame data samples will also be removed.

[0036] in, 、 、 These are the minimum average threshold, the maximum average threshold, and the amplitude threshold, respectively. After removing the aforementioned video frame data samples, the remaining data serves as the first type of data and can be used to construct the dataset.

[0037] In step S120 above, data is collected from low-frame-rate 3D movie sources to construct a second type of data containing spatial information. This includes extracting left and right eye video frame pairs corresponding to the same moment from the low-frame-rate (e.g., 24 frames per second) 3D movie sources as the second type of data. Unlike motion information, the spatial information contained in 3D videos is relatively stable and does not change due to changes in motion, content, etc. Therefore, based on left and right eye video frame pairs at the same moment (ensuring temporal consistency), the associated spatial information can be reflected, and these can be used as the second type of data for constructing the dataset. For example, during the data collection process, the same method as in step S120 can be used to capture 3D video data within a predetermined time.

[0038] In step S130 above, a third type of data containing motion and spatial information is obtained from the 3D high frame rate movie source. Because there are few sample data of 3D high frame rate videos, a small amount of the third type of data is selected as fine-tuning data to reflect the interaction of motion and spatial information between actual high frame rate 3D movies. This helps the algorithm to further learn the intrinsic relationship between motion estimation and spatial estimation after completing the motion estimation and spatial estimation modules separately, thereby further improving the frame interpolation effect.

[0039] For example, in step S140 above, a dataset for training the 3D movie frame interpolation model is obtained by fusing the first type of data, the second type of data, and the third type of data. The fusion process can involve integrating the three types of data to form a new dataset and labeling the data to indicate which type it corresponds to. The first and second types of data reflect motion and spatial information, respectively. Based on this, the third type of data represents the interaction of motion and spatial information between actual high frame rate 3D movies, which is then used to train the network model.

[0040] The aforementioned 3D movie frame interpolation model includes an interpolation network and a spatial information estimation module. The interpolation network performs inter-frame interpolation on a single video signal and includes at least an optical flow estimation module, a pixel extraction module, and a pixel fusion module. The spatial information estimation module estimates the 3D spatial information between left and right video frame pairs and embeds this 3D spatial information into the interpolation network.

[0041] When training the motion estimation capability of a 3D movie frame interpolation model using the first type of data, the method further includes: disconnecting the spatial information estimation module, setting the output of the spatial information estimation module to zero, and allowing the frame interpolation network to share the video signals used to process the left and right eyes. Figure 2 As shown, during the training of motion estimation capability, the spatial information estimation module does not participate in the training, and its output is set to zero. Furthermore, the video signals from both the left and right channels are input into the frame interpolation network (i.e.,...). Figure 2The motion estimation network in the dataset is trained to demonstrate its motion estimation capabilities. Based on the input data, the final output (frame interpolation result) is obtained.

[0042] For example, the second type of data is used to train the spatial information estimation capability of the 3D movie frame interpolation model, such as... Figure 3 As shown, during training, the spatial information estimation capability of the 3D movie frame interpolation model is trained. The parameters of the trained frame interpolation network are fixed, and the spatial information estimation module is enabled to output the estimated 3D spatial information for training. Figure 3 In this model, the parameters of the motion estimation networks on both the left and right sides are frozen and share weights, thus highlighting the training of the ability to fix spatial information. After the input data passes through the model, the interpolated frame results are output. Based on the comparison between the video frame pairs on the left and right sides, the 3D spatial information of the input is confirmed, thereby training the spatial information estimation module.

[0043] For example, when fine-tuning using the third type of data, the learning rate is lower than that used in the previous training phase, while updating the parameters of the interpolation network and spatial information estimation module of the 3D movie interpolation model. In this phase, a very small amount of 3D high frame rate movie video is used for network fine-tuning (because such data is less abundant and more difficult to obtain), prompting the 3D interpolation algorithm, which has already learned motion and spatial information, to further optimize its performance. All network weights participate in the update in this phase, but to ensure that the network performance does not overfit and degrade due to specific data inputs, a very small learning rate is used to update the network (e.g., 1 / 10 of that used in the motion and spatial information estimation phase). Figure 4 As shown, in this stage, both the motion estimation network and the spatial information estimation network (corresponding to the frame interpolation network and the spatial information estimation module) participate in the training to adjust the parameters. The left and right motion estimation networks share weights.

[0044] The segmented training described above solves the problems of pattern conflict and performance degradation that are prone to occur when training with multi-source data, thus ensuring the stability of the training process.

[0045] This application provides a data processing method and computer device for 3D movie frame interpolation. It decomposes the target data—high frame rate 3D movies—into two dimensions: motion information and spatial information. These are replaced by readily available high frame rate 2D movie data and widely available low frame rate 3D movie data, respectively, to construct a large-scale dataset suitable for training. By constructing a second type of data containing stable spatial information and establishing an independent spatial information estimation stage during training, the model can deeply learn the geometric structure of 3D scenes, avoiding flickering and ghosting in 3D effects caused by parallax errors, and improving the stereoscopic visual stability of the generated results. Through a segmented training strategy, it solves the problems of mode conflict and performance degradation that easily occur during joint training with multi-source data, ensuring the stability of the training process. This provides reasonable dataset support for 3D movie frame interpolation, improving the consistency and stability of the 3D movie frame interpolation process.

[0046] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0047] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0048] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0050] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0051] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0052] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method for frame interpolation in 3D movies, characterized in that, The data processing method for 3D movie frame interpolation includes: Acquire high frame rate 2D movie sources, perform data acquisition and motion processing on the 2D movie sources, and construct a first type of data containing inter-frame motion information; Acquire low-frame-rate 3D movie sources, collect data from the 3D movie sources, and construct a second type of data containing spatial information; Obtain 3D high frame rate movie source, and obtain a third type of data containing motion information and spatial information based on the 3D high frame rate movie source, wherein the amount of the third type of data is less than the first type of data and the second type of data; The dataset used to train the 3D movie frame interpolation model is obtained by fusing the first type of data, the second type of data, and the third type of data; wherein the process of training the 3D movie frame interpolation model includes at least: using the first type of data to train the motion estimation ability of the 3D movie frame interpolation model, using the second type of data to train the spatial information estimation ability of the 3D movie frame interpolation model, and using the third type of data for fine-tuning.

2. The data processing method for 3D movie frame interpolation according to claim 1, characterized in that, Data acquisition of the 2D movie source includes: Obtain the original data source by acquiring 2D movie sources of different film types; Video is captured from the original data source, and the captured video is decoded to obtain video frame data.

3. The data processing method for 3D movie frame interpolation according to claim 2, characterized in that, Motion processing of the 2D movie source includes: The video frame data is scaled using bilinear sampling. Optical flow data is obtained by estimating the video frame data based on an optical flow estimation algorithm; Motion cleaning is performed on the video frame data based on the optical flow data to remove data that does not meet preset conditions, so as to obtain the first type of data.

4. The data processing method for 3D movie frame interpolation according to claim 3, characterized in that, Motion cleaning is performed on the video frame data based on the optical flow data to remove data that does not meet preset conditions, including: Inter-frame motion information is confirmed based on optical flow data, and the inter-frame motion information includes the average inter-frame motion value and the inter-frame motion amplitude. The average inter-frame motion value is compared with the lowest and highest average thresholds, and the inter-frame motion amplitude is compared with the amplitude threshold. Video frame data with average inter-frame motion value less than the lowest average threshold and inter-frame motion amplitude less than the amplitude threshold are removed; video frame data with average inter-frame motion value greater than the highest average threshold are also removed.

5. The data processing method for 3D movie frame interpolation according to claim 1, characterized in that, Data is collected from the 3D movie source to construct a second type of data containing spatial information, including: extracting left and right video frame pairs corresponding to the same moment from the 3D movie source with a low frame rate, as the second type of data.

6. The data processing method for 3D movie frame interpolation according to claim 5, characterized in that, The 3D movie frame interpolation model includes a frame interpolation network and a spatial information estimation module; The frame interpolation network is used to perform inter-frame interpolation on a single video signal, and the frame interpolation network includes at least an optical flow estimation module, a pixel capture module, and a pixel fusion module. The spatial information estimation module is used to estimate the 3D spatial information between the left and right video frame pairs and embed the 3D spatial information into the frame interpolation network.

7. The data processing method for 3D movie frame interpolation according to claim 6, characterized in that, When training the motion estimation capability of the 3D movie frame interpolation model using the first type of data, the method further includes: disconnecting the spatial information estimation module, setting the output of the spatial information estimation module to zero, and making the frame interpolation network share the video signals used to process the left and right eyes.

8. The data processing method for 3D movie frame interpolation according to claim 6, characterized in that, When using the second type of data to train the spatial information estimation capability of the 3D movie frame interpolation model, the method further includes: fixing the parameters of the trained frame interpolation network and enabling the spatial information estimation module to output the estimated 3D spatial information for training.

9. The data processing method for 3D movie frame interpolation according to claim 6, characterized in that, When fine-tuning using the third type of data, the method further includes: updating the parameters of the interpolation network and spatial information estimation module of the 3D movie interpolation model at a learning rate lower than that of the previous training phase.

10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the data processing method for 3D movie frame interpolation as described in any one of claims 1-9.

Images