Training data acquisition method and device, electronic equipment and storage medium
By filtering three-frame images with similarity in the target video with the threshold as training data, the problems of slow acquisition speed and low coverage in the prior art are solved, and the training data in multiple scenarios are quickly acquired, which is suitable for interpolation processing of multiple video scenarios.
Patent Information
- Application Number
- CN202410075063.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-18
- Publication Date
- 2025-07-18
AI Technical Summary
In the prior art, the acquisition of training data is slow and the coverage is small, so it is impossible to obtain training data in many special video scenarios.
By obtaining three consecutive frames of images in the target video, based on similarity and interpolation model processing, alternative sample data with similarity matching the threshold are filtered as training data for training interpolation model.
It quickly obtains training data in a wide range of scenarios, improves the coverage of training data, and is suitable for interpolation processing of multiple video scenarios.
Smart Images
Figure CN120339739A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technologies, and in particular, to a method, apparatus, electronic device, and storage medium for obtaining training data. Background Art
[0002] As the refresh rate of display devices such as screens continues to increase, people's requirements for the refresh rate of videos are also getting higher and higher. Video frame interpolation technology increases the frame rate of videos by inserting new frames between video frames, making the videos look smoother and more natural. Performing frame interpolation processing on a video based on a frame interpolation model is the most commonly used implementation manner of video frame interpolation technology. Regardless of the type of frame interpolation model, training data is required to train the frame interpolation model.
[0003] In related technologies, it is usually necessary to construct an ideal environment to capture specific training data in this environment. In this way, the speed of obtaining training data is slow, and moreover, the coverage is small, and it is impossible to obtain training data in many special video scenarios. Summary of the Invention
[0004] To overcome the problems existing in related technologies, the present disclosure provides a method, apparatus, electronic device, and storage medium for obtaining training data.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a method for obtaining training data, the method including:
[0006] Obtaining alternative sample data, where the alternative sample data includes at least three consecutive frames of images in a target video;
[0007] Performing frame interpolation processing on a to-be-processed image in the alternative sample data to obtain an interpolated image corresponding to a reference image, where the reference image is at least one frame of image other than the first frame of image in the alternative sample data, and the to-be-processed image is an image other than the reference image in the alternative sample data;
[0008] When the similarity between the reference image and the interpolated image corresponding to the reference image is greater than a first threshold, determining the alternative sample data as training data, where the training data is used to train a frame interpolation model.
[0009] In some embodiments, the obtaining alternative sample data includes:
[0010] Obtaining the target video, where the target video includes multiple frames of images;
[0011] Determining the alternative sample data based on the similarity between every two adjacent frames of images.
[0012] In some embodiments, the determining the alternative sample data based on the similarity between every two adjacent frames of images includes:
[0013] When the similarity between every two adjacent frames among at least three consecutive frames in the target video is greater than a second threshold and less than a third threshold, determine the at least three consecutive frames as the alternative sample data; wherein, the third threshold is greater than the second threshold.
[0014] In some embodiments, before determining the alternative sample data based on the similarity between every two adjacent frames, the method further includes:
[0015] Determine the peak signal-to-noise ratio between two adjacent frames, and determine the peak signal-to-noise ratio as the similarity between every two adjacent frames.
[0016] In some embodiments, the performing interpolation processing on the image to be processed in the alternative sample data to obtain an interpolated image corresponding to a reference image includes:
[0017] Call a trained preset interpolation model to perform interpolation processing on the image to be processed to obtain an interpolated image corresponding to the reference image; wherein, the preset interpolation model is a model for performing interpolation processing.
[0018] In some embodiments, the alternative sample data includes N frames of images, where N is an integer greater than 2; the performing interpolation processing on the image to be processed in the alternative sample data to obtain an interpolated image corresponding to a reference image includes:
[0019] The reference image is an image in the alternative sample data except the first frame image and the Nth frame image, and perform interpolation processing on the first frame image and the Nth frame image to obtain an interpolated image corresponding to the reference image; or,
[0020] The reference image is an image in the alternative sample data except the first M frame images, and perform interpolation processing on the first M frame images to obtain an interpolated image corresponding to the reference image, where M is an integer less than N.
[0021] In some embodiments, the method further includes:
[0022] Save the training data to a lightweight memory-mapped database.
[0023] According to a second aspect of the embodiments of the present disclosure, there is provided a training data acquisition device, the device includes:
[0024] An alternative data acquisition module, configured to acquire alternative sample data, the alternative sample data including at least three consecutive frames in a target video;
[0025] An interpolation processing module, configured to perform interpolation processing on the image to be processed in the alternative sample data to obtain an interpolated image corresponding to the reference image, where the reference image is at least one frame of image in the alternative sample data except the first frame of image, and the image to be processed is an image in the alternative sample data except the reference image;
[0026] A training data determination module, configured to determine the alternative sample data as training data when the similarity between the reference image and the interpolated image corresponding to the reference image is greater than a first threshold, and the training data is used to train an interpolation model.
[0027] In some embodiments, the alternative data acquisition module is configured to:
[0028] Obtain the target video, where the target video includes multiple frames of images;
[0029] Determine the alternative sample data based on the similarity between every two adjacent frames of images.
[0030] In some embodiments, the alternative data acquisition module is configured to:
[0031] When the similarity between every two adjacent frames of at least three consecutive frames of images in the target video is greater than a second threshold and less than a third threshold, determine the at least three consecutive frames of images as the alternative sample data; where the third threshold is greater than the second threshold.
[0032] In some embodiments, the apparatus further includes:
[0033] A similarity determination module, configured to determine the peak signal-to-noise ratio between two adjacent frames of images, and determine the peak signal-to-noise ratio as the similarity between every two adjacent frames of images.
[0034] In some embodiments, the interpolation processing module is configured to call a trained preset interpolation model to perform interpolation processing on the image to be processed to obtain an interpolated image corresponding to the reference image; where the preset interpolation model is a model for performing interpolation processing.
[0035] In some embodiments, the alternative sample data includes N frames of images, where N is an integer greater than 2; the interpolation processing module is configured to:
[0036] The reference image is an image in the alternative sample data except the first frame of image and the Nth frame of image, and perform interpolation processing on the first frame of image and the Nth frame of image to obtain an interpolated image corresponding to the reference image; or,
[0037] The reference image is an image in the alternative sample data other than the first to Mth frames of images. Interpolation processing is performed on the first to Mth frames of images to obtain an interpolated image corresponding to the reference image, where M is an integer less than N.
[0038] In some embodiments, the apparatus further includes:
[0039] A data storage module configured to store the training data in a lightweight memory-mapped database.
[0040] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:
[0041] A processor;
[0042] A memory for storing instructions executable by the processor;
[0043] Wherein, the processor is configured to execute the method described in the first aspect of the embodiments of the present disclosure.
[0044] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method described in the first aspect of the embodiments of the present disclosure.
[0045] Adopting the above method of the present disclosure has the following beneficial effects:
[0046] If the method provided by the embodiments of the present disclosure performs interpolation processing on alternative sample data, an interpolated image similar to the reference image in the alternative sample data can be obtained, indicating that the alternative sample data is suitable for interpolation processing. Therefore, the alternative sample data can be used as training data for training an interpolation model. Moreover, since the alternative sample data is data in the target video and there is no need to construct an ideal environment for shooting, training data can be quickly obtained. And there is no restriction on the scene of the target video, and this method can be used to obtain training data in many scenarios, thereby improving the coverage of the training data.
[0047] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.
[0049] Figure 1 It is a flowchart of a method for obtaining training data shown according to an exemplary embodiment;
[0050] Figure 2 is a flowchart of a training data acquisition method shown according to an exemplary embodiment;
[0051] Figure 3 is a schematic diagram of an application scenario shown according to an exemplary embodiment;
[0052] Figure 4 is a block diagram of a training data acquisition device shown according to an exemplary embodiment;
[0053] Figure 5 is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners
[0054] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0055] In the related art, most video frame interpolation algorithms are deep learning-based algorithms. For example, the video frame interpolation algorithm based on optical flow estimation estimates the intermediate flow and uses the intermediate flow to estimate the intermediate frame; the video frame interpolation algorithm based on pure convolution calculates the pixel values of the new frame according to the pixel values between adjacent frames to implement the intermediate frame estimation. The deep learning-based video frame interpolation algorithm (i.e., the frame interpolation model) depends on a large amount of training data.
[0056] The method provided by the embodiments of the present disclosure can quickly obtain a large amount of suitable training data, and the coverage of the training data is wide.
[0057] The method provided by the embodiments of the present disclosure is executed by an electronic device, and the electronic device can be a device such as a mobile phone, a tablet computer, a notebook computer, a wearable device, a server, etc.
[0058] Figure 1 is a flowchart of a training data acquisition method shown according to an exemplary embodiment, which is executed by an electronic device. Refer to Figure 1 , and the method includes the following steps:
[0059] Step S101, obtain alternative sample data, where the alternative sample data includes at least three consecutive frames of images in the target video.
[0060] Among them, the target video is any video, which can be a video shot by the user or a video downloaded from the Internet. The embodiments of the present disclosure do not limit the source, type, etc. of the target video. The alternative sample data is the data cut from the target video, and the alternative sample data includes at least three consecutive frames of images.
[0061] Step S102: Perform frame interpolation on the image to be processed in the alternative sample data to obtain the interpolated image corresponding to the reference image. The reference image is at least one frame of image other than the first frame in the alternative sample data, and the image to be processed is the image other than the reference image in the alternative sample data.
[0062] There is a basic assumption in frame interpolation that the motion of an object is fixed and linear in at least three consecutive frames. However, in a real video, there may be rapid scene changes and object motions in at least three consecutive frames of images, resulting in a non-linear motion relationship between the objects in at least three consecutive frames of images. Such at least three consecutive frames of images cannot be used as training data. The alternative sample data is at least three consecutive frames of images in the target video, and the above situation may occur. When this situation occurs, the alternative sample data cannot be used as training data.
[0063] In the embodiments of the present disclosure, in order to verify whether the alternative sample data can be used as training data, the images in the alternative sample data are divided into the image to be processed and the reference image. The interpolated image corresponding to the reference image is the image obtained by performing frame interpolation on the image to be processed. That is, frame interpolation is performed on the image to be processed to obtain the interpolated image corresponding to the reference image. Among them, the reference image is at least one frame of image other than the first frame in the alternative sample data, and the image to be processed is the image other than the reference image in the alternative sample data.
[0064] Step S103: When the similarity between the reference image and the interpolated image corresponding to the reference image is greater than the first threshold, determine the alternative sample data as training data, and the training data is used to train the frame interpolation model.
[0065] When the similarity between the reference image and the interpolated image corresponding to the reference image is large, it indicates that the interpolated image obtained by frame interpolation is similar to the original reference image. When performing frame interpolation on the image to be processed, the motion relationship between the images to be processed can be captured. Therefore, the alternative sample data can be used as training data. Among them, the first threshold is pre-set data.
[0066] If the method provided by the embodiments of the present disclosure performs frame interpolation processing on alternative sample data and can obtain a frame interpolation image similar to the reference image in the alternative sample data, it indicates that the alternative sample data is suitable for frame interpolation processing. Therefore, the alternative sample data can be used as training data for training a frame interpolation model. Moreover, since the alternative sample data is data in the target video and there is no need to construct an ideal environment for shooting, training data can be obtained quickly. Additionally, there is no restriction on the scene of the target video, and this method can be used to obtain training data in many scenarios, thereby improving the coverage of the training data.
[0067] Figure 2 is a flowchart of a method for obtaining training data shown according to an exemplary embodiment, which is executed by an electronic device. Refer to Figure 2 This method includes the following steps:
[0068] Step S201, obtain a target video, where the target video includes multiple frames of images.
[0069] In some embodiments, the electronic device shoots the target video, or the electronic device downloads the target video from the network. The embodiments of the present disclosure do not limit the source of the target video.
[0070] In some embodiments, the target video can be a video in various video scenarios such as a TV drama, a movie, a short video, a 2D animation, a 3D animation, a game video, etc. The embodiments of the present disclosure do not limit the type of the target video and the video content.
[0071] In some embodiments, the frame rate of the target video is greater than a preset frame rate. For example, the preset frame rate is 60, so as to facilitate obtaining better-quality training data.
[0072] Step S202, determine the similarity between each adjacent two frames of images.
[0073] Among them, the similarity is used to characterize the similarity degree between two frames of images. The greater the similarity, the more similar the two frames of images are; the smaller the similarity, the greater the difference between the two frames of images.
[0074] In some embodiments, determine the peak signal-to-noise ratio (PSNR) between two adjacent frames of images, and determine the peak signal-to-noise ratio as the similarity. For example, use the following formula to determine the peak signal-to-noise ratio:
[0075] PSNR(f1,f2)=-10log 10 (mean((f1-f2) 2 )
[0076] Wherein, PSNR(f1, f2) represents the peak signal-to-noise ratio between the first frame image and the second frame image, f1 represents the first frame image, f2 represents the second frame image, f1 - f2 represents subtracting the pixel values of the pixel points at the same position in the first frame image and the second frame image respectively, and mean(·) represents taking the mean of the squares of multiple differences.
[0077] In some embodiments, the structural similarity index (Structural Similarity Index Measure, SSIM) between two adjacent frame images is determined, and this structural similarity index is determined as the similarity between the two adjacent frame images.
[0078] In some embodiments, a trained similarity determination model is called to determine the similarity between two adjacent frame images. Among them, the similarity determination model is used to determine the similarity between two frame images.
[0079] Of course, other methods can also be used to calculate the similarity, and the embodiments of the present disclosure do not limit the implementation methods of calculating the similarity.
[0080] Step S203, determine the alternative sample data based on the similarity between each two adjacent frame images.
[0081] The target video will include scene switching and situations where objects move too fast. Images in such situations are not suitable for the video interpolation scenario. Therefore, these images need to be screened out. When scene switching or objects move too fast, the similarity between two adjacent frame images will be too small; the target video will also include situations where the scene hardly changes. In such situations, the learning value of the images is low. Therefore, these images need to be screened out. When the scene hardly changes, the similarity between two adjacent frame images will be too large. Therefore, determine the alternative sample images based on the similarity between each two adjacent frame images.
[0082] In some embodiments, when the similarity between each two adjacent frame images among at least three consecutive frame images in the target video is greater than a second threshold and less than a third threshold, determine the at least three consecutive frame images as alternative sample data; wherein, the third threshold is greater than the second threshold. For example, taking three frame images as an example, when the similarity between the first frame image and the second frame image is greater than the second threshold and less than the third threshold, and the similarity between the second frame image and the third frame image is greater than the second threshold and less than the third threshold, determine these three frame images as alternative sample data.
[0083] It should be noted that the embodiments of the present disclosure only take determining a group of alternative sample data as an example. In another embodiment, this method of determining alternative sample data can be used to screen out multiple groups of alternative sample data from the target video.
[0084] Step S204: Invoke the trained preset interpolation model to perform interpolation processing on the image to be processed in the alternative sample data, and obtain an interpolated image corresponding to the reference image.
[0085] Among them, the preset interpolation model is a trained interpolation model. For example, the preset interpolation model can be the RIFE (Real-time Intermediate Flow Estimation) model, or it can also be other interpolation models. The embodiments of the present disclosure do not limit this. The number of interpolated images obtained by performing interpolation processing on the image to be processed is the same as that of the reference image.
[0086] In some embodiments, when performing interpolation processing, interpolation can be performed between two frames of images to obtain at least one interpolated image located between the two frames of images. Alternatively, interpolation can also be performed after at least two frames of images to obtain at least one image located after the at least two frames of images. When the alternative sample data includes N frames of images, where N is an integer greater than 2, performing interpolation processing on the image to be processed in the alternative sample data to obtain an interpolated image corresponding to the reference image includes: the reference image is an image in the alternative sample data except for the first frame of image and the Nth frame of image, and performing interpolation processing on the first frame of image and the Nth frame of image to obtain an interpolated image corresponding to the reference image; or, the reference image is an image in the alternative sample data except for the first to Mth frames of images, and performing interpolation processing on the first to Mth frames of images to obtain an interpolated image corresponding to the reference image, where M is an integer less than N. It should be noted that the above process of interpolation processing can be executed by the preset interpolation model.
[0087] Step S205: Determine the similarity between the reference image and the interpolated image corresponding to the reference image.
[0088] The implementation manner of calculating the similarity between the reference image and the interpolated image is the same as the implementation manner of calculating the similarity between two adjacent frames of images in step S202 above, and will not be elaborated here.
[0089] Step S206: When the similarity between the reference image and the interpolated image corresponding to the parameter image is greater than the first threshold, determine the alternative sample data as training data.
[0090] When the similarity between the reference image and the interpolated image corresponding to the reference image is relatively large, it indicates that the interpolated image obtained by interpolation processing is similar to the original reference image. When performing interpolation processing on the image to be processed, the motion relationship between the images to be processed can be captured. Therefore, the alternative sample data can be used as training data. Among them, the first threshold is pre-set data.
[0091] It should be noted that when the similarity between the reference image and the interpolated image corresponding to the reference image is less than or equal to the first threshold, it indicates that the running relationship between the images to be processed cannot be captured during the interpolation process of the images to be processed. Therefore, this alternative sample data cannot be used as training data.
[0092] Another point to note is that after using the alternative sample data as training data, an interpolation model can be trained based on this training data. The interpolation model can be the above-mentioned preset interpolation model or other interpolation models, and the embodiments of the present disclosure do not limit this.
[0093] Step S207, save the training data to a lightweight memory-mapped database.
[0094] After obtaining the training data, in order to facilitate the management of the training data and facilitate the rapid reading of the training data during the subsequent process of training the model, the training data is saved to a lightweight memory-mapped database (Lightning Memory-Mapped Database, LMDB).
[0095] In some embodiments, the training data and the identifier corresponding to the training data are saved to LMDB. When reading the training data, the corresponding training data can be quickly queried from LMDB according to the identifier, thereby improving the acceleration speed of the interpolation model.
[0096] It should be noted that in the embodiments of the present disclosure, only one set of alternative sample data is taken as an example. In another embodiment, for the target video, the above steps S202 and S203 can be used to determine multiple sets of alternative sample data. Each set of alternative sample data can adopt the implementation manners of the above steps S204 - S206 to determine whether it can be used as training data. For example, based on the target video, a sample data set is determined, and the sample data set includes multiple sets of alternative sample data. Then, for each set of alternative sample data, it is determined whether it can be used as training data, so as to screen out the training data from multiple sets of alternative sample data.
[0097] For the method provided by the embodiments of the present disclosure, if an interpolated image similar to the reference image in the alternative sample data can be obtained through interpolation processing of the alternative sample data, it indicates that the alternative sample data is suitable for interpolation processing. Therefore, this alternative sample data can be used as training data for training the interpolation model. And since the alternative sample data is data in the target video, there is no need to construct an ideal environment for shooting. Therefore, training data can be quickly obtained, and the scenes of the target video are not restricted. This way can be used to obtain training data in many scenarios, thereby improving the coverage of the training data.
[0098] Moreover, in the actual application process of the frame interpolation algorithm, if it is found that the effect of the frame interpolation algorithm is not good for some scenes, the method for determining training data provided in this embodiment of the present disclosure can be used to directly and quickly obtain the training data for specific scenes, and the frame interpolation algorithm can be tuned specifically, and the algorithm can be optimized using the training data.
[0099] In one example, referring to Figure 3 the schematic diagram of the application scenario shown in the figure, first obtain the real video (target video), then through sample segmentation, obtain multiple video frames in the real video, and then perform preliminary screening on the multiple video frames to obtain alternative sample data, so as to obtain the preliminary screening samples, and the preliminary screening samples include at least one group of alternative sample data. Then obtain the trained RIFE model to perform fine screening on the preliminary screening samples, and eliminate the alternative sample data that is too fast or too blurred in motion and not suitable for frame interpolation training to obtain the fine screening samples. Finally, construct an LMDB based on the fine screening samples to obtain a frame interpolation dataset, and the frame interpolation dataset includes the fine screening samples. Finally, when training the model is required, optimize the frame interpolation model based on the fine screening samples in the LMDB. Among them, the preliminary screening process refers to the above steps S202 and S203, and the fine screening process refers to the above steps S204-S206.
[0100] Figure 4 is a block diagram of a training data acquisition device shown according to an exemplary embodiment, which is configured in an electronic device, referring to Figure 4 and the device includes:
[0101] An alternative data acquisition module 401, configured to acquire alternative sample data, and the alternative sample data includes at least three consecutive images in the target video;
[0102] A frame interpolation processing module 402, configured to perform frame interpolation processing on the image to be processed in the alternative sample data to obtain a frame interpolated image corresponding to the reference image, where the reference image is at least one frame of image other than the first frame of image in the alternative sample data, and the image to be processed is the image other than the reference image in the alternative sample data;
[0103] A training data determination module 403, configured to determine the alternative sample data as training data when the similarity between the reference image and the frame interpolated image corresponding to the reference image is greater than a first threshold, and the training data is used to train the frame interpolation model.
[0104] In some embodiments, the alternative data acquisition module 401 is configured to:
[0105] Acquire a target video, where the target video includes multiple images;
[0106] Determine alternative sample data based on the similarity between every two adjacent images.
[0107] In some embodiments, the alternative data acquisition module 401 is configured to:
[0108] When the similarity between every two adjacent frames among at least three consecutive frames in the target video is greater than a second threshold and less than a third threshold, determine the at least three consecutive frames as alternative sample data; wherein, the third threshold is greater than the second threshold.
[0109] In some embodiments, the apparatus further includes:
[0110] A similarity determination module, configured to determine the peak signal-to-noise ratio between two adjacent frames of images, and determine the peak signal-to-noise ratio as the similarity between every two adjacent frames of images.
[0111] In some embodiments, the frame interpolation processing module 402 is configured to call a trained preset frame interpolation model to perform frame interpolation processing on the image to be processed, and obtain an interpolated image corresponding to the reference image; wherein, the preset frame interpolation model is a model for performing frame interpolation processing.
[0112] In some embodiments, the alternative sample data includes N frames of images, where N is an integer greater than 2; the frame interpolation processing module 402 is configured to:
[0113] The reference image is an image other than the first frame image and the Nth frame image in the alternative sample data, and perform frame interpolation processing on the first frame image and the Nth frame image to obtain an interpolated image corresponding to the reference image; or,
[0114] The reference image is an image other than the first M frame images in the alternative sample data, and perform frame interpolation processing on the first M frame images to obtain an interpolated image corresponding to the reference image, where M is an integer less than N.
[0115] In some embodiments, the apparatus further includes:
[0116] A data storage module, configured to store the training data in a lightweight memory-mapped database.
[0117] Regarding the apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.
[0118] Embodiments of the present disclosure further provide an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to execute the training data acquisition method in the above embodiments.
[0119] Figure 5 It is a block diagram of an electronic device 500 shown according to an exemplary embodiment.
[0120] Referring to Figure 5, the electronic device 500 may include one or more of the following components: a processing component 502, a memory 504, a power component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.
[0121] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.
[0122] The memory 504 is configured to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.
[0123] The power component 506 provides power to the various components of the electronic device 500. The power component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 500.
[0124] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.
[0125] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 further includes a speaker for outputting audio signals.
[0126] The I / O interface 512 provides an interface between the processing component 502 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.
[0127] The sensor component 514 includes one or more sensors for providing a status assessment of various aspects of the electronic device 500. For example, the sensor component 514 can detect the on / off state of the electronic device 500, the relative positioning of components, such as the display and the keypad of the electronic device 500. The sensor component 514 can also detect a change in the position of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor component 514 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 514 can also include a light sensor, such as a CMOS or a CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 514 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0128] The communication component 516 is configured to facilitate communication between the electronic device 500 and other devices in a wired or wireless manner. The electronic device 500 can access a communication standard-based wireless network, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0129] In an exemplary embodiment, the electronic device 500 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.
[0130] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the above instructions can be executed by a processor 520 of the electronic device 500 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0131] The embodiments of the present disclosure also provide a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enabling the electronic device to execute the training data acquisition method in the above embodiments.
[0132] Those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed in this disclosure. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present invention are pointed out by the following claims.
[0133] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.
Claims
1. A method for obtaining training data, characterized in that, The method includes: Obtain alternative sample data, where the alternative sample data includes at least three consecutive frames of images in the target video; Perform frame interpolation on the image to be processed in the alternative sample data to obtain an interpolated image corresponding to the reference image, where the reference image is at least one frame of image other than the first frame of image in the alternative sample data, and the image to be processed is the image other than the reference image in the alternative sample data; When the similarity between the reference image and the interpolated image corresponding to the reference image is greater than a first threshold, determine the alternative sample data as training data, and the training data is used to train the frame interpolation model.
2. The method according to claim 1, characterized in that, The obtaining of the alternative sample data includes: Obtain the target video, where the target video includes multiple frames of images; Determine the alternative sample data based on the similarity between every two adjacent frames of images.
3. The method according to claim 2, wherein The determining of the alternative sample data based on the similarity between every two adjacent frames of images includes: When the similarity between every two adjacent frames of images in at least three consecutive frames of images in the target video is greater than a second threshold and less than a third threshold, determine the at least three consecutive frames of images as the alternative sample data; where the third threshold is greater than the second threshold.
4. The method according to claim 2, wherein Before the determining of the alternative sample data based on the similarity between every two adjacent frames of images, the method further includes: Determine the peak signal-to-noise ratio between two adjacent frames of images, and determine the peak signal-to-noise ratio as the similarity between every two adjacent frames of images.
5. The method according to claim 1, wherein The performing of frame interpolation on the image to be processed in the alternative sample data to obtain an interpolated image corresponding to the reference image includes: Call a pre-trained preset frame interpolation model to perform frame interpolation on the image to be processed to obtain an interpolated image corresponding to the reference image; where the preset frame interpolation model is a model for performing frame interpolation.
6. The method according to claim 1, characterized in that The alternative sample data includes N frames of images, where N is an integer greater than 2; the performing of frame interpolation on the image to be processed in the alternative sample data to obtain an interpolated image corresponding to the reference image includes: The reference image is the image other than the first frame of image and the Nth frame of image in the alternative sample data, perform frame interpolation on the first frame of image and the Nth frame of image to obtain an interpolated image corresponding to the reference image; or The reference image is the image other than the first to Mth frames of images in the alternative sample data, perform frame interpolation on the first to Mth frames of images to obtain an interpolated image corresponding to the reference image, where M is an integer less than N.
7. The method according to claim 1, characterized in that, The method further includes: Save the training data to a lightweight memory-mapped database.
8. A training data acquisition device, characterized in that, The apparatus includes: An alternative data acquisition module configured to obtain alternative sample data, where the alternative sample data includes at least three consecutive frames of images in the target video; An interpolation processing module, configured to perform interpolation processing on a to-be-processed image in the alternative sample data to obtain an interpolated image corresponding to a reference image, where the reference image is at least one frame of image in the alternative sample data other than the first frame image, and the to-be-processed image is an image in the alternative sample data other than the reference image; A training data determination module, configured to determine the alternative sample data as training data when the similarity between the reference image and the interpolated image corresponding to the reference image is greater than a first threshold, and the training data is used to train an interpolation model.
9. An electronic device, characterized in that, Comprising: A processor; A memory for storing processor-executable instructions; Wherein, the processor is configured to execute the method according to any one of claims 1-7.
10. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1-7.