Method, apparatus and computer program product for processing video

Abstract

Embodiments of the present disclosure relate to a method, device and computer program product for processing a video. The method comprises: selecting, from a first instance of the video, frames having a first type of feature as a first candidate set, the first instance having a first resolution; generating a set of training frames based at least on the first candidate set; obtaining a corresponding set of frames of the set of training frames in a second instance of the video, the second instance having a second resolution lower than the first resolution; and determining, with the set of training frames and the corresponding set of frames, conversion parameters for converting from the second resolution to a third resolution. This scheme provides a smaller scale and higher quality training set for the training of a video conversion model, thereby improving the quality of the training while saving computing resources and improving the training speed.

Description

Technical Field

[0001] Embodiments of this disclosure relate to video technology, and more specifically, to methods, apparatus, and computer program products for processing video. Background Technology

[0002] In the transmission and storage of video content, it is often necessary to convert the original video content into lower resolution video using compression and other technologies to save transmission and storage resources. Furthermore, the video can be reconstructed into higher resolution video when needed. However, improving the quality and efficiency of video reconstruction remains a challenge across diverse video content and application scenarios. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for processing video is provided, comprising: selecting frames having features of a first type from a first instance of the video as a first candidate set, the first instance having a first resolution; generating a training frame set based at least on the first candidate set; obtaining a corresponding frame set of the training frame set in a second instance of the video, the second instance having a second resolution lower than the first resolution; and using the training frame set and the corresponding frame set to determine conversion parameters for conversion from the second resolution to a third resolution.

[0004] In a second aspect of this disclosure, an electronic device is provided, including a processor and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the device to perform an action, the action including: selecting frames having features of a first type from a first instance of a video as a first candidate set, the first instance having a first resolution; generating a training frame set based at least on the first candidate set; obtaining a corresponding frame set of the training frame set in a second instance of the video, the second instance having a second resolution lower than the first resolution; and using the training frame set and the corresponding frame set to determine conversion parameters for conversion from the second resolution to a third resolution.

[0005] In a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the method according to the first aspect of the invention.

[0006] Please note that this summary is provided to present a simplified version of the chosen concepts, which will be further described in the detailed embodiments below. This summary is not intended to identify key or principal features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0007] The above and other objects, features, and advantages of this disclosure will become more apparent from a more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein:

[0008] Figure 1 The illustration shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0009] Figure 2 The illustration shows a flowchart of a method for processing video according to some embodiments of the present disclosure;

[0010] Figure 3 The illustration shows a non-limiting example pipeline for generating a set of training frames according to some embodiments of the present disclosure;

[0011] Figure 4 The illustration shows a method for updating a set of training frames according to some embodiments of the present disclosure; and

[0012] Figure 5 A schematic block diagram of a device that can be used to implement embodiments of the present disclosure is shown.

[0013] In all the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are merely exemplary and not intended to limit the scope of protection of this disclosure.

[0015] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment". Definitions of other terms will be given in the description below.

[0016] The term "Video Super-Resolution (VSR)" refers to the use of techniques to construct or reconstruct higher-resolution (such as original resolution) video data from lower-resolution video data (such as compressed video data). Compressed video is much smaller than the original video (for example, reducing video frames by 4 can roughly reduce the video to 1 / 16 of the original video size). Therefore, in many video-related fields (e.g., video transmission and streaming, video conferencing, remote gaming, etc.), lower-resolution video is often transmitted or stored and then combined with VSR techniques to reconstruct higher-resolution video when needed.

[0017] In applications such as Video Streaming Responsives (VSR), conversion models (e.g., deep neural network-based models) are typically used for conversion between different resolution versions of videos. To train the model and determine its conversion parameters, video frames containing scenes and visual patterns are needed as training data, allowing the model to learn how to convert (e.g., reconstruct) various features. A common strategy is to extract all frames from high-resolution (e.g., the original version) and low-resolution (e.g., compressed) versions of the same video content and generate pairs of high-resolution and low-resolution frames as samples in the training data. However, directly using the original frames from the video for training requires significant computational resources and a long training time. For example, a medium-sized video contains tens of thousands of frames (e.g., 10 minutes at 60 frames / second = 36,000 frames), while a large-sized video / movie contains hundreds of thousands of frames. The raw format of these frames can occupy up to 100GB of storage space. Training just one pass / epoch of all the original frames extracted from the video can take hours, and a complete training cycle typically requires dozens or even hundreds of passes. This necessitates clusters equipped with high-end graphics processors, large amounts of memory, and ample storage space.

[0018] On the other hand, the original frame set is often imbalanced and contains a lot of redundant content (e.g., the same scene), which is especially common in frame sets of various real-life videos. This redundancy cannot be easily detected and eliminated using methods such as hash-based methods as with duplicate data. For example, in a video used to train a VSR model, a character's face might appear in thousands of frames, but no two frames are exactly alike. However, selecting some representative frames may be sufficient for model training.

[0019] The aforementioned imbalance can also affect the training quality of the model, causing it to unnecessarily learn excessive amounts of redundant content while failing to adequately learn less frequently occurring content. For example, almost all videos contain elements or scenes that appear infrequently. Data containing these elements or scenes is often referred to as long-tail data. Long-tail data constitutes only a small portion of the entire video and is therefore often insufficiently learned by the model when training it using the original frame set, resulting in poor performance when the model transforms the various features contained in the long-tail data. For example, a VSR model may not be able to reconstruct the structure on a Coke can that rarely appears in the video very well. However, long-tail data may contain key information or important elements. For example, certain characters may only appear for a short time in the training video (e.g., a road sign for 2 seconds, an index for 3 seconds, etc.), but these characters are easily detected when the reconstructed video is of poor quality, leading to a poor perceptual experience.

[0020] To at least partially address the aforementioned problems and other potential issues, embodiments of this disclosure propose a scheme for video processing. This scheme uses a series of feature selection functions to select the frames most relevant to the training objective from a set of video frames, and generates a training set based on these frames for training a video conversion model. The summarization pipeline formed by these functions can identify features relevant / irrelevant to the training objective and retain / discard them in the training data. This scheme can significantly reduce the number of training frames (depending on the characteristics of the video, in some cases, by up to 95%); simultaneously, important features that occur less frequently in the original video can be preserved, thereby avoiding imbalanced learning. Thus, this scheme improves training quality while saving computational resources required for preprocessing and training and increasing training speed.

[0021] Figure 1 A schematic diagram of an environment 100 in which embodiments of the present disclosure can be implemented is shown. Environment 100 includes a computing device 101 configured to select training data from a set of frames of video content with different resolutions and to train a video conversion model (e.g., a VSR model) using the selected training data. The computing device 101 may be a device on a platform such as that of a video provider or a device of a third party providing services to a video provider. In some cases, the computing device 101 may also be a device associated with a video receiver. Although shown as a single device, the computing device 101 may also be multiple devices, a virtual device, or any other form of device suitable for implementing embodiments of the present disclosure.

[0022] For the sake of explanation, Figure 1 The diagram also illustrates different video instances in environment 100, namely, a first instance 110 and a second instance 120. The first instance 110 is a first version of the video content with a higher first resolution; the second instance 120 is another version of the same video content with a lower second resolution. For example, in some cases, the first instance 110 may be the original version of the video, while the second instance 120 may be a version obtained from the first instance 110 using downsampling.

[0023] According to embodiments of this disclosure, computing device 101 can use frames selected from the training frame set 115 of the first instance 110 and pairs of corresponding frames from the second instance 120 as training samples, and use these training samples to train a video conversion model to determine various conversion parameters of the model. Furthermore, in some embodiments, as will be described below, computing device 101 may optionally utilize the determined conversion parameters to convert the second instance 120 into a third instance 130 for use or to evaluate the quality of the conversion parameters. Computing device 101 can perform the above processes using methods according to embodiments of this disclosure, which will be described in more detail below in conjunction with other accompanying drawings.

[0024] It should be understood that environment 100 is described for illustrative purposes only and does not imply any limitation on the scope of this disclosure. Environment 100 may also include Figure 1 Devices, components, and other entities not shown. Furthermore, embodiments of this disclosure can also be applied to environments different from environment 100.

[0025] Figure 2 A flowchart of an example method 200 for processing video according to some embodiments of the present disclosure is illustrated. Method 200 can be executed, for example, by a computing device 101 (more specifically, the processor of computing device 101). It should be understood that method 200 may also include additional actions not shown. References below Figure 1 Let me describe method 200 in detail.

[0026] At box 210, computing device 101 selects frames with features of a first type from first instance 110 as a first candidate set. This is for selecting training data for training the video conversion model. In some embodiments, computing device 101 may use a corresponding pattern recognition method to select frames with features of the first type. For example, in some embodiments, computing device 101 may use a face recognition algorithm to select frames with faces from first instance 110.

[0027] In some embodiments, the specific type of the features of the first type can be determined based on factors such as the content type (e.g., movies, games, etc.) contained in the first instance 120, characteristics (e.g., the style of the movie), and the application domain of the model to be trained (e.g., online games, teleconferences, etc.).

[0028] For example, suppose the first instance 110 is a movie video, and in which subtitles are embedded directly as part of the image within the frame (rather than as separate metadata). In this case, computing device 101 can use an embedded subtitle recognition method to select frames in the first instance 110 that include subtitles. Embedded subtitle recognition can be considered a specific example of optical character recognition (OCR). In some other embodiments, computing device 101 may also use other OCR methods to select frames with other types of characters (such as road signs).

[0029] As another example, the first instance 110 may be a game video used for training game video stream reconstruction. In this case, the regions of interest (ROIs) that the game player is most interested in are highly correlated with the training objective. Therefore, in some such embodiments, the computing device 101 can use a ROI identification method to select frames that include ROIs from the first instance 110. The scope of the invention is not limited to specific feature types and specific methods for selecting frames with these types.

[0030] At block 220, computing device 101 generates training frame set 115 based at least on a first candidate set. In the simplest embodiment, computing device 101 may use the first candidate set as training frame set 115. In various embodiments, method 200 may further include computing device 101 further filtering the first candidate set before using it as training frame set 115 or a portion thereof, as detailed below.

[0031] In some embodiments, computing device 101 may identify frames with another type of feature (referred to herein as a second type) from a first candidate set and remove frames without the second type of feature from the first candidate set. For example, after selecting a first candidate set with faces as a first type of feature, computing device 101 may further identify frames with blue eyes from it and retain only these frames in the first candidate set. It will be understood that, in the same manner, computing device 101 may also retain only frames with more types of features simultaneously in the first candidate set.

[0032] In some embodiments, computing device 101 may also identify frames with another type of feature (referred to herein as a third type) from the first candidate set and remove these frames from the first candidate set. This operation is particularly advantageous in embodiments where the video contains a large amount of redundant content. For example, computing device 101 may use the most common object / background recognition methods to identify one or more features that appear frequently in the first candidate set (such as the face of the same person with only subtle differences appearing in thousands of frames) and remove redundant frames that include these features from the first candidate set. In this way, computing device 101 can avoid providing too many identical features in the training frame set 115 generated based on the first candidate set, which could lead to imbalanced training later.

[0033] The training frame set 115 often needs to include multiple types of features for the model to learn. For example, a model used to reconstruct a video stream of an online meeting may need to fully learn the faces of participants in the video and the whiteboard used for writing at the meeting. In some such embodiments, method 200 may further include computing device 101 selecting frames with a fourth type of features from the first instance 110 as a second candidate set, and generating the training frame set 115 based on the first and second candidate sets. Computing device 101 may select the second candidate set and further filter the second candidate set in a manner similar to that described above for selecting the first candidate set.

[0034] In some embodiments, computing device 101 may further select a larger set of candidates from the first instance 110 as part of the training frame set 115, in the manner described above. In these embodiments, computing device 101 may select the training frame set 115 from the first instance 110 using a variety of application-specific identification methods and appropriate combinations of the selection steps described above. At a high level, the entire process of generating the training frame set 115 can be viewed as a summary pipeline formed by a series of cascading selection functions. At each step of the pipeline (i.e., a selection function), computing device 101 may identify and select a portion of the frames from the first instance 110 in various ways as described above. Reference will be made later. Figure 3 A sample pipeline 300 for generating a training frame set 115 according to embodiments of the present disclosure is described in more detail.

[0035] Continue to refer to Figure 2At box 230, computing device 101 acquires the corresponding set of training frames 115 in the second instance 120. In some cases, to improve training quality, computing device 101 may associate one (higher resolution) frame in training frame set 115 with multiple (lower resolution) frames in the second instance 120. As an example, computing device 101 may use a moving window to select the corresponding frame in the frame time series of the second instance 120. For example, for the Nth frame in the frame time series of the first instance 110 in training frame set 115, computing device 110 may acquire the frame in the time series of the second instance 120 that is sequentially in the interval [Na, N+b] as its corresponding frame. The integers a and b that specify the size of the moving window may vary depending on the specific implementation. It should be understood that the method of selecting the corresponding set of frames may also vary depending on the specific implementation, and the method disclosed herein is not limited thereto.

[0036] At box 240, computing device 101 uses training frame set 115 and corresponding frame set to determine conversion parameters for converting from a second resolution to a third resolution. The third resolution is a resolution different from the second resolution. For example, in VSR reconstruction, computing device 101 can use pairs of frames from training frame set 115 and their corresponding frames as training samples to train a video reconstruction model to determine its parameters. This video reconstruction model is used to reconstruct a video with a second resolution (e.g., compressed video) to a video with a third resolution (e.g., the original resolution) higher than the second resolution. The third resolution can be the same as or different from the first resolution. The method of this disclosure is not limited to the specific form of the conversion model or the specific numerical value of the resolution. In some embodiments, computing device 101 can shuffle the training samples and feed training pairs to the video model to be trained in mini-batches until the model converges, thereby determining the conversion parameters of the model.

[0037] Figure 3 The illustration shows a non-limiting example pipeline 300 according to some embodiments of the present disclosure, which is used for drawing from... Figure 1 The first instance 110 generates a training frame set 115. Example pipeline 300 can be generated by system 101 in, for example... Figure 2 Method 200 is used in boxes 210 and 220. The following will refer to... Figure 1 Let's describe the example pipeline 300 in detail.

[0038] Using selection function 1310, computing device 101 selects frames with a first type of feature from the frames of first instance 110 as a first candidate set. Taking video conferencing as an example, these frames could be frames with faces.

[0039] Then, computing device 101 passes the first instance 110 and the first candidate set selected from selection function 1 to selection function 2 320. Depending on the specific implementation, selection function 2 320 may be any of the following operations performed by computing device 101: identifying frames with a second type of feature (e.g., blue eyes) from the first candidate set and removing frames without the second type of feature from the first candidate set; identifying frames with a third type of feature (e.g., a person's face appearing repeatedly) from the first candidate set and removing these frames from the first candidate set; or selecting frames with a fourth type of feature (e.g., a whiteboard) from the first instance 110 as the second candidate set.

[0040] The ellipsis 330 indicates that pipeline 300 may also include Figure 3 Multiple selection functions, not shown, such as selection function 3, selection function 4, ..., and selection function N-1, where the number of N depends on the specific application. During the generation of the training frame set 115, the computing device 101 uses each selection function to further identify frames with additional features from the selected candidate set for retention or removal from the candidate set, or to select additional candidate sets from the first instance 110 to add to the selected candidate set. For example, the system 101 can use selection function 3 to remove frames without characters on the whiteboard from a second candidate set that includes whiteboard features. For example, the system 101 can use selection function 4 to select frames with a display screen from the first instance 110. Finally, after selection using the final selection function N, the computing device 101 generates the training frame set based on the output of pipeline 300. For example, the computing device 101 can use the combined set of multiple candidate sets from the final output of pipeline 300, after removing duplicate frames, as the training frame set.

[0041] It should be understood that the various selection functions in pipeline 300 mentioned above and the order in which these functions are used are merely examples. Depending on the type and nature of the first instance 115, and the domain in which the model to be trained will be applied, system 101 may construct pipelines with different selection functions and / or different orders of use of the selection functions. It should also be understood that in pipeline 300, multiple functions for selecting multiple related features can be implemented as one function, multiple functions for retaining or removing features of the same type from multiple candidate sets can also be implemented as one function, and a function may include multiple sub-functions. In addition to selection functions, pipeline 300 may also include additional functions not shown, such as performing a preliminary deredundancy operation on the first instance 110 before initially selecting each candidate set from the first instance 110.

[0042] The summarization pipeline constructed according to embodiments of this disclosure enables the selection of frames for training from an initial set of video frames using multiple pattern recognition methods. Unlike conventional recognition methods used to select content considered "most informative" from a human perspective, in this disclosure, these methods are used in the summarization pipeline to select the set of frames that are most likely to enable the model to learn all the features required for the transformation and are as concise as possible. This pipeline allows for the preservation of unique features that occur less frequently in the training data, the addition of desired features that were not previously selected, and the removal of frames that do not affect or negatively impact the model's training, thereby improving the quality of the training data.

[0043] After determining the conversion parameters of the video conversion model using the method according to embodiments of the present disclosure, the determined model can be quality evaluated, and the training frame set can be updated based on the evaluation results to iteratively train the model. Figure 4 An example method 400 for updating a selected set of training frames according to some embodiments of the present disclosure is illustrated. Method 400 can be, for example, by... Figure 1 The computing device 101 shown is used to execute the commands, and the computing device 101 can be used via... Figure 2 Method 200 is executed after the transformation parameters are determined. See below for reference. Figure 1 Let me describe method 400 in detail.

[0044] At box 410, computing device 101 can use predetermined transformation parameters to transform second instance 120 into a third instance 130 of video with a third resolution. Taking VSR reconstruction as an example, computing device can set predetermined transformation parameters for the corresponding super-resolution reconstruction model, feed the lower-resolution second instance 120 to be reconstructed into the model, and run the model to obtain a higher-resolution reconstructed third instance 130.

[0045] At block 420, computing device 101 can determine whether the number of frames in third instance 130 with quality below a threshold exceeds a threshold number. In some embodiments, computing device 101 can use structural similarity (SSIM), PSNR (peak signal-to-noise ratio), MSE (mean square error), or any other applicable algorithm to measure the relationship between frames in second instance 120 and frames in third instance 130 to evaluate the quality of frames in third instance 130.

[0046] If, at box 420, it is determined that the number of frames in the third instance 130 with quality below a threshold exceeds a threshold number, then method 400 proceeds to box 430. At box 430, device 101 may update training frame set 115. In some embodiments, computing device 101 may determine the corresponding frames in the first instance 110 for the frames in the third instance 130 with quality below the threshold, and add these corresponding frames to training frame set 115 as the updated training frame set. For example, computing device 101 may utilize the combination of the above description... Figure 2 The moving window is used to determine the corresponding frame in the first instance 110. The method disclosed herein is not limited to the specific method for determining the corresponding frame. Next, the updated training frame set can be used by the computing device 101 to regenerate training pairs to train the model, thereby determining the updated transformation parameters.

[0047] In some embodiments, if the number of frames in the third instance 130 with quality below a threshold quality is determined, the computing device 101 may reselect the training frame set 115 from the first instance 110 using a method according to embodiments of the present disclosure for retraining the video conversion model. For example, the computing device 101 may reselect the training frame set 115 from the first instance 110 using a different recognition method and / or pipeline than previously used. The evaluation information obtained at block 420 may be used to adjust the model and / or pipeline used.

[0048] Figure 5 A schematic block diagram of a device 500 that can be used to implement embodiments of the present disclosure is shown. Device 500 may be the device or apparatus described in the embodiments of the present disclosure. Figure 5 As shown, device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 502 or loaded from storage unit 508 into random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of device 500. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504. Although not shown in... Figure 5 As shown, device 500 may also include a coprocessor.

[0049] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0050] The various methods or processes described above can be executed by processing unit 501. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by CPU 501, one or more steps or actions in the methods or processes described above can be performed.

[0051] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0052] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0053] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media within the respective computing / processing device.

[0054] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0055] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0056] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0057] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0058] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing video, comprising: Frames with features of a first type are selected from a first instance of the video as a first candidate set, the first instance having a first resolution; At least a set of training frames should be generated based on the first candidate set; Obtain the corresponding frame set of the training frame set in the second instance of the video, wherein the second instance has a second resolution lower than the first resolution; The training frame set and the corresponding frame set are used to determine the conversion parameters for converting from the second resolution to the third resolution; The second instance is converted into a third instance of the video with the third resolution using the conversion parameters. as well as If it is determined that the number of frames with quality below the threshold in the third instance exceeds the threshold number, then the training frame set is updated.

2. The method according to claim 1, wherein generating the training frame set comprises: The training frame set is generated by removing frames that do not have the second type of features from the first candidate set.

3. The method according to claim 1, wherein generating the training frame set comprises: The training frame set is generated by removing frames with features of the third type from the first candidate set.

4. The method of claim 1, wherein generating the training frame set comprises: Frames with features of the fourth type are selected from the first instance as the second candidate set; as well as The training frame set is generated based on the first candidate set and the second candidate set.

5. The method according to claim 1, wherein updating the training frame set comprises: Determine the corresponding frame in the first instance for the frame in the third instance whose quality is lower than the threshold quality; as well as The corresponding frame from the first instance is added to the training frame set to update the training frame set.

6. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor, the actions including: Frames with features of a first type are selected from a first instance of the video as a first candidate set, the first instance having a first resolution; At least a set of training frames should be generated based on the first candidate set; Obtain the corresponding frame set of the training frame set in the second instance of the video, wherein the second instance has a second resolution lower than the first resolution; The training frame set and the corresponding frame set are used to determine the conversion parameters for converting from the second resolution to the third resolution; Using the conversion parameters, the second instance is converted into a third instance of the video having the third resolution; and If it is determined that the number of frames with quality below the threshold in the third instance exceeds the threshold number, then the training frame set is updated.

7. The electronic device of claim 6, wherein generating the training frame set comprises: The training frame set is generated by removing frames that do not have the second type of features from the first candidate set.

8. The electronic device of claim 7, wherein generating the training frame set comprises: The training frame set is generated by removing frames with third-type features from the first candidate set.

9. The electronic device of claim 6, wherein generating the training frame set comprises: Frames with features of the fourth type are selected from the first instance as the second candidate set; as well as The training frame set is generated based on the first candidate set and the second candidate set.

10. The electronic device of claim 6, wherein updating the training frame set comprises: Determine the corresponding frame in the first instance for the frame in the third instance whose quality is lower than the threshold quality; as well as The corresponding frame from the first instance is added to the training frame set to update the training frame set.

11. A computer program product tangibly stored on a computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 5.

Images