Video processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a video processing method, device, electronic equipment and storage medium. The video processing method comprises: downsampling a target video frame to obtain a downsampled video frame, compressing and decoding the downsampled video frame to reconstruct a reconstructed video frame, upsampling the reconstructed video frame to obtain a first upsampled video frame, upsampling the downsampled video frame to obtain a second upsampled video frame; determining a first residual energy of a residual error of the target video frame and the first upsampled video frame in a frequency domain, and a second residual energy of a residual error of the target video frame and the second upsampled video frame in the frequency domain; and determining an encoding resolution when encoding the target video frame based on the first residual energy and the second residual energy. The present disclosure determines the encoding resolution of the target video frame by combining the content characteristics and the encoding characteristics of the target video frame, thereby improving the encoding performance.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a video processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Video, such as game footage, typically requires encoding before transmission and decoding upon receipt. The efficiency of video encoding directly impacts transmission efficiency. Summary of the Invention

[0003] This disclosure provides a video processing method, apparatus, electronic device, and storage medium.

[0004] The following technical solution is adopted in this disclosure.

[0005] In some embodiments, this disclosure provides a video processing method, including:

[0006] The target video frame is downsampled to obtain a downsampled video frame. The downsampled video frame is compressed and decoded to reconstruct a reconstructed video frame. The reconstructed video frame is upsampled to obtain a first upsampled video frame. The downsampled video frame is upsampled to obtain a second upsampled video frame.

[0007] Determine the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and the second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame;

[0008] The encoding resolution for encoding the target video frame is determined based on the first residual energy and the second residual energy.

[0009] In some embodiments, this disclosure provides a video processing apparatus, including:

[0010] The processing unit is configured to downsample the target video frame to obtain a downsampled video frame, compress and decode the downsampled video frame to reconstruct the reconstructed video frame, upsample the reconstructed video frame to obtain a first upsampled video frame, and upsample the downsampled video frame to obtain a second upsampled video frame.

[0011] The determining unit is configured to determine a first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and a second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame; and to determine the encoding resolution for encoding the target video frame based on the first residual energy and the second residual energy.

[0012] In some embodiments, this disclosure provides an electronic device, including: at least one memory and at least one processor;

[0013] The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above method.

[0014] In some embodiments, this disclosure provides a computer-readable storage medium for storing program code that, when run by a processor, causes the processor to perform the methods described above.

[0015] The video processing method provided in this disclosure involves downsampling a target video frame to obtain a downsampled video frame, compressing and decoding the downsampled video frame to reconstruct it into a reconstructed video frame, upsampling the reconstructed video frame to obtain a first upsampled video frame, and upsampling the downsampled video frame to obtain a second upsampled video frame. It also involves determining a first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and a second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame. Based on the first and second residual energies, it determines the encoding resolution for encoding the target video frame. This disclosure improves encoding performance by combining the content characteristics and encoding characteristics of the target video frame to determine its encoding resolution. Attached Figure Description

[0016] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0017] Figure 1 This is a schematic diagram of a video processing method in the related technology of this disclosure.

[0018] Figure 2 This is a flowchart of a video processing method according to an embodiment of the present disclosure.

[0019] Figure 3 This is a schematic diagram of a video processing method according to an embodiment of the present disclosure.

[0020] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] It should be understood that the various steps described in the method embodiments of this disclosure can be performed in sequence and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0023] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0024] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0025] It should be noted that the use of the word "a" in this disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as "one or more" unless otherwise expressly indicated in the context.

[0026] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0027] The solutions provided by the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0028] With the development of communication technology, cloud gaming has attracted increasing attention from the industry. Because real-time rendering of game visuals requires significant computing resources, cloud gaming systems can render game visuals on the server side and compress and transmit them to the client terminal. Therefore, users do not need high-end processors and graphics cards to enjoy a high-quality gaming experience. Thus, cloud gaming systems can greatly reduce the computational resource overhead on the client side. Despite the obvious advantages of cloud gaming, the latency of the server during computation or the end-to-end latency during transmission can cause stuttering, jerking, distortion, and a decrease in image quality, which will greatly affect the client experience. Improving the continuity and smoothness of game scenes and enhancing the quality of game visuals are key issues that need to be addressed in cloud gaming research and deployment. In cloud gaming systems, since the game videos rendered by the server need to be transmitted to the client in a highly efficient, low-latency, and high-quality manner, efficient encoding of cloud game videos is a core aspect of cloud gaming systems.

[0029] Since video encoding plays a crucial role in cloud gaming systems and platforms, it is necessary to research efficient adaptive resolution encoding methods to reduce encoding complexity, alleviate system latency, and improve user experience, given the unique characteristics of cloud gaming visuals and content. Adaptive resolution encoding can adjust the resolution of the video frame to be encoded. The variable resolution decision can be applied in the preprocessing analysis stage of video encoding, where the decoder parses the variable resolution flag to restore the resolution of the reconstructed video. Variable resolution encoding schemes in related technologies are mainly implemented in two ways. The first method is achieved through two-stage encoding, using both the original resolution frame and a downsampled resolution frame for encoding. The decision to use the original resolution frame or the downsampled resolution frame is made by comparing rate-distortion performance. This method has high accuracy and can effectively improve compression performance, but the decision-making process introduces high computational overhead. The second method directly decides whether to perform variable resolution encoding on the current frame based on the temporal correlation between the current frame and the reference frame, but the accuracy is lower, thus affecting compression performance.

[0030] Please refer to Figure 1 This describes the process of the first scheme. In the video coding preprocessing, the current frame of the original video is I, and the reconstructed frame of the reconstructed video 1 after encoding and decoding is I′. Then the coding rate distortion cost J1 can be expressed as:

[0031] J1=D(I,I′)+λ·R1

[0032] Where λ is the Lagrange daily number, R1 is the frame rate of the current frame, and D is the coding distortion, usually measured by the mean square error between the reconstructed frame I′ and the current frame I of the original video. Further, S... dRepresenting a downsampling filter, the current frame I of the original video frame is downsampled to a lower resolution video frame. It can be represented as:

[0033]

[0034] After encoding, the bitrate can be represented as R². The video frames of a low-resolution video, after encoding and decoding, can be reconstructed as the video frames of the reconstructed low-resolution video. Then pass through the upsampling filter S u After upsampling, it can be represented as video frame I of reconstructed video 2. du :

[0035]

[0036] At this point, the coding rate distortion cost J2 can be expressed as:

[0037] J2=(U,U du )+λ·R2

[0038] The variable resolution coding process in related technologies is as follows: Figure 1 As shown, it is necessary to complete the encoding and decoding reconstruction of the input frame I of the original video, as well as the downsampled frame. The encoding, decoding, reconstruction, and upsampling restoration are performed, and then the size of J1 and J2 is compared to decide whether the current frame should maintain the original resolution encoding or adopt down-resolution encoding to obtain better rate-distortion performance. This method requires two encoding and decoding operations, which has a large computational overhead and is difficult to meet the requirements in scenarios with high timeliness requirements (such as the cloud gaming video application scenario mentioned above).

[0039] In some embodiments of this disclosure, a video processing method is proposed that can be used in the preprocessing or analysis stage of encoding, for example, in the preprocessing or analysis stage of video frames of cloud video. This video processing method can achieve adaptive resolution encoding with low computational complexity.

[0040] like Figure 2 As shown, Figure 2 This is a flowchart of a video processing method according to an embodiment of the present disclosure, including:

[0041] S11: Downsample the target video frame to obtain a downsampled video frame, compress and decode the downsampled video frame to reconstruct the reconstructed video frame, upsample the reconstructed video frame to obtain a first upsampled video frame, and upsample the downsampled video frame to obtain a second upsampled video frame.

[0042] S12: Determine the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and the second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame.

[0043] S13: Determine the coding resolution when coding the target video frame based on the first residual energy and the second residual energy.

[0044] In some embodiments, the target video frame may be, for example, a cloud gaming video frame. Steps S11 to S13 may be performed before encoding the target video frame. The target video frame may be a single video frame or multiple video frames. When downsampling the target video frame, the Bicubic method (bicubic interpolation algorithm) may be used for downsampling. Downsampling, also known as subsampling, will reduce the resolution of the target video frame, meaning the resolution of the downsampled video frame will be lower than the resolution of the target video frame. It should be noted that the steps of obtaining the first upsampled video frame and obtaining the second upsampled video frame do not have a specific order. Both steps only need to be performed after obtaining the downsampled video frame. This disclosure does not limit the order of the steps of "compressing and decoding the downsampled video frame to reconstruct the reconstructed video frame, and upsampling the reconstructed video frame to obtain the first upsampled video frame" and "upsampling the downsampled video frame to obtain the second upsampled video frame." They can be performed sequentially or simultaneously as needed. Furthermore, it should be noted that after obtaining the first upsampled video frame, the step of determining the first residual energy in step S12 can be performed. After obtaining the second upsampled video frame, step S12, which determines the second residual energy, can be executed. That is, step S12 does not need to wait for step S11 to complete completely before execution. The first and second residual energies are frequency domain residual energies. Based on these, the encoding resolution for encoding the target video frame is determined. For example, it can be determined whether the encoding resolution used during subsequent encoding is the original resolution of the target video or a resolution lower than the original resolution. The resolution lower than the original resolution can be the resolution of the downsampled video frame.

[0045] In some embodiments of this disclosure, during the preprocessing or analysis stage before encoding the target video frame, the encoding resolution of the target video frame is first determined. At this time, the encoding resolution can be determined using the method proposed in the embodiments of this disclosure. The method proposed in the embodiments of this disclosure does not perform two encoding and decoding reconstructions on the target video frame, but only one encoding and decoding reconstruction. This reduces the amount of computation and improves the response speed. Furthermore, encoding and decoding reconstruction are performed during the process of obtaining the first upsampled video frame, that is, the encoding characteristics of the target video frame are reflected in the first residual energy. Downsampling and upsampling are performed respectively during the process of obtaining the second upsampled video frame, that is, the content characteristics of the target video frame are reflected in the second residual energy. In other words, the encoding resolution of the target video frame is determined by combining the content characteristics and encoding characteristics of the target video frame.

[0046] In some embodiments of this disclosure, determining the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame includes: determining the first spatial domain residual between the first upsampled video frame and the target video frame; transforming the first spatial domain residual to the frequency domain to obtain the first frequency domain residual; and obtaining the first residual energy based on the first frequency domain residual at each position on the target video frame.

[0047] In some embodiments, the target video frame can be represented as I, and a downsampled video frame is generated by downsampling the target video frame. Then compression is performed, with the compression quantization parameter denoted as QP. The reconstructed video frame obtained after decoding and reconstruction is denoted as... Subsequently, the video frames were reconstructed. The first upsampled video frame is obtained by upsampling. At this time, the first upsampled video frame First spatial residual between the target video frame I It can be represented as:

[0048]

[0049] Furthermore, the residual in the first airspace... After being transformed into the frequency domain by discrete cosine transform T, the first frequency domain residual is obtained. It can be represented as:

[0050]

[0051] The target video frame has one or more locations, each location can have one or more pixels, and each location has a corresponding first frequency domain residual. When determining the first residual energy of the target video frame, the first frequency domain residuals of each location need to be considered. In some embodiments, the first residual energy can be expressed as... The following formula can be used for calculation:

[0052]

[0053] Where i represents the coefficient at each position of the target video frame. The first residual energy of the target video frame can be obtained in the above way.

[0054] In some embodiments of this disclosure, determining a second spatial domain residual between a target video frame and a second upsampled video frame, and transforming the second spatial domain residual to the frequency domain to obtain a second residual energy, includes: determining a second frequency domain residual between the second upsampled video frame and the target video frame in the frequency domain; and obtaining the second residual energy based on the second frequency domain residual at each position on the target video frame.

[0055] In some embodiments, the target video frame I is downsampled to generate a downsampled video frame. At this point, no encoding or decoding is required; the video frame is upsampled to obtain the second upsampled video frame. Upsampling method for generating the first upsampled video frame and method for generating the second upsampled video frame The upsampling method can be the same, both can be the Bicubic method, and the second upsampled video frame The resolution is the same as that of the first upsampled video frame. Similar to the method for generating the first spatial domain residual and the first frequency domain residual, the second spatial domain residual can be expressed as: The second spatial residual corresponds to the second frequency domain residual in the frequency domain. It can be represented as:

[0056]

[0057] Where T represents the discrete cosine transform, and further, the second residual energy... It can be represented as:

[0058]

[0059] Where i represents the coefficient at each position of the target video frame. The second residual energy of the target video frame can be obtained using the above method.

[0060] In some embodiments of this disclosure, determining the encoding resolution for encoding the target video includes: determining whether to use the original resolution of the target video frame during encoding, or to use a resolution lower than the original resolution during encoding. In some embodiments, the resolution lower than the original resolution may be the resolution of a downsampled video frame. In some embodiments, the encoding resolution needs to be determined before encoding the target video frame. Different resolutions result in different levels of distortion and computational consumption for the target video frame. The degree of distortion is reflected in the first residual energy and the second residual energy. Therefore, the difference in the degree of distortion is used to consider whether to use the original resolution or a resolution lower than the original resolution for encoding, thereby balancing the degree of distortion and computational consumption, ensuring both the quality of the video frame and the encoding efficiency.

[0061] In some embodiments of this disclosure, determining the encoding resolution for encoding a target video frame based on a first residual energy and a second residual energy includes: determining the energy difference between the first residual energy and the second residual energy and a target threshold; if the energy difference is greater than the target threshold, determining that the original resolution of the target video frame is used during encoding; or, if the energy difference is not greater than the target threshold, determining that a resolution lower than the original resolution is used during encoding.

[0062] In some embodiments, a target threshold and a first residual energy are preset. With the second residual energy The difference Δ() can be expressed as:

[0063]

[0064] If Δ() is greater than the target threshold, it indicates that the encoding process has introduced higher distortion compared to the loss during downsampling and resolution scaling. The decoder will find it difficult to recover the lost information through upsampling. This corresponds to complex content and scenes in the target video frame, and downsampling encoding should be avoided. Encoding should be performed at the original resolution. A resolution scaling flag can be set in the target video frame. If this flag is enabled, it indicates that downsampling encoding is required; if it is disabled, it indicates that encoding should be performed at the original resolution. In this case, the resolution scaling flag should be disabled. Similarly, if Δ() is less than or equal to the target threshold, it indicates that the loss introduced during encoding is below a certain level, and the decoder is capable of recovering the information of the target video frame. This corresponds to simple content and scenes in the target video frame, and therefore, a low-resolution encoding method should be used, employing a resolution lower than the original resolution to skip the higher-resolution encoding and decoding process. In this case, the resolution scaling flag should be enabled.

[0065] In some embodiments of this disclosure, determining the target threshold includes: determining the target threshold based on the compression quantization parameters used when compressing downsampled video frames. In some embodiments, the target threshold is associated with the compression quantization parameters (which can be represented as QP) used when compressing downsampled video frames. For example, the larger the compression quantization parameter QP, the larger the target threshold. This is because a larger compression quantization parameter QP indicates higher compression efficiency, which can significantly improve the compression ratio and thus improve transmission timeliness. Therefore, the tolerance for information loss due to encoding is also higher. However, if the compression efficiency is low and a lot of information is lost, it should not be allowed. Therefore, the compression quantization parameter QP needs to be considered when determining the target threshold. Specifically, in some embodiments, when the compression quantization parameter QP is set to 22, the corresponding target threshold can be set to 2. When the compression quantization parameter QP is set to 27, the corresponding target threshold can be set to 7. When the compression quantization parameter QP is set to 32, the corresponding target threshold can be set to 16. When the compression quantization parameter QP is set to 37, the corresponding target threshold can be set to 40.

[0066] In some embodiments of this disclosure, the method further includes: encoding the target video frame based on the determined encoding resolution. After the encoding resolution is determined, the target video frame is encoded; if encoding is performed at a resolution lower than that of the original video frame, the process of encoding at the original resolution is skipped.

[0067] In some embodiments of this disclosure, the target video frame can be a cloud gaming video frame, and the method in the embodiments of this disclosure can be used in a cloud gaming server. The proposed video processing method can reduce the computational complexity of adaptive resolution encoding for cloud gaming and reduce the latency of cloud gaming. Compared with fixed resolution encoding, the proposed method can improve encoding performance and reduce the storage and transmission overhead of cloud gaming videos.

[0068] In some embodiments of this disclosure, the encoding resolution can be determined during the encoding preprocessing / pre-analysis stage. The content characteristics and encoding characteristics of the target video frame guide whether resolution reduction (variable resolution) encoding is needed. The content characteristics of the target video frame can be represented by the total energy (second residual energy) in the frequency domain after downsampling and upsampling. The encoding characteristics of the target video frame can be represented by the total energy (first residual energy) in the frequency domain of the residual after downsampling, encoding, decoding reconstruction, and upsampling restoration. The encoding resolution of the target video frame is determined by combining its content characteristics and encoding characteristics; if the encoded video frame is determined to use low-resolution encoding (below the original resolution), the original resolution encoding process is skipped, and the variable resolution encoding flag remains enabled; if the encoded video frame is determined to use the original resolution, original resolution encoding is required, and the variable resolution encoding flag is disabled.

[0069] The following is in conjunction with the appendix Figure 3 A specific embodiment of this disclosure will be described. In this embodiment, the method can be used on a cloud gaming server, and the target video frame can be a video frame from a cloud gaming game. Figure 3 The original video in the process is the target video frame. The original video is downsampled to obtain a low-resolution video (downsampled video frame). This low-resolution video frame is then encoded and decoded to reconstruct a low-resolution reconstructed video (reconstructed video frame). This reconstructed video frame is then upsampled to obtain a reconstructed video (first upsampled video frame). Finally, the low-resolution video is upsampled again to recover the original video (second upsampled video frame). The first frequency domain residual is obtained from the original video frame and the upsampled reconstructed video. The second frequency domain residual is obtained from the original video and the reconstructed original video. The first frequency domain residual Second frequency domain residual The adaptive resolution determination model is used to judge whether to use the original resolution or a reduced resolution for encoding. Specifically, this is based on the first frequency domain residual. Second frequency domain residual The energy difference between the first and second residual energies is determined. A target threshold is then determined based on the compression quantization parameter QP. Finally, based on the relationship between the energy difference and the target threshold, it is determined whether to use the original resolution or a resolution lower than the original resolution for encoding. The cloud gaming server can then encode the target video frames according to the determined encoding resolution and transmit them to the client for decoding and playback.

[0070] Some embodiments of this disclosure also propose a video processing apparatus, comprising:

[0071] The processing unit is used to downsample the target video frame to obtain a downsampled video frame, compress and decode the downsampled video frame to reconstruct the reconstructed video frame, upsample the reconstructed video frame to obtain a first upsampled video frame, and upsample the downsampled video frame to obtain a second upsampled video frame.

[0072] The determining unit is used to determine the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and the second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame; and to determine the coding resolution when coding the target video frame based on the first residual energy and the second residual energy.

[0073] In some embodiments, the determining unit determines the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, including: determining the first spatial domain residual between the first upsampled video frame and the target video frame; transforming the first spatial domain residual to the frequency domain to obtain the first frequency domain residual; and obtaining the first residual energy based on the first frequency domain residual at each position on the target video frame.

[0074] In some embodiments, the determining unit determines the second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame, including: determining the second spatial domain residual between the second upsampled video frame and the target video frame; transforming the second spatial domain residual to the frequency domain to obtain the second frequency domain residual; and obtaining the second residual energy based on the second frequency domain residual at each position on the target video frame.

[0075] In some embodiments, the determining unit determines the encoding resolution when encoding the target video, including: determining that the original resolution of the target video frame is used during encoding, or that a resolution lower than the original resolution is used during encoding.

[0076] In some embodiments, the determining unit determines the encoding resolution for encoding the target video frame based on the first residual energy and the second residual energy, including: determining the energy difference between the first residual energy and the second residual energy and a target threshold; if the energy difference is greater than the target threshold, determining that the original resolution of the target video frame is used during encoding; or, if the energy difference is not greater than the target threshold, determining that a resolution lower than the original resolution is used during encoding.

[0077] In some embodiments, the determining unit determines the target threshold by: determining the target threshold based on compression quantization parameters when compressing the downsampled video frame.

[0078] In some embodiments, the target video frame is a cloud gaming video frame.

[0079] For embodiments of the apparatus, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments. The apparatus embodiments described above are merely illustrative, and the modules described as separate modules may or may not be separate. Some or all of the modules can be selected based on actual needs to achieve the purpose of this embodiment. Those skilled in the art can understand and implement this without any creative effort.

[0080] The methods and apparatus of this disclosure have been described above based on embodiments and application examples. Furthermore, this disclosure also provides an electronic device and a computer-readable storage medium, which are described below.

[0081] The following is for reference. Figure 4 The figure illustrates a structural schematic of an electronic device (e.g., a terminal device or server) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in the figure is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0082] Electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 802 or a program loaded from storage device 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0083] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although an electronic device 800 with various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0084] In particular, based on embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0085] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0086] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0087] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0088] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods of the present disclosure.

[0089] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0091] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0092] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0093] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] Based on one or more embodiments of this disclosure, a video processing method is provided, comprising:

[0095] The target video frame is downsampled to obtain a downsampled video frame. The downsampled video frame is compressed and decoded to reconstruct the reconstructed video frame. The reconstructed video frame is upsampled to obtain a first upsampled video frame. The downsampled video frame is upsampled to obtain a second upsampled video frame.

[0096] Determine the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and the second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame;

[0097] The encoding resolution for encoding the target video frame is determined based on the first residual energy and the second residual energy.

[0098] Based on one or more embodiments of this disclosure, a video processing method is provided to determine the first residual energy in the frequency domain of the residual between a target video frame and a first upsampled video frame, including:

[0099] Determine the first spatial residual between the first upsampled video frame and the target video frame;

[0100] The first spatial domain residual is transformed to the frequency domain to obtain the first frequency domain residual;

[0101] The first residual energy is obtained based on the first frequency domain residual at each position on the target video frame.

[0102] Based on one or more embodiments of this disclosure, a video processing method is provided to determine the second residual energy in the frequency domain of the residual between a target video frame and a second upsampled video frame, comprising:

[0103] Determine the second spatial residual between the second upsampled video frame and the target video frame;

[0104] The second spatial domain residual is transformed to the frequency domain to obtain the second frequency domain residual;

[0105] The second residual energy is obtained based on the second frequency domain residual at each position on the target video frame.

[0106] Based on one or more embodiments of this disclosure, a video processing method is provided to determine the encoding resolution of a target video during encoding, including:

[0107] Determine whether to use the original resolution of the target video frame during encoding, or to use a resolution lower than the original resolution during encoding.

[0108] Based on one or more embodiments of this disclosure, a video processing method is provided, which determines the encoding resolution when encoding a target video frame based on a first residual energy and a second residual energy, including:

[0109] Determine the energy difference between the first residual energy and the second residual energy, as well as the target threshold;

[0110] If the energy difference is greater than the target threshold, the original resolution of the target video frame is used during encoding; or,

[0111] If the energy difference is not greater than the target threshold, a resolution lower than the original resolution is used when determining the encoding.

[0112] Based on one or more embodiments of this disclosure, a video processing method is provided to determine a target threshold, including: determining the target threshold based on compression quantization parameters when compressing downsampled video frames.

[0113] Based on one or more embodiments of this disclosure, a video processing method is provided, wherein the target video frame is a cloud gaming video frame.

[0114] Based on one or more embodiments of this disclosure, a video processing apparatus is provided, comprising:

[0115] The processing unit is used to downsample the target video frame to obtain a downsampled video frame, compress and decode the downsampled video frame to reconstruct the reconstructed video frame, upsample the reconstructed video frame to obtain a first upsampled video frame, and upsample the downsampled video frame to obtain a second upsampled video frame.

[0116] The determining unit is used to determine the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and the second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame; and to determine the coding resolution when coding the target video frame based on the first residual energy and the second residual energy.

[0117] Based on one or more embodiments of this disclosure, an electronic device is provided, including: at least one memory and at least one processor;

[0118] The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method described in any one of the above.

[0119] Based on one or more embodiments of the present disclosure, a computer-readable storage medium is provided for storing program code that, when executed by a processor, causes the processor to perform the methods described above.

[0120] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0121] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0122] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video processing method, characterized in that, include: The target video frame is downsampled to obtain a downsampled video frame. The downsampled video frame is compressed and decoded to reconstruct a reconstructed video frame. The reconstructed video frame is upsampled to obtain a first upsampled video frame. The downsampled video frame is upsampled to obtain a second upsampled video frame. Determine the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and the second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame; The encoding resolution for encoding the target video frame is determined based on the first residual energy and the second residual energy. The determination of the encoding resolution for encoding the target video frame based on the first residual energy and the second residual energy includes: Determine the energy difference between the first residual energy and the second residual energy, as well as the target threshold. If the energy difference is greater than the target threshold, the original resolution of the target video frame is used during encoding; or, If the energy difference is not greater than the target threshold, a resolution lower than the original resolution is used for encoding.

2. The method according to claim 1, characterized in that, Determining the first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame includes: Determine the first spatial residual between the first upsampled video frame and the target video frame; The first spatial domain residual is transformed to the frequency domain to obtain the first frequency domain residual; The first residual energy is obtained based on the first frequency domain residual at each position on the target video frame.

3. The method based on claim 1, characterized in that, Determining the second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame includes: Determine the second spatial residual between the second upsampled video frame and the target video frame; The second spatial domain residual is transformed to the frequency domain to obtain the second frequency domain residual; The second residual energy is obtained based on the second frequency domain residual at each position on the target video frame.

4. The method based on claim 1, characterized in that, Determining the encoding resolution when encoding the target video includes: The original resolution of the target video frame is used during encoding, or a resolution lower than the original resolution is used during encoding.

5. The method based on claim 1, characterized in that, Determining the target threshold includes: The target threshold is determined based on the compression quantization parameters used when compressing the downsampled video frames.

6. The method based on claim 1, characterized in that, The target video frame is a cloud gaming video frame.

7. A video processing apparatus, comprising: The processing unit is configured to downsample the target video frame to obtain a downsampled video frame, compress and decode the downsampled video frame to reconstruct the reconstructed video frame, upsample the reconstructed video frame to obtain a first upsampled video frame, and upsample the downsampled video frame to obtain a second upsampled video frame. A determining unit is configured to determine a first residual energy in the frequency domain of the residual between the target video frame and the first upsampled video frame, and a second residual energy in the frequency domain of the residual between the target video frame and the second upsampled video frame; and to determine the encoding resolution for encoding the target video frame based on the first residual energy and the second residual energy. The determination of the encoding resolution for encoding the target video frame based on the first residual energy and the second residual energy includes: Determine the energy difference between the first residual energy and the second residual energy, as well as the target threshold. If the energy difference is greater than the target threshold, the original resolution of the target video frame is used during encoding; or, If the energy difference is not greater than the target threshold, a resolution lower than the original resolution is used for encoding.

8. An electronic device, comprising: At least one memory and at least one processor; The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method of any one of claims 1 to 6.

9. A computer-readable storage medium for storing program code that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 6.

Images