Video processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a video processing method and device, equipment and medium, relating to the technical field of image and video processing. The implementation scheme is: determining a first frame and a second frame in a video; determining an optical flow between the first frame and the second frame; determining an estimated frame corresponding to the second frame based on the first frame and the optical flow; determining a predicted frame corresponding to the second frame based on a difference between the estimated frame and the second frame; and inter-frame encoding the video based on the first frame and the predicted frame.

Description

Technical Field

[0001] This disclosure relates to the field of image and video processing technology, specifically to a video processing method, a video processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] After video images are digitized, the data volume is extremely large. If transmitted directly without processing, it will waste a lot of bandwidth and storage, and also consume a lot of processing time. Therefore, video needs to be compressed and encoded before transmission, which requires video encoding and decoding technology. Video encoding and decoding is the process of compressing, storing, and decompressing digital video signals. The encoder converts the original video signal into compressed video data, and the decoder restores the compressed video data to the original video signal.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a video processing method, a video processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a video processing method is provided, comprising: determining a first frame and a second frame consecutively in a video; determining an optical flow between the first frame and the second frame; determining an estimated frame corresponding to the second frame based on the first frame and the optical flow; determining a predicted frame corresponding to the second frame based on the difference between the estimated frame and the second frame; and performing inter-frame coding on the video based on the first frame and the predicted frame.

[0006] According to another aspect of this disclosure, a video processing apparatus is also provided, comprising: a video frame determination unit configured to determine a first frame and a second frame consecutively in a video; an optical flow determination unit configured to determine an optical flow between the first frame and the second frame; an estimation unit configured to determine an estimated frame corresponding to the second frame based on the first frame and the optical flow; a prediction unit configured to determine a predicted frame corresponding to the second frame based on the difference between the estimated frame and the second frame; and an encoding unit configured to perform inter-frame encoding on the video based on the first frame and the predicted frame.

[0007] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein said memory stores a computer program that, when executed by said at least one processor, implements the method described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the method described above when executed by a processor.

[0009] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method described above.

[0010] The embodiments provided in this disclosure can improve the temporal correlation between frames, thereby reducing the bitrate required for video encoding while ensuring video quality.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 An exemplary flowchart of a video processing method according to an embodiment of the present disclosure is shown;

[0014] Figure 2 An exemplary block diagram of a video processing apparatus according to embodiments of the present disclosure is shown; and

[0015] Figure 3 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0018] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0019] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0020] The following is an explanation of the terminology used in the embodiments of this disclosure.

[0021] Video bitrate: refers to the rate at which video data is transmitted over a period of time, usually measured in bits per second (bps). Video bitrate determines the size and quality of a video file; a higher bitrate results in better video quality, but also a larger file size.

[0022] Optical flow estimation refers to the process of calculating the velocity and direction of motion of each pixel by analyzing its temporal changes between adjacent frames in a video sequence. The velocity of a pixel in the x and y directions is typically referred to as horizontal optical flow and vertical optical flow, respectively.

[0023] Dense optical flow: refers to calculating the speed and direction of motion of each pixel in an image sequence in adjacent frames.

[0024] Intra-frame coding and inter-frame coding are two important techniques in video encoding and decoding. Intra-frame coding compresses a single frame of image into a series of blocks, each of which is independently encoded into a bitstream. This method requires a high bit rate because every frame needs to be encoded. Inter-frame coding, on the other hand, performs differential compression on consecutive frames. It utilizes the redundancy between frames to represent the video sequence with less data. Because inter-frame coding uses the correlation between consecutive frames for compression, it achieves better compression ratios compared to intra-frame coding.

[0025] MEMC (Motion Estimation and Motion Compensation) is an inter-frame coding technique commonly used in video coding and compression. Its basic principle is to reduce redundant information in video data through motion estimation and motion compensation, thereby achieving video compression and lowering the bitrate.

[0026] Existing inter-frame coding methods typically use motion estimation to find motion vectors between adjacent frames, but this method has limitations. First, motion estimation can be affected by factors such as noise and compression artifacts, leading to prediction errors. Furthermore, motion estimation struggles to handle complex motion scenarios, such as rapid motion, rotational motion, and deformation motion, which can increase prediction errors and degrade coding quality.

[0027] To address the aforementioned problems, this disclosure provides a novel video processing method.

[0028] Figure 1 An exemplary flowchart of a video processing method according to an embodiment of the present disclosure is shown.

[0029] like Figure 1 In step S102, a first frame and a second frame consecutively in the video are determined. In step S104, the optical flow between the first frame and the second frame is determined. In step S106, an estimated frame corresponding to the second frame is determined based on the first frame and the optical flow. In step S108, a predicted frame corresponding to the second frame is determined based on the difference between the estimated frame and the second frame. In step S110, inter-frame coding is performed on the video based on the first frame and the predicted frame.

[0030] The video processing method provided in this disclosure can improve the temporal correlation between frames and reduce the bitrate required for video encoding while ensuring video quality.

[0031] The embodiments of this disclosure will now be described in detail.

[0032] In step S102, a first frame and a second frame consecutively in the video can be determined. In some embodiments, the first frame and the second frame come from a consecutive temporal video segment belonging to the same shot.

[0033] In step S104, the optical flow between the first frame and the second frame can be determined.

[0034] Various optical flow estimation models (such as FlowNet) can be used to process the first and second frames to obtain the optical flow estimation results between the first and second frames.

[0035] The optical flow between the first and second frames can be dense optical flow, which means that the speed and direction of motion of each pixel between the first and second frames are calculated.

[0036] In some embodiments, when estimating the optical flow between the first frame and the second frame in step S104, various means can be used to remove redundant information in the video frames in order to obtain a uniform and smooth optical flow.

[0037] In some implementations, step S104 may include denoising the first frame and the second frame to obtain a denoised first frame and a denoised second frame, and then determining the optical flow between the denoised first frame and the denoised second frame as the optical flow between the first frame and the second frame. In some examples, the first frame and the second frame may be denoised by filtering (such as median filtering). It is understood that any other available denoising method may be used to process the first frame and the second frame without departing from the principles of this disclosure. The denoised video frame has less redundant information, therefore, optical flow estimation using the denoised video frame can yield a more uniform and smooth optical flow.

[0038] In some implementations, step S104 may include optical flow estimation on the scaled-down first and second frames. The first and second frames can be downsampled to obtain downsampled first and second frames. For example, the first and second frames can be downsampled by a factor of 2. The optical flow between the downsampled first and second frames can be determined using an optical flow estimation model. It is understood that when the video frame used for optical flow estimation is downsampled, the size of the resulting optical flow is consistent with the size of the downsampled video frame. Therefore, the optical flow between the downsampled first and second frames can be amplified to obtain the optical flow between the first and second frames. In the case of downsampling the first and second frames by a factor of 2, the optical flow with the same size as the downsampled video frame can be amplified by a factor of 2, thereby obtaining an optical flow consistent with the size of the first and second frames. When amplifying the optical flow, the length, width, and optical flow value can be amplified by a corresponding factor (e.g., 2). A more uniform and smooth optical flow can also be obtained by estimating the optical flow on the scaled-down video frames.

[0039] In some implementations, optical flow estimation can be performed simultaneously by denoising and downsampling video frames. For example, the first and second frames can be denoised first, then downsampled, and optical flow estimation can be performed on the downsampled frames. In this case, the obtained optical flow can be amplified to obtain the optical flow between the first and second frames. By applying denoising and size reduction methods simultaneously to the optical flow estimation of video frames, a uniform and smooth optical flow with less redundant information can be obtained.

[0040] In step S106, an estimated frame corresponding to the second frame can be determined based on the first frame and the optical flow.

[0041] In inter-frame coding, MEMC is typically performed on a pixel-block (i.e., macroblock) basis. The basic process is as follows: The macroblock with the smallest matching error is searched between two frames. Motion vectors and residuals of the matched macroblock pairs are obtained based on the positional offsets between the macroblocks. Encoding the motion vectors and residuals completes the encoding of a macroblock in the video frame. During this process, the amount of data contained in the residuals between macroblock pairs directly affects the bitrate of the encoded video. Therefore, it is necessary to minimize the residuals between macroblock pairs. If the residuals between macroblock pairs are 0, then encoding this macroblock requires no data.

[0042] The optical flow determined in step S104 can represent the amount of movement of a pixel in the first frame to the second frame. The first frame can be sampled based on the optical flow using a warp operation to obtain an estimated frame corresponding to the second frame.

[0043] Each pixel in the estimated frame obtained using this method originates from the previous frame, so the difference in pixel values ​​between corresponding pixels in two frames is 0. Since optical flow can represent the motion of a moving object composed of multiple pixels, the estimated frame obtained in this way contains many macroblock pairs with a matching error of 0 (or a small matching error) with the previous frame. This results in encoding such macroblocks requiring almost no data or only a small amount of data. Furthermore, because optical flow can express the basic laws of pixel motion, differences in the appearance of the same object between two frames due to various factors (such as instantaneous brightness differences) are eliminated, effectively reducing noise, thus ensuring no loss of video quality.

[0044] In step S108, a predicted frame corresponding to the second frame is determined based on the difference between the estimated frame and the second frame.

[0045] Since optical flow estimation is not 100% accurate, the difference between the estimated frame obtained in step S106 and the original second frame can be evaluated to determine the predicted frame corresponding to the second frame. The estimated frame obtained by sampling the first frame based on optical flow has roughly the same image content as the second frame. However, due to errors in optical flow estimation, some pixels in the estimated frame may have significant differences from their corresponding pixels in the second frame. In this case, to eliminate the error in optical flow estimation, the pixel values ​​of the pixels in the estimated frame can be adjusted based on the difference between the estimated frame and the second frame.

[0046] In some embodiments, the pixel values ​​of pixels in the estimated frame can be adjusted based on the difference between the estimated frame and the second frame according to predetermined rules. For example, when it is determined that the difference between the pixel values ​​of pixels in the estimated frame and the corresponding pixel values ​​in the second frame exceeds a predetermined threshold, the pixel values ​​of pixels in the estimated frame can be adjusted based on predetermined rules to make them closer to the pixel values ​​of the corresponding pixels in the second frame.

[0047] In some implementations, the pixel values ​​of pixels in the estimated frame can be adjusted by directly adding or subtracting a certain value. The added or subtracted value can be determined based on the difference between the pixel values ​​of the pixels in the estimated frame and the corresponding pixel values ​​in the second frame. This adjustment ensures that the difference between the pixel values ​​of the pixels in the estimated frame and the corresponding pixel values ​​in the second frame does not exceed a predetermined threshold.

[0048] In some implementations, the value of a pixel in the predicted frame can be determined by a weighted sampling of the pixel value of the corresponding pixel in the second frame and the pixel value of the estimated frame, wherein the weight of each pixel is determined based on the difference between the estimated frame and the second frame.

[0049] In some examples, the difference between the estimated frame and the second frame refers to the difference in pixel values ​​at individual pixels. Weighted sampling weights for the estimated frame or the second frame can be obtained by comparing the pixel values ​​of corresponding pixels in the estimated frame and the second frame pixel-by-pixel. A mask of the same size as the second frame can be obtained through pixel-by-pixel comparison. The predicted frame is then obtained by weighted sampling of the estimated frame and the second frame based on the obtained mask, where the values ​​of each pixel in the mask can be determined as the weighted sampling weights for the estimated frame or the second frame.

[0050] For example, the weighted sampling weight can be determined to be 0 or 1. In the obtained mask, the positions of pixels with a difference less than the first threshold are recorded as 1 (or other possible representations), and the positions of pixels with a difference not less than the first threshold are recorded as 0 (or other possible representations). When the difference is equal to the first threshold, those skilled in the art can record the corresponding position as 1 or 0 according to the actual situation. When the pixel in the mask is recorded as 0 when the difference is equal to the first threshold, for each pixel in the prediction frame, the difference between the corresponding pixel value in the second frame and the corresponding pixel value in the estimation frame is determined. If the difference is determined to be less than the first threshold, the pixel value of the pixel is determined to be the pixel value of the corresponding pixel in the estimation frame; if the difference is determined to be not less than the first threshold, the pixel value of the pixel is determined to be the pixel value of the corresponding pixel in the second frame. The above describes the principle of this disclosure by using the values ​​of the pixels in the mask as the weights of the estimation frame. It can be understood that the values ​​of the pixels in the mask can also be used as the weights of the second frame, as long as the positions of pixels with a difference less than the first threshold are recorded as 0 and the positions of pixels with a difference greater than the first threshold are recorded as 1. As mentioned above, when the difference is equal to the first threshold, those skilled in the art can record the corresponding position as 1 or 0 according to the actual situation.

[0051] In other examples, the difference between the estimated frame and the second frame can refer to the statistical difference within a window of a predetermined size centered on each pixel. This window can be an N*M pixel block centered on the pixel being processed, such as a 7*7 block or any other suitable size. The appropriate window size can be determined based on hyperparameter experiments and the size of the video frames to avoid distortion caused by retaining too many estimated frame pixels if the window is too large, or difficulty in saving bitrate if the window is too small and retains too many original frame pixels. For pixels at the image edges, the required pixels can be padded to fill the window, and the pixel values ​​of the padded pixels can be determined based on any suitable method.

[0052] The correlation between a pixel window in the estimation frame and its corresponding pixel window in the second frame can be determined to determine the weighted sampling weights used for the estimation frame or the second frame. The correlation can indicate the statistical difference between the individual pixels of the pixel window. The correlation can be a normalized correlation coefficient. For example, the correlation coefficient between pixel windows can be determined based on the following equation (1):

[0053]

[0054] Where X represents the average value of N*M pixels within the estimated pixel window of the frame. This represents the average value of N*M pixels within a pixel window in the second frame, where i and j represent index coefficients, and X... i,jY represents the pixel value of the pixel in the i-th row and j-th column within the pixel window of the estimated frame. i,j This represents the pixel value of the pixel in the i-th row and j-th column within the pixel window of the second frame, where sqrt represents the square root operation.

[0055] The correlation coefficient P obtained using the above method is a result with a value range of [-1, 1]. The larger the value, the stronger the correlation between the corresponding pixels in the estimated frame and the second frame, indicating that the pixels in the current estimated frame are more reliable.

[0056] The weighted sampling weights used for estimating the frame or the second frame can be determined based on the correlation coefficient P. A mask of the same size as the second frame can be obtained by calculating the correlation coefficient pixel by pixel based on Equation (1). The predicted frame is obtained by weighted sampling of the estimated frame and the second frame based on the obtained mask, where the value of each pixel in the mask can be determined as the weighted sampling weight for estimating the frame or the second frame.

[0057] For example, the weighted sampling weights can be determined to be 0 or 1. In the obtained mask, the positions of pixels with a correlation greater than the second threshold (e.g., 0) are recorded as 1 (or other possible representations), and the positions of pixels with a correlation less than the second threshold are recorded as 0 (or other possible representations). When the correlation equals the second threshold, those skilled in the art can record the corresponding positions as 1 or 0 according to the actual situation. When the pixels in the mask are recorded as 0 when the correlation equals the second threshold, for each pixel in the prediction frame, the correlation between the corresponding pixel in the second frame and the pixel window of the corresponding pixel in the estimation frame is determined. If the correlation is determined to be greater than the second threshold, the pixel value of the pixel is determined to be the pixel value of the corresponding pixel in the estimation frame; if the correlation is determined to be less than the second threshold, the pixel value of the pixel is determined to be the pixel value of the corresponding pixel in the second frame. The above describes the principle of this disclosure by using the values ​​of the pixels in the mask as the weights of the estimation frame. It can be understood that the values ​​of the pixels in the mask can also be used as the weights of the second frame, as long as the positions of pixels with a difference greater than the second threshold in the mask are recorded as 0, and the positions of pixels with a difference less than the second threshold are recorded as 1. As mentioned above, when the correlation equals the second threshold, those skilled in the art can record the corresponding position as 1 or 0 according to the actual situation.

[0058] Using this method, when there is a large difference between the estimated frame obtained based on optical flow estimation and the original frame, it is considered that this result may be introduced by inaccurate optical flow estimation. In this case, the pixel values ​​in the original frame are used as the pixel values ​​of the predicted frame to avoid inaccurate optical flow estimation introducing unnecessary errors in the image.

[0059] For example, the weighted sampling weights can also be determined to other values. Without departing from the principles of this disclosure, those skilled in the art can flexibly set the weighted sampling weights so that when the difference between the estimated frame and the original second frame is large, more information from the original frame is used; when the difference between the estimated frame and the original second frame is small, more information from the estimated frame is used. By adjusting the weighted sampling weights of the estimated frame and the second frame using the above method, the estimated information based on optical flow and the original true information can be comprehensively considered in the prediction frame to obtain a more balanced result, thereby improving the compression ratio of inter-frame coding while retaining more true information. Furthermore, by using the optical flow estimation results of dense optical flow to obtain the estimated frame and the corresponding prediction frame, the inter-frame correlation between the prediction frame and the previous frame can be further improved.

[0060] In step S110, the video is inter-frame coded based on the first frame and the predicted frame.

[0061] By predicting the second frame before inter-frame coding, the temporal correlation between the first and second frames can be improved, thereby reducing the bitrate required for video inter-frame coding without sacrificing image quality.

[0062] The first frame can be either the original video frame or a frame predicted based on optical flow. For example, steps S102 to S108 can be used to process the frame preceding the first frame and the first frame to obtain the predicted frame of the original frame of the first frame.

[0063] Steps S102-S108 can be performed sequentially on each video frame before inter-frame encoding as video preprocessing, or steps S102-S108 can be performed on the video frame to be processed before inter-frame encoding for each frame during video encoding. Those skilled in the art can make the choice based on the actual situation.

[0064] Figure 2 An exemplary block diagram of a video processing apparatus according to an embodiment of the present disclosure is shown.

[0065] like Figure 2 As shown, the video processing apparatus 200 may include a video frame determination unit 210, an optical flow determination unit 220, an estimation unit 230, a prediction unit 240, and an encoding unit 250.

[0066] The video frame determination unit 210 can be configured to determine a first frame and a second frame consecutively in the video. The optical flow determination unit 220 can be configured to determine the optical flow between the first frame and the second frame. The estimation unit 230 can be configured to determine an estimated frame corresponding to the second frame based on the first frame and the optical flow. The prediction unit 240 can be configured to determine a predicted frame corresponding to the second frame based on the difference between the estimated frame and the second frame. The coding unit 250 can be configured to perform inter-frame coding on the video based on the first frame and the predicted frame.

[0067] The video processing apparatus provided in this disclosure can improve the temporal correlation between frames and reduce the bit rate required for video encoding while ensuring video quality.

[0068] It should be understood that Figure 2 Each unit of the device 200 shown can be connected to a reference. Figure 1 The steps in method 100 described correspond to each other. Therefore, the operations, features, and advantages described above for method 100 also apply to apparatus 200 and its constituent units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0069] It should also be understood that this article can describe various technologies in the general context of software and hardware components or program modules. The above regarding... Figure 2 The described units can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuit. For example, in some embodiments, one or more of the video frame determination unit 210, optical flow determination unit 220, estimation unit 230, prediction unit 240, and encoding unit 250 can be implemented together in a System-on-Chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components of other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0070] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein said memory stores a computer program that, when executed by said at least one processor, implements the method described above.

[0071] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the method described above when executed by a processor.

[0072] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method described above.

[0073] See Figure 3 The present invention describes a structural block diagram of an electronic device 300 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0074] like Figure 3 As shown, the electronic device 300 may include at least one processor 301, working memory 302, input unit 304, display unit 305, speaker 306, storage unit 307, communication unit 308 and other output units 306 that are capable of communicating with each other via system bus 303.

[0075] Processor 301 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 301 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Processor 301 may be configured to acquire and execute computer-readable instructions stored in working memory 302, storage unit 307, or other computer-readable media, such as program code of operating system 302a, program code of application program 302b, etc.

[0076] Working memory 302 and storage unit 307 are examples of computer-readable storage media for storing instructions that are executed by processor 301 to perform the various functions described above. Working memory 302 may include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, storage unit 307 may include hard disk drives, solid-state drives, removable media including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Working memory 302 and storage unit 307 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by processor 301 as a specific machine configured to perform the operations and functions described in the examples herein.

[0077] Input unit 306 can be any type of device capable of inputting information to electronic device 300. Input unit 306 can receive input digital or character information and generate key signal input related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and / or remote control. Output unit can be any type of device capable of presenting information, and can include, but is not limited to, display unit 305, speaker 306 and other output units 309. Other output units 309 can include, but are not limited to, video / audio output terminals, vibrators and / or printers. Communication unit 308 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.3 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0078] The application program 302b in working register 302 can be loaded to execute the various methods and processes described above, for example... Figure 1Steps S102-S110 are described above. For example, in some embodiments, the method 100 described above may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 307. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via storage unit 307 and / or communication unit 308. When the computer program is loaded and executed by processor 301, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, processor 301 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0079] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0080] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0081] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0082] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0083] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0084] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0085] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0086] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A video processing method, comprising: Determine the first and second consecutive frames in the video; Determine the optical flow between the first frame and the second frame; Based on the first frame and the optical flow, an estimated frame corresponding to the second frame is determined; A prediction frame corresponding to the second frame is determined based on the difference between the estimated frame and the second frame, wherein the value of a pixel in the prediction frame is determined based on a weighted sampling of the pixel value of the corresponding pixel in the second frame and the pixel value of the pixel in the estimated frame, wherein the weighted sampling weight of each pixel is determined based on the difference of the pixel values ​​of each pixel between the estimated frame and the second frame; as well as The video is subjected to differential compression based on inter-frame coding based on the first frame and the predicted frame.

2. The video processing method as described in claim 1, wherein, Determining the optical flow between the first frame and the second frame includes: The first frame and the second frame are denoised to obtain the denoised first frame and the denoised second frame; The optical flow between the denoised first frame and the denoised second frame is determined as the optical flow between the first frame and the second frame.

3. The video processing method as described in claim 2, wherein, Denoising the first frame and the second frame includes performing median filtering on the first frame and the second frame.

4. The video processing method as described in claim 1, wherein, Determining the optical flow between the first frame and the second frame includes: The first frame and the second frame are downsampled to obtain the downsampled first frame and the downsampled second frame; Determine the optical flow between the downsampled first frame and the downsampled second frame; The optical flow between the downsampled first frame and the downsampled second frame is amplified to obtain the optical flow between the first frame and the second frame.

5. The video processing method according to any one of claims 1-4, wherein, The optical flow between the first frame and the second frame is a dense optical flow.

6. The video processing method as described in claim 1, wherein, Determining the predicted frame corresponding to the second frame based on the difference between the estimated frame and the second frame includes: For each pixel in the predicted frame, Determine the difference between the pixel values ​​of the corresponding pixel in the second frame and the corresponding pixel in the estimated frame. If the difference is determined to be less than a first threshold, the pixel value of that pixel is determined to be the pixel value of the corresponding pixel in the estimated frame. Based on the determination that the difference is not less than the first threshold, the pixel value of the pixel is determined to be the pixel value of the corresponding pixel in the second frame.

7. The video processing method as described in claim 1, wherein, Determining the predicted frame corresponding to the second frame based on the difference between the estimated frame and the second frame includes: For each pixel in the predicted frame, Define a pixel window centered on the given pixel, wherein the pixel window comprises multiple pixels. Determine the correlation between the corresponding pixel window in the second frame and the corresponding pixel window in the estimated frame. Based on the determination that the correlation is greater than the second threshold, the pixel value of that pixel is determined to be the pixel value of the corresponding pixel in the estimated frame. Based on the determination that the correlation is not greater than the second threshold, the pixel value of the pixel is determined to be the pixel value of the corresponding pixel in the second frame.

8. The video processing method of claim 7, wherein the correlation is a normalized correlation coefficient.

9. A video processing apparatus, comprising: The video frame determination unit is configured to determine the first and second consecutive frames in the video. An optical flow determination unit is configured to determine the optical flow between the first frame and the second frame; The estimation unit is configured to determine an estimated frame corresponding to the second frame based on the first frame and the optical flow; The prediction unit is configured to determine a prediction frame corresponding to the second frame based on the difference between the estimated frame and the second frame, wherein the value of a pixel in the prediction frame is determined based on a weighted sampling of the pixel value of the corresponding pixel in the second frame and the pixel value of the pixel in the estimated frame, wherein the weighted sampling weight of each pixel is determined based on the difference of the pixel values ​​of each pixel between the estimated frame and the second frame; as well as The encoding unit is configured to perform inter-frame-based differential compression on the video based on the first frame and the predicted frame.

10. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores a computer program that, when executed by the at least one processor, implements the method according to any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-8.

12. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-8.

Images